From a020dc62cb0ced3f0a679f697a2ff3892bef4168 Mon Sep 17 00:00:00 2001 From: Olly99999 Date: Thu, 18 Jun 2026 12:39:41 +0100 Subject: [PATCH 1/8] Add di.grafana module --- di/grafana/grafana.md | 147 ++++++++++++++++++++++++ di/grafana/grafana.q | 259 ++++++++++++++++++++++++++++++++++++++++++ di/grafana/init.q | 3 + di/grafana/test.csv | 52 +++++++++ 4 files changed, 461 insertions(+) create mode 100644 di/grafana/grafana.md create mode 100644 di/grafana/grafana.q create mode 100644 di/grafana/init.q create mode 100644 di/grafana/test.csv diff --git a/di/grafana/grafana.md b/di/grafana/grafana.md new file mode 100644 index 00000000..ccf06823 --- /dev/null +++ b/di/grafana/grafana.md @@ -0,0 +1,147 @@ +# `grafana.q` – Grafana JSON datasource adaptor for kdb-x + +A library that lets a [Grafana](https://grafana.com/) server query a kdb+ +process directly, using the +[SimPod JSON datasource](https://github.com/simPod/GrafanaJsonDatasource) API. + +Once initialised, the module installs HTTP handlers on the process so that +requests carrying the `X-Grafana-Org-Id` header are intercepted and answered +with Grafana-shaped JSON, while all other HTTP requests fall through to any +existing handler. Point a SimPod JSON datasource at the process's HTTP port and +its tables become available as Grafana panels. + +--- + +## :sparkles: Features + +- Serves the `/search` endpoint – populates Grafana's metric dropdowns from the + tables in the process. +- Serves the `/query` endpoint – returns either **timeseries** or **table** + panel data. +- Answers the Grafana test-connection `GET` with a `200 OK`. +- Wraps any pre-existing `.z.pp`/`.z.ph` handlers rather than overwriting them. +- No connection to other modules required – loads standalone with an injected + logger. + +--- + +## :inbox_tray: Import + +```q +q)grafana:use`di.grafana +``` + +Only `init` is exported; everything else runs automatically via the installed +HTTP handlers. + +--- + +## :electric_plug: Dependencies + +| Dependency | Required | Description | +|---|---|---| +| `log` | yes | A dictionary of `` `info`warn`error `` logging functions, each with the `{[c;m]}` signature (context symbol; message string). | + +The logger is **mandatory** – `init` errors immediately if it is not supplied. +A ready-made logger can be taken from `di.log`, or you can pass your own: + +```q +/ custom logger +mylog:`info`warn`error!( + {[c;m] -1 "INFO [",string[c],"] ",m;}; + {[c;m] -1 "WARN [",string[c],"] ",m;}; + {[c;m] -2 "ERROR [",string[c],"] ",m;}); +``` + +--- + +## :gear: Configuration + +Configuration is optional and passed to `init` under the `config` key. Any +omitted value keeps its default. + +| Key | Type | Default | Description | +|---|---|---|---| +| `timecol` | symbol | `` `time `` | Name of the time column used for timeseries queries. | +| `sym` | symbol | `` `sym `` | Name of the column used to split data by instrument. | +| `timebackdate` | timespan | `2D` | How far back to look when finding distinct syms for the dropdowns. | +| `ticks` | long | `1000` | Number of rows returned for a table request. | +| `del` | char | `"."` | Delimiter separating the arguments within a query target. | + +--- + +## :memo: Initialisation + +`init` takes a single dictionary of dependencies (and optional config), then +installs the HTTP handlers. The handlers are installed only once, so `init` may +be called again to update the logger or configuration without re-wrapping. + +```q +/ logger only, defaults for everything else +q)grafana.init[enlist[`log]!enlist mylog] + +/ logger plus configuration overrides +q)grafana.init[`log`config!(mylog;`timecol`ticks!(`ts;500))] +``` + +--- + +## :wrench: Exported functions + +### `init` + +``` +init[deps] +``` + +- `deps` – a dictionary, one of: + - `` enlist[`log]!enlist logdict `` – inject the logger, use config defaults. + - `` `log`config!(logdict;configdict) `` – inject the logger and override config. +- `logdict` – `` `info`warn`error `` ! three `{[c;m]}` functions (required). +- `configdict` – any subset of the keys in the [configuration](#gear-configuration) table. + +Errors if no logger is supplied. Returns nothing; its effect is wiring the +logger, applying configuration, and installing the `.z.pp`/`.z.ph` handlers. + +--- + +## :mag: Query target syntax + +The Grafana metric strings produced by `/search` encode the table, panel type +and arguments, separated by `del` (default `"."`): + +| Prefix | Meaning | +|---|---| +| `t.` | table panel for the whole table | +| `t.
.` | table panel filtered to one sym | +| `g.
` | graph panel, one series per numeric column | +| `g.
.` | graph panel, one series per sym for a column | +| `o.
...` | "other" panels (single-stat, gauge, etc.) | +| `f.<...>` | the target is a function call rather than a table | + +--- + +## :rocket: Example usage + +```q +q)grafana:use`di.grafana +q)log:use`di.log / or define your own logger +q)grafana.init[enlist[`log]!enlist `info`warn`error!(log.info;log.warn;log.error)] + +q)trade:([]time:.z.p-0D00:00:01*til 100;sym:100#`a`b`c;price:100?100f) +``` + +Add a SimPod JSON datasource in Grafana pointing at this process's HTTP port, +then build panels using the metrics offered in the dropdowns (`t.trade`, +`g.trade.price`, …). + +--- + +## :test_tube: Tests + +Tests are written for k4unit and run with: + +```q +q)k4unit:use`di.k4unit +q)k4unit.moduletest`di.grafana +``` diff --git a/di/grafana/grafana.q b/di/grafana/grafana.q new file mode 100644 index 00000000..c960d5ee --- /dev/null +++ b/di/grafana/grafana.q @@ -0,0 +1,259 @@ +/ grafana json datasource adaptor for kdb-x +/ implements the simpod json datasource api so a grafana server can query +/ kdb+ tables and timeseries data over http - the /search endpoint populates +/ the grafana dropdowns and the /query endpoint returns timeseries or table data + +/ ----------------------------------------------------------------------------- +/ configuration - defaults applied here, overridden via the config dict in init +/ ----------------------------------------------------------------------------- + +/ name of the time column used for timeseries queries +timecol:`time; +/ name of the sym column used to split data by instrument +sym:`sym; +/ how far back to look when finding distinct syms +timebackdate:2D; +/ number of ticks to return for table requests +ticks:1000; +/ delimiter separating the arguments within a query target +del:"."; + +/ json type for each kdb datatype, keyed by .Q.t character +types:.Q.t!`array`boolean,(3#`null),(5#`number),11#`string; +/ milliseconds between 1970.01.01 and 2000.01.01 +epoch:946684800000; + +/ ----------------------------------------------------------------------------- +/ request handling +/ ----------------------------------------------------------------------------- + +zpp:{[x] + / parse a grafana http post request and dispatch to the matching handler + / cut at the first whitespace to isolate the api url from any function params + r:(0;n?" ")cut n:first x; + rqt:.j.k r 1; + .z.m.loginfo[`grafana;"received ",(r 0)," request"]; + handler:$["query"~r 0;query;"search"~r 0;search;annotation]; + :.[handler;enlist rqt;{[e].z.m.logerr[`grafana;"failed to process request: ",e];'e}]; + }; + +annotation:{[rqt] + / annotation endpoint is not yet implemented + .z.m.logwarn[`grafana;"annotation url not yet implemented"]; + :`$"Annotation url nyi"; + }; + +query:{[rqt] + / dispatch a /query request to the timeseries or table builder by target type + rqtype:raze rqt[`targets]`type; + :.h.hy[`json]$[rqtype~"timeserie";tsfunc rqt;tbfunc rqt]; + }; + +search:{[rqt] + / build the grafana dropdown options from the tables available in the process + tabs:tables[]; + symtabs:tabs where sym in'cols each tabs; + timetabs:tabs where timecol in'cols each tabs; + rsp:string tabs; + if[count timetabs; + rsp,:s1:prefix["t";string timetabs]; + rsp,:s2:prefix["g";string timetabs]; + / suffix the numeric columns for the graph and other panel options + rsp,:raze(s2,'del),/:'c1:string {cols[x] where`number=types(0!meta x)`t}each timetabs; + rsp,:raze(prefix["o";string timetabs],'del),/:'c1; + if[count symtabs; + / suffix the distinct syms for the timeseries and other panel options + rsp,:raze(s1,'del),/:'c2:string each finddistinctsyms'[timetabs]; + rsp,:raze(prefix["o";string timetabs],'del),/:'{x[0]cross del,'string finddistinctsyms x 1}each(enlist each c1),'timetabs; + ]; + ]; + :.h.hy[`json].j.j rsp; + }; + +finddistinctsyms:{[x] + / distinct syms seen in table x within the configured lookback window + :?[x;enlist(>;timecol;(-;.z.p;timebackdate));1b;{x!x}enlist sym]sym; + }; + +prefix:{[c;s] + / prefix string c and the delimiter to each string in s + :(c,del),/:s; + }; + +/ ----------------------------------------------------------------------------- +/ fetching the last n ticks +/ ----------------------------------------------------------------------------- + +diskvals:{[x] + / last `ticks` rows of an on-disk partitioned table + c:(count[x]-ticks)+til ticks; + :get'[.Q.ind[x;c]]; + }; + +memvals:{[x] + / last `ticks` rows of an in-memory table + :get'[?[x;enlist(within;`i;count[x]-ticks,0);0b;()]]; + }; + +catchvals:{[x] + / try the on-disk path first, fall back to the in-memory path on error + :@[diskvals;x;{[x;y]memvals x}[x]]; + }; + +/ ----------------------------------------------------------------------------- +/ target parsing helpers +/ ----------------------------------------------------------------------------- + +istype:{[targ;char] + / test whether the target is prefixed with char followed by the delimiter + :(char,del)~2#targ; + }; +isfunc:istype[;"f"]; +istab:istype[;"t"]; + +/ ----------------------------------------------------------------------------- +/ building json responses +/ ----------------------------------------------------------------------------- + +tabresponse:{[colname;coltype;rqt] + / build a table response in the json datasource schema + :.j.j enlist`columns`rows`type!(flip`text`type!(colname;coltype);catchvals rqt;`table); + }; + +tbfunc:{[rqt] + / process a table request and return the json datasource table response + rqt:raze rqt[`targets]`target; + symname:0b; + / strip the type prefix: f.t.func drops 4, f.func drops 2, t.tab leaves the + / table name plus an optional sym + rqt:0!value $[isfunc[rqt]&istab 2_rqt;4_rqt; + isfunc rqt;2_rqt; + istab rqt;[rqt:`$del vs rqt;if[2meta[rqt][timecol;`t];rqt:@[rqt;timecol;+;.z.D]]; + / restrict to the time range requested by grafana + range:"P"$-1_'x[`range]`from`to; + rqt:?[rqt;enlist(within;timecol;range);0b;()]; + / add the milliseconds-since-epoch column grafana expects + rqt:@[rqt;`msec;:;mil rqt timecol]; + / dispatch on the number of arguments and the panel type + $[(2 Date: Wed, 24 Jun 2026 10:35:01 +0100 Subject: [PATCH 2/8] Refactoring code to match new spec with Jonny's feedback --- di/grafana/grafana.md | 17 +++++++++++++++++ di/grafana/grafana.q | 35 +++++++++++++++++++++-------------- di/grafana/test.csv | 1 + 3 files changed, 39 insertions(+), 14 deletions(-) diff --git a/di/grafana/grafana.md b/di/grafana/grafana.md index ccf06823..3f3e1708 100644 --- a/di/grafana/grafana.md +++ b/di/grafana/grafana.md @@ -53,6 +53,23 @@ mylog:`info`warn`error!( {[c;m] -2 "ERROR [",string[c],"] ",m;}); ``` +### Handler wiring + +`init` installs the adaptor by closure-wrapping the process's `.z.pp` (POST) and +`.z.ph` (GET) callbacks: it captures any pre-existing handler, then installs a +wrapper that routes Grafana requests (those carrying the `X-Grafana-Org-Id` +header) to the adaptor and **falls through to the captured handler for +everything else** — so adding the module to a process that already serves HTTP +does not break existing endpoints. This follows the `di.timer` precedent of a +module owning a `.z.*` callback as its core mechanism. + +The intended end-state under the modularisation guide is to register these +through the injected `di.handlers` dependency rather than assigning `.z.*` +directly. That migration is deferred until `di.handlers` exists and supports +**return-yielding** handler chains — `.z.pp`/`.z.ph` must *return* the HTTP +response and conditionally defer, which a side-effect-only chain cannot express. +`di.handlers` is not implemented on any branch yet. + --- ## :gear: Configuration diff --git a/di/grafana/grafana.q b/di/grafana/grafana.q index c960d5ee..8c0d433f 100644 --- a/di/grafana/grafana.q +++ b/di/grafana/grafana.q @@ -23,6 +23,9 @@ types:.Q.t!`array`boolean,(3#`null),(5#`number),11#`string; / milliseconds between 1970.01.01 and 2000.01.01 epoch:946684800000; +/ internal flag - set true once the http handlers have been installed +wired:0b; + / ----------------------------------------------------------------------------- / request handling / ----------------------------------------------------------------------------- @@ -32,14 +35,14 @@ zpp:{[x] / cut at the first whitespace to isolate the api url from any function params r:(0;n?" ")cut n:first x; rqt:.j.k r 1; - .z.m.loginfo[`grafana;"received ",(r 0)," request"]; + .z.m.log[`info][`grafana;"received ",(r 0)," request"]; handler:$["query"~r 0;query;"search"~r 0;search;annotation]; - :.[handler;enlist rqt;{[e].z.m.logerr[`grafana;"failed to process request: ",e];'e}]; + :.[handler;enlist rqt;{[e].z.m.log[`error][`grafana;"failed to process request: ",e];'e}]; }; annotation:{[rqt] / annotation endpoint is not yet implemented - .z.m.logwarn[`grafana;"annotation url not yet implemented"]; + .z.m.log[`warn][`grafana;"annotation url not yet implemented"]; :`$"Annotation url nyi"; }; @@ -237,20 +240,24 @@ init:{[deps] / examples: / grafana.init[enlist[`log]!enlist mylog] / grafana.init[`log`config!(mylog;enlist[`ticks]!enlist 500)] - logdict:$[99h=type deps;$[(`log in key deps)and not(::)~deps`log;deps`log;()!()];()!()]; - if[not count logdict;'"di.grafana: log dependency is required; pass `info`warn`error functions - see di.log for a default implementation"]; - .z.m.loginfo:logdict`info; - .z.m.logwarn:logdict`warn; - .z.m.logerr:logdict`error; + / validate the required log dependency - nested guards, no eager `and` + if[99h<>type deps;'"di.grafana: deps must be a dict with `log key"]; + if[not`log in key deps;'"di.grafana: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log;'"di.grafana: log value must be a dict; pass `info`warn`error functions"]; + if[not all`info`warn`error in key deps`log;'"di.grafana: log dict must have `info`warn`error keys; got: ",", " sv string key deps`log]; + .z.m.log:deps`log; / overlay any supplied configuration onto the defaults - config:$[99h=type deps;$[(`config in key deps)and not(::)~deps`config;deps`config;()!()];()!()]; - if[count config; - vars:`timecol`sym`timebackdate`ticks`del inter key config; - (.Q.dd[.z.M]each vars)set'config vars; + config:$[`config in key deps;deps`config;()!()]; + if[99h=type config; + if[`timecol in key config;.z.m.timecol:config`timecol]; + if[`sym in key config;.z.m.sym:config`sym]; + if[`timebackdate in key config;.z.m.timebackdate:config`timebackdate]; + if[`ticks in key config;.z.m.ticks:config`ticks]; + if[`del in key config;.z.m.del:config`del]; ]; / install the http handlers once, preserving any existing definitions - if[not`wired in key .z.M;sethandlers[]]; - .z.m.loginfo[`grafana;"initialised grafana json datasource adaptor"]; + if[not wired;sethandlers[]]; + .z.m.log[`info][`grafana;"initialised grafana json datasource adaptor"]; }; getconfig:{ diff --git a/di/grafana/test.csv b/di/grafana/test.csv index bb64c3ba..07d59f88 100644 --- a/di/grafana/test.csv +++ b/di/grafana/test.csv @@ -7,6 +7,7 @@ comment,,,,,,,Export surface and required dependency true,0,0,q,(asc key grafana)~`getconfig`init,1,,only init and getconfig are exported fail,0,0,q,grafana.init[(::)],1,,init errors when no log dependency is supplied fail,0,0,q,grafana.init[enlist[`log]!enlist(::)],1,,init errors when log dependency is null +fail,0,0,q,grafana.init[enlist[`log]!enlist `info`warn!({[c;m]};{[c;m]})],1,,init errors when log dict is missing keys comment,,,,,,,Initialisation wires handlers and applies defaults run,0,0,q,grafana.init[enlist[`log]!enlist mocklog],1,,initialise with mock logger From 8f57ee25193461d269350ef8d30812382a2f1742 Mon Sep 17 00:00:00 2001 From: Olly99999 Date: Wed, 24 Jun 2026 14:53:53 +0100 Subject: [PATCH 3/8] Make changes to grafana.md --- di/grafana/grafana.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/di/grafana/grafana.md b/di/grafana/grafana.md index 3f3e1708..6a7b542a 100644 --- a/di/grafana/grafana.md +++ b/di/grafana/grafana.md @@ -31,8 +31,8 @@ its tables become available as Grafana panels. q)grafana:use`di.grafana ``` -Only `init` is exported; everything else runs automatically via the installed -HTTP handlers. +`init` and `getconfig` are exported; the request handlers run automatically +once `init` has installed them. --- @@ -117,9 +117,20 @@ init[deps] - `logdict` – `` `info`warn`error `` ! three `{[c;m]}` functions (required). - `configdict` – any subset of the keys in the [configuration](#gear-configuration) table. -Errors if no logger is supplied. Returns nothing; its effect is wiring the +Errors immediately if the `log` dependency is missing, is not a dictionary, or +lacks the `info`/`warn`/`error` keys. Returns nothing; its effect is wiring the logger, applying configuration, and installing the `.z.pp`/`.z.ph` handlers. +### `getconfig` + +``` +getconfig[] +``` + +Returns the currently active configuration as a dictionary (`timecol`, `sym`, +`timebackdate`, `ticks`, `del`) — useful for confirming how the running module +is tuned without reaching into module internals. + --- ## :mag: Query target syntax From c817a74f776a00ae5dfff7bbce335b2e724d5f8a Mon Sep 17 00:00:00 2001 From: Olly99999 Date: Fri, 26 Jun 2026 10:59:21 +0100 Subject: [PATCH 4/8] Changes to follow peer review format --- di/grafana/grafana.md | 215 +++++++++++++++++++++--------------------- di/grafana/grafana.q | 41 +++++--- di/grafana/test.csv | 11 ++- 3 files changed, 141 insertions(+), 126 deletions(-) diff --git a/di/grafana/grafana.md b/di/grafana/grafana.md index 6a7b542a..845ad743 100644 --- a/di/grafana/grafana.md +++ b/di/grafana/grafana.md @@ -1,144 +1,114 @@ -# `grafana.q` – Grafana JSON datasource adaptor for kdb-x +# di.grafana -A library that lets a [Grafana](https://grafana.com/) server query a kdb+ -process directly, using the -[SimPod JSON datasource](https://github.com/simPod/GrafanaJsonDatasource) API. - -Once initialised, the module installs HTTP handlers on the process so that -requests carrying the `X-Grafana-Org-Id` header are intercepted and answered -with Grafana-shaped JSON, while all other HTTP requests fall through to any -existing handler. Point a SimPod JSON datasource at the process's HTTP port and -its tables become available as Grafana panels. +Grafana JSON datasource adaptor for kdb+. Implements the +[SimPod JSON datasource](https://github.com/simPod/GrafanaJsonDatasource) API so +a Grafana server can query a kdb+ process directly: the module installs HTTP +handlers that answer Grafana's `/search` and `/query` requests with the JSON it +expects, serving the process's tables as timeseries and table panels. --- -## :sparkles: Features +## Features -- Serves the `/search` endpoint – populates Grafana's metric dropdowns from the - tables in the process. -- Serves the `/query` endpoint – returns either **timeseries** or **table** - panel data. -- Answers the Grafana test-connection `GET` with a `200 OK`. -- Wraps any pre-existing `.z.pp`/`.z.ph` handlers rather than overwriting them. -- No connection to other modules required – loads standalone with an injected - logger. +- Serves the `/search` endpoint - populates Grafana's metric dropdowns from the tables in the process +- Serves the `/query` endpoint - returns either timeseries or table panel data +- Answers the Grafana test-connection `GET` with a `200 OK` +- Wraps any pre-existing `.z.pp`/`.z.ph` handlers rather than overwriting them, so other HTTP endpoints keep working +- Accepts a `kx.log` instance directly, or a plain `info`warn`error` dict +- Loads standalone - no hard dependencies on other modules --- -## :inbox_tray: Import - -```q -q)grafana:use`di.grafana -``` - -`init` and `getconfig` are exported; the request handlers run automatically -once `init` has installed them. +## Dependencies ---- +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | `info`, `warn`, and `error` - each a binary `{[c;m]}` where `c` is a symbol context and `m` is a string. All three are required because this module calls all three. | -## :electric_plug: Dependencies +**Hard dependency:** none - the module loads standalone. -| Dependency | Required | Description | -|---|---|---| -| `log` | yes | A dictionary of `` `info`warn`error `` logging functions, each with the `{[c;m]}` signature (context symbol; message string). | +The `log` dependency must be passed to `init` inside the `deps` dict keyed on +`` `log ``. The module throws immediately if `log` is absent, is not a +dictionary, or is missing any of `info`/`warn`/`error`. -The logger is **mandatory** – `init` errors immediately if it is not supplied. -A ready-made logger can be taken from `di.log`, or you can pass your own: +A `kx.log` instance can be passed directly - the module normalises its monadic +functions to the binary `{[c;m]}` contract automatically (detected by the +`getlvl`/`sinks`/`fmts` keys). Plain `info`warn`error` dicts pass through +unchanged. ```q -/ custom logger +/ plain custom logger mylog:`info`warn`error!( {[c;m] -1 "INFO [",string[c],"] ",m;}; {[c;m] -1 "WARN [",string[c],"] ",m;}; {[c;m] -2 "ERROR [",string[c],"] ",m;}); -``` +grafana:use`di.grafana +grafana.init[enlist[`log]!enlist mylog] -### Handler wiring - -`init` installs the adaptor by closure-wrapping the process's `.z.pp` (POST) and -`.z.ph` (GET) callbacks: it captures any pre-existing handler, then installs a -wrapper that routes Grafana requests (those carrying the `X-Grafana-Org-Id` -header) to the adaptor and **falls through to the captured handler for -everything else** — so adding the module to a process that already serves HTTP -does not break existing endpoints. This follows the `di.timer` precedent of a -module owning a `.z.*` callback as its core mechanism. +/ or a kx.log instance directly +kxlog:use`kx.log +grafana.init[enlist[`log]!enlist kxlog.createLog[]] +``` -The intended end-state under the modularisation guide is to register these -through the injected `di.handlers` dependency rather than assigning `.z.*` -directly. That migration is deferred until `di.handlers` exists and supports -**return-yielding** handler chains — `.z.pp`/`.z.ph` must *return* the HTTP -response and conditionally defer, which a side-effect-only chain cannot express. -`di.handlers` is not implemented on any branch yet. +Configuration keys `timecol`, `sym`, `timebackdate`, `ticks`, and `del` are all +optional - omit any or all of them and the module falls back to sensible +defaults. See Initialisation for full details. --- -## :gear: Configuration +## Initialisation -Configuration is optional and passed to `init` under the `config` key. Any -omitted value keeps its default. +`init[deps]` takes a single dictionary combining the `log` dependency with any +configuration overrides. -| Key | Type | Default | Description | -|---|---|---|---| -| `timecol` | symbol | `` `time `` | Name of the time column used for timeseries queries. | -| `sym` | symbol | `` `sym `` | Name of the column used to split data by instrument. | -| `timebackdate` | timespan | `2D` | How far back to look when finding distinct syms for the dropdowns. | -| `ticks` | long | `1000` | Number of rows returned for a table request. | -| `del` | char | `"."` | Delimiter separating the arguments within a query target. | - ---- - -## :memo: Initialisation +| Key | Required | Description | +|---|---|---| +| `` `log `` | yes | Log dep - `info`/`warn`/`error` `{[c;m]}` functions, or a `kx.log` instance | +| `` `timecol `` | no | Name of the time column used for timeseries queries. Default: `` `time `` | +| `` `sym `` | no | Name of the column used to split data by instrument. Default: `` `sym `` | +| `` `timebackdate `` | no | How far back to look when finding distinct syms for the dropdowns. Default: `2D` | +| `` `ticks `` | no | Number of rows returned for a table request. Default: `1000` | +| `` `del `` | no | Delimiter separating the arguments within a query target. Default: `"."` | -`init` takes a single dictionary of dependencies (and optional config), then -installs the HTTP handlers. The handlers are installed only once, so `init` may -be called again to update the logger or configuration without re-wrapping. +`init` must be called before the module will serve any requests. It validates +the logger, applies config, and installs the `.z.pp`/`.z.ph` handlers - once; +re-calling `init` updates the logger/config without re-wrapping the handlers. ```q -/ logger only, defaults for everything else -q)grafana.init[enlist[`log]!enlist mylog] +/ defaults only +grafana.init[enlist[`log]!enlist logdep] -/ logger plus configuration overrides -q)grafana.init[`log`config!(mylog;`timecol`ticks!(`ts;500))] +/ with config overrides (config keys sit alongside `log) +grafana.init[`log`timecol`ticks!(logdep;`ts;500)] ``` --- -## :wrench: Exported functions +## Exported Functions -### `init` - -``` -init[deps] +### `init[deps]` +Initialise the module: validate the log dependency, apply config, and install the HTTP handlers. Errors immediately if `log` is missing, is not a dict, or lacks `info`/`warn`/`error`. +```q +grafana.init[enlist[`log]!enlist logdep] +/ or with config: +grafana.init[`log`ticks!(logdep;500)] ``` -- `deps` – a dictionary, one of: - - `` enlist[`log]!enlist logdict `` – inject the logger, use config defaults. - - `` `log`config!(logdict;configdict) `` – inject the logger and override config. -- `logdict` – `` `info`warn`error `` ! three `{[c;m]}` functions (required). -- `configdict` – any subset of the keys in the [configuration](#gear-configuration) table. - -Errors immediately if the `log` dependency is missing, is not a dictionary, or -lacks the `info`/`warn`/`error` keys. Returns nothing; its effect is wiring the -logger, applying configuration, and installing the `.z.pp`/`.z.ph` handlers. - -### `getconfig` - -``` -getconfig[] +### `getconfig[]` +Return the currently active configuration as a dictionary - useful for confirming how the running module is tuned. +```q +grafana.getconfig[] +/ `timecol`sym`timebackdate`ticks`del!(`time;`sym;2D;1000;".") ``` -Returns the currently active configuration as a dictionary (`timecol`, `sym`, -`timebackdate`, `ticks`, `del`) — useful for confirming how the running module -is tuned without reaching into module internals. - --- -## :mag: Query target syntax +## Query target syntax -The Grafana metric strings produced by `/search` encode the table, panel type +The Grafana metric strings produced by `/search` encode the table, panel type, and arguments, separated by `del` (default `"."`): -| Prefix | Meaning | +| Target | Meaning | |---|---| | `t.
` | table panel for the whole table | | `t.
.` | table panel filtered to one sym | @@ -149,27 +119,52 @@ and arguments, separated by `del` (default `"."`): --- -## :rocket: Example usage +## Usage Example ```q -q)grafana:use`di.grafana -q)log:use`di.log / or define your own logger -q)grafana.init[enlist[`log]!enlist `info`warn`error!(log.info;log.warn;log.error)] +/ load and initialise with a printing logger +mylog:`info`warn`error!( + {[c;m] -1 "INFO [",string[c],"] ",m;}; + {[c;m] -1 "WARN [",string[c],"] ",m;}; + {[c;m] -2 "ERROR [",string[c],"] ",m;}); + +grafana:use`di.grafana +grafana.init[enlist[`log]!enlist mylog] -q)trade:([]time:.z.p-0D00:00:01*til 100;sym:100#`a`b`c;price:100?100f) +/ open an HTTP port and create some data with time + sym columns +\p 5000 +trade:([]time:.z.p-0D00:00:01*til 100;sym:100?`AAPL`MSFT`GOOG;price:100?100f) ``` -Add a SimPod JSON datasource in Grafana pointing at this process's HTTP port, +Add a SimPod JSON datasource in Grafana pointing at the process's HTTP port, then build panels using the metrics offered in the dropdowns (`t.trade`, -`g.trade.price`, …). +`g.trade.price`, ...). --- -## :test_tube: Tests - -Tests are written for k4unit and run with: +## Running Tests ```q -q)k4unit:use`di.k4unit -q)k4unit.moduletest`di.grafana +k4unit:use`di.k4unit +k4unit.moduletest`di.grafana ``` + +The suite injects a no-op mock logger and covers: the export surface; dependency +validation (non-dict deps, missing `log` key, null log value, missing log keys); +handler wiring and the install-once `wired` guard; config defaults and overrides +via `getconfig`; the target-parsing helpers (`isfunc`/`istab`/`istype`/`prefix`); +`finddistinctsyms`/`memvals`/`catchvals`; the `/query` table and timeseries +builders (`tbfunc`, and `tsfunc` including the per-column and per-sym graph +paths); the live `.z.pp`/`.z.ph` routes including non-Grafana fall-through; and a +`kx.log` integration test confirming the `normlog` wrapping works end-to-end. + +--- + +## Notes + +- The module installs `.z.pp`/`.z.ph` by closure-wrapping any pre-existing handler: Grafana requests (those carrying the `X-Grafana-Org-Id` header) are routed to the adaptor, and every other request falls through to the captured handler - so adding the module to a process that already serves HTTP does not break existing endpoints. This follows the `di.timer` precedent of a module owning a `.z.*` callback. +- The intended end-state is to register the handlers through an injected `di.handlers` dependency rather than assigning `.z.*` directly. That is deferred until `di.handlers` exists and supports return-yielding `.z.pp`/`.z.ph` chains - a side-effect-only chain cannot return the HTTP response. `di.handlers` is not implemented yet. +- Table panels return the last `ticks` rows (default 1000); raise `ticks` via config if a panel needs more +- Sym dropdowns list only syms seen within `timebackdate` (default 2 days) +- Data tables must expose the configured `timecol` and `sym` columns (defaults `` `time ``/`` `sym ``) +- The `/annotations` endpoint returns a "not yet implemented" marker, matching the TorQ original diff --git a/di/grafana/grafana.q b/di/grafana/grafana.q index 8c0d433f..f475ca36 100644 --- a/di/grafana/grafana.q +++ b/di/grafana/grafana.q @@ -222,6 +222,18 @@ tablesym:{[coln;rqt;symname] / initialisation / ----------------------------------------------------------------------------- +normlog:{[logdict] + / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[any `getlvl`sinks`fmts in key logdict; + `info`warn`error!( + {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); + logdict] + }; + sethandlers:{ / wrap any existing .z.pp/.z.ph so grafana requests are intercepted and every / other request falls through to the original handler @@ -234,27 +246,26 @@ sethandlers:{ init:{[deps] / wire the logging dependency and any configuration, then install the handlers - / deps - `log!enlist logdict or `log`config!(logdict;configdict) - / logdict - `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) - required - / configdict - optional overrides for `timecol`sym`timebackdate`ticks`del + / deps - a single dict holding the log dependency and any config overrides + / `log - `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) - required + / config keys - optional `timecol`sym`timebackdate`ticks`del, alongside `log / examples: / grafana.init[enlist[`log]!enlist mylog] - / grafana.init[`log`config!(mylog;enlist[`ticks]!enlist 500)] + / grafana.init[`log`ticks!(mylog;500)] / validate the required log dependency - nested guards, no eager `and` if[99h<>type deps;'"di.grafana: deps must be a dict with `log key"]; if[not`log in key deps;'"di.grafana: log dependency is required; pass `info`warn`error functions keyed on `log"]; if[99h<>type deps`log;'"di.grafana: log value must be a dict; pass `info`warn`error functions"]; - if[not all`info`warn`error in key deps`log;'"di.grafana: log dict must have `info`warn`error keys; got: ",", " sv string key deps`log]; - .z.m.log:deps`log; - / overlay any supplied configuration onto the defaults - config:$[`config in key deps;deps`config;()!()]; - if[99h=type config; - if[`timecol in key config;.z.m.timecol:config`timecol]; - if[`sym in key config;.z.m.sym:config`sym]; - if[`timebackdate in key config;.z.m.timebackdate:config`timebackdate]; - if[`ticks in key config;.z.m.ticks:config`ticks]; - if[`del in key config;.z.m.del:config`del]; - ]; + / normalise a kx.log instance into the binary {[c;m]} contract; plain dicts pass through + lg:normlog deps`log; + if[not all`info`warn`error in key lg;'"di.grafana: log dict must have `info`warn`error keys; got: ",", " sv string key lg]; + .z.m.log:lg; + / overlay any supplied config - config values sit alongside `log in deps + if[`timecol in key deps;.z.m.timecol:deps`timecol]; + if[`sym in key deps;.z.m.sym:deps`sym]; + if[`timebackdate in key deps;.z.m.timebackdate:deps`timebackdate]; + if[`ticks in key deps;.z.m.ticks:deps`ticks]; + if[`del in key deps;.z.m.del:deps`del]; / install the http handlers once, preserving any existing definitions if[not wired;sethandlers[]]; .z.m.log[`info][`grafana;"initialised grafana json datasource adaptor"]; diff --git a/di/grafana/test.csv b/di/grafana/test.csv index 07d59f88..3afa2c1f 100644 --- a/di/grafana/test.csv +++ b/di/grafana/test.csv @@ -38,6 +38,9 @@ comment,,,,,,,Timeseries request run,0,0,q,.test.tsrqt:`targets`range!((enlist `target`type!("g.trade";"timeserie"));`from`to!(("" sv (string .z.p-1D;enlist"Z"));("" sv (string .z.p+1D;enlist"Z")))),1,,build a graph timeseries request run,0,0,q,.test.ts:.m.di.0grafana.tsfunc .test.tsrqt,1,,run the timeseries builder true,0,0,q,.test.ts like "*price*",1,,timeseries response returns numeric column datapoints +run,0,0,q,.test.tsgrqt:`targets`range!((enlist `target`type!("g.trade.price";"timeserie"));.test.tsrqt`range),1,,build a per-sym graph request +run,0,0,q,.test.tsg:.m.di.0grafana.tsfunc .test.tsgrqt,1,,run the timeseries builder for graphsym +true,0,0,q,.test.tsg like "*\"target\":\"a\"*",1,,graphsym returns one datapoints series per sym comment,,,,,,,Handlers route grafana requests run,0,0,q,.test.sr:.z.pp (("search {}");(enlist `$"X-Grafana-Org-Id")!enlist ""),1,,post routed through installed .z.pp to search @@ -48,6 +51,12 @@ run,0,0,q,.test.ph:.z.ph (`;(enlist `$"X-Grafana-Org-Id")!enlist ""),1,,get rout true,0,0,q,.test.ph~"HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n",1,,get returns an alive response comment,,,,,,,Configuration override -run,0,0,q,grafana.init[`log`config!(mocklog;enlist[`ticks]!enlist 5)],1,,re-init with a config override +run,0,0,q,grafana.init[`log`ticks!(mocklog;5)],1,,re-init with a config override true,0,0,q,5=.m.di.0grafana.ticks,1,,ticks config overridden true,0,0,q,5=grafana.getconfig[][`ticks],1,,getconfig reflects the overridden ticks + +comment,,,,,,,kx.log instance normalisation +run,0,0,q,kxlog:`getlvl`sinks`fmts`info`warn`error!((::);(::);(::);{[m]};{[m]};{[m]}),1,,build a fake kx.log instance with monadic loggers +run,0,0,q,grafana.init[enlist[`log]!enlist kxlog],1,,init accepts a kx.log instance via normlog +true,0,0,q,(`error`info`warn)~asc key .m.di.0grafana.log,1,,normlog wrapped kx.log into the info/warn/error contract +true,0,0,q,(::)~.m.di.0grafana.log[`info][`ctx;"msg"],1,,wrapped logger is callable as a binary {[c;m]} From 0d282121f94b0b174ab4324d3fd4f459431a7456 Mon Sep 17 00:00:00 2001 From: Olly99999 Date: Mon, 29 Jun 2026 13:41:53 +0100 Subject: [PATCH 5/8] Making changes following PR auto-reviewer whilst keeping in line with peer reviews --- di/grafana/grafana.md | 26 +++++++++++-- di/grafana/grafana.q | 87 +++++++++++++++++++++++++++---------------- di/grafana/test.csv | 14 ++++++- 3 files changed, 90 insertions(+), 37 deletions(-) diff --git a/di/grafana/grafana.md b/di/grafana/grafana.md index 845ad743..db4a1c38 100644 --- a/di/grafana/grafana.md +++ b/di/grafana/grafana.md @@ -50,9 +50,9 @@ kxlog:use`kx.log grafana.init[enlist[`log]!enlist kxlog.createLog[]] ``` -Configuration keys `timecol`, `sym`, `timebackdate`, `ticks`, and `del` are all -optional - omit any or all of them and the module falls back to sensible -defaults. See Initialisation for full details. +Configuration keys `timecol`, `sym`, `timebackdate`, `ticks`, `del`, and +`allowfunctions` are all optional - omit any or all of them and the module falls +back to sensible defaults. See Initialisation for full details. --- @@ -69,6 +69,7 @@ configuration overrides. | `` `timebackdate `` | no | How far back to look when finding distinct syms for the dropdowns. Default: `2D` | | `` `ticks `` | no | Number of rows returned for a table request. Default: `1000` | | `` `del `` | no | Delimiter separating the arguments within a query target. Default: `"."` | +| `` `allowfunctions `` | no | Enable `f.` function targets, which evaluate arbitrary q (see Security note). Default: `0b` (disabled) | `init` must be called before the module will serve any requests. It validates the logger, applies config, and installs the `.z.pp`/`.z.ph` handlers - once; @@ -115,7 +116,7 @@ and arguments, separated by `del` (default `"."`): | `g.
` | graph panel, one series per numeric column | | `g.
.` | graph panel, one series per sym for a column | | `o.
...` | "other" panels (single-stat, gauge, etc.) | -| `f.<...>` | the target is a function call rather than a table | +| `f.<...>` | the target is a q expression to evaluate rather than a table - **disabled unless `allowfunctions` is set** (see Security) | --- @@ -160,6 +161,23 @@ paths); the live `.z.pp`/`.z.ph` routes including non-Grafana fall-through; and --- +## Security + +The module answers any HTTP request carrying the `X-Grafana-Org-Id` header, so +treat its port as a trust boundary - put it behind Grafana's authentication and +firewall it from untrusted networks. + +- **Table targets are resolved by name, never evaluated.** `t.`/`g.`/`o.` and + bare targets are looked up as symbols against `tables[]`; an unknown name is + rejected. A target string can therefore never be executed as q code through + these paths. +- **`f.` function targets evaluate arbitrary q and are disabled by default.** + They run only when `allowfunctions` is set to `1b` in `init`. Enable it solely + on processes where every client able to reach the port is trusted, since an + `f.` target is, by design, remote code execution. + +--- + ## Notes - The module installs `.z.pp`/`.z.ph` by closure-wrapping any pre-existing handler: Grafana requests (those carrying the `X-Grafana-Org-Id` header) are routed to the adaptor, and every other request falls through to the captured handler - so adding the module to a process that already serves HTTP does not break existing endpoints. This follows the `di.timer` precedent of a module owning a `.z.*` callback. diff --git a/di/grafana/grafana.q b/di/grafana/grafana.q index f475ca36..4ae790e9 100644 --- a/di/grafana/grafana.q +++ b/di/grafana/grafana.q @@ -4,7 +4,8 @@ / the grafana dropdowns and the /query endpoint returns timeseries or table data / ----------------------------------------------------------------------------- -/ configuration - defaults applied here, overridden via the config dict in init +/ configuration - load-time defaults; overridden via the deps dict in init and +/ read back through .z.m at every call site / ----------------------------------------------------------------------------- / name of the time column used for timeseries queries @@ -17,6 +18,8 @@ timebackdate:2D; ticks:1000; / delimiter separating the arguments within a query target del:"."; +/ allow f. function targets to evaluate arbitrary q (off by default - see grafana.md) +allowfunctions:0b; / json type for each kdb datatype, keyed by .Q.t character types:.Q.t!`array`boolean,(3#`null),(5#`number),11#`string; @@ -55,19 +58,19 @@ query:{[rqt] search:{[rqt] / build the grafana dropdown options from the tables available in the process tabs:tables[]; - symtabs:tabs where sym in'cols each tabs; - timetabs:tabs where timecol in'cols each tabs; + symtabs:tabs where .z.m.sym in'cols each tabs; + timetabs:tabs where .z.m.timecol in'cols each tabs; rsp:string tabs; if[count timetabs; rsp,:s1:prefix["t";string timetabs]; rsp,:s2:prefix["g";string timetabs]; / suffix the numeric columns for the graph and other panel options - rsp,:raze(s2,'del),/:'c1:string {cols[x] where`number=types(0!meta x)`t}each timetabs; - rsp,:raze(prefix["o";string timetabs],'del),/:'c1; + rsp,:raze(s2,'.z.m.del),/:'c1:string {cols[x] where`number=types(0!meta x)`t}each timetabs; + rsp,:raze(prefix["o";string timetabs],'.z.m.del),/:'c1; if[count symtabs; / suffix the distinct syms for the timeseries and other panel options - rsp,:raze(s1,'del),/:'c2:string each finddistinctsyms'[timetabs]; - rsp,:raze(prefix["o";string timetabs],'del),/:'{x[0]cross del,'string finddistinctsyms x 1}each(enlist each c1),'timetabs; + rsp,:raze(s1,'.z.m.del),/:'c2:string each finddistinctsyms'[timetabs]; + rsp,:raze(prefix["o";string timetabs],'.z.m.del),/:'{x[0]cross .z.m.del,'string finddistinctsyms x 1}each(enlist each c1),'timetabs; ]; ]; :.h.hy[`json].j.j rsp; @@ -75,12 +78,12 @@ search:{[rqt] finddistinctsyms:{[x] / distinct syms seen in table x within the configured lookback window - :?[x;enlist(>;timecol;(-;.z.p;timebackdate));1b;{x!x}enlist sym]sym; + :?[x;enlist(>;.z.m.timecol;(-;.z.p;.z.m.timebackdate));1b;{x!x}enlist .z.m.sym] .z.m.sym; }; prefix:{[c;s] / prefix string c and the delimiter to each string in s - :(c,del),/:s; + :(c,.z.m.del),/:s; }; / ----------------------------------------------------------------------------- @@ -89,13 +92,13 @@ prefix:{[c;s] diskvals:{[x] / last `ticks` rows of an on-disk partitioned table - c:(count[x]-ticks)+til ticks; + c:(count[x]-.z.m.ticks)+til .z.m.ticks; :get'[.Q.ind[x;c]]; }; memvals:{[x] / last `ticks` rows of an in-memory table - :get'[?[x;enlist(within;`i;count[x]-ticks,0);0b;()]]; + :get'[?[x;enlist(within;`i;count[x]-.z.m.ticks,0);0b;()]]; }; catchvals:{[x] @@ -109,11 +112,29 @@ catchvals:{[x] istype:{[targ;char] / test whether the target is prefixed with char followed by the delimiter - :(char,del)~2#targ; + :(char,.z.m.del)~2#targ; }; isfunc:istype[;"f"]; istab:istype[;"t"]; +resolvetab:{[t] + / resolve a table-name symbol to its unkeyed table, rejecting anything that is + / not a known table - the name is looked up, never evaluated as code + if[not t in tables[]; + .z.m.log[`error][`grafana;"unknown table: ",string t]; + '"di.grafana: unknown table ",string t]; + :0!value t; + }; + +evalfunc:{[s] + / evaluate an f. function-target expression - gated behind allowfunctions + / because it executes arbitrary q (disabled by default) + if[not .z.m.allowfunctions; + .z.m.log[`error][`grafana;"function target rejected; allowfunctions is disabled"]; + '"di.grafana: function targets are disabled (set allowfunctions to enable)"]; + :value s; + }; + / ----------------------------------------------------------------------------- / building json responses / ----------------------------------------------------------------------------- @@ -127,16 +148,15 @@ tbfunc:{[rqt] / process a table request and return the json datasource table response rqt:raze rqt[`targets]`target; symname:0b; - / strip the type prefix: f.t.func drops 4, f.func drops 2, t.tab leaves the - / table name plus an optional sym - rqt:0!value $[isfunc[rqt]&istab 2_rqt;4_rqt; - isfunc rqt;2_rqt; - istab rqt;[rqt:`$del vs rqt;if[2meta[rqt][timecol;`t];rqt:@[rqt;timecol;+;.z.D]]; + if["p"<>meta[rqt][.z.m.timecol;`t];rqt:@[rqt;.z.m.timecol;+;.z.D]]; / restrict to the time range requested by grafana range:"P"$-1_'x[`range]`from`to; - rqt:?[rqt;enlist(within;timecol;range);0b;()]; + rqt:?[rqt;enlist(within;.z.m.timecol;range);0b;()]; / add the milliseconds-since-epoch column grafana expects - rqt:@[rqt;`msec;:;mil rqt timecol]; + rqt:@[rqt;`msec;:;mil rqt .z.m.timecol]; / dispatch on the number of arguments and the panel type $[(2 Date: Tue, 30 Jun 2026 16:36:25 +0100 Subject: [PATCH 6/8] made further changes according to auto PR reviewer --- di/grafana/grafana.md | 1 + di/grafana/grafana.q | 23 +++++++++++++++++------ di/grafana/test.csv | 12 +++++++++++- 3 files changed, 29 insertions(+), 7 deletions(-) diff --git a/di/grafana/grafana.md b/di/grafana/grafana.md index db4a1c38..2ec9e366 100644 --- a/di/grafana/grafana.md +++ b/di/grafana/grafana.md @@ -182,6 +182,7 @@ firewall it from untrusted networks. - The module installs `.z.pp`/`.z.ph` by closure-wrapping any pre-existing handler: Grafana requests (those carrying the `X-Grafana-Org-Id` header) are routed to the adaptor, and every other request falls through to the captured handler - so adding the module to a process that already serves HTTP does not break existing endpoints. This follows the `di.timer` precedent of a module owning a `.z.*` callback. - The intended end-state is to register the handlers through an injected `di.handlers` dependency rather than assigning `.z.*` directly. That is deferred until `di.handlers` exists and supports return-yielding `.z.pp`/`.z.ph` chains - a side-effect-only chain cannot return the HTTP response. `di.handlers` is not implemented yet. +- A panel with multiple queries works: each target in the request is dispatched by its own type and the per-target results are merged into one JSON array - Table panels return the last `ticks` rows (default 1000); raise `ticks` via config if a panel needs more - Sym dropdowns list only syms seen within `timebackdate` (default 2 days) - Data tables must expose the configured `timecol` and `sym` columns (defaults `` `time ``/`` `sym ``) diff --git a/di/grafana/grafana.q b/di/grafana/grafana.q index 4ae790e9..2e245ea7 100644 --- a/di/grafana/grafana.q +++ b/di/grafana/grafana.q @@ -35,7 +35,9 @@ wired:0b; zpp:{[x] / parse a grafana http post request and dispatch to the matching handler - / cut at the first whitespace to isolate the api url from any function params + / kdb passes .z.pp (requeststring;headers); requeststring is " " + / (method and leading / already stripped), so cut at the first space gives + / (endpoint;body) - e.g. "query {...}" -> ("query";"{...}") r:(0;n?" ")cut n:first x; rqt:.j.k r 1; .z.m.log[`info][`grafana;"received ",(r 0)," request"]; @@ -50,9 +52,17 @@ annotation:{[rqt] }; query:{[rqt] - / dispatch a /query request to the timeseries or table builder by target type - rqtype:raze rqt[`targets]`type; - :.h.hy[`json]$[rqtype~"timeserie";tsfunc rqt;tbfunc rqt]; + / dispatch each target to the timeseries or table builder by its own type and + / merge the per-target json arrays - grafana can send several targets at once + tgts:rqt`targets; + / a single target may arrive as a bare dict; normalise to a list of targets + tgts:$[99h=type tgts;enlist tgts;tgts]; + / iterate by index so a list or a table of targets is handled the same way + build:{[rqt;tgts;i]t:tgts i;r:@[rqt;`targets;:;t];$[t[`type]~"timeserie";tsfunc r;tbfunc r]}; + / strip the [ ] from each per-target array, drop empties, re-wrap as one array + inners:{1_-1_x}each build[rqt;tgts]each til count tgts; + inners:inners where 0meta[rqt][.z.m.timecol;`t];rqt:@[rqt;.z.m.timecol;+;.z.D]]; - / restrict to the time range requested by grafana - range:"P"$-1_'x[`range]`from`to; + / restrict to the time range requested by grafana - grafana sends iso-8601 utc; + / strip a trailing Z only if present rather than blindly dropping the last char + range:"P"${$["Z"=last x;-1_x;x]}each x[`range]`from`to; rqt:?[rqt;enlist(within;.z.m.timecol;range);0b;()]; / add the milliseconds-since-epoch column grafana expects rqt:@[rqt;`msec;:;mil rqt .z.m.timecol]; diff --git a/di/grafana/test.csv b/di/grafana/test.csv index 9ce95ec4..81685375 100644 --- a/di/grafana/test.csv +++ b/di/grafana/test.csv @@ -35,7 +35,7 @@ true,0,0,q,.test.tb like "*columns*",1,,table response carries column metadata true,0,0,q,.test.tb like "*table*",1,,table response declares table type comment,,,,,,,Timeseries request -run,0,0,q,.test.tsrqt:`targets`range!((enlist `target`type!("g.trade";"timeserie"));`from`to!(("" sv (string .z.p-1D;enlist"Z"));("" sv (string .z.p+1D;enlist"Z")))),1,,build a graph timeseries request +run,0,0,q,.test.tsrqt:`targets`range!((enlist `target`type!("g.trade";"timeserie"));`from`to!("2000-01-01T00:00:00.000Z";"2099-01-01T00:00:00.000Z")),1,,build a graph timeseries request with a real grafana ISO-8601 range run,0,0,q,.test.ts:.m.di.0grafana.tsfunc .test.tsrqt,1,,run the timeseries builder true,0,0,q,.test.ts like "*price*",1,,timeseries response returns numeric column datapoints run,0,0,q,.test.tsgrqt:`targets`range!((enlist `target`type!("g.trade.price";"timeserie"));.test.tsrqt`range),1,,build a per-sym graph request @@ -45,6 +45,10 @@ true,0,0,q,.test.tsg like "*\"target\":\"a\"*",1,,graphsym returns one datapoint comment,,,,,,,Handlers route grafana requests run,0,0,q,.test.sr:.z.pp (("search {}");(enlist `$"X-Grafana-Org-Id")!enlist ""),1,,post routed through installed .z.pp to search true,0,0,q,.test.sr like "*trade*",1,,search response lists the test table +run,0,0,q,.test.qtgt:`target`type!("t.trade";"table"),1,,build a query target +run,0,0,q,.test.qbody:.j.j enlist[`targets]!enlist .test.qtgt,1,,serialise the query body as a client would +run,0,0,q,.test.qresp:.z.pp (("" sv ("query ";.test.qbody));(enlist `$"X-Grafana-Org-Id")!enlist "1"),1,,real-shape query posted through installed .z.pp +true,0,0,q,.test.qresp like "*columns*",1,,query dispatched to the table builder not annotation run,0,0,q,.m.di.0grafana.prevpp:{"fellthrough"},1,,install a sentinel as the pre-existing post handler true,0,0,q,.z.pp[(("ignored");()!())]~"fellthrough",1,,non-grafana post falls through to the pre-existing handler run,0,0,q,.test.ph:.z.ph (`;(enlist `$"X-Grafana-Org-Id")!enlist ""),1,,get routed through installed .z.ph @@ -72,3 +76,9 @@ true,0,0,q,.m.di.0grafana.allowfunctions,1,,allowfunctions enabled run,0,0,q,.test.fnrqt:enlist[`targets]!enlist (enlist[`target]!enlist "f.trade"),1,,build a function target returning a table run,0,0,q,.test.fnr:.m.di.0grafana.tbfunc .test.fnrqt,1,,function target evaluates when enabled true,0,0,q,.test.fnr like "*columns*",1,,enabled function target returns a table response + +comment,,,,,,,Multi-target query requests (grafana sends targets as an array) +run,0,0,q,.test.mt:`target`type!("t.trade";"table"),1,,a single target object +run,0,0,q,.test.mtrqt:enlist[`targets]!enlist (.test.mt;.test.mt),1,,a request carrying two targets +run,0,0,q,.test.mtresp:.m.di.0grafana.query .test.mtrqt,1,,run the multi-target query +true,0,0,q,1 Date: Wed, 1 Jul 2026 11:58:02 +0100 Subject: [PATCH 7/8] Ran a live test and confirmed working, added a LIVE_TESTING.md for instructions --- di/grafana/LIVE_TESTING.md | 196 +++++++++++++++++++++++++++++++++++++ di/grafana/grafana.q | 10 +- di/grafana/test.csv | 8 ++ 3 files changed, 212 insertions(+), 2 deletions(-) create mode 100644 di/grafana/LIVE_TESTING.md diff --git a/di/grafana/LIVE_TESTING.md b/di/grafana/LIVE_TESTING.md new file mode 100644 index 00000000..d6185dbd --- /dev/null +++ b/di/grafana/LIVE_TESTING.md @@ -0,0 +1,196 @@ +# di.grafana — Live verification (manual) + +An end-to-end check of `di.grafana` against a **real** Grafana instance: browser → +Grafana → HTTP (with the `X-Grafana-Org-Id` header) → the module's `.z.pp`/`.z.ph` +handlers → `search`/`query` → JSON → a rendered panel. + +This is a **manual** procedure for confidence/demos — it is **not** part of the +automated test suite (`k4unit.moduletest\`di.grafana`), which already covers the +module's logic in-process. Use this to see it work in the real UI. + +> **Verified working:** 2026-07-01 — Grafana **10.4.14** + `grafana-simple-json-datasource` +> v1.4.2, KDB-X 5.0, module on `di.grafana-pr`. Live panel rendered a moving +> per-sym price series driven by a kdb feed. + +--- + +## ⚠️ Grafana version compatibility (read this first) + +The module implements the **classic Grafana SimpleJSON protocol** — `/search`, +`/query`, `/annotations`, and a `GET /` health check. The only Grafana datasource +plugin that speaks this protocol natively is **`grafana-simple-json-datasource`**, +which is **Angular-based and deprecated**. Grafana has been phasing Angular out: + +| Grafana version | Status of the plugin | +|---|---| +| **≤ 10.x** | Angular enabled by default — plugin works out of the box ✅ | +| **11.x** | Angular disabled by default — plugin loads **only** if Angular is re-enabled (`[plugins] angular_support_enabled = true`, or env `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true`) ⚠️ | +| **≥ 12.x** | Angular support **removed entirely** — the plugin will not load at all ❌ | + +**This guide pins Grafana 10.4.14** because Angular is on by default there — the +least friction. On Grafana 11 you must re-enable Angular (see Troubleshooting). + +### Future note — when you must move to Grafana ≥ 12 + +`grafana-simple-json-datasource` is a dead end from Grafana 12 onward. Options then: + +- **Infinity** (`yesoreyeram-infinity-datasource`, React, actively maintained): + point it at `POST http://:/query` with a JSON body and a custom + `X-Grafana-Org-Id: 1` header, and configure its parser to read the `datapoints`. + Works, but it's manual per-panel config — no metric dropdown from `/search`. +- **SimPod** (`simpod-json-datasource`, React): non-Angular, but newer versions + call `/metrics` instead of `/search`, so the metric dropdown won't populate + (the module serves `/search`); a manually-typed target may still hit `/query`. +- **Extend the module** to also serve the endpoints a modern React plugin expects + (e.g. `/metrics`) — a code change; weigh against keeping the module lean. + +The module itself is protocol-correct; this is purely about which Grafana plugin +can consume it. If this doc stops working after a Grafana upgrade, the cause is +almost certainly the Angular removal above, not the module. + +--- + +## Overview + +Two terminals plus a browser. Ports used below: **kdb HTTP = `6702`**, **Grafana +UI = `3000`** (change either if busy). + +--- + +## 1. kdb process (Terminal 1) + +Start kdb-x from the module repo so `use\`di.grafana` resolves, then wire it up as +a datasource with a small live feed: + +```bash +cd ~/kdb_x/kdbx-modules-memstats-rename +export QPATH=$(pwd):$QPATH +q # your KDB-X launcher +``` + +```q +\p 6702 / listen for HTTP + +/ a logger that prints, so incoming requests are visible +mylog:`info`warn`error!( + {[c;m] -1 "INFO [",string[c],"] ",m;}; + {[c;m] -1 "WARN [",string[c],"] ",m;}; + {[c;m] -2 "ERROR [",string[c],"] ",m;}); + +grafana:use`di.grafana +grafana.init[enlist[`log]!enlist mylog] + +/ ~16 min of recent data + a 1/sec live feed so the graph moves +trade:([]time:.z.p-0D00:00:01*til 1000;sym:1000?`AAPL`MSFT`GOOG;price:1000?100f;size:1000?500) +.z.ts:{`trade insert (.z.p;rand`AAPL`MSFT`GOOG;rand 100f;rand 500)}; system"t 1000" +``` + +Leave this session running. (`.z.ts` is yours; the module only owns `.z.pp`/`.z.ph`.) + +--- + +## 2. Grafana 10.4 (Terminal 2, no root required) + +Runs entirely from your home directory — no Docker, no `sudo`: + +```bash +cd ~ +wget https://dl.grafana.com/oss/release/grafana-10.4.14.linux-amd64.tar.gz +tar -zxf grafana-10.4.14.linux-amd64.tar.gz +cd grafana-v10.4.14 +mkdir -p data/plugins + +# install the SimpleJson plugin into a local (writable) dir +./bin/grafana-cli --pluginsDir "$PWD/data/plugins" plugins install grafana-simple-json-datasource + +# run it, all paths under $HOME +GF_SERVER_HTTP_PORT=3000 \ +GF_PATHS_DATA="$PWD/data" \ +GF_PATHS_LOGS="$PWD/data/log" \ +GF_PATHS_PLUGINS="$PWD/data/plugins" \ +GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=grafana-simple-json-datasource \ +./bin/grafana-server +``` + +Wait for both of these in the log (and **no** "Refusing to initialize … Angular"): + +``` +Plugin registered … pluginId=grafana-simple-json-datasource +HTTP Server Listen … :3000 +``` + +--- + +## 3. If Grafana is on a remote server (SSH / VS Code Remote) + +Grafana listens on the **server's** `localhost:3000`, which your local browser +can't reach directly. Forward the UI port: + +- **VS Code Remote:** open the **PORTS** panel → **Forward a Port** → `3000` → + open the resulting local address. +- **Plain SSH:** `ssh -L 3000:localhost:3000 @` then browse `http://localhost:3000`. + +Only the **Grafana UI port** needs forwarding. The datasource URL +`http://localhost:6702` is resolved by Grafana's **backend on the server** (that's +what "Access: Server" means), so kdb's port never leaves the server — do **not** +forward 6702. + +--- + +## 4. Add the datasource + +1. `http://localhost:3000` → login **admin / admin** (skip the password prompt). +2. **Connections → Data sources → Add data source →** search **"SimpleJson"** → select it. +3. **URL:** `http://localhost:6702` · **Access: Server (default)**. + - **Access must be Server** — that is what makes Grafana's backend attach the + `X-Grafana-Org-Id` header the module keys on. +4. **Save & Test** → expect green **"Data source is working"** (a `GET /` → `.z.ph` + → `200 OK`; visible in the q console). + +--- + +## 5. Build a panel + +1. **Dashboards → New → New dashboard → + Add visualization →** pick the datasource. +2. Open the **Metric** dropdown (populated by `/search`): `trade`, `t.trade`, + `g.trade`, `g.trade.price`, `g.trade.size`, `t.trade.AAPL`, … +3. Select **`g.trade.price`**; set the time range to **Last 15 minutes** + (optionally auto-refresh 5s to watch the live feed). + +Exercise the other paths too: +- **`g.trade`** — one line per numeric column +- **`t.trade`** — switch the panel type to **Table** for the whole table +- **`t.trade.AAPL`** — table filtered to one sym + +--- + +## What success looks like + +| Signal | Confirms | +|---|---| +| **Save & Test** green | GET handler + connectivity | +| Metric **dropdown populates** | `/search` | +| Panel shows **live lines** (AAPL/MSFT/GOOG) | `/query` → `tsfunc` → per-sym builder, end-to-end | +| `INFO [grafana] received … request` in Terminal 1 | requests flowing through `.z.pp` | + +--- + +## Teardown + +- Grafana: **Ctrl-C** in Terminal 2. +- kdb: `system"t 0"` to stop the feed, then exit q. + +--- + +## Troubleshooting + +| Symptom | Cause / fix | +|---|---| +| `Refusing to initialize plugin … Angular` in the Grafana log | You're on Grafana 11 — use the 10.4 tarball here, or set `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true` (config `[plugins] angular_support_enabled = true`). On Grafana ≥ 12 it can't be enabled — see "Future note" above. | +| `docker: permission denied … docker.sock` | Not in the `docker` group. Use the **binary** method here (no Docker), or `sudo docker …`. | +| `bind: address already in use` on start | UI port taken — set `GF_SERVER_HTTP_PORT` to a free port (e.g. `6905`). | +| Plugin install: `permission denied … /var/lib/grafana/plugins` | The CLI defaulted to the system dir — pass `--pluginsDir "$PWD/data/plugins"`. | +| Browser: page won't load (remote server) | Forward the UI port over SSH — see section 3. | +| **Save & Test: "Bad Gateway"** | Grafana's backend couldn't reach/parse kdb. From the server: `curl -i -H "X-Grafana-Org-Id: 1" http://localhost:6702/` — `Connection refused` means the kdb session isn't listening on 6702 (restart it); anything non-`200` means the wrong handler is wired. | +| Panel empty / "No data" | Time range doesn't overlap the data — use **Last 15 minutes** (data is `.z.p`-recent). | +| Metric dropdown empty | Wrong plugin — must be **SimpleJson** (`/search`+`/query`), not the newer SimPod (`/metrics`). | diff --git a/di/grafana/grafana.q b/di/grafana/grafana.q index 2e245ea7..883bf4c3 100644 --- a/di/grafana/grafana.q +++ b/di/grafana/grafana.q @@ -267,13 +267,19 @@ normlog:{[logdict] logdict] }; +isgrafana:{[x] + / true if the request's header dict carries the X-Grafana-Org-Id header; guard + / the type so a non-dict last element (malformed/non-http arg) falls through + $[99h=type h:last x;(`$"X-Grafana-Org-Id")in key h;0b] + }; + sethandlers:{ / wrap any existing .z.pp/.z.ph so grafana requests are intercepted and every / other request falls through to the original handler .z.m.prevpp:$[@[{value x;1b};`.z.pp;0b];.z.pp;{[x]}]; - .z.pp:{[x]$[(`$"X-Grafana-Org-Id")in key last x;.z.m.zpp x;.z.m.prevpp x]}; + .z.pp:{[x]$[.z.m.isgrafana x;.z.m.zpp x;.z.m.prevpp x]}; .z.m.prevph:$[@[{value x;1b};`.z.ph;0b];.z.ph;{[x]}]; - .z.ph:{[x]$[(`$"X-Grafana-Org-Id")in key last x;"HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n";.z.m.prevph x]}; + .z.ph:{[x]$[.z.m.isgrafana x;"HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n";.z.m.prevph x]}; .z.m.wired:1b; }; diff --git a/di/grafana/test.csv b/di/grafana/test.csv index 81685375..0c13aabf 100644 --- a/di/grafana/test.csv +++ b/di/grafana/test.csv @@ -51,6 +51,7 @@ run,0,0,q,.test.qresp:.z.pp (("" sv ("query ";.test.qbody));(enlist `$"X-Grafana true,0,0,q,.test.qresp like "*columns*",1,,query dispatched to the table builder not annotation run,0,0,q,.m.di.0grafana.prevpp:{"fellthrough"},1,,install a sentinel as the pre-existing post handler true,0,0,q,.z.pp[(("ignored");()!())]~"fellthrough",1,,non-grafana post falls through to the pre-existing handler +true,0,0,q,.z.pp["rawstring"]~"fellthrough",1,,malformed (non-dict header) request falls through without erroring run,0,0,q,.test.ph:.z.ph (`;(enlist `$"X-Grafana-Org-Id")!enlist ""),1,,get routed through installed .z.ph true,0,0,q,.test.ph~"HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n",1,,get returns an alive response @@ -77,6 +78,13 @@ run,0,0,q,.test.fnrqt:enlist[`targets]!enlist (enlist[`target]!enlist "f.trade") run,0,0,q,.test.fnr:.m.di.0grafana.tbfunc .test.fnrqt,1,,function target evaluates when enabled true,0,0,q,.test.fnr like "*columns*",1,,enabled function target returns a table response +comment,,,,,,,Bare (unprefixed) targets resolve by name and are never evaluated +run,0,0,q,.test.bareok:.m.di.0grafana.tbfunc enlist[`targets]!enlist (enlist[`target]!enlist "trade"),1,,bare table name resolves via lookup +true,0,0,q,.test.bareok like "*columns*",1,,bare valid target returns the table +run,0,0,q,.test.pwned:0b,1,,sentinel to detect code execution +run,0,0,q,@[.m.di.0grafana.tbfunc;enlist[`targets]!enlist (enlist[`target]!enlist ".test.pwned:1b");{x}],1,,bare q-expression target is rejected +true,0,0,q,not .test.pwned,1,,the bare target string was not executed + comment,,,,,,,Multi-target query requests (grafana sends targets as an array) run,0,0,q,.test.mt:`target`type!("t.trade";"table"),1,,a single target object run,0,0,q,.test.mtrqt:enlist[`targets]!enlist (.test.mt;.test.mt),1,,a request carrying two targets From dfb448c584138ef09c670f98a39b1ae51661cf1f Mon Sep 17 00:00:00 2001 From: Olly99999 Date: Wed, 1 Jul 2026 16:35:12 +0100 Subject: [PATCH 8/8] update md --- di/grafana/LIVE_TESTING.md | 48 +++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/di/grafana/LIVE_TESTING.md b/di/grafana/LIVE_TESTING.md index d6185dbd..027da2ba 100644 --- a/di/grafana/LIVE_TESTING.md +++ b/di/grafana/LIVE_TESTING.md @@ -1,14 +1,14 @@ -# di.grafana — Live verification (manual) +# di.grafana - Live verification (manual) An end-to-end check of `di.grafana` against a **real** Grafana instance: browser → Grafana → HTTP (with the `X-Grafana-Org-Id` header) → the module's `.z.pp`/`.z.ph` handlers → `search`/`query` → JSON → a rendered panel. -This is a **manual** procedure for confidence/demos — it is **not** part of the +This is a **manual** procedure for confidence/demos - it is **not** part of the automated test suite (`k4unit.moduletest\`di.grafana`), which already covers the module's logic in-process. Use this to see it work in the real UI. -> **Verified working:** 2026-07-01 — Grafana **10.4.14** + `grafana-simple-json-datasource` +> **Verified working:** 2026-07-01 - Grafana **10.4.14** + `grafana-simple-json-datasource` > v1.4.2, KDB-X 5.0, module on `di.grafana-pr`. Live panel rendered a moving > per-sym price series driven by a kdb feed. @@ -16,33 +16,33 @@ module's logic in-process. Use this to see it work in the real UI. ## ⚠️ Grafana version compatibility (read this first) -The module implements the **classic Grafana SimpleJSON protocol** — `/search`, +The module implements the **classic Grafana SimpleJSON protocol** - `/search`, `/query`, `/annotations`, and a `GET /` health check. The only Grafana datasource plugin that speaks this protocol natively is **`grafana-simple-json-datasource`**, which is **Angular-based and deprecated**. Grafana has been phasing Angular out: | Grafana version | Status of the plugin | |---|---| -| **≤ 10.x** | Angular enabled by default — plugin works out of the box ✅ | -| **11.x** | Angular disabled by default — plugin loads **only** if Angular is re-enabled (`[plugins] angular_support_enabled = true`, or env `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true`) ⚠️ | -| **≥ 12.x** | Angular support **removed entirely** — the plugin will not load at all ❌ | +| **≤ 10.x** | Angular enabled by default - plugin works out of the box ✅ | +| **11.x** | Angular disabled by default - plugin loads **only** if Angular is re-enabled (`[plugins] angular_support_enabled = true`, or env `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true`) ⚠️ | +| **≥ 12.x** | Angular support **removed entirely** - the plugin will not load at all ❌ | -**This guide pins Grafana 10.4.14** because Angular is on by default there — the +**This guide pins Grafana 10.4.14** because Angular is on by default there - the least friction. On Grafana 11 you must re-enable Angular (see Troubleshooting). -### Future note — when you must move to Grafana ≥ 12 +### Future note - when you must move to Grafana ≥ 12 `grafana-simple-json-datasource` is a dead end from Grafana 12 onward. Options then: - **Infinity** (`yesoreyeram-infinity-datasource`, React, actively maintained): point it at `POST http://:/query` with a JSON body and a custom `X-Grafana-Org-Id: 1` header, and configure its parser to read the `datapoints`. - Works, but it's manual per-panel config — no metric dropdown from `/search`. + Works, but it's manual per-panel config - no metric dropdown from `/search`. - **SimPod** (`simpod-json-datasource`, React): non-Angular, but newer versions call `/metrics` instead of `/search`, so the metric dropdown won't populate (the module serves `/search`); a manually-typed target may still hit `/query`. - **Extend the module** to also serve the endpoints a modern React plugin expects - (e.g. `/metrics`) — a code change; weigh against keeping the module lean. + (e.g. `/metrics`) - a code change; weigh against keeping the module lean. The module itself is protocol-correct; this is purely about which Grafana plugin can consume it. If this doc stops working after a Grafana upgrade, the cause is @@ -91,7 +91,7 @@ Leave this session running. (`.z.ts` is yours; the module only owns `.z.pp`/`.z. ## 2. Grafana 10.4 (Terminal 2, no root required) -Runs entirely from your home directory — no Docker, no `sudo`: +Runs entirely from your home directory - no Docker, no `sudo`: ```bash cd ~ @@ -132,7 +132,7 @@ can't reach directly. Forward the UI port: Only the **Grafana UI port** needs forwarding. The datasource URL `http://localhost:6702` is resolved by Grafana's **backend on the server** (that's -what "Access: Server" means), so kdb's port never leaves the server — do **not** +what "Access: Server" means), so kdb's port never leaves the server - do **not** forward 6702. --- @@ -142,7 +142,7 @@ forward 6702. 1. `http://localhost:3000` → login **admin / admin** (skip the password prompt). 2. **Connections → Data sources → Add data source →** search **"SimpleJson"** → select it. 3. **URL:** `http://localhost:6702` · **Access: Server (default)**. - - **Access must be Server** — that is what makes Grafana's backend attach the + - **Access must be Server** - that is what makes Grafana's backend attach the `X-Grafana-Org-Id` header the module keys on. 4. **Save & Test** → expect green **"Data source is working"** (a `GET /` → `.z.ph` → `200 OK`; visible in the q console). @@ -158,9 +158,9 @@ forward 6702. (optionally auto-refresh 5s to watch the live feed). Exercise the other paths too: -- **`g.trade`** — one line per numeric column -- **`t.trade`** — switch the panel type to **Table** for the whole table -- **`t.trade.AAPL`** — table filtered to one sym +- **`g.trade`** - one line per numeric column +- **`t.trade`** - switch the panel type to **Table** for the whole table +- **`t.trade.AAPL`** - table filtered to one sym --- @@ -186,11 +186,11 @@ Exercise the other paths too: | Symptom | Cause / fix | |---|---| -| `Refusing to initialize plugin … Angular` in the Grafana log | You're on Grafana 11 — use the 10.4 tarball here, or set `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true` (config `[plugins] angular_support_enabled = true`). On Grafana ≥ 12 it can't be enabled — see "Future note" above. | +| `Refusing to initialize plugin … Angular` in the Grafana log | You're on Grafana 11 - use the 10.4 tarball here, or set `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true` (config `[plugins] angular_support_enabled = true`). On Grafana ≥ 12 it can't be enabled - see "Future note" above. | | `docker: permission denied … docker.sock` | Not in the `docker` group. Use the **binary** method here (no Docker), or `sudo docker …`. | -| `bind: address already in use` on start | UI port taken — set `GF_SERVER_HTTP_PORT` to a free port (e.g. `6905`). | -| Plugin install: `permission denied … /var/lib/grafana/plugins` | The CLI defaulted to the system dir — pass `--pluginsDir "$PWD/data/plugins"`. | -| Browser: page won't load (remote server) | Forward the UI port over SSH — see section 3. | -| **Save & Test: "Bad Gateway"** | Grafana's backend couldn't reach/parse kdb. From the server: `curl -i -H "X-Grafana-Org-Id: 1" http://localhost:6702/` — `Connection refused` means the kdb session isn't listening on 6702 (restart it); anything non-`200` means the wrong handler is wired. | -| Panel empty / "No data" | Time range doesn't overlap the data — use **Last 15 minutes** (data is `.z.p`-recent). | -| Metric dropdown empty | Wrong plugin — must be **SimpleJson** (`/search`+`/query`), not the newer SimPod (`/metrics`). | +| `bind: address already in use` on start | UI port taken - set `GF_SERVER_HTTP_PORT` to a free port (e.g. `6905`). | +| Plugin install: `permission denied … /var/lib/grafana/plugins` | The CLI defaulted to the system dir - pass `--pluginsDir "$PWD/data/plugins"`. | +| Browser: page won't load (remote server) | Forward the UI port over SSH - see section 3. | +| **Save & Test: "Bad Gateway"** | Grafana's backend couldn't reach/parse kdb. From the server: `curl -i -H "X-Grafana-Org-Id: 1" http://localhost:6702/` - `Connection refused` means the kdb session isn't listening on 6702 (restart it); anything non-`200` means the wrong handler is wired. | +| Panel empty / "No data" | Time range doesn't overlap the data - use **Last 15 minutes** (data is `.z.p`-recent). | +| Metric dropdown empty | Wrong plugin - must be **SimpleJson** (`/search`+`/query`), not the newer SimPod (`/metrics`). |