Feature grafana

di/grafana/LIVE_TESTING.md

@@ -0,0 +1,196 @@
+# di.grafana - Live verification (manual)
+
+An end-to-end check of `di.grafana` against a **real** Grafana instance: browser →
+Grafana → HTTP (with the `X-Grafana-Org-Id` header) → the module's `.z.pp`/`.z.ph`
+handlers → `search`/`query` → JSON → a rendered panel.
+
+This is a **manual** procedure for confidence/demos - it is **not** part of the
+automated test suite (`k4unit.moduletest\`di.grafana`), which already covers the
+module's logic in-process. Use this to see it work in the real UI.
+
+> **Verified working:** 2026-07-01 - Grafana **10.4.14** + `grafana-simple-json-datasource`
+> v1.4.2, KDB-X 5.0, module on `di.grafana-pr`. Live panel rendered a moving
+> per-sym price series driven by a kdb feed.
+
+---
+
+## ⚠️ Grafana version compatibility (read this first)
+
+The module implements the **classic Grafana SimpleJSON protocol** - `/search`,
+`/query`, `/annotations`, and a `GET /` health check. The only Grafana datasource
+plugin that speaks this protocol natively is **`grafana-simple-json-datasource`**,
+which is **Angular-based and deprecated**. Grafana has been phasing Angular out:
+
+| Grafana version | Status of the plugin |
+|---|---|
+| **≤ 10.x** | Angular enabled by default - plugin works out of the box ✅ |
+| **11.x** | Angular disabled by default - plugin loads **only** if Angular is re-enabled (`[plugins] angular_support_enabled = true`, or env `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true`) ⚠️ |
+| **≥ 12.x** | Angular support **removed entirely** - the plugin will not load at all ❌ |
+
+**This guide pins Grafana 10.4.14** because Angular is on by default there - the
+least friction. On Grafana 11 you must re-enable Angular (see Troubleshooting).
+
+### Future note - when you must move to Grafana ≥ 12
+
+`grafana-simple-json-datasource` is a dead end from Grafana 12 onward. Options then:
+
+- **Infinity** (`yesoreyeram-infinity-datasource`, React, actively maintained):
+  point it at `POST http://<host>:<port>/query` with a JSON body and a custom
+  `X-Grafana-Org-Id: 1` header, and configure its parser to read the `datapoints`.
+  Works, but it's manual per-panel config - no metric dropdown from `/search`.
+- **SimPod** (`simpod-json-datasource`, React): non-Angular, but newer versions
+  call `/metrics` instead of `/search`, so the metric dropdown won't populate
+  (the module serves `/search`); a manually-typed target may still hit `/query`.
+- **Extend the module** to also serve the endpoints a modern React plugin expects
+  (e.g. `/metrics`) - a code change; weigh against keeping the module lean.
+
+The module itself is protocol-correct; this is purely about which Grafana plugin
+can consume it. If this doc stops working after a Grafana upgrade, the cause is
+almost certainly the Angular removal above, not the module.
+
+---
+
+## Overview
+
+Two terminals plus a browser. Ports used below: **kdb HTTP = `6702`**, **Grafana
+UI = `3000`** (change either if busy).
+
+---
+
+## 1. kdb process (Terminal 1)
+
+Start kdb-x from the module repo so `use\`di.grafana` resolves, then wire it up as
+a datasource with a small live feed:
+
+```bash
+cd ~/kdb_x/kdbx-modules-memstats-rename
+export QPATH=$(pwd):$QPATH
+q                                   # your KDB-X launcher
+```
+
+```q
+\p 6702                             / listen for HTTP
+
+/ a logger that prints, so incoming requests are visible
+mylog:`info`warn`error!(
+  {[c;m] -1 "INFO  [",string[c],"] ",m;};
+  {[c;m] -1 "WARN  [",string[c],"] ",m;};
+  {[c;m] -2 "ERROR [",string[c],"] ",m;});
+
+grafana:use`di.grafana
+grafana.init[enlist[`log]!enlist mylog]
+
+/ ~16 min of recent data + a 1/sec live feed so the graph moves
+trade:([]time:.z.p-0D00:00:01*til 1000;sym:1000?`AAPL`MSFT`GOOG;price:1000?100f;size:1000?500)
+.z.ts:{`trade insert (.z.p;rand`AAPL`MSFT`GOOG;rand 100f;rand 500)}; system"t 1000"
+```
+
+Leave this session running. (`.z.ts` is yours; the module only owns `.z.pp`/`.z.ph`.)
+
+---
+
+## 2. Grafana 10.4 (Terminal 2, no root required)
+
+Runs entirely from your home directory - no Docker, no `sudo`:
+
+```bash
+cd ~
+wget https://dl.grafana.com/oss/release/grafana-10.4.14.linux-amd64.tar.gz
+tar -zxf grafana-10.4.14.linux-amd64.tar.gz
+cd grafana-v10.4.14
+mkdir -p data/plugins
+
+# install the SimpleJson plugin into a local (writable) dir
+./bin/grafana-cli --pluginsDir "$PWD/data/plugins" plugins install grafana-simple-json-datasource
+
+# run it, all paths under $HOME
+GF_SERVER_HTTP_PORT=3000 \
+GF_PATHS_DATA="$PWD/data" \
+GF_PATHS_LOGS="$PWD/data/log" \
+GF_PATHS_PLUGINS="$PWD/data/plugins" \
+GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=grafana-simple-json-datasource \
+./bin/grafana-server
+```
+
+Wait for both of these in the log (and **no** "Refusing to initialize … Angular"):
+
+```
+Plugin registered … pluginId=grafana-simple-json-datasource
+HTTP Server Listen … :3000
+```
+
+---
+
+## 3. If Grafana is on a remote server (SSH / VS Code Remote)
+
+Grafana listens on the **server's** `localhost:3000`, which your local browser
+can't reach directly. Forward the UI port:
+
+- **VS Code Remote:** open the **PORTS** panel → **Forward a Port** → `3000` →
+  open the resulting local address.
+- **Plain SSH:** `ssh -L 3000:localhost:3000 <user>@<server>` then browse `http://localhost:3000`.
+
+Only the **Grafana UI port** needs forwarding. The datasource URL
+`http://localhost:6702` is resolved by Grafana's **backend on the server** (that's
+what "Access: Server" means), so kdb's port never leaves the server - do **not**
+forward 6702.
+
+---
+
+## 4. Add the datasource
+
+1. `http://localhost:3000` → login **admin / admin** (skip the password prompt).
+2. **Connections → Data sources → Add data source →** search **"SimpleJson"** → select it.
+3. **URL:** `http://localhost:6702` · **Access: Server (default)**.
+   - **Access must be Server** - that is what makes Grafana's backend attach the
+     `X-Grafana-Org-Id` header the module keys on.
+4. **Save & Test** → expect green **"Data source is working"** (a `GET /` → `.z.ph`
+   → `200 OK`; visible in the q console).
+
+---
+
+## 5. Build a panel
+
+1. **Dashboards → New → New dashboard → + Add visualization →** pick the datasource.
+2. Open the **Metric** dropdown (populated by `/search`): `trade`, `t.trade`,
+   `g.trade`, `g.trade.price`, `g.trade.size`, `t.trade.AAPL`, …
+3. Select **`g.trade.price`**; set the time range to **Last 15 minutes**
+   (optionally auto-refresh 5s to watch the live feed).
+
+Exercise the other paths too:
+- **`g.trade`** - one line per numeric column
+- **`t.trade`** - switch the panel type to **Table** for the whole table
+- **`t.trade.AAPL`** - table filtered to one sym
+
+---
+
+## What success looks like
+
+| Signal | Confirms |
+|---|---|
+| **Save & Test** green | GET handler + connectivity |
+| Metric **dropdown populates** | `/search` |
+| Panel shows **live lines** (AAPL/MSFT/GOOG) | `/query` → `tsfunc` → per-sym builder, end-to-end |
+| `INFO [grafana] received … request` in Terminal 1 | requests flowing through `.z.pp` |
+
+---
+
+## Teardown
+
+- Grafana: **Ctrl-C** in Terminal 2.
+- kdb: `system"t 0"` to stop the feed, then exit q.
+
+---
+
+## Troubleshooting
+
+| Symptom | Cause / fix |
+|---|---|
+| `Refusing to initialize plugin … Angular` in the Grafana log | You're on Grafana 11 - use the 10.4 tarball here, or set `GF_PLUGINS_ANGULAR_SUPPORT_ENABLED=true` (config `[plugins] angular_support_enabled = true`). On Grafana ≥ 12 it can't be enabled - see "Future note" above. |
+| `docker: permission denied … docker.sock` | Not in the `docker` group. Use the **binary** method here (no Docker), or `sudo docker …`. |
+| `bind: address already in use` on start | UI port taken - set `GF_SERVER_HTTP_PORT` to a free port (e.g. `6905`). |
+| Plugin install: `permission denied … /var/lib/grafana/plugins` | The CLI defaulted to the system dir - pass `--pluginsDir "$PWD/data/plugins"`. |
+| Browser: page won't load (remote server) | Forward the UI port over SSH - see section 3. |
+| **Save & Test: "Bad Gateway"** | Grafana's backend couldn't reach/parse kdb. From the server: `curl -i -H "X-Grafana-Org-Id: 1" http://localhost:6702/` - `Connection refused` means the kdb session isn't listening on 6702 (restart it); anything non-`200` means the wrong handler is wired. |
+| Panel empty / "No data" | Time range doesn't overlap the data - use **Last 15 minutes** (data is `.z.p`-recent). |
+| Metric dropdown empty | Wrong plugin - must be **SimpleJson** (`/search`+`/query`), not the newer SimPod (`/metrics`). |

di/grafana/grafana.md

@@ -0,0 +1,189 @@
+# di.grafana
+
+Grafana JSON datasource adaptor for kdb+. Implements the
+[SimPod JSON datasource](https://github.com/simPod/GrafanaJsonDatasource) API so
+a Grafana server can query a kdb+ process directly: the module installs HTTP
+handlers that answer Grafana's `/search` and `/query` requests with the JSON it
+expects, serving the process's tables as timeseries and table panels.
+
+---
+
+## Features
+
+- Serves the `/search` endpoint - populates Grafana's metric dropdowns from the tables in the process
+- Serves the `/query` endpoint - returns either timeseries or table panel data
+- Answers the Grafana test-connection `GET` with a `200 OK`
+- Wraps any pre-existing `.z.pp`/`.z.ph` handlers rather than overwriting them, so other HTTP endpoints keep working
+- Accepts a `kx.log` instance directly, or a plain `info`warn`error` dict
+- Loads standalone - no hard dependencies on other modules
+
+---
+
+## Dependencies
+
+| Dependency | Key | Required | Description |
+|---|---|---|---|
+| logger | `` `log `` | yes | `info`, `warn`, and `error` - each a binary `{[c;m]}` where `c` is a symbol context and `m` is a string. All three are required because this module calls all three. |
+
+**Hard dependency:** none - the module loads standalone.
+
+The `log` dependency must be passed to `init` inside the `deps` dict keyed on
+`` `log ``. The module throws immediately if `log` is absent, is not a
+dictionary, or is missing any of `info`/`warn`/`error`.
+
+A `kx.log` instance can be passed directly - the module normalises its monadic
+functions to the binary `{[c;m]}` contract automatically (detected by the
+`getlvl`/`sinks`/`fmts` keys). Plain `info`warn`error` dicts pass through
+unchanged.
+
+```q
+/ plain custom logger
+mylog:`info`warn`error!(
+  {[c;m] -1 "INFO  [",string[c],"] ",m;};
+  {[c;m] -1 "WARN  [",string[c],"] ",m;};
+  {[c;m] -2 "ERROR [",string[c],"] ",m;});
+grafana:use`di.grafana
+grafana.init[enlist[`log]!enlist mylog]
+
+/ or a kx.log instance directly
+kxlog:use`kx.log
+grafana.init[enlist[`log]!enlist kxlog.createLog[]]
+```
+
+Configuration keys `timecol`, `sym`, `timebackdate`, `ticks`, `del`, and
+`allowfunctions` are all optional - omit any or all of them and the module falls
+back to sensible defaults. See Initialisation for full details.
+
+---
+
+## Initialisation
+
+`init[deps]` takes a single dictionary combining the `log` dependency with any
+configuration overrides.
+
+| Key | Required | Description |
+|---|---|---|
+| `` `log `` | yes | Log dep - `info`/`warn`/`error` `{[c;m]}` functions, or a `kx.log` instance |
+| `` `timecol `` | no | Name of the time column used for timeseries queries. Default: `` `time `` |
+| `` `sym `` | no | Name of the column used to split data by instrument. Default: `` `sym `` |
+| `` `timebackdate `` | no | How far back to look when finding distinct syms for the dropdowns. Default: `2D` |
+| `` `ticks `` | no | Number of rows returned for a table request. Default: `1000` |
+| `` `del `` | no | Delimiter separating the arguments within a query target. Default: `"."` |
+| `` `allowfunctions `` | no | Enable `f.` function targets, which evaluate arbitrary q (see Security note). Default: `0b` (disabled) |
+
+`init` must be called before the module will serve any requests. It validates
+the logger, applies config, and installs the `.z.pp`/`.z.ph` handlers - once;
+re-calling `init` updates the logger/config without re-wrapping the handlers.
+
+```q
+/ defaults only
+grafana.init[enlist[`log]!enlist logdep]
+
+/ with config overrides (config keys sit alongside `log)
+grafana.init[`log`timecol`ticks!(logdep;`ts;500)]
+```
+
+---
+
+## Exported Functions
+
+### `init[deps]`
+Initialise the module: validate the log dependency, apply config, and install the HTTP handlers. Errors immediately if `log` is missing, is not a dict, or lacks `info`/`warn`/`error`.
+```q
+grafana.init[enlist[`log]!enlist logdep]
+/ or with config:
+grafana.init[`log`ticks!(logdep;500)]
+```
+
+### `getconfig[]`
+Return the currently active configuration as a dictionary - useful for confirming how the running module is tuned.
+```q
+grafana.getconfig[]
+/ `timecol`sym`timebackdate`ticks`del!(`time;`sym;2D;1000;".")
+```
+
+---
+
+## Query target syntax
+
+The Grafana metric strings produced by `/search` encode the table, panel type,
+and arguments, separated by `del` (default `"."`):
+
+| Target | Meaning |
+|---|---|
+| `t.<table>` | table panel for the whole table |
+| `t.<table>.<sym>` | table panel filtered to one sym |
+| `g.<table>` | graph panel, one series per numeric column |
+| `g.<table>.<col>` | graph panel, one series per sym for a column |
+| `o.<table>...` | "other" panels (single-stat, gauge, etc.) |
+| `f.<...>` | the target is a q expression to evaluate rather than a table - **disabled unless `allowfunctions` is set** (see Security) |
+
+---
+
+## Usage Example
+
+```q
+/ load and initialise with a printing logger
+mylog:`info`warn`error!(
+  {[c;m] -1 "INFO  [",string[c],"] ",m;};
+  {[c;m] -1 "WARN  [",string[c],"] ",m;};
+  {[c;m] -2 "ERROR [",string[c],"] ",m;});
+
+grafana:use`di.grafana
+grafana.init[enlist[`log]!enlist mylog]
+
+/ open an HTTP port and create some data with time + sym columns
+\p 5000
+trade:([]time:.z.p-0D00:00:01*til 100;sym:100?`AAPL`MSFT`GOOG;price:100?100f)
+```
+
+Add a SimPod JSON datasource in Grafana pointing at the process's HTTP port,
+then build panels using the metrics offered in the dropdowns (`t.trade`,
+`g.trade.price`, ...).
+
+---
+
+## Running Tests
+
+```q
+k4unit:use`di.k4unit
+k4unit.moduletest`di.grafana
+```
+
+The suite injects a no-op mock logger and covers: the export surface; dependency
+validation (non-dict deps, missing `log` key, null log value, missing log keys);
+handler wiring and the install-once `wired` guard; config defaults and overrides
+via `getconfig`; the target-parsing helpers (`isfunc`/`istab`/`istype`/`prefix`);
+`finddistinctsyms`/`memvals`/`catchvals`; the `/query` table and timeseries
+builders (`tbfunc`, and `tsfunc` including the per-column and per-sym graph
+paths); the live `.z.pp`/`.z.ph` routes including non-Grafana fall-through; and a
+`kx.log` integration test confirming the `normlog` wrapping works end-to-end.
+
+---
+
+## Security
+
+The module answers any HTTP request carrying the `X-Grafana-Org-Id` header, so
+treat its port as a trust boundary - put it behind Grafana's authentication and
+firewall it from untrusted networks.
+
+- **Table targets are resolved by name, never evaluated.** `t.`/`g.`/`o.` and
+  bare targets are looked up as symbols against `tables[]`; an unknown name is
+  rejected. A target string can therefore never be executed as q code through
+  these paths.
+- **`f.` function targets evaluate arbitrary q and are disabled by default.**
+  They run only when `allowfunctions` is set to `1b` in `init`. Enable it solely
+  on processes where every client able to reach the port is trusted, since an
+  `f.` target is, by design, remote code execution.
+
+---
+
+## Notes
+
+- The module installs `.z.pp`/`.z.ph` by closure-wrapping any pre-existing handler: Grafana requests (those carrying the `X-Grafana-Org-Id` header) are routed to the adaptor, and every other request falls through to the captured handler - so adding the module to a process that already serves HTTP does not break existing endpoints. This follows the `di.timer` precedent of a module owning a `.z.*` callback.
+- The intended end-state is to register the handlers through an injected `di.handlers` dependency rather than assigning `.z.*` directly. That is deferred until `di.handlers` exists and supports return-yielding `.z.pp`/`.z.ph` chains - a side-effect-only chain cannot return the HTTP response. `di.handlers` is not implemented yet.
+- A panel with multiple queries works: each target in the request is dispatched by its own type and the per-target results are merged into one JSON array
+- Table panels return the last `ticks` rows (default 1000); raise `ticks` via config if a panel needs more
+- Sym dropdowns list only syms seen within `timebackdate` (default 2 days)
+- Data tables must expose the configured `timecol` and `sym` columns (defaults `` `time ``/`` `sym ``)
+- The `/annotations` endpoint returns a "not yet implemented" marker, matching the TorQ original