Feature HTML

di/html/html.md

@@ -0,0 +1,161 @@
+# di.html
+
+WebSocket pub/sub and HTML page serving module for kdb+.
+
+## Features
+
+- Register kdb+ tables for live pub/sub over WebSockets
+- Push updates to all subscribed browser handles via `pub`
+- Serve static assets (HTML, JS, CSS) from a configurable directory using kdb's built-in HTTP handler — no separate web server required
+- Browser clients subscribe via the `sub` evaluate dispatch and receive an initial snapshot on connection
+- Custom per-table modifier functions to control serialisation (c.js binary or plain JSON text)
+- Connections cleaned up automatically on WebSocket close (`.z.wc`) or IPC port close (`.z.pc`)
+
+## How it works
+
+A single q process does two jobs on the same port:
+
+1. **Serves the HTML page** — kdb's built-in HTTP handler serves static files from the directory pointed to by `KDBHTML`. When a browser requests `http://host:port/index.html`, kdb reads and returns the file.
+2. **Handles WebSocket connections** — once the page loads, the browser opens a WebSocket back to the same `host:port`. The `.z.ws` handler receives messages, dispatches via `evaluate`, and sends replies. Live updates are pushed to subscribed handles via `pub`.
+
+You need two things:
+
+- A **q process** started with `-p PORT`, with di.html loaded and `init` called
+- **HTML/JS/CSS files** in the directory pointed to by `KDBHTML`
+
+## Dependencies
+
+| Key | Type | Required | Description |
+|---|---|---|---|
+| `` `log `` | dict | yes | Logging functions keyed on at minimum `` `info ``, each `{[ctx;msg]}` where `ctx` is a symbol and `msg` is a string. kx.log instances are normalised automatically. |
+
+The HTML home directory is configured via the `KDBHTML` environment variable (not a deps key). kdb-x sets `KDBHTML` by default to the `analyst/html` directory in the kdb-x installation, which contains the KX Analyst UI. If `KDBHTML` is unset, `"html"` (relative to the working directory) is used as a fallback. Set or override `KDBHTML` before calling `init` to point at your own HTML files.
+
+## Initialisation
+
+```q
+html.init[deps]
+```
+
+`deps` is a dictionary containing the `log` key. `init`:
+
+- Normalises the log dependency (wraps kx.log monadic functions into the dyadic contract)
+- Sets `.h.HOME` from `KDBHTML` so the default HTTP handler serves static assets
+- Registers `.z.ws`, `.z.wc`, and `.z.pc` handlers — wired once only; any existing `.z.wc`/`.z.pc` handlers are preserved by wrapping
+- Safe to call multiple times; handler wiring is skipped on subsequent calls
+
+## Exported Functions
+
+### addtables
+
+```q
+html.addtables[tablelist]
+```
+
+Registers a list of table names for pub/sub. Can be called multiple times to add new tables. Sets a default modifier per table that JSON-encodes updates before sending to subscribers.
+
+### pub
+
+```q
+html.pub[tbl;data]
+```
+
+Publishes `data` for `tbl` to all currently subscribed handles, applying the per-table modifier before sending.
+
+### sub
+
+```q
+html.sub[tbl;syms]
+```
+
+Subscribes the current handle (`.z.w`) to `tbl`. Pass `` ` `` as `syms` to receive all data. Pass `` ` `` as `tbl` to subscribe to all registered tables. Returns `(tablename; current data)` so the subscriber can initialise their local copy before live updates arrive.
+
+Browser clients call `sub` via the evaluate dispatch:
+
+```json
+{"func": "sub", "arg1": "trades", "arg2": ""}
+```
+
+`arg1` is the table name (empty string = all tables), `arg2` is the sym filter (empty string = all syms). The caller is responsible for converting strings to q symbols before passing to `evaluate`.
+
+### setmodifier
+
+```q
+html.setmodifier[tbl;fn]
+```
+
+Sets a custom modifier function for `tbl`. `fn` receives `(\`upd;tbl;data)` and must return bytes or a string to send directly to subscribers.
+
+The default modifier uses the c.js binary protocol (kdb+ IPC-wrapped JSON). Switch to plain JSON text for clients without c.js:
+
+```q
+html.setmodifier[`trades;{.j.j `name`data!("upd";`tablename`tabledata!(x 1;x 2))}]
+```
+
+Signals an error if `tbl` has not been registered via `addtables`.
+
+### dataformat
+
+```q
+html.dataformat[msgtype;msgdata]
+```
+
+Wraps a message into a `` `name`data `` dictionary, javascript-formatting each table in `msgdata` (a list or dictionary of tables). Useful for request/reply calls where the front end asks for several tables at once.
+
+### evaluate
+
+```q
+html.evaluate[inputdict]
+```
+
+Extracts the `func` key from a q dictionary, calls the named function with any additional keys as arguments, and returns the result. Used internally by `.z.ws` — the handler deserialises JSON before calling this. Functions in `funcmap` (`sub`, `addtables`, `pub`, `dataformat`) are resolved first; other global functions are reachable by name.
+
+## Usage Example
+
+```q
+html:use`di.html
+log:use`di.log
+
+logdep:`info`warn`error!(log.info;log.warn;log.error)
+
+/ set static file directory before init
+setenv[`KDBHTML;"/opt/app/html"]
+html.init[enlist[`log]!enlist logdep]
+
+/ register tables
+html.addtables[`trades`quotes]
+
+/ publish from a timer or upd handler
+html.pub[`trades;newdata]
+```
+
+With kx.log, pass the log instance directly — it is normalised automatically:
+
+```q
+logger:use`kx.log
+kxlog:logger.createLog[]
+html.init[enlist[`log]!enlist kxlog]
+```
+
+## Running Tests
+
+```q
+k4unit:use`di.k4unit
+k4unit.moduletest`di.html
+```
+
+All exported functions are covered. The one path not unit-tested is `pub` physically delivering a message over a live WebSocket connection, which requires a real client handle.
+
+For interactive browser testing, use the included integration test script:
+
+```bash
+q di/html/integrationtest.q -p 5678
+```
+
+This starts a process with a plain-text JSON websocket handler (no c.js required). Navigate to `http://localhost:5678/test.html` to subscribe to tables and push live data via `tick[\`trades;5]` in the q session.
+
+## Notes
+
+- The default `.z.ws` handler sends kdb+ binary IPC (`-8!`) and requires the KX c.js library in the browser. Use `setmodifier` to switch individual tables to plain JSON text for clients without c.js.
+- `.z.wc` fires on WebSocket close; `.z.pc` fires on any IPC port close. Both are wired so that `sub` works over plain IPC handles as well as WebSocket connections.
+- `evaluate` resolves function names first from the module `funcmap`, then from the global namespace. Restrict which globals are reachable by controlling what is defined in the process.

di/html/html.q

@@ -0,0 +1,182 @@
+/ di.html - websocket pub/sub and html page serving
+
+/ list of table names registered for pub/sub
+subtables:`symbol$();
+
+/ subscriber list per table: each entry is (handle; sym-filter)
+subs:()!();
+
+/ modifier function per table: transforms data before sending to subscriber
+modifier:()!();
+
+/ flag so direct .z handler wiring happens only once across repeated init calls
+zwired:0b;
+
+normlog:{[logdict]
+  / normalise a log dict: if it looks like a kx.log instance (has getlvl/sinks/fmts keys),
+  / wrap each level into a dyadic {[c;m]} function that embeds the context symbol into the message
+  $[any `getlvl`sinks`fmts in key logdict;
+    `info`warn`error!(
+      {[fn;c;m] fn[string[c],": ",m]}[logdict`info;];
+      {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;];
+      {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]);
+    logdict];
+  };
+
+jstsiso8601:{[x]
+  / converts a list of timestamps or datetimes to iso 8601 strings e.g. "2024-01-02T12:00:00Z"
+  / vectorised: stringifies the date and time parts in bulk rather than per element; nulls return ""
+  i:where 10=count each d:string `date$x;
+  r:(count d)#enlist "";
+  if[not count i;:r];
+  dd:d i;
+  dd[;4 7]:"-";
+  r[i]:dd,'("T",/:string `second$x i),\:"Z";
+  :r;
+  };
+
+/ converts a list of dates to javascript epoch milliseconds
+/ kdb dates are days since 2000-01-01; 10957 is the day offset from 1970-01-01 to 2000-01-01
+jstsfromd:{[x] "j"$86400000 * 10957 + `long$x};
+
+/ converts a list of time, second, or minute values to milliseconds since midnight
+jstsfromt:{[x] "j"$"t"$x};
+
+/ converts a list of months to javascript epoch milliseconds via the first day of each month
+jstsfromm:{[x] jstsfromd `date$x};
+
+/ maps kdb type shorts to their javascript converter function
+/ types not listed here are left unchanged by jsformat
+typemap:12 13 14 15 16 17 18 19h!(jstsiso8601;jstsfromm;jstsfromd;jstsiso8601;jstsfromt;jstsfromt;jstsfromt;jstsfromt);
+
+jsformat:{[tbl]
+  / applies the correct javascript converter to each column of a table that needs it
+  / columns whose type is not in the typemap are passed through unchanged (via ::)
+  coldict:flip 0!tbl;
+  k:key coldict;
+  colvals:value coldict;
+  converters:typemap type each colvals;
+  :flip k!converters@'colvals;
+  };
+
+dataformat:{[msgtype;msgdata]
+  / wraps a message into a name/data dictionary, javascript-formatting each table in msgdata
+  / msgdata is a list or dictionary of tables - used by host data functions requested from the front end
+  :(`name`data)!(msgtype;jsformat each msgdata);
+  };
+
+/ filter applied before sending data to a subscriber - returns full table (no filtering)
+sel:{[tbl;syms] tbl};
+
+del:{[tbl;handle]
+  / removes a handle from the subscriber list for a table
+  / if handle is not found, ? returns count of list and drop has no effect
+  idx:subs[tbl;;0]?handle;
+  .z.m.subs:@[subs;tbl;_;idx];
+  };
+
+add:{[tbl;syms]
+  / adds the current handle to the subscriber list for a table
+  / if already subscribed, updates the sym filter by taking union with new syms
+  / if not subscribed, appends a new (handle; syms) pair to the list
+  / returns (tablename; current data) so the subscriber can initialise their local copy
+  i:subs[tbl;;0]?.z.w;
+  .z.m.subs:$[(count subs tbl)>i;
+    .[subs;(tbl;i;1);union;syms];
+    @[subs;tbl;,;enlist(.z.w;syms)]
+    ];
+  :(tbl;$[99=type v:value tbl;sel[v;syms];@[0#v;`sym;`g#]]);
+  };
+
+closehandle:{[handle]
+  / removes the given handle from all subscriber lists when a connection closes
+  del[;handle] each subtables;
+  };
+
+execdict:{[inputdict]
+  / extracts the func key and any additional args from a dictionary and calls the function
+  / args are passed to the function in the order the keys appear after func
+  / checks the module funcmap first (module-local functions), then falls back to value for globals
+  if[not `func in key inputdict;'"no func in dictionary"];
+  fname:`$inputdict`func;
+  f:$[fname in key funcmap;
+    funcmap fname;
+    @[value;inputdict`func;{'"unknown function: ",x}]];
+  args:value inputdict _ `func;
+  :$[1=count key inputdict;f @ 1;f . args];
+  };
+
+/ websocket message handler - module-level so it carries the module context for evaluate
+wshandler:{neg[.z.w] -8!.j.j[evaluate[.j.k -9!x]];};
+
+addtables:{[tablelist]
+  / registers a list of tables for pub/sub and sets their default modifier
+  / can be called multiple times; already-registered tables are ignored
+  tablelist,:();
+  new:tablelist except subtables;
+  .z.m.subtables:subtables,new;
+  .z.m.subs:subs,new!(count new)#();
+  / default modifier applies jsformat then sends kdb+ IPC binary wrapping a json string (c.js binary protocol)
+  .z.m.modifier:modifier,new!(count new)#{-8!.j.j `name`data!("upd";`tablename`tabledata!(x 1;jsformat x 2))};
+  if[count new;.z.m.log[`info][`di.html;"registered tables: ",", " sv string new]];
+  };
+
+pub:{[tbl;data]
+  / publishes data to all current subscribers of a table
+  / applies the table modifier before sending (default modifier json-encodes the data)
+  {[tbl;data;s]
+    if[count data:sel[data;s 1];
+      (neg first s) modifier[tbl]@(`upd;tbl;data)];
+    }[tbl;data] each subs tbl;
+  };
+
+sub:{[tbl;syms]
+  / subscribes the current handle to a table with an optional sym filter
+  / pass backtick as tbl to subscribe to all registered tables
+  / removes any existing subscription for this handle before re-adding
+  if[tbl~`;:sub[;syms] each subtables];
+  if[not tbl in subtables;'tbl];
+  del[tbl;.z.w];
+  :add[tbl;syms];
+  };
+
+setmodifier:{[tbl;fn]
+  / sets a custom modifier function for tbl; fn receives (`upd;tbl;data) and must return bytes or a string to send
+  if[not tbl in subtables;'tbl];
+  .z.m.modifier:@[modifier;tbl;:;fn];
+  };
+
+evaluate:{[inputdict]
+  / safely calls execdict on the input, logging then re-throwing any errors with context
+  :@[execdict;inputdict;{[d;e] m:"failed to execute ",(-3!d)," : ",e;.z.m.log[`error][`di.html;m];'"di.html: ",m}[inputdict]];
+  };
+
+/ module-local functions callable via websocket evaluate
+/ execdict checks here first before falling back to value for global lookups
+funcmap:`sub`addtables`pub`dataformat!(sub;addtables;pub;dataformat);
+
+init:{[deps]
+  / initialise the module; deps must contain a log key
+  / deps: dict with `log key -> dict with at minimum `info!{[c;m]} (dyadic: ctx symbol, msg string)
+  / kx.log instances are normalised automatically via normlog
+  / wires .z.ws/.z.wc/.z.pc handlers once; .h.HOME set from KDBHTML env var (else "html")
+  if[99h<>type deps;
+    '"di.html: deps must be a dict with `log key"];
+  if[not `log in key deps;
+    '"di.html: log dependency is required; pass at minimum `info!{[c;m]} keyed on `log"];
+  if[99h<>type deps`log;
+    '"di.html: log value must be a dict; pass at minimum `info!{[c;m]}"];
+  if[not `info in key deps`log;
+    '"di.html: log dict must have at minimum an `info key; got: ",(", " sv string key deps`log)];
+  .z.m.log:normlog deps`log;
+  hd:$[count e:getenv`KDBHTML;e;"html"];
+  / .h.HOME lets the default http handler serve static assets; set from KDBHTML env var (else "html")
+  @[{.h.HOME:x;.h.tx[`non]:{enlist x};.h.ty[`non]:"text/html"};hd;{[e]}];
+  .z.m.log[`info][`di.html;"initialised"];
+  / wire handlers once; wrap any existing .z.wc/.z.pc to preserve them
+  if[zwired;:()];
+  .z.ws:wshandler;
+  .z.wc:{[existing;h] closehandle h; existing h}[@[value;`.z.wc;{{[x]}}];];
+  .z.pc:{[existing;h] closehandle h; existing h}[@[value;`.z.pc;{{[x]}}];];
+  .z.m.zwired:1b;
+  };

di/html/init.q

@@ -0,0 +1,4 @@
+/ entry point for di.html module
+\l ::html.q
+
+export:([init;addtables;pub;sub;setmodifier;dataformat;evaluate])