From e59a53caa7f059419e011cbaa06ec380d30339d4 Mon Sep 17 00:00:00 2001 From: Matthew Date: Tue, 9 Jun 2026 09:58:08 +0100 Subject: [PATCH 01/10] Independent module using log dependency. With passing unit tests --- di/html/html.md | 165 ++++++++++++++++++++++++++++++++++++++ di/html/html.q | 201 +++++++++++++++++++++++++++++++++++++++++++++++ di/html/init.q | 4 + di/html/test.csv | 40 ++++++++++ 4 files changed, 410 insertions(+) create mode 100644 di/html/html.md create mode 100644 di/html/html.q create mode 100644 di/html/init.q create mode 100644 di/html/test.csv diff --git a/di/html/html.md b/di/html/html.md new file mode 100644 index 00000000..c1d98d59 --- /dev/null +++ b/di/html/html.md @@ -0,0 +1,165 @@ +# di.html + +WebSocket pub/sub and HTML page serving module, extracted from TorQ. + +## Overview + +This module does two things: + +1. **Pub/sub over WebSockets** — allows browser clients to subscribe to kdb+ tables and receive live updates as data is published. +2. **HTML page serving** — reads HTML files from a configured directory and serves them over HTTP, optionally replacing server/port placeholders so the page can connect back to itself. + +## Usage + +```q +html:use`di.html + +/ minimal setup - log dep is required, handlers dep is optional +logdep:`info`warn`error!( + {[c;m] -1 "INFO ",(string c)," ",m;}; + {[c;m] -1 "WARN ",(string c)," ",m;}; + {[c;m] -2 "ERROR ",(string c)," ",m;}) +html.init[(enlist `homedir)!enlist "/opt/app/html";enlist[`log]!enlist logdep] + +/ register tables for pub/sub +html.addtables[`trades`quotes] + +/ publish data to subscribers +html.pub[`trades;newdata] +``` + +## init + +```q +html.init[config;deps] +``` + +### config keys + +| Key | Type | Description | Required | +|---|---|---|---| +| `homedir` | string | Path to the directory containing HTML files | Yes | + +### deps keys + +| Key | Description | If absent | +|---|---|---| +| `` `log `` | Logging function dict with keys `` `info`warn`error `` | **Required** — `init` signals an error | +| `` `handlers `` | Handler registration dict with key `` `register `` | Assigns `.z.wc` and `.z.ws` directly | + +## Exported functions + +### addtables + +```q +html.addtables[tablelist] +``` + +Registers a list of table names for pub/sub. Can be called multiple times to add new tables. Sets a default modifier that JSON-encodes updates before sending to subscribers. + +### pub + +```q +html.pub[tbl;data] +``` + +Publishes `data` for `tbl` to all currently subscribed handles. + +### sub + +```q +html.sub[tbl;syms] +``` + +Subscribes the current handle (`.z.w`) to `tbl`. Pass `` ` `` as `syms` to receive all data. Pass `` ` `` as `tbl` to subscribe to all registered tables. Returns `(tablename; current data)` so the subscriber can initialise their local copy of the table. + +### wssub + +```q +html.wssub[tbl] +``` + +Calls `sub[tbl;`` ` ``]`. Pass `` ` `` as `tbl` to subscribe to all registered tables. Returns nothing. + +### end + +```q +html.end[eodval] +``` + +Broadcasts an end-of-day message to all subscriber handles. + +### readpage + +```q +html.readpage[filename] +``` + +Reads the file at `homedir/filename` and returns its contents as a string. Returns an error message string if the file is not found. + +### readpagereplaceHP + +```q +html.readpagereplaceHP[filename] +``` + +Reads the file at `homedir/filename` and replaces `MYKDBSERVER` and `MYKDBPORT` tokens with the process's current IP address and port. Used to serve self-referencing HTML pages over HTTP. + +### evaluate + +```q +html.evaluate[inputdict] +``` + +Takes a q dictionary (already deserialised from JSON), extracts the `func` key, calls the named function with any additional keys as arguments, and returns the result. Used internally by the `.z.ws` handler — the handler does the JSON deserialisation before calling this function. + +## WebSocket handler + +The module registers a `.z.ws` handler that receives bytes from the browser, deserialises them to a q dict, calls `evaluate`, JSON-encodes the result, and sends it back. A `.z.wc` handler cleans up subscriptions when a connection closes. + +## Logging + +The module logs at three points: + +- On `init`: confirms the module started and which `homedir` was set +- On `readpage`: warns if a requested file is not found +- On `evaluate`: logs at error level when a WebSocket-invoked function fails (the error is also re-thrown to the caller) + +The `log` dependency is required — `init` signals an error if it is absent. Internally the loggers are stored on `.z.m` as `lginfo`/`lgwarn`/`lgerr` (not `log`, which is a q built-in) and every call site invokes them through `.z.m`. + +## Example with injected dependencies + +The `log` dep is required and the `handlers` dep is optional; both are supplied by +the host application. The `log` functions are called as `(ctx;msg)` and the +`handlers` registry's `register` is called as `(.z event name; label; handler)`. + +```q +/ wire up logging on top of the kx.log module +/ kx.log loggers take a single message, so wrap them to the (ctx;msg) shape +logger:use`kx.log +kxlog:logger.createLog[] +logdep:`info`warn`error!( + {[c;m] kxlog.info[(string c),": ",m]}; + {[c;m] kxlog.warn[(string c),": ",m]}; + {[c;m] kxlog.error[(string c),": ",m]}) + +/ wire up handlers via a host-provided registry +/ a real registry composes handlers so several modules can share .z.ws / .z.wc +/ omit this dep entirely to have the module assign .z.ws / .z.wc directly +hnddep:enlist[`register]!enlist {[zname;label;fn] zname set fn} + +/ initialise html module +html:use`di.html +html.init[(enlist `homedir)!enlist "/opt/app/html";`log`handlers!(logdep;hnddep)] +``` + +## Testing + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.html +``` + +All exported functions are covered. The one path not unit-tested is `pub` physically +delivering a message over a live WebSocket connection, which requires a real client +handle. diff --git a/di/html/html.q b/di/html/html.q new file mode 100644 index 00000000..57c0e384 --- /dev/null +++ b/di/html/html.q @@ -0,0 +1,201 @@ +/ di.html - websocket pub/sub and html page serving + +/ list of table names registered for pub/sub +subtables:`symbol$(); + +/ subscriber list per table: each entry is (handle; sym-filter) +subs:()!(); + +/ modifier function per table: transforms data before sending to subscriber +modifier:()!(); + +/ cache of resolved ip addresses: int -> symbol +ipacache:(`int$())!`symbol$(); + +/ home directory for html files - set by init +homedir:""; + +/ logging functions - required injected deps, wired by init via .z.m + +/ converts a list of timestamps or dates to iso 8601 strings e.g. "2024-01-02T12:00:00Z" +jstsiso8601:{[x] {("-" sv "." vs string `date$x),"T",string[`second$x],"Z"} each x}; + +/ converts a list of dates to javascript epoch milliseconds +/ kdb dates are days since 2000-01-01; 10957 is the day offset from 1970-01-01 to 2000-01-01 +jstsfromd:{[x] "j"$86400000 * 10957 + `long$x}; + +/ converts a list of time, second, or minute values to milliseconds since midnight +jstsfromt:{[x] "j"$"t"$x}; + +/ converts a list of months to javascript epoch milliseconds via the first day of each month +jstsfromm:{[x] jstsfromd `date$x}; + +/ maps kdb type shorts to their javascript converter function +/ types not listed here are left unchanged by jsformat +typemap:12 13 14 16 17 18 19h!(jstsiso8601;jstsfromm;jstsfromd;jstsfromt;jstsfromt;jstsfromt;jstsfromt); + +jsformat:{[tbl] + / applies the correct javascript converter to each column of a table that needs it + / columns whose type is not in the typemap are passed through unchanged (via ::) + coldict:flip 0!tbl; + k:key coldict; + colvals:value coldict; + converters:typemap type each colvals; + :flip k!converters@'colvals; + }; + +updformat:{[msgtype;msgdata] + / wraps an upd message into a name/data dictionary with javascript-formatted table data + formatteddata:(key msgdata)!(msgdata`tablename;jsformat msgdata`tabledata); + :(`name`data)!(msgtype;formatteddata); + }; + +/ filter applied before sending data to a subscriber - returns full table (no filtering) +sel:{[tbl;syms] tbl}; + +del:{[tbl;handle] + / removes a handle from the subscriber list for a table + / if handle is not found, ? returns count of list and drop has no effect + idx:subs[tbl;;0]?handle; + .z.m.subs:@[subs;tbl;_;idx]; + }; + +add:{[tbl;syms] + / adds the current handle to the subscriber list for a table + / if already subscribed, updates the sym filter by taking union with new syms + / if not subscribed, appends a new (handle; syms) pair to the list + / returns (tablename; current data) so the subscriber can initialise their local copy + i:subs[tbl;;0]?.z.w; + .z.m.subs:$[(count subs tbl)>i; + .[subs;(tbl;i;1);union;syms]; + @[subs;tbl;,;enlist(.z.w;syms)] + ]; + :(tbl;$[99=type v:value tbl;sel[v;syms];@[0#v;`sym;`g#]]); + }; + +closehandle:{[handle] + / removes the given handle from all subscriber lists when a connection closes + del[;handle] each subtables; + }; + +replace:{[str;findreplace] + / applies each find->replace pair to the string in sequence using ssr + :(ssr/)[str;string key findreplace;value findreplace]; + }; + +ipa:{[ipint] + / resolves an ip address integer to a hostname symbol, caching results + / tries .Q.host first; falls back to converting the ip bytes manually if that fails + if[not `~r:ipacache ipint;:r]; + hostname:.Q.host ipint; + r:$[`~hostname;`$"."sv string "i"$0x0 vs ipint;hostname]; + .z.m.ipacache:@[ipacache;ipint;:;r]; + :r; + }; + +/ returns the current listen port as a string +getport:{[] string system "p"}; + +execdict:{[inputdict] + / extracts the func key and any additional args from a dictionary and calls the function + / args are passed to the function in the order the keys appear after func + if[not `func in key inputdict;'"no func in dictionary"]; + f:value inputdict`func; + args:value inputdict _ `func; + :$[1=count key inputdict;f @ 1;f . args]; + }; + +/ websocket message handler - module-level so it carries the module context for evaluate +wshandler:{neg[.z.w] -8!.j.j[evaluate[.j.k -9!x]];}; + +addtables:{[tablelist] + / registers a list of tables for pub/sub and sets their default modifier + / can be called multiple times; already-registered tables are ignored + tablelist,:(); + new:tablelist except subtables; + .z.m.subtables:subtables,new; + .z.m.subs:subs,new!(count new)#(); + .z.m.modifier:modifier,new!(count new)#{-8!.j.j updformat["upd";`tablename`tabledata!(x 1;x 2)]}; + if[count new;.z.m.lginfo[`html;"registered tables: ",", " sv string new]]; + }; + +pub:{[tbl;data] + / publishes data to all current subscribers of a table + / applies the table modifier before sending (default modifier json-encodes the data) + {[tbl;data;s] + if[count data:sel[data;s 1]; + (neg first s) modifier[tbl]@(`upd;tbl;data)]; + }[tbl;data] each subs tbl; + }; + +sub:{[tbl;syms] + / subscribes the current handle to a table with an optional sym filter + / pass backtick as tbl to subscribe to all registered tables + / removes any existing subscription for this handle before re-adding + if[tbl~`;:sub[;syms] each subtables]; + if[not tbl in subtables;'tbl]; + del[tbl;.z.w]; + :add[tbl;syms]; + }; + +wssub:{[tbl] + / subscribes via websocket, no return value + sub[tbl;`]; + }; + +end:{[eodval] + / broadcasts end-of-day message to all subscriber handles across all tables + (neg union/[subs[;;0]])@\:(`.u.end;eodval); + }; + +readpage:{[filename] + / reads an html file from the configured home directory and returns it as a string + / returns a "not found" message string if the file does not exist + p:homedir,"/",filename; + r:@[read1;`$":",p;""]; + if[not count r;.z.m.lgwarn[`html;p,": not found"]]; + :$[count r;"c"$r;p,": not found"]; + }; + +readpagereplaceHP:{[filename] + / reads a page and replaces MYKDBSERVER and MYKDBPORT tokens with live server values + :replace[readpage filename;`MYKDBSERVER`MYKDBPORT!("\"",(string ipa .z.a),"\"";getport[])]; + }; + +evaluate:{[inputdict] + / safely calls execdict on the input, logging then re-throwing any errors with context + :@[execdict;inputdict;{[d;e] m:"failed to execute ",(-3!d)," : ",e;.z.m.lgerr[`html;m];'m}[inputdict]]; + }; + +init:{[config;deps] + / sets up module state from config and registers websocket handlers + / log dep is required; handlers dep is optional + + / log dependency is required - guard that deps is a dict, log key is present and not (::) + logdict:$[99h=type deps;$[(`log in key deps)and not(::)~deps`log;deps`log;()!()];()!()]; + if[not count logdict; + '"di.html: log dependency is required; pass `info`warn`error functions - see di.log for a default implementation"; + ]; + .z.m.lginfo:logdict`info; + .z.m.lgwarn:logdict`warn; + .z.m.lgerr:logdict`error; + + / store config + .z.m.homedir:config`homedir; + + / register .h content type handlers - protected in case not available in kdb-x + @[{.h.tx[`non]:{enlist x};.h.ty[`non]:"text/html"};`;{[e]}]; + + / register handlers via dep if provided + if[`handlers in key deps; + deps[`handlers][`register][`.z.ws;`html.ws;wshandler]; + deps[`handlers][`register][`.z.wc;`html.close;closehandle]; + :()]; + + / no handlers dep: assign directly, wrapping any existing .z.wc to preserve it + .z.ws:wshandler; + .z.wc:@[value;`.z.wc;{{}}]; + .z.wc:{[existing;h] closehandle h; existing h}[.z.wc;]; + + .z.m.lginfo[`html;"initialised with homedir: ",homedir]; + }; diff --git a/di/html/init.q b/di/html/init.q new file mode 100644 index 00000000..c86244a9 --- /dev/null +++ b/di/html/init.q @@ -0,0 +1,4 @@ +/ entry point for di.html module +\l ::html.q + +export:([init;addtables;pub;sub;wssub;end;readpage;readpagereplaceHP;evaluate]) diff --git a/di/html/test.csv b/di/html/test.csv new file mode 100644 index 00000000..583aa277 --- /dev/null +++ b/di/html/test.csv @@ -0,0 +1,40 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,html:use`di.html,1,,load di.html module +before,0,0,q,"mocklog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,,mock log dep that discards all messages +before,0,0,q,"html.init[(enlist `homedir)!enlist ""/tmp"";enlist[`log]!enlist[mocklog]]",1,,initialise module with /tmp as html home directory +before,0,0,q,"(`:/tmp/di_html_test.html) 1: ""test""",1,,write test html file using q file io +before,0,0,q,"(`:/tmp/di_html_hp_test.html) 1: ""MYKDBSERVER:MYKDBPORT""",1,,write test file containing server and port tokens +before,0,0,q,htmltestfn:{x-y},1,,helper function for multi-arg evaluate test +before,0,0,q,"trades:([]sym:`g#`symbol$();px:`float$())",1,,real global table so sub success path can read its contents +before,0,0,q,"quotes:([]sym:`g#`symbol$();bid:`float$())",1,,real global table so subscribing to all tables can read its contents +before,0,0,q,"hns:` sv `.m.di,first (key `.m.di) where (key `.m.di) like ""*html""",1,,resolve the html module's internal namespace (avoids hardcoding the load-order prefix) +before,0,0,q,"jsf:get ` sv hns,`jsformat",1,,grab the internal jsformat converter so its date logic can be tested directly +before,0,0,q,"convtest:([]ts:enlist 2024.01.02D12:00:00.000000000;d:enlist 2024.01.02;tm:enlist 12:00:00.000;sym:enlist `AAPL)",1,,sample table with one column of each time type plus an untyped sym column + +run,0,0,q,"html.addtables[`trades`quotes]",1,,register two tables for pub/sub +run,0,0,q,"html.addtables[`trades]",1,,re-registering an existing table does not error +run,0,0,q,"html.pub[`trades;([]a:enlist 1i)]",1,,pub with no subscribers completes without error + +fail,0,0,q,"html.sub[`notregistered;`]",1,,subscribing to an unregistered table throws an error +run,0,0,q,"html.end[`eod]",1,,end is a clean no-op when there are no subscribers (tested before any console sub) +true,0,0,q,"`trades~first html.sub[`trades;`]",1,,sub success path returns the table name as first element +true,0,0,q,"0=count last html.sub[`trades;`]",1,,sub success path returns the current (empty) table contents +run,0,0,q,"html.wssub[`trades]",1,,wssub delegates to sub without error +run,0,0,q,"html.sub[`;`]",1,,subscribing with backtick subscribes to all registered tables + +true,0,0,q,"-42~html.evaluate[`func`arg1!(""neg"";42)]",1,,evaluate calls a function with one arg correctly +true,0,0,q,"-1~html.evaluate[`func`arg1`arg2!(""htmltestfn"";1;2)]",1,,evaluate calls a function with multiple args correctly +true,0,0,q,"-1~html.evaluate[(enlist `func)!enlist ""neg""]",1,,evaluate with only func key calls f@1 - preserving TorQ behaviour +fail,0,0,q,"html.evaluate[(enlist `arg1)!enlist 1]",1,,evaluate throws when the func key is missing + +true,0,0,q,"""test""~html.readpage[""di_html_test.html""]",1,,readpage returns file contents as a string +true,0,0,q,"""/tmp/di_html_notfound.html: not found""~html.readpage[""di_html_notfound.html""]",1,,readpage returns a not-found message string for a missing file +true,0,0,q,"0=count html.readpagereplaceHP[""di_html_hp_test.html""] ss ""MYKDBSERVER""",1,,readpagereplaceHP replaces the MYKDBSERVER token +true,0,0,q,"0=count html.readpagereplaceHP[""di_html_hp_test.html""] ss ""MYKDBPORT""",1,,readpagereplaceHP replaces the MYKDBPORT token + +true,0,0,q,"""2024-01-02T12:00:00Z""~first (jsf convtest)`ts",1,,jsformat converts a timestamp column to an iso 8601 string +true,0,0,q,"1704153600000~first (jsf convtest)`d",1,,jsformat converts a date column to javascript epoch milliseconds +true,0,0,q,"43200000~first (jsf convtest)`tm",1,,jsformat converts a time column to milliseconds since midnight +true,0,0,q,"`AAPL~first (jsf convtest)`sym",1,,jsformat leaves columns whose type is not in the typemap unchanged + +after,0,0,q,"system ""rm /tmp/di_html_test.html /tmp/di_html_hp_test.html""",1,,remove test html files created during setup From 259738e0670bf085f06d1954b5a8a2ff422d9b40 Mon Sep 17 00:00:00 2001 From: Matthew Date: Thu, 11 Jun 2026 15:02:39 +0100 Subject: [PATCH 02/10] Changed html to be in line with other modules --- di/html/html.md | 59 ++++++++++++++++++++++----------------- di/html/html.q | 72 ++++++++++++++++++++++++++++++++---------------- di/html/init.q | 2 +- di/html/test.csv | 14 ++++++++-- 4 files changed, 96 insertions(+), 51 deletions(-) diff --git a/di/html/html.md b/di/html/html.md index c1d98d59..d538af4d 100644 --- a/di/html/html.md +++ b/di/html/html.md @@ -14,12 +14,9 @@ This module does two things: ```q html:use`di.html -/ minimal setup - log dep is required, handlers dep is optional -logdep:`info`warn`error!( - {[c;m] -1 "INFO ",(string c)," ",m;}; - {[c;m] -1 "WARN ",(string c)," ",m;}; - {[c;m] -2 "ERROR ",(string c)," ",m;}) -html.init[(enlist `homedir)!enlist "/opt/app/html";enlist[`log]!enlist logdep] +/ minimal setup - all config is optional +/ homedir defaults to the KDBHTML env var (else "html"), logging defaults to the console +html.init[] / register tables for pub/sub html.addtables[`trades`quotes] @@ -28,24 +25,29 @@ html.addtables[`trades`quotes] html.pub[`trades;newdata] ``` +This mirrors the original TorQ deployment: set `KDBHTML`, initialise, register tables. + ## init ```q -html.init[config;deps] +html.init[] +html.init[configs] ``` -### config keys +`configs` is an optional dictionary — call `init[]` to use the defaults. Only recognised keys are picked up: -| Key | Type | Description | Required | +| Key | Type | Description | Default | |---|---|---|---| -| `homedir` | string | Path to the directory containing HTML files | Yes | +| `homedir` | string | Path to the directory containing HTML files | `KDBHTML` env var, else `"html"` (TorQ behaviour) | +| `` `log `` | dict | Logging functions with keys `` `info`warn`error ``, each called as `(ctx;msg)` | Console loggers — info/warn to stdout, error to stderr | +| `` `handlers `` | dict | Handler registry with key `` `register `` | Assigns `.z.ws`, `.z.wc` and `.z.pc` directly | -### deps keys +```q +/ override config explicitly +html.init[`homedir`log!("/opt/app/html";logdict)] +``` -| Key | Description | If absent | -|---|---|---| -| `` `log `` | Logging function dict with keys `` `info`warn`error `` | **Required** — `init` signals an error | -| `` `handlers `` | Handler registration dict with key `` `register `` | Assigns `.z.wc` and `.z.ws` directly | +`init` also sets `.h.HOME` to `homedir` (protected, skipped if `.h` is unavailable) so the default HTTP handler serves static assets (css/js/img) from the same directory — the equivalent of TorQ's `KDBHTML` behaviour. ## Exported functions @@ -89,6 +91,14 @@ html.end[eodval] Broadcasts an end-of-day message to all subscriber handles. +### dataformat + +```q +html.dataformat[msgtype;msgdata] +``` + +Wraps a message into a `` `name`data `` dictionary, javascript-formatting each table in `msgdata` (a list or dictionary of tables). Used by host data functions that the front end requests over the websocket, e.g. TorQ's monitor `start` call returning several tables at once. + ### readpage ```q @@ -115,7 +125,7 @@ Takes a q dictionary (already deserialised from JSON), extracts the `func` key, ## WebSocket handler -The module registers a `.z.ws` handler that receives bytes from the browser, deserialises them to a q dict, calls `evaluate`, JSON-encodes the result, and sends it back. A `.z.wc` handler cleans up subscriptions when a connection closes. +The module registers a `.z.ws` handler that receives bytes from the browser, deserialises them to a q dict, calls `evaluate`, JSON-encodes the result, and sends it back. Subscriptions are cleaned up when a connection closes via both `.z.wc` (websocket) and `.z.pc` (IPC), as in TorQ — `sub` can also be called over a plain IPC handle. In the direct-assignment path (no `handlers` config) any existing `.z.wc`/`.z.pc` handlers are preserved by wrapping, and the wiring happens only once across repeated `init` calls. ## Logging @@ -125,12 +135,11 @@ The module logs at three points: - On `readpage`: warns if a requested file is not found - On `evaluate`: logs at error level when a WebSocket-invoked function fails (the error is also re-thrown to the caller) -The `log` dependency is required — `init` signals an error if it is absent. Internally the loggers are stored on `.z.m` as `lginfo`/`lgwarn`/`lgerr` (not `log`, which is a q built-in) and every call site invokes them through `.z.m`. +Logging defaults to the console (info/warn to stdout, error to stderr); pass a `log` dict to `init` to integrate with a real logging module. Internally the loggers are stored on `.z.m` as `lginfo`/`lgwarn`/`lgerr` (not `log`, which is a q built-in) and every call site invokes them through `.z.m`. -## Example with injected dependencies +## Example with custom log and handlers config -The `log` dep is required and the `handlers` dep is optional; both are supplied by -the host application. The `log` functions are called as `(ctx;msg)` and the +Both keys are optional overrides. The `log` functions are called as `(ctx;msg)` and the `handlers` registry's `register` is called as `(.z event name; label; handler)`. ```q @@ -138,19 +147,19 @@ the host application. The `log` functions are called as `(ctx;msg)` and the / kx.log loggers take a single message, so wrap them to the (ctx;msg) shape logger:use`kx.log kxlog:logger.createLog[] -logdep:`info`warn`error!( +logdict:`info`warn`error!( {[c;m] kxlog.info[(string c),": ",m]}; {[c;m] kxlog.warn[(string c),": ",m]}; {[c;m] kxlog.error[(string c),": ",m]}) / wire up handlers via a host-provided registry -/ a real registry composes handlers so several modules can share .z.ws / .z.wc -/ omit this dep entirely to have the module assign .z.ws / .z.wc directly -hnddep:enlist[`register]!enlist {[zname;label;fn] zname set fn} +/ a real registry composes handlers so several modules can share .z.ws / .z.wc / .z.pc +/ omit this key entirely to have the module assign .z.ws / .z.wc / .z.pc directly +hnddict:enlist[`register]!enlist {[zname;label;fn] zname set fn} / initialise html module html:use`di.html -html.init[(enlist `homedir)!enlist "/opt/app/html";`log`handlers!(logdep;hnddep)] +html.init[`homedir`log`handlers!("/opt/app/html";logdict;hnddict)] ``` ## Testing diff --git a/di/html/html.q b/di/html/html.q index 57c0e384..c981d048 100644 --- a/di/html/html.q +++ b/di/html/html.q @@ -15,7 +15,16 @@ ipacache:(`int$())!`symbol$(); / home directory for html files - set by init homedir:""; -/ logging functions - required injected deps, wired by init via .z.m +/ flag so direct .z handler wiring happens only once across repeated init calls +zwired:0b; + +/ logging functions - default to the console, overridable via the log config key on init + +/ default console loggers, called as (ctx;msg) +deflog:`info`warn`error!( + {[c;m] -1 "INFO ",(string c)," ",m;}; + {[c;m] -1 "WARN ",(string c)," ",m;}; + {[c;m] -2 "ERROR ",(string c)," ",m;}); / converts a list of timestamps or dates to iso 8601 strings e.g. "2024-01-02T12:00:00Z" jstsiso8601:{[x] {("-" sv "." vs string `date$x),"T",string[`second$x],"Z"} each x}; @@ -32,7 +41,7 @@ jstsfromm:{[x] jstsfromd `date$x}; / maps kdb type shorts to their javascript converter function / types not listed here are left unchanged by jsformat -typemap:12 13 14 16 17 18 19h!(jstsiso8601;jstsfromm;jstsfromd;jstsfromt;jstsfromt;jstsfromt;jstsfromt); +typemap:12 13 14 15 16 17 18 19h!(jstsiso8601;jstsfromm;jstsfromd;jstsiso8601;jstsfromt;jstsfromt;jstsfromt;jstsfromt); jsformat:{[tbl] / applies the correct javascript converter to each column of a table that needs it @@ -50,6 +59,12 @@ updformat:{[msgtype;msgdata] :(`name`data)!(msgtype;formatteddata); }; +dataformat:{[msgtype;msgdata] + / wraps a message into a name/data dictionary, javascript-formatting each table in msgdata + / msgdata is a list or dictionary of tables - used by host data functions requested from the front end + :(`name`data)!(msgtype;jsformat each msgdata); + }; + / filter applied before sending data to a subscriber - returns full table (no filtering) sel:{[tbl;syms] tbl}; @@ -167,35 +182,46 @@ evaluate:{[inputdict] :@[execdict;inputdict;{[d;e] m:"failed to execute ",(-3!d)," : ",e;.z.m.lgerr[`html;m];'m}[inputdict]]; }; -init:{[config;deps] - / sets up module state from config and registers websocket handlers - / log dep is required; handlers dep is optional +init:{[configs] + / sets up module state and registers websocket handlers + / configs is an optional dict - recognised keys are homedir, log and handlers + / defaults: homedir from the KDBHTML env var (else "html"), console logging, direct .z handler assignment + + / default configuration values + hd:$[count e:getenv`KDBHTML;e;"html"]; + logdict:deflog; + hnd:(::); + + / set custom config values - only recognised keys are picked up + if[not configs~(::); + if[`homedir in key configs;hd:configs`homedir]; + if[`log in key configs;logdict:configs`log]; + if[`handlers in key configs;hnd:configs`handlers]]; - / log dependency is required - guard that deps is a dict, log key is present and not (::) - logdict:$[99h=type deps;$[(`log in key deps)and not(::)~deps`log;deps`log;()!()];()!()]; - if[not count logdict; - '"di.html: log dependency is required; pass `info`warn`error functions - see di.log for a default implementation"; - ]; .z.m.lginfo:logdict`info; .z.m.lgwarn:logdict`warn; .z.m.lgerr:logdict`error; + .z.m.homedir:hd; - / store config - .z.m.homedir:config`homedir; + / register .h content type handlers and static file root - protected in case not available in kdb-x + / .h.HOME lets the default http handler serve static assets (css/js/img) from homedir, as TorQ does via KDBHTML + @[{.h.HOME:x;.h.tx[`non]:{enlist x};.h.ty[`non]:"text/html"};homedir;{[e]}]; - / register .h content type handlers - protected in case not available in kdb-x - @[{.h.tx[`non]:{enlist x};.h.ty[`non]:"text/html"};`;{[e]}]; + .z.m.lginfo[`html;"initialised with homedir: ",homedir]; - / register handlers via dep if provided - if[`handlers in key deps; - deps[`handlers][`register][`.z.ws;`html.ws;wshandler]; - deps[`handlers][`register][`.z.wc;`html.close;closehandle]; + / register handlers via the handlers config if provided + / closehandle is registered for both websocket (.z.wc) and ipc (.z.pc) closes, as in TorQ + if[not hnd~(::); + hnd[`register][`.z.ws;`html.ws;wshandler]; + hnd[`register][`.z.wc;`html.close;closehandle]; + hnd[`register][`.z.pc;`html.close;closehandle]; :()]; - / no handlers dep: assign directly, wrapping any existing .z.wc to preserve it + / no handlers config: assign directly, wrapping any existing handlers to preserve them + / wire only once so repeated init calls do not stack the wrappers + if[zwired;:()]; .z.ws:wshandler; - .z.wc:@[value;`.z.wc;{{}}]; - .z.wc:{[existing;h] closehandle h; existing h}[.z.wc;]; - - .z.m.lginfo[`html;"initialised with homedir: ",homedir]; + .z.wc:{[existing;h] closehandle h; existing h}[@[value;`.z.wc;{{[x]}}];]; + .z.pc:{[existing;h] closehandle h; existing h}[@[value;`.z.pc;{{[x]}}];]; + .z.m.zwired:1b; }; diff --git a/di/html/init.q b/di/html/init.q index c86244a9..b13d5faf 100644 --- a/di/html/init.q +++ b/di/html/init.q @@ -1,4 +1,4 @@ / entry point for di.html module \l ::html.q -export:([init;addtables;pub;sub;wssub;end;readpage;readpagereplaceHP;evaluate]) +export:([init;addtables;pub;sub;wssub;end;dataformat;readpage;readpagereplaceHP;evaluate]) diff --git a/di/html/test.csv b/di/html/test.csv index 583aa277..6974123c 100644 --- a/di/html/test.csv +++ b/di/html/test.csv @@ -1,7 +1,7 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,html:use`di.html,1,,load di.html module -before,0,0,q,"mocklog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,,mock log dep that discards all messages -before,0,0,q,"html.init[(enlist `homedir)!enlist ""/tmp"";enlist[`log]!enlist[mocklog]]",1,,initialise module with /tmp as html home directory +before,0,0,q,"mocklog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,,mock log config that discards all messages +before,0,0,q,"html.init[`homedir`log!(""/tmp"";mocklog)]",1,,initialise module with /tmp as html home directory before,0,0,q,"(`:/tmp/di_html_test.html) 1: ""test""",1,,write test html file using q file io before,0,0,q,"(`:/tmp/di_html_hp_test.html) 1: ""MYKDBSERVER:MYKDBPORT""",1,,write test file containing server and port tokens before,0,0,q,htmltestfn:{x-y},1,,helper function for multi-arg evaluate test @@ -32,9 +32,19 @@ true,0,0,q,"""/tmp/di_html_notfound.html: not found""~html.readpage[""di_html_no true,0,0,q,"0=count html.readpagereplaceHP[""di_html_hp_test.html""] ss ""MYKDBSERVER""",1,,readpagereplaceHP replaces the MYKDBSERVER token true,0,0,q,"0=count html.readpagereplaceHP[""di_html_hp_test.html""] ss ""MYKDBPORT""",1,,readpagereplaceHP replaces the MYKDBPORT token +true,0,0,q,"""/tmp""~.h.HOME",1,,init sets .h.HOME to homedir so the default http handler serves static assets +true,0,0,q,"""start""~html.dataformat[""start"";enlist convtest][`name]",1,,dataformat returns the message type as name +true,0,0,q,"1704153600000~first (first html.dataformat[""start"";enlist convtest][`data])[`d]",1,,dataformat javascript-formats each table in a list +true,0,0,q,"`t1`t2~key html.dataformat[""start"";`t1`t2!(convtest;convtest)][`data]",1,,dataformat preserves table names when given a dictionary of tables +true,0,0,q,"""2024-01-02T12:00:00Z""~first html.dataformat[""start"";`t1`t2!(convtest;convtest)][`data][`t1][`ts]",1,,dataformat javascript-formats each table in a dictionary true,0,0,q,"""2024-01-02T12:00:00Z""~first (jsf convtest)`ts",1,,jsformat converts a timestamp column to an iso 8601 string true,0,0,q,"1704153600000~first (jsf convtest)`d",1,,jsformat converts a date column to javascript epoch milliseconds true,0,0,q,"43200000~first (jsf convtest)`tm",1,,jsformat converts a time column to milliseconds since midnight true,0,0,q,"`AAPL~first (jsf convtest)`sym",1,,jsformat leaves columns whose type is not in the typemap unchanged +run,0,0,q,"setenv[`KDBHTML;""/tmp""]",1,,set KDBHTML so default init picks up the TorQ env var +run,0,0,q,"html.init[enlist[`log]!enlist mocklog]",1,,init without homedir config applies defaults without error +true,0,0,q,"""/tmp""~.h.HOME",1,,default init takes homedir from the KDBHTML env var like TorQ +true,0,0,q,"""test""~html.readpage[""di_html_test.html""]",1,,readpage works against the env var derived homedir + after,0,0,q,"system ""rm /tmp/di_html_test.html /tmp/di_html_hp_test.html""",1,,remove test html files created during setup From 5f9a3fe79e904f57de0754d902ae5ad1e73a6cd4 Mon Sep 17 00:00:00 2001 From: Matthew Date: Fri, 12 Jun 2026 13:58:04 +0100 Subject: [PATCH 03/10] Changed in line with dexters comment about required logging dependency --- di/html/html.md | 19 ++++++++++--------- di/html/html.q | 41 ++++++++++++++++++++++++----------------- di/html/test.csv | 8 ++++++++ 3 files changed, 42 insertions(+), 26 deletions(-) diff --git a/di/html/html.md b/di/html/html.md index d538af4d..a5064892 100644 --- a/di/html/html.md +++ b/di/html/html.md @@ -13,10 +13,12 @@ This module does two things: ```q html:use`di.html +log:use`di.log -/ minimal setup - all config is optional -/ homedir defaults to the KDBHTML env var (else "html"), logging defaults to the console -html.init[] +/ minimal setup - the log dependency is required +/ homedir defaults to the KDBHTML env var (else "html") +logdep:`info`warn`error!(log.info;log.warn;log.error) +html.init[enlist[`log]!enlist logdep] / register tables for pub/sub html.addtables[`trades`quotes] @@ -30,16 +32,15 @@ This mirrors the original TorQ deployment: set `KDBHTML`, initialise, register t ## init ```q -html.init[] html.init[configs] ``` -`configs` is an optional dictionary — call `init[]` to use the defaults. Only recognised keys are picked up: +`configs` is a dictionary. The `` `log `` key is required — `init` signals an error if it is missing or does not contain `` `info`warn`error `` functions. Only recognised keys are picked up: | Key | Type | Description | Default | |---|---|---|---| +| `` `log `` | dict | **Required.** Logging functions with keys `` `info`warn`error ``, each called as `(ctx;msg)` — e.g. from `di.log` | — | | `homedir` | string | Path to the directory containing HTML files | `KDBHTML` env var, else `"html"` (TorQ behaviour) | -| `` `log `` | dict | Logging functions with keys `` `info`warn`error ``, each called as `(ctx;msg)` | Console loggers — info/warn to stdout, error to stderr | | `` `handlers `` | dict | Handler registry with key `` `register `` | Assigns `.z.ws`, `.z.wc` and `.z.pc` directly | ```q @@ -135,12 +136,12 @@ The module logs at three points: - On `readpage`: warns if a requested file is not found - On `evaluate`: logs at error level when a WebSocket-invoked function fails (the error is also re-thrown to the caller) -Logging defaults to the console (info/warn to stdout, error to stderr); pass a `log` dict to `init` to integrate with a real logging module. Internally the loggers are stored on `.z.m` as `lginfo`/`lgwarn`/`lgerr` (not `log`, which is a q built-in) and every call site invokes them through `.z.m`. +There is no built-in default logger — the `log` dict must be injected via `init`, typically from `di.log`. Internally the loggers are stored on `.z.m` as `lginfo`/`lgwarn`/`lgerr` (not `log`, which is a q built-in) and every call site invokes them through `.z.m`. ## Example with custom log and handlers config -Both keys are optional overrides. The `log` functions are called as `(ctx;msg)` and the -`handlers` registry's `register` is called as `(.z event name; label; handler)`. +The `log` functions are called as `(ctx;msg)` and the `handlers` registry's `register` +is called as `(.z event name; label; handler)`. ```q / wire up logging on top of the kx.log module diff --git a/di/html/html.q b/di/html/html.q index c981d048..e1c3cbd7 100644 --- a/di/html/html.q +++ b/di/html/html.q @@ -18,16 +18,17 @@ homedir:""; / flag so direct .z handler wiring happens only once across repeated init calls zwired:0b; -/ logging functions - default to the console, overridable via the log config key on init - -/ default console loggers, called as (ctx;msg) -deflog:`info`warn`error!( - {[c;m] -1 "INFO ",(string c)," ",m;}; - {[c;m] -1 "WARN ",(string c)," ",m;}; - {[c;m] -2 "ERROR ",(string c)," ",m;}); - -/ converts a list of timestamps or dates to iso 8601 strings e.g. "2024-01-02T12:00:00Z" -jstsiso8601:{[x] {("-" sv "." vs string `date$x),"T",string[`second$x],"Z"} each x}; +jstsiso8601:{[x] + / converts a list of timestamps or datetimes to iso 8601 strings e.g. "2024-01-02T12:00:00Z" + / vectorised: stringifies the date and time parts in bulk rather than per element; nulls return "" + i:where 10=count each d:string `date$x; + r:(count d)#enlist ""; + if[not count i;:r]; + dd:d i; + dd[;4 7]:"-"; + r[i]:dd,'("T",/:string `second$x i),\:"Z"; + :r; + }; / converts a list of dates to javascript epoch milliseconds / kdb dates are days since 2000-01-01; 10957 is the day offset from 1970-01-01 to 2000-01-01 @@ -184,19 +185,25 @@ evaluate:{[inputdict] init:{[configs] / sets up module state and registers websocket handlers - / configs is an optional dict - recognised keys are homedir, log and handlers - / defaults: homedir from the KDBHTML env var (else "html"), console logging, direct .z handler assignment + / configs is a dict - recognised keys are homedir, log and handlers + / log is required: `info`warn`error!(...) where each is a {[ctx;msg]} function, e.g. from di.log + / defaults: homedir from the KDBHTML env var (else "html"), direct .z handler assignment + + / log dependency is required - fail loudly rather than falling back to a default logger + logdict:$[99h=type configs;$[(`log in key configs) and not (::)~configs`log;configs`log;()!()];()!()]; + if[not count logdict; + '"di.html: log dependency is required; pass `info`warn`error functions - see di.log or refer to confluence documentation"]; + if[not 99h=type logdict;'"di.html: log must be a dict of `info`warn`error!(logging functions)"]; + if[count missing:`info`warn`error except key logdict; + '"di.html: log dict missing key(s): ",", " sv string missing]; / default configuration values hd:$[count e:getenv`KDBHTML;e;"html"]; - logdict:deflog; hnd:(::); / set custom config values - only recognised keys are picked up - if[not configs~(::); - if[`homedir in key configs;hd:configs`homedir]; - if[`log in key configs;logdict:configs`log]; - if[`handlers in key configs;hnd:configs`handlers]]; + if[`homedir in key configs;hd:configs`homedir]; + if[`handlers in key configs;hnd:configs`handlers]; .z.m.lginfo:logdict`info; .z.m.lgwarn:logdict`warn; diff --git a/di/html/test.csv b/di/html/test.csv index 6974123c..519e8490 100644 --- a/di/html/test.csv +++ b/di/html/test.csv @@ -41,10 +41,18 @@ true,0,0,q,"""2024-01-02T12:00:00Z""~first (jsf convtest)`ts",1,,jsformat conver true,0,0,q,"1704153600000~first (jsf convtest)`d",1,,jsformat converts a date column to javascript epoch milliseconds true,0,0,q,"43200000~first (jsf convtest)`tm",1,,jsformat converts a time column to milliseconds since midnight true,0,0,q,"`AAPL~first (jsf convtest)`sym",1,,jsformat leaves columns whose type is not in the typemap unchanged +true,0,0,q,"""""~first (jsf ([]ts:enlist 0Np))`ts",1,,jsformat converts null timestamps to empty strings +true,0,0,q,"0=count (jsf 0#convtest)`ts",1,,jsformat handles empty tables run,0,0,q,"setenv[`KDBHTML;""/tmp""]",1,,set KDBHTML so default init picks up the TorQ env var run,0,0,q,"html.init[enlist[`log]!enlist mocklog]",1,,init without homedir config applies defaults without error true,0,0,q,"""/tmp""~.h.HOME",1,,default init takes homedir from the KDBHTML env var like TorQ true,0,0,q,"""test""~html.readpage[""di_html_test.html""]",1,,readpage works against the env var derived homedir +fail,0,0,q,"html.init[(::)]",1,,init without a configs dict throws - the log dependency is required +fail,0,0,q,"html.init[enlist[`homedir]!enlist ""/tmp""]",1,,init with a configs dict missing the log key throws +fail,0,0,q,"html.init[enlist[`log]!enlist 42]",1,,init throws when the log value is not a dict +fail,0,0,q,"html.init[enlist[`log]!enlist enlist[`info]!enlist {[c;m]}]",1,,init throws when the log dict is missing the warn and error keys +true,0,0,q,"(@[html.init;(::);{x}]) like ""*log*""",1,,the missing log dependency error message names the log key + after,0,0,q,"system ""rm /tmp/di_html_test.html /tmp/di_html_hp_test.html""",1,,remove test html files created during setup From 4556fb19398012ae54441348ee61abc9c714573f Mon Sep 17 00:00:00 2001 From: Matthew Date: Mon, 15 Jun 2026 10:58:56 +0100 Subject: [PATCH 04/10] Removed camel case and fixed remaining unit tests --- di/html/html.md | 4 ++-- di/html/html.q | 2 +- di/html/init.q | 2 +- di/html/test.csv | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/di/html/html.md b/di/html/html.md index a5064892..571039b1 100644 --- a/di/html/html.md +++ b/di/html/html.md @@ -108,10 +108,10 @@ html.readpage[filename] Reads the file at `homedir/filename` and returns its contents as a string. Returns an error message string if the file is not found. -### readpagereplaceHP +### readpagereplacehostport ```q -html.readpagereplaceHP[filename] +html.readpagereplacehostport[filename] ``` Reads the file at `homedir/filename` and replaces `MYKDBSERVER` and `MYKDBPORT` tokens with the process's current IP address and port. Used to serve self-referencing HTML pages over HTTP. diff --git a/di/html/html.q b/di/html/html.q index e1c3cbd7..6584a82b 100644 --- a/di/html/html.q +++ b/di/html/html.q @@ -173,7 +173,7 @@ readpage:{[filename] :$[count r;"c"$r;p,": not found"]; }; -readpagereplaceHP:{[filename] +readpagereplacehostport:{[filename] / reads a page and replaces MYKDBSERVER and MYKDBPORT tokens with live server values :replace[readpage filename;`MYKDBSERVER`MYKDBPORT!("\"",(string ipa .z.a),"\"";getport[])]; }; diff --git a/di/html/init.q b/di/html/init.q index b13d5faf..a31e306e 100644 --- a/di/html/init.q +++ b/di/html/init.q @@ -1,4 +1,4 @@ / entry point for di.html module \l ::html.q -export:([init;addtables;pub;sub;wssub;end;dataformat;readpage;readpagereplaceHP;evaluate]) +export:([init;addtables;pub;sub;wssub;end;dataformat;readpage;readpagereplacehostport;evaluate]) diff --git a/di/html/test.csv b/di/html/test.csv index 519e8490..a5a78946 100644 --- a/di/html/test.csv +++ b/di/html/test.csv @@ -29,8 +29,8 @@ fail,0,0,q,"html.evaluate[(enlist `arg1)!enlist 1]",1,,evaluate throws when the true,0,0,q,"""test""~html.readpage[""di_html_test.html""]",1,,readpage returns file contents as a string true,0,0,q,"""/tmp/di_html_notfound.html: not found""~html.readpage[""di_html_notfound.html""]",1,,readpage returns a not-found message string for a missing file -true,0,0,q,"0=count html.readpagereplaceHP[""di_html_hp_test.html""] ss ""MYKDBSERVER""",1,,readpagereplaceHP replaces the MYKDBSERVER token -true,0,0,q,"0=count html.readpagereplaceHP[""di_html_hp_test.html""] ss ""MYKDBPORT""",1,,readpagereplaceHP replaces the MYKDBPORT token +true,0,0,q,"0=count html.readpagereplacehostport[""di_html_hp_test.html""] ss ""MYKDBSERVER""",1,,readpagereplacehostport replaces the MYKDBSERVER token +true,0,0,q,"0=count html.readpagereplacehostport[""di_html_hp_test.html""] ss ""MYKDBPORT""",1,,readpagereplacehostport replaces the MYKDBPORT token true,0,0,q,"""/tmp""~.h.HOME",1,,init sets .h.HOME to homedir so the default http handler serves static assets true,0,0,q,"""start""~html.dataformat[""start"";enlist convtest][`name]",1,,dataformat returns the message type as name From dd2fac3ee3bf3cdb17dbe6d584b762f362583bee Mon Sep 17 00:00:00 2001 From: Ruairi-wq2 Date: Thu, 25 Jun 2026 14:44:18 +0100 Subject: [PATCH 05/10] Refine di.html exports and fix default pub modifier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Drop wssub (thin wrapper over sub) and end (TorQ-specific .u.end broadcast) - Add setmodifier[tbl;fn] so callers can override per-table encoding - Fix default modifier to apply jsformat — timestamps and dates now convert to JS-friendly formats out of the box rather than sending raw kdb+ values - Remove updformat (dead code; jsformat is now called directly in the modifier) - Remove readpage/readpagereplacehostport from docs and test.html (TorQ relics) - Update modulesetup.q to cast JSON numeric arg2 to long for tick[] calls - All 41 tests passing Co-Authored-By: Claude Sonnet 4.6 --- di/html/html.md | 90 ++++++--------- di/html/html.q | 136 ++++++---------------- di/html/init.q | 2 +- di/html/modulesetup.q | 54 +++++++++ di/html/test.csv | 52 ++++----- di/html/test.html | 262 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 413 insertions(+), 183 deletions(-) create mode 100644 di/html/modulesetup.q create mode 100644 di/html/test.html diff --git a/di/html/html.md b/di/html/html.md index 571039b1..3ae76848 100644 --- a/di/html/html.md +++ b/di/html/html.md @@ -4,10 +4,7 @@ WebSocket pub/sub and HTML page serving module, extracted from TorQ. ## Overview -This module does two things: - -1. **Pub/sub over WebSockets** — allows browser clients to subscribe to kdb+ tables and receive live updates as data is published. -2. **HTML page serving** — reads HTML files from a configured directory and serves them over HTTP, optionally replacing server/port placeholders so the page can connect back to itself. +This module provides pub/sub over WebSockets — browser clients subscribe to kdb+ tables and receive live updates as data is published. The module also sets `.h.HOME` so the default HTTP handler serves static assets (HTML, JS, CSS) from the configured directory without any additional code. ## Usage @@ -27,28 +24,23 @@ html.addtables[`trades`quotes] html.pub[`trades;newdata] ``` -This mirrors the original TorQ deployment: set `KDBHTML`, initialise, register tables. +Set `KDBHTML` before calling `init` so static files are served from the right directory. ## init ```q -html.init[configs] +html.init[deps] ``` -`configs` is a dictionary. The `` `log `` key is required — `init` signals an error if it is missing or does not contain `` `info`warn`error `` functions. Only recognised keys are picked up: +`deps` is a dictionary. The `` `log `` key is required — `init` signals an error if it is missing or malformed. | Key | Type | Description | Default | |---|---|---|---| -| `` `log `` | dict | **Required.** Logging functions with keys `` `info`warn`error ``, each called as `(ctx;msg)` — e.g. from `di.log` | — | -| `homedir` | string | Path to the directory containing HTML files | `KDBHTML` env var, else `"html"` (TorQ behaviour) | -| `` `handlers `` | dict | Handler registry with key `` `register `` | Assigns `.z.ws`, `.z.wc` and `.z.pc` directly | +| `` `log `` | dict | **Required.** Logging functions keyed `` `info`warn`error ``, each called as `{[msg]}` — e.g. from `di.log` | — | -```q -/ override config explicitly -html.init[`homedir`log!("/opt/app/html";logdict)] -``` +The HTML home directory comes from the `KDBHTML` environment variable. If the variable is unset `"html"` is used. There is no `homedir` config key. -`init` also sets `.h.HOME` to `homedir` (protected, skipped if `.h` is unavailable) so the default HTTP handler serves static assets (css/js/img) from the same directory — the equivalent of TorQ's `KDBHTML` behaviour. +`init` also sets `.h.HOME` to the resolved home directory (protected, skipped if `.h` is unavailable) so the default HTTP handler serves static assets (css/js/img) from the same directory. Calling `init` multiple times is safe — the `.z.ws`/`.z.wc`/`.z.pc` handlers are wired only once. ## Exported functions @@ -76,21 +68,22 @@ html.sub[tbl;syms] Subscribes the current handle (`.z.w`) to `tbl`. Pass `` ` `` as `syms` to receive all data. Pass `` ` `` as `tbl` to subscribe to all registered tables. Returns `(tablename; current data)` so the subscriber can initialise their local copy of the table. -### wssub +### setmodifier ```q -html.wssub[tbl] +html.setmodifier[tbl;fn] ``` -Calls `sub[tbl;`` ` ``]`. Pass `` ` `` as `tbl` to subscribe to all registered tables. Returns nothing. +Sets a custom modifier function for `tbl`. `fn` is called on every `pub` as `fn[(\`upd;tbl;data)]` and must return bytes or a string suitable to send directly to a subscriber handle. -### end +The default modifier encodes data using the c.js binary protocol (kdb+ IPC-wrapped JSON). Use `setmodifier` to switch a table to plain JSON text — for example when subscribers are plain WebSocket clients without c.js: ```q -html.end[eodval] +html.setmodifier[`trades;{-8!.j.j `name`data!("upd";`tablename`tabledata!(x 1;x 2))}] / default (c.js binary) +html.setmodifier[`trades;{.j.j `name`data!("upd";`tablename`tabledata!(x 1;x 2))}] / plain json text ``` -Broadcasts an end-of-day message to all subscriber handles. +Signals an error if `tbl` has not been registered via `addtables`. ### dataformat @@ -100,22 +93,6 @@ html.dataformat[msgtype;msgdata] Wraps a message into a `` `name`data `` dictionary, javascript-formatting each table in `msgdata` (a list or dictionary of tables). Used by host data functions that the front end requests over the websocket, e.g. TorQ's monitor `start` call returning several tables at once. -### readpage - -```q -html.readpage[filename] -``` - -Reads the file at `homedir/filename` and returns its contents as a string. Returns an error message string if the file is not found. - -### readpagereplacehostport - -```q -html.readpagereplacehostport[filename] -``` - -Reads the file at `homedir/filename` and replaces `MYKDBSERVER` and `MYKDBPORT` tokens with the process's current IP address and port. Used to serve self-referencing HTML pages over HTTP. - ### evaluate ```q @@ -133,36 +110,37 @@ The module registers a `.z.ws` handler that receives bytes from the browser, des The module logs at three points: - On `init`: confirms the module started and which `homedir` was set -- On `readpage`: warns if a requested file is not found - On `evaluate`: logs at error level when a WebSocket-invoked function fails (the error is also re-thrown to the caller) -There is no built-in default logger — the `log` dict must be injected via `init`, typically from `di.log`. Internally the loggers are stored on `.z.m` as `lginfo`/`lgwarn`/`lgerr` (not `log`, which is a q built-in) and every call site invokes them through `.z.m`. - -## Example with custom log and handlers config +There is no built-in default logger — the `log` dict must be injected via `init`, typically from `di.log`. The log functions must be monadic (`{[msg]}`) to match the `kx.log` contract. They are stored on `.z.m` as a single `log` dict and called as `.z.m.log[\`info]["message"]`. -The `log` functions are called as `(ctx;msg)` and the `handlers` registry's `register` -is called as `(.z event name; label; handler)`. +## Example with kx.log ```q -/ wire up logging on top of the kx.log module -/ kx.log loggers take a single message, so wrap them to the (ctx;msg) shape +/ wire up logging from the kx.log module logger:use`kx.log kxlog:logger.createLog[] -logdict:`info`warn`error!( - {[c;m] kxlog.info[(string c),": ",m]}; - {[c;m] kxlog.warn[(string c),": ",m]}; - {[c;m] kxlog.error[(string c),": ",m]}) +logdep:`info`warn`error!( + {[m] kxlog.info[m]}; + {[m] kxlog.warn[m]}; + {[m] kxlog.error[m]}) -/ wire up handlers via a host-provided registry -/ a real registry composes handlers so several modules can share .z.ws / .z.wc / .z.pc -/ omit this key entirely to have the module assign .z.ws / .z.wc / .z.pc directly -hnddict:enlist[`register]!enlist {[zname;label;fn] zname set fn} - -/ initialise html module +/ initialise html module — set KDBHTML before calling init +setenv[`KDBHTML;"/opt/app/html"] html:use`di.html -html.init[`homedir`log`handlers!("/opt/app/html";logdict;hnddict)] +html.init[enlist[`log]!enlist logdep] ``` +## Browser testing (development) + +The default `.z.ws` handler uses kdb+ binary IPC format (`-9!`/`-8!`) and is designed to work with the KX c.js WebSocket library. For development testing without c.js, use the included `modulesetup.q`: + +```bash +q di/html/modulesetup.q +``` + +This starts a process on port 5678 with a plain-text JSON websocket handler. Open `test.html` in a browser (or navigate to `http://localhost:5678/test.html` once the process is running) to connect and interact with the module. Run `tick[\`trades;5]` in the q session to push live data to browser subscribers. + ## Testing ```q diff --git a/di/html/html.q b/di/html/html.q index 6584a82b..f3566faa 100644 --- a/di/html/html.q +++ b/di/html/html.q @@ -9,12 +9,6 @@ subs:()!(); / modifier function per table: transforms data before sending to subscriber modifier:()!(); -/ cache of resolved ip addresses: int -> symbol -ipacache:(`int$())!`symbol$(); - -/ home directory for html files - set by init -homedir:""; - / flag so direct .z handler wiring happens only once across repeated init calls zwired:0b; @@ -54,12 +48,6 @@ jsformat:{[tbl] :flip k!converters@'colvals; }; -updformat:{[msgtype;msgdata] - / wraps an upd message into a name/data dictionary with javascript-formatted table data - formatteddata:(key msgdata)!(msgdata`tablename;jsformat msgdata`tabledata); - :(`name`data)!(msgtype;formatteddata); - }; - dataformat:{[msgtype;msgdata] / wraps a message into a name/data dictionary, javascript-formatting each table in msgdata / msgdata is a list or dictionary of tables - used by host data functions requested from the front end @@ -94,29 +82,15 @@ closehandle:{[handle] del[;handle] each subtables; }; -replace:{[str;findreplace] - / applies each find->replace pair to the string in sequence using ssr - :(ssr/)[str;string key findreplace;value findreplace]; - }; - -ipa:{[ipint] - / resolves an ip address integer to a hostname symbol, caching results - / tries .Q.host first; falls back to converting the ip bytes manually if that fails - if[not `~r:ipacache ipint;:r]; - hostname:.Q.host ipint; - r:$[`~hostname;`$"."sv string "i"$0x0 vs ipint;hostname]; - .z.m.ipacache:@[ipacache;ipint;:;r]; - :r; - }; - -/ returns the current listen port as a string -getport:{[] string system "p"}; - execdict:{[inputdict] / extracts the func key and any additional args from a dictionary and calls the function / args are passed to the function in the order the keys appear after func + / checks the module funcmap first (module-local functions), then falls back to value for globals if[not `func in key inputdict;'"no func in dictionary"]; - f:value inputdict`func; + fname:`$inputdict`func; + f:$[fname in key funcmap; + funcmap fname; + @[value;inputdict`func;{'"unknown function: ",x}]]; args:value inputdict _ `func; :$[1=count key inputdict;f @ 1;f . args]; }; @@ -131,8 +105,9 @@ addtables:{[tablelist] new:tablelist except subtables; .z.m.subtables:subtables,new; .z.m.subs:subs,new!(count new)#(); - .z.m.modifier:modifier,new!(count new)#{-8!.j.j updformat["upd";`tablename`tabledata!(x 1;x 2)]}; - if[count new;.z.m.lginfo[`html;"registered tables: ",", " sv string new]]; + / default modifier applies jsformat then sends kdb+ IPC binary wrapping a json string (c.js binary protocol) + .z.m.modifier:modifier,new!(count new)#{-8!.j.j `name`data!("upd";`tablename`tabledata!(x 1;jsformat x 2))}; + if[count new;.z.m.log[`info]["di.html: registered tables: ",", " sv string new]]; }; pub:{[tbl;data] @@ -154,78 +129,39 @@ sub:{[tbl;syms] :add[tbl;syms]; }; -wssub:{[tbl] - / subscribes via websocket, no return value - sub[tbl;`]; - }; - -end:{[eodval] - / broadcasts end-of-day message to all subscriber handles across all tables - (neg union/[subs[;;0]])@\:(`.u.end;eodval); - }; - -readpage:{[filename] - / reads an html file from the configured home directory and returns it as a string - / returns a "not found" message string if the file does not exist - p:homedir,"/",filename; - r:@[read1;`$":",p;""]; - if[not count r;.z.m.lgwarn[`html;p,": not found"]]; - :$[count r;"c"$r;p,": not found"]; - }; - -readpagereplacehostport:{[filename] - / reads a page and replaces MYKDBSERVER and MYKDBPORT tokens with live server values - :replace[readpage filename;`MYKDBSERVER`MYKDBPORT!("\"",(string ipa .z.a),"\"";getport[])]; +setmodifier:{[tbl;fn] + / sets a custom modifier function for tbl; fn receives (`upd;tbl;data) and must return bytes or a string to send + if[not tbl in subtables;'tbl]; + .z.m.modifier:@[modifier;tbl;:;fn]; }; evaluate:{[inputdict] / safely calls execdict on the input, logging then re-throwing any errors with context - :@[execdict;inputdict;{[d;e] m:"failed to execute ",(-3!d)," : ",e;.z.m.lgerr[`html;m];'m}[inputdict]]; - }; - -init:{[configs] - / sets up module state and registers websocket handlers - / configs is a dict - recognised keys are homedir, log and handlers - / log is required: `info`warn`error!(...) where each is a {[ctx;msg]} function, e.g. from di.log - / defaults: homedir from the KDBHTML env var (else "html"), direct .z handler assignment - - / log dependency is required - fail loudly rather than falling back to a default logger - logdict:$[99h=type configs;$[(`log in key configs) and not (::)~configs`log;configs`log;()!()];()!()]; - if[not count logdict; - '"di.html: log dependency is required; pass `info`warn`error functions - see di.log or refer to confluence documentation"]; - if[not 99h=type logdict;'"di.html: log must be a dict of `info`warn`error!(logging functions)"]; - if[count missing:`info`warn`error except key logdict; - '"di.html: log dict missing key(s): ",", " sv string missing]; - - / default configuration values + :@[execdict;inputdict;{[d;e] m:"di.html: failed to execute ",(-3!d)," : ",e;.z.m.log[`error][m];'m}[inputdict]]; + }; + +/ module-local functions callable via websocket evaluate +/ execdict checks here first before falling back to value for global lookups +funcmap:`sub`addtables`pub`dataformat!(sub;addtables;pub;dataformat); + +init:{[deps] + / initialise the module; deps must contain a log key with info/warn/error functions + / deps: dict with `log key -> `info`warn`error!(fn;fn;fn) where each fn is {[msg]} + / wires .z.ws/.z.wc/.z.pc handlers once; .h.HOME set from KDBHTML env var (else "html") + if[99h<>type deps; + '"di.html: deps must be a dict with a `log key"]; + if[not `log in key deps; + '"di.html: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.html: log value must be a dict of `info`warn`error functions"]; + if[not all `info`warn`error in key deps`log; + '"di.html: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:deps`log; hd:$[count e:getenv`KDBHTML;e;"html"]; - hnd:(::); - - / set custom config values - only recognised keys are picked up - if[`homedir in key configs;hd:configs`homedir]; - if[`handlers in key configs;hnd:configs`handlers]; - - .z.m.lginfo:logdict`info; - .z.m.lgwarn:logdict`warn; - .z.m.lgerr:logdict`error; - .z.m.homedir:hd; - - / register .h content type handlers and static file root - protected in case not available in kdb-x - / .h.HOME lets the default http handler serve static assets (css/js/img) from homedir, as TorQ does via KDBHTML - @[{.h.HOME:x;.h.tx[`non]:{enlist x};.h.ty[`non]:"text/html"};homedir;{[e]}]; - - .z.m.lginfo[`html;"initialised with homedir: ",homedir]; - - / register handlers via the handlers config if provided - / closehandle is registered for both websocket (.z.wc) and ipc (.z.pc) closes, as in TorQ - if[not hnd~(::); - hnd[`register][`.z.ws;`html.ws;wshandler]; - hnd[`register][`.z.wc;`html.close;closehandle]; - hnd[`register][`.z.pc;`html.close;closehandle]; - :()]; - - / no handlers config: assign directly, wrapping any existing handlers to preserve them - / wire only once so repeated init calls do not stack the wrappers + / .h.HOME lets the default http handler serve static assets; set from KDBHTML env var (else "html") + @[{.h.HOME:x;.h.tx[`non]:{enlist x};.h.ty[`non]:"text/html"};hd;{[e]}]; + .z.m.log[`info]["di.html: initialised"]; + / wire handlers once; wrap any existing .z.wc/.z.pc to preserve them if[zwired;:()]; .z.ws:wshandler; .z.wc:{[existing;h] closehandle h; existing h}[@[value;`.z.wc;{{[x]}}];]; diff --git a/di/html/init.q b/di/html/init.q index a31e306e..6785c10c 100644 --- a/di/html/init.q +++ b/di/html/init.q @@ -1,4 +1,4 @@ / entry point for di.html module \l ::html.q -export:([init;addtables;pub;sub;wssub;end;dataformat;readpage;readpagereplacehostport;evaluate]) +export:([init;addtables;pub;sub;setmodifier;dataformat;evaluate]) diff --git a/di/html/modulesetup.q b/di/html/modulesetup.q new file mode 100644 index 00000000..e6c47321 --- /dev/null +++ b/di/html/modulesetup.q @@ -0,0 +1,54 @@ +/ di.html development test server +/ starts a plain-text-mode websocket process for browser testing without c.js +/ usage: q di/html/modulesetup.q (from kdbx-modules root, after setting QPATH) + +\p 5678 + +html:use`di.html; + +logdep:`info`warn`error!( + {-1 "[INFO] ",x}; + {-1 "[WARN] ",x}; + {-2 "[ERROR] ",x}); + +/ set KDBHTML to this file's directory so test.html can be served over http +/ .z.f is the script path; split on "/" and drop the filename component +setenv[`KDBHTML;"/" sv -1_"/" vs string .z.f]; +html.init[enlist[`log]!enlist logdep]; + +/ sample tables +trades:([]time:`timestamp$();sym:`symbol$();px:`float$();sz:`long$()); +quotes:([]time:`timestamp$();sym:`symbol$();bid:`float$();ask:`float$()); +html.addtables[`trades`quotes]; + +/ resolve the html module namespace for direct state access +hns:` sv `.m.di,first (key `.m.di) where (key `.m.di) like "*html"; + +/ override .z.ws with a plain-text json handler +/ converts string arg1 to symbol and json numeric arg2 to long for functions that need it +.z.ws:{ + d:.j.k x; + if[(`func in key d) and (d[`func] in ("sub";"wssub";"tick")); + if[`arg1 in key d; d:@[d;`arg1;`$]]]; + if[(`func in key d) and (d[`func]~"tick"); + if[`arg2 in key d; d:@[d;`arg2;"j"$]]]; + neg[.z.w] .j.j html.evaluate d; + }; + +/ generate n rows of fake data and send as plain json text directly to subscribed browser handles +/ bypasses the module modifier (which sends c.js binary) so plain browsers can receive the data +tick:{[t;n] + tm:n#.z.p; + syms:n?`AAPL`MSFT`GOOG`AMZN; + data:$[t=`trades; + ([]time:tm;sym:syms;px:100f+n?1f;sz:n?100); + ([]time:tm;sym:syms;bid:99f+n?1f;ask:101f+n?1f) + ]; + msg:.j.j `name`data!("upd";`tablename`tabledata!(t;data)); + tsubs:(get ` sv hns,`subs) t; + if[count tsubs; {[m;s] (neg s 0) m}[msg;] each tsubs]; + }; + +-1 "di.html dev server ready on port 5678"; +-1 "open test.html in browser, or browse to http://localhost:5678/test.html"; +-1 "q commands: tick[`trades;5] , tick[`quotes;5]"; diff --git a/di/html/test.csv b/di/html/test.csv index a5a78946..a8c8e71c 100644 --- a/di/html/test.csv +++ b/di/html/test.csv @@ -1,9 +1,8 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,html:use`di.html,1,,load di.html module -before,0,0,q,"mocklog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,,mock log config that discards all messages -before,0,0,q,"html.init[`homedir`log!(""/tmp"";mocklog)]",1,,initialise module with /tmp as html home directory -before,0,0,q,"(`:/tmp/di_html_test.html) 1: ""test""",1,,write test html file using q file io -before,0,0,q,"(`:/tmp/di_html_hp_test.html) 1: ""MYKDBSERVER:MYKDBPORT""",1,,write test file containing server and port tokens +before,0,0,q,"mocklog:`info`warn`error!({[m]};{[m]};{[m]})",1,,mock log dep that discards all messages (monadic {[msg]} matching kx.log contract) +before,0,0,q,"setenv[`KDBHTML;""/tmp""]",1,,set KDBHTML so init picks up homedir from env +before,0,0,q,"html.init[enlist[`log]!enlist mocklog]",1,,initialise module with log dep only before,0,0,q,htmltestfn:{x-y},1,,helper function for multi-arg evaluate test before,0,0,q,"trades:([]sym:`g#`symbol$();px:`float$())",1,,real global table so sub success path can read its contents before,0,0,q,"quotes:([]sym:`g#`symbol$();bid:`float$())",1,,real global table so subscribing to all tables can read its contents @@ -12,27 +11,28 @@ before,0,0,q,"jsf:get ` sv hns,`jsformat",1,,grab the internal jsformat converte before,0,0,q,"convtest:([]ts:enlist 2024.01.02D12:00:00.000000000;d:enlist 2024.01.02;tm:enlist 12:00:00.000;sym:enlist `AAPL)",1,,sample table with one column of each time type plus an untyped sym column run,0,0,q,"html.addtables[`trades`quotes]",1,,register two tables for pub/sub +true,0,0,q,"`trades`quotes~get ` sv hns,`subtables",1,,addtables stores both table names in module subtables list +true,0,0,q,"2=count get ` sv hns,`subs",1,,addtables initialises a subscriber list for each registered table +true,0,0,q,"2=count get ` sv hns,`modifier",1,,addtables creates a default modifier for each registered table run,0,0,q,"html.addtables[`trades]",1,,re-registering an existing table does not error +true,0,0,q,"2=count get ` sv hns,`subtables",1,,re-registering a known table leaves subtables count unchanged run,0,0,q,"html.pub[`trades;([]a:enlist 1i)]",1,,pub with no subscribers completes without error fail,0,0,q,"html.sub[`notregistered;`]",1,,subscribing to an unregistered table throws an error -run,0,0,q,"html.end[`eod]",1,,end is a clean no-op when there are no subscribers (tested before any console sub) true,0,0,q,"`trades~first html.sub[`trades;`]",1,,sub success path returns the table name as first element true,0,0,q,"0=count last html.sub[`trades;`]",1,,sub success path returns the current (empty) table contents -run,0,0,q,"html.wssub[`trades]",1,,wssub delegates to sub without error +true,0,0,q,"1=count(get ` sv hns,`subs)[`trades]",1,,sub adds the current handle to the trades subscriber list run,0,0,q,"html.sub[`;`]",1,,subscribing with backtick subscribes to all registered tables +true,0,0,q,"all 1=count each(get ` sv hns,`subs)each `trades`quotes",1,,subscribing with backtick registers handle across all tables -true,0,0,q,"-42~html.evaluate[`func`arg1!(""neg"";42)]",1,,evaluate calls a function with one arg correctly -true,0,0,q,"-1~html.evaluate[`func`arg1`arg2!(""htmltestfn"";1;2)]",1,,evaluate calls a function with multiple args correctly +true,0,0,q,"-42~html.evaluate[`func`arg1!(""neg"";42)]",1,,evaluate calls a global function with one arg correctly +true,0,0,q,"-1~html.evaluate[`func`arg1`arg2!(""htmltestfn"";1;2)]",1,,evaluate calls a global function with multiple args correctly true,0,0,q,"-1~html.evaluate[(enlist `func)!enlist ""neg""]",1,,evaluate with only func key calls f@1 - preserving TorQ behaviour +true,0,0,q,"all `sub`addtables`pub`dataformat in key get ` sv hns,`funcmap",1,,funcmap contains the module functions callable via websocket dispatch +true,0,0,q,"not any `wssub`end in key get ` sv hns,`funcmap",1,,wssub and end are not in funcmap fail,0,0,q,"html.evaluate[(enlist `arg1)!enlist 1]",1,,evaluate throws when the func key is missing -true,0,0,q,"""test""~html.readpage[""di_html_test.html""]",1,,readpage returns file contents as a string -true,0,0,q,"""/tmp/di_html_notfound.html: not found""~html.readpage[""di_html_notfound.html""]",1,,readpage returns a not-found message string for a missing file -true,0,0,q,"0=count html.readpagereplacehostport[""di_html_hp_test.html""] ss ""MYKDBSERVER""",1,,readpagereplacehostport replaces the MYKDBSERVER token -true,0,0,q,"0=count html.readpagereplacehostport[""di_html_hp_test.html""] ss ""MYKDBPORT""",1,,readpagereplacehostport replaces the MYKDBPORT token - -true,0,0,q,"""/tmp""~.h.HOME",1,,init sets .h.HOME to homedir so the default http handler serves static assets +true,0,0,q,"""/tmp""~.h.HOME",1,,init sets .h.HOME so the default http handler serves static assets true,0,0,q,"""start""~html.dataformat[""start"";enlist convtest][`name]",1,,dataformat returns the message type as name true,0,0,q,"1704153600000~first (first html.dataformat[""start"";enlist convtest][`data])[`d]",1,,dataformat javascript-formats each table in a list true,0,0,q,"`t1`t2~key html.dataformat[""start"";`t1`t2!(convtest;convtest)][`data]",1,,dataformat preserves table names when given a dictionary of tables @@ -43,16 +43,16 @@ true,0,0,q,"43200000~first (jsf convtest)`tm",1,,jsformat converts a time column true,0,0,q,"`AAPL~first (jsf convtest)`sym",1,,jsformat leaves columns whose type is not in the typemap unchanged true,0,0,q,"""""~first (jsf ([]ts:enlist 0Np))`ts",1,,jsformat converts null timestamps to empty strings true,0,0,q,"0=count (jsf 0#convtest)`ts",1,,jsformat handles empty tables - -run,0,0,q,"setenv[`KDBHTML;""/tmp""]",1,,set KDBHTML so default init picks up the TorQ env var -run,0,0,q,"html.init[enlist[`log]!enlist mocklog]",1,,init without homedir config applies defaults without error -true,0,0,q,"""/tmp""~.h.HOME",1,,default init takes homedir from the KDBHTML env var like TorQ -true,0,0,q,"""test""~html.readpage[""di_html_test.html""]",1,,readpage works against the env var derived homedir - -fail,0,0,q,"html.init[(::)]",1,,init without a configs dict throws - the log dependency is required -fail,0,0,q,"html.init[enlist[`homedir]!enlist ""/tmp""]",1,,init with a configs dict missing the log key throws -fail,0,0,q,"html.init[enlist[`log]!enlist 42]",1,,init throws when the log value is not a dict -fail,0,0,q,"html.init[enlist[`log]!enlist enlist[`info]!enlist {[c;m]}]",1,,init throws when the log dict is missing the warn and error keys +run,0,0,q,"setenv[`KDBHTML;""/tmp""]",1,,re-set KDBHTML and call init a second time to verify repeated init is safe +run,0,0,q,"html.init[enlist[`log]!enlist mocklog]",1,,repeated init with valid deps does not error +true,0,0,q,"""/tmp""~.h.HOME",1,,homedir from KDBHTML env var is applied on each init + +fail,0,0,q,"html.setmodifier[`notregistered;{x}]",1,,setmodifier throws for a table that has not been registered +run,0,0,q,"html.setmodifier[`trades;{42}]",1,,set a custom modifier for trades +true,0,0,q,"42~(get ` sv hns,`modifier)[`trades]@()",1,,setmodifier replaces the per-table modifier function + +fail,0,0,q,"html.init[(::)]",1,,init rejects a non-dict deps arg +fail,0,0,q,"html.init[enlist[`homedir]!enlist ""/tmp""]",1,,init rejects a dict missing the log key +fail,0,0,q,"html.init[enlist[`log]!enlist 42]",1,,init rejects a non-dict log value +fail,0,0,q,"html.init[enlist[`log]!enlist enlist[`info]!enlist {[m]}]",1,,init rejects a log dict missing the warn and error keys true,0,0,q,"(@[html.init;(::);{x}]) like ""*log*""",1,,the missing log dependency error message names the log key - -after,0,0,q,"system ""rm /tmp/di_html_test.html /tmp/di_html_hp_test.html""",1,,remove test html files created during setup diff --git a/di/html/test.html b/di/html/test.html new file mode 100644 index 00000000..9dbc61b6 --- /dev/null +++ b/di/html/test.html @@ -0,0 +1,262 @@ + + + + +di.html module test + + + +

di.html module test disconnected

+ +
+ + +
+

1. connection

+ + +
click connect — opens ws://localhost:PORT
+
+ + +
+

2. sub / wssub

+ + + +
+
subscribe → returns (tablename; snapshot)
+
+ + +
+

3. pub — live updates

+ + + + calls tick[] in modulesetup.q via evaluate — subscribe first +
+
no updates yet — subscribe to a table first, then click a tick button above
+
+
+ + +
+

4. evaluate — call q functions

+ +
+ + +
+
result appears here
+
+ + +
+

log

+
+
+ +
+ + + + From aac12b3cd0215b8f1acb05c75c746b1c613a9372 Mon Sep 17 00:00:00 2001 From: Ruairi-wq2 Date: Tue, 30 Jun 2026 14:05:52 +0100 Subject: [PATCH 06/10] Align di.html log contract with PR #108 standard and fix wssub - Log functions updated to dyadic {[ctx;msg]} contract; kx.log instances normalised via normlog to wrap monadic functions into the required form - init validates log at minimum an `info key; warn/error not required - modulesetup.q renamed to integrationtest.q with dynamic port via -p flag - test.html subscription fixed to call sub via evaluate (wssub removed); snapshot now rendered into table panel on subscribe - Docs updated to reflect log contract, normlog behaviour, and sub dispatch Co-Authored-By: Claude Sonnet 4.6 --- di/html/html.md | 22 ++++++++++--- di/html/html.q | 34 +++++++++++++------- di/html/{modulesetup.q => integrationtest.q} | 22 +++++++------ di/html/test.csv | 6 ++-- di/html/test.html | 11 +++---- 5 files changed, 61 insertions(+), 34 deletions(-) rename di/html/{modulesetup.q => integrationtest.q} (72%) diff --git a/di/html/html.md b/di/html/html.md index 3ae76848..eae343ed 100644 --- a/di/html/html.md +++ b/di/html/html.md @@ -36,7 +36,7 @@ html.init[deps] | Key | Type | Description | Default | |---|---|---|---| -| `` `log `` | dict | **Required.** Logging functions keyed `` `info`warn`error ``, each called as `{[msg]}` — e.g. from `di.log` | — | +| `` `log `` | dict | **Required.** Logging functions keyed on at minimum `` `info ``, each called as `{[ctx;msg]}` where `ctx` is a symbol and `msg` is a string. kx.log instances are normalised automatically. | — | The HTML home directory comes from the `KDBHTML` environment variable. If the variable is unset `"html"` is used. There is no `homedir` config key. @@ -103,7 +103,19 @@ Takes a q dictionary (already deserialised from JSON), extracts the `func` key, ## WebSocket handler -The module registers a `.z.ws` handler that receives bytes from the browser, deserialises them to a q dict, calls `evaluate`, JSON-encodes the result, and sends it back. Subscriptions are cleaned up when a connection closes via both `.z.wc` (websocket) and `.z.pc` (IPC), as in TorQ — `sub` can also be called over a plain IPC handle. In the direct-assignment path (no `handlers` config) any existing `.z.wc`/`.z.pc` handlers are preserved by wrapping, and the wiring happens only once across repeated `init` calls. +The module registers a `.z.ws` handler that receives bytes from the browser, deserialises them to a q dict, calls `evaluate`, JSON-encodes the result, and sends it back. + +Browser clients subscribe by calling `sub` through the evaluate dispatch — the same mechanism used for any funcmap function. There is no separate `wssub` function; `sub` handles both websocket and IPC callers: + +```json +{"func": "sub", "arg1": "trades", "arg2": ""} +``` + +`arg1` is the table name (empty string subscribes to all tables), `arg2` is the sym filter (empty string means all syms). The caller is responsible for converting these to q symbols before calling `evaluate` — `integrationtest.q` shows the pattern. + +`sub` returns `(tablename; current data)` as the reply, giving the subscriber an initial snapshot to render before live updates arrive. + +Subscriptions are cleaned up when a connection closes via both `.z.wc` (websocket) and `.z.pc` (IPC). Any existing `.z.wc`/`.z.pc` handlers are preserved by wrapping, and the wiring happens only once across repeated `init` calls. ## Logging @@ -112,7 +124,7 @@ The module logs at three points: - On `init`: confirms the module started and which `homedir` was set - On `evaluate`: logs at error level when a WebSocket-invoked function fails (the error is also re-thrown to the caller) -There is no built-in default logger — the `log` dict must be injected via `init`, typically from `di.log`. The log functions must be monadic (`{[msg]}`) to match the `kx.log` contract. They are stored on `.z.m` as a single `log` dict and called as `.z.m.log[\`info]["message"]`. +The `log` dict must be injected via `init`. Log functions must be dyadic (`{[ctx;msg]}`) — `ctx` is a symbol tag and `msg` is a string. kx.log instances (detected by the presence of `getlvl`/`sinks`/`fmts` keys) are wrapped automatically into the dyadic contract by prepending `string[ctx],": "` to the message. Stored on `.z.m` and called as `.z.m.log[\`info][\`di.html;"message"]`. The module uses all three levels (`info`, `warn`, `error`) internally, though only `info` is required by validation. ## Example with kx.log @@ -136,10 +148,10 @@ html.init[enlist[`log]!enlist logdep] The default `.z.ws` handler uses kdb+ binary IPC format (`-9!`/`-8!`) and is designed to work with the KX c.js WebSocket library. For development testing without c.js, use the included `modulesetup.q`: ```bash -q di/html/modulesetup.q +q di/html/integrationtest.q -p 5678 ``` -This starts a process on port 5678 with a plain-text JSON websocket handler. Open `test.html` in a browser (or navigate to `http://localhost:5678/test.html` once the process is running) to connect and interact with the module. Run `tick[\`trades;5]` in the q session to push live data to browser subscribers. +This starts a process on the given port with a plain-text JSON websocket handler. Open `test.html` in a browser (or navigate to `http://localhost:5678/test.html`) to connect and interact with the module. Run `tick[\`trades;5]` in the q session to push live data to browser subscribers. ## Testing diff --git a/di/html/html.q b/di/html/html.q index f3566faa..42bdf2bb 100644 --- a/di/html/html.q +++ b/di/html/html.q @@ -12,6 +12,17 @@ modifier:()!(); / flag so direct .z handler wiring happens only once across repeated init calls zwired:0b; +/ normalise a log dict: if it looks like a kx.log instance (has getlvl/sinks/fmts keys), +/ wrap each level into a dyadic {[c;m]} function that embeds the context symbol into the message +normlog:{[logdict] + $[any `getlvl`sinks`fmts in key logdict; + `info`warn`error!( + {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); + logdict]; + }; + jstsiso8601:{[x] / converts a list of timestamps or datetimes to iso 8601 strings e.g. "2024-01-02T12:00:00Z" / vectorised: stringifies the date and time parts in bulk rather than per element; nulls return "" @@ -107,7 +118,7 @@ addtables:{[tablelist] .z.m.subs:subs,new!(count new)#(); / default modifier applies jsformat then sends kdb+ IPC binary wrapping a json string (c.js binary protocol) .z.m.modifier:modifier,new!(count new)#{-8!.j.j `name`data!("upd";`tablename`tabledata!(x 1;jsformat x 2))}; - if[count new;.z.m.log[`info]["di.html: registered tables: ",", " sv string new]]; + if[count new;.z.m.log[`info][`di.html;"registered tables: ",", " sv string new]]; }; pub:{[tbl;data] @@ -137,7 +148,7 @@ setmodifier:{[tbl;fn] evaluate:{[inputdict] / safely calls execdict on the input, logging then re-throwing any errors with context - :@[execdict;inputdict;{[d;e] m:"di.html: failed to execute ",(-3!d)," : ",e;.z.m.log[`error][m];'m}[inputdict]]; + :@[execdict;inputdict;{[d;e] m:"failed to execute ",(-3!d)," : ",e;.z.m.log[`error][`di.html;m];'"di.html: ",m}[inputdict]]; }; / module-local functions callable via websocket evaluate @@ -145,22 +156,23 @@ evaluate:{[inputdict] funcmap:`sub`addtables`pub`dataformat!(sub;addtables;pub;dataformat); init:{[deps] - / initialise the module; deps must contain a log key with info/warn/error functions - / deps: dict with `log key -> `info`warn`error!(fn;fn;fn) where each fn is {[msg]} + / initialise the module; deps must contain a log key + / deps: dict with `log key -> dict with at minimum `info!{[c;m]} (dyadic: ctx symbol, msg string) + / kx.log instances are normalised automatically via normlog / wires .z.ws/.z.wc/.z.pc handlers once; .h.HOME set from KDBHTML env var (else "html") if[99h<>type deps; - '"di.html: deps must be a dict with a `log key"]; + '"di.html: deps must be a dict with `log key"]; if[not `log in key deps; - '"di.html: log dependency is required; pass `info`warn`error functions keyed on `log"]; + '"di.html: log dependency is required; pass at minimum `info!{[c;m]} keyed on `log"]; if[99h<>type deps`log; - '"di.html: log value must be a dict of `info`warn`error functions"]; - if[not all `info`warn`error in key deps`log; - '"di.html: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:deps`log; + '"di.html: log value must be a dict; pass at minimum `info!{[c;m]}"]; + if[not `info in key deps`log; + '"di.html: log dict must have at minimum an `info key; got: ",(", " sv string key deps`log)]; + .z.m.log:normlog deps`log; hd:$[count e:getenv`KDBHTML;e;"html"]; / .h.HOME lets the default http handler serve static assets; set from KDBHTML env var (else "html") @[{.h.HOME:x;.h.tx[`non]:{enlist x};.h.ty[`non]:"text/html"};hd;{[e]}]; - .z.m.log[`info]["di.html: initialised"]; + .z.m.log[`info][`di.html;"initialised"]; / wire handlers once; wrap any existing .z.wc/.z.pc to preserve them if[zwired;:()]; .z.ws:wshandler; diff --git a/di/html/modulesetup.q b/di/html/integrationtest.q similarity index 72% rename from di/html/modulesetup.q rename to di/html/integrationtest.q index e6c47321..dadf9a83 100644 --- a/di/html/modulesetup.q +++ b/di/html/integrationtest.q @@ -1,15 +1,16 @@ -/ di.html development test server +/ di.html integration test / starts a plain-text-mode websocket process for browser testing without c.js -/ usage: q di/html/modulesetup.q (from kdbx-modules root, after setting QPATH) +/ usage: q di/html/integrationtest.q (from kdbx-modules root, after setting QPATH) +/ pass -p PORT to use a specific port; otherwise the OS assigns a free port -\p 5678 +if[0=system"p"; system"p 0"]; html:use`di.html; logdep:`info`warn`error!( - {-1 "[INFO] ",x}; - {-1 "[WARN] ",x}; - {-2 "[ERROR] ",x}); + {[c;m] -1 "[INFO] ",string[c]," ",m}; + {[c;m] -1 "[WARN] ",string[c]," ",m}; + {[c;m] -2 "[ERROR] ",string[c]," ",m}); / set KDBHTML to this file's directory so test.html can be served over http / .z.f is the script path; split on "/" and drop the filename component @@ -28,8 +29,10 @@ hns:` sv `.m.di,first (key `.m.di) where (key `.m.di) like "*html"; / converts string arg1 to symbol and json numeric arg2 to long for functions that need it .z.ws:{ d:.j.k x; - if[(`func in key d) and (d[`func] in ("sub";"wssub";"tick")); + if[(`func in key d) and (d[`func] in ("sub";"tick")); if[`arg1 in key d; d:@[d;`arg1;`$]]]; + if[(`func in key d) and (d[`func]~"sub"); + if[`arg2 in key d; d:@[d;`arg2;`$]]]; if[(`func in key d) and (d[`func]~"tick"); if[`arg2 in key d; d:@[d;`arg2;"j"$]]]; neg[.z.w] .j.j html.evaluate d; @@ -49,6 +52,7 @@ tick:{[t;n] if[count tsubs; {[m;s] (neg s 0) m}[msg;] each tsubs]; }; --1 "di.html dev server ready on port 5678"; --1 "open test.html in browser, or browse to http://localhost:5678/test.html"; +p:system"p"; +-1 "di.html integration test ready on port ",p; +-1 "open test.html in browser, or browse to http://localhost:",p,"/test.html"; -1 "q commands: tick[`trades;5] , tick[`quotes;5]"; diff --git a/di/html/test.csv b/di/html/test.csv index a8c8e71c..b00de1e6 100644 --- a/di/html/test.csv +++ b/di/html/test.csv @@ -1,6 +1,6 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,html:use`di.html,1,,load di.html module -before,0,0,q,"mocklog:`info`warn`error!({[m]};{[m]};{[m]})",1,,mock log dep that discards all messages (monadic {[msg]} matching kx.log contract) +before,0,0,q,"mocklog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,,mock log dep that discards all messages (dyadic {[ctx;msg]} matching di.* log contract) before,0,0,q,"setenv[`KDBHTML;""/tmp""]",1,,set KDBHTML so init picks up homedir from env before,0,0,q,"html.init[enlist[`log]!enlist mocklog]",1,,initialise module with log dep only before,0,0,q,htmltestfn:{x-y},1,,helper function for multi-arg evaluate test @@ -54,5 +54,5 @@ true,0,0,q,"42~(get ` sv hns,`modifier)[`trades]@()",1,,setmodifier replaces the fail,0,0,q,"html.init[(::)]",1,,init rejects a non-dict deps arg fail,0,0,q,"html.init[enlist[`homedir]!enlist ""/tmp""]",1,,init rejects a dict missing the log key fail,0,0,q,"html.init[enlist[`log]!enlist 42]",1,,init rejects a non-dict log value -fail,0,0,q,"html.init[enlist[`log]!enlist enlist[`info]!enlist {[m]}]",1,,init rejects a log dict missing the warn and error keys -true,0,0,q,"(@[html.init;(::);{x}]) like ""*log*""",1,,the missing log dependency error message names the log key +run,0,0,q,"html.init[enlist[`log]!enlist enlist[`info]!enlist {[c;m]}]",1,,log dict with only an info key is accepted (warn/error not required by validation) +true,0,0,q,"(@[html.init;(::);{x}]) like ""*log*""",1,,init with a non-dict arg signals an error naming log diff --git a/di/html/test.html b/di/html/test.html index 9dbc61b6..0f896697 100644 --- a/di/html/test.html +++ b/di/html/test.html @@ -66,7 +66,7 @@

3. pub — live updates

- calls tick[] in modulesetup.q via evaluate — subscribe first + calls tick[] in integrationtest.q via evaluate — subscribe first
no updates yet — subscribe to a table first, then click a tick button above
@@ -169,17 +169,16 @@

log

// section 2: subscribe function wssubTo(tbl) { - send({func:"wssub", arg1:tbl}, function(r) { + send({func:"sub", arg1:tbl, arg2:""}, function(r) { var tables = Array.isArray(r[0]) ? r : [r]; tables.forEach(function(pair) { if (!pair || !pair[0]) return; var name = pair[0]; subscribed[name] = true; updateSubBadges(); - var snap = pair[1]; - setResult("sub-result", - name+": snapshot "+JSON.stringify(snap).slice(0,80)+(JSON.stringify(snap).length>80?"...":""), - "ok"); + var snap = Array.isArray(pair[1]) ? pair[1] : []; + renderTable({name:"upd", data:{tablename:name, tabledata:snap}}); + setResult("sub-result", name+": subscribed (snapshot: "+snap.length+" rows)", "ok"); }); }); } From 20d04ed6840f6a2d7a7e639e56f5476658c1098c Mon Sep 17 00:00:00 2001 From: Ruairi-wq2 Date: Tue, 30 Jun 2026 14:06:15 +0100 Subject: [PATCH 07/10] Remove remaining wssub label references from test.html Co-Authored-By: Claude Sonnet 4.6 --- di/html/test.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/di/html/test.html b/di/html/test.html index 0f896697..4ec6872f 100644 --- a/di/html/test.html +++ b/di/html/test.html @@ -52,10 +52,10 @@

1. connection

-

2. sub / wssub

- - - +

2. sub

+ + +
subscribe → returns (tablename; snapshot)
From c33198a29b1ca43ac038d7cdec4f24dd11ea8beb Mon Sep 17 00:00:00 2001 From: Ruairi-wq2 Date: Tue, 30 Jun 2026 14:15:25 +0100 Subject: [PATCH 08/10] Restructure html.md to match eodtime module doc format Sections now follow: Features, How it works, Dependencies, Initialisation, Exported Functions, Usage Example, Running Tests, Notes. Added practical setup explanation (q process + HTML files on same port). Co-Authored-By: Claude Sonnet 4.6 --- di/html/html.md | 150 +++++++++++++++++++++++------------------------- 1 file changed, 73 insertions(+), 77 deletions(-) diff --git a/di/html/html.md b/di/html/html.md index eae343ed..09384f0a 100644 --- a/di/html/html.md +++ b/di/html/html.md @@ -1,48 +1,50 @@ # di.html -WebSocket pub/sub and HTML page serving module, extracted from TorQ. +WebSocket pub/sub and HTML page serving module for kdb+. -## Overview +## Features -This module provides pub/sub over WebSockets — browser clients subscribe to kdb+ tables and receive live updates as data is published. The module also sets `.h.HOME` so the default HTTP handler serves static assets (HTML, JS, CSS) from the configured directory without any additional code. +- Register kdb+ tables for live pub/sub over WebSockets +- Push updates to all subscribed browser handles via `pub` +- Serve static assets (HTML, JS, CSS) from a configurable directory using kdb's built-in HTTP handler — no separate web server required +- Browser clients subscribe via the `sub` evaluate dispatch and receive an initial snapshot on connection +- Custom per-table modifier functions to control serialisation (c.js binary or plain JSON text) +- Connections cleaned up automatically on WebSocket close (`.z.wc`) or IPC port close (`.z.pc`) -## Usage +## How it works -```q -html:use`di.html -log:use`di.log +A single q process does two jobs on the same port: -/ minimal setup - the log dependency is required -/ homedir defaults to the KDBHTML env var (else "html") -logdep:`info`warn`error!(log.info;log.warn;log.error) -html.init[enlist[`log]!enlist logdep] +1. **Serves the HTML page** — kdb's built-in HTTP handler serves static files from the directory pointed to by `KDBHTML`. When a browser requests `http://host:port/index.html`, kdb reads and returns the file. +2. **Handles WebSocket connections** — once the page loads, the browser opens a WebSocket back to the same `host:port`. The `.z.ws` handler receives messages, dispatches via `evaluate`, and sends replies. Live updates are pushed to subscribed handles via `pub`. -/ register tables for pub/sub -html.addtables[`trades`quotes] +You need two things: -/ publish data to subscribers -html.pub[`trades;newdata] -``` +- A **q process** started with `-p PORT`, with di.html loaded and `init` called +- **HTML/JS/CSS files** in the directory pointed to by `KDBHTML` + +## Dependencies + +| Key | Type | Required | Description | +|---|---|---|---| +| `` `log `` | dict | yes | Logging functions keyed on at minimum `` `info ``, each `{[ctx;msg]}` where `ctx` is a symbol and `msg` is a string. kx.log instances are normalised automatically. | -Set `KDBHTML` before calling `init` so static files are served from the right directory. +The HTML home directory is configured via the `KDBHTML` environment variable (not a deps key). If unset, `"html"` is used. Set it before calling `init`. -## init +## Initialisation ```q html.init[deps] ``` -`deps` is a dictionary. The `` `log `` key is required — `init` signals an error if it is missing or malformed. - -| Key | Type | Description | Default | -|---|---|---|---| -| `` `log `` | dict | **Required.** Logging functions keyed on at minimum `` `info ``, each called as `{[ctx;msg]}` where `ctx` is a symbol and `msg` is a string. kx.log instances are normalised automatically. | — | +`deps` is a dictionary containing the `log` key. `init`: -The HTML home directory comes from the `KDBHTML` environment variable. If the variable is unset `"html"` is used. There is no `homedir` config key. +- Normalises the log dependency (wraps kx.log monadic functions into the dyadic contract) +- Sets `.h.HOME` from `KDBHTML` so the default HTTP handler serves static assets +- Registers `.z.ws`, `.z.wc`, and `.z.pc` handlers — wired once only; any existing `.z.wc`/`.z.pc` handlers are preserved by wrapping +- Safe to call multiple times; handler wiring is skipped on subsequent calls -`init` also sets `.h.HOME` to the resolved home directory (protected, skipped if `.h` is unavailable) so the default HTTP handler serves static assets (css/js/img) from the same directory. Calling `init` multiple times is safe — the `.z.ws`/`.z.wc`/`.z.pc` handlers are wired only once. - -## Exported functions +## Exported Functions ### addtables @@ -50,7 +52,7 @@ The HTML home directory comes from the `KDBHTML` environment variable. If the va html.addtables[tablelist] ``` -Registers a list of table names for pub/sub. Can be called multiple times to add new tables. Sets a default modifier that JSON-encodes updates before sending to subscribers. +Registers a list of table names for pub/sub. Can be called multiple times to add new tables. Sets a default modifier per table that JSON-encodes updates before sending to subscribers. ### pub @@ -58,7 +60,7 @@ Registers a list of table names for pub/sub. Can be called multiple times to add html.pub[tbl;data] ``` -Publishes `data` for `tbl` to all currently subscribed handles. +Publishes `data` for `tbl` to all currently subscribed handles, applying the per-table modifier before sending. ### sub @@ -66,7 +68,15 @@ Publishes `data` for `tbl` to all currently subscribed handles. html.sub[tbl;syms] ``` -Subscribes the current handle (`.z.w`) to `tbl`. Pass `` ` `` as `syms` to receive all data. Pass `` ` `` as `tbl` to subscribe to all registered tables. Returns `(tablename; current data)` so the subscriber can initialise their local copy of the table. +Subscribes the current handle (`.z.w`) to `tbl`. Pass `` ` `` as `syms` to receive all data. Pass `` ` `` as `tbl` to subscribe to all registered tables. Returns `(tablename; current data)` so the subscriber can initialise their local copy before live updates arrive. + +Browser clients call `sub` via the evaluate dispatch: + +```json +{"func": "sub", "arg1": "trades", "arg2": ""} +``` + +`arg1` is the table name (empty string = all tables), `arg2` is the sym filter (empty string = all syms). The caller is responsible for converting strings to q symbols before passing to `evaluate`. ### setmodifier @@ -74,13 +84,12 @@ Subscribes the current handle (`.z.w`) to `tbl`. Pass `` ` `` as `syms` to recei html.setmodifier[tbl;fn] ``` -Sets a custom modifier function for `tbl`. `fn` is called on every `pub` as `fn[(\`upd;tbl;data)]` and must return bytes or a string suitable to send directly to a subscriber handle. +Sets a custom modifier function for `tbl`. `fn` receives `(\`upd;tbl;data)` and must return bytes or a string to send directly to subscribers. -The default modifier encodes data using the c.js binary protocol (kdb+ IPC-wrapped JSON). Use `setmodifier` to switch a table to plain JSON text — for example when subscribers are plain WebSocket clients without c.js: +The default modifier uses the c.js binary protocol (kdb+ IPC-wrapped JSON). Switch to plain JSON text for clients without c.js: ```q -html.setmodifier[`trades;{-8!.j.j `name`data!("upd";`tablename`tabledata!(x 1;x 2))}] / default (c.js binary) -html.setmodifier[`trades;{.j.j `name`data!("upd";`tablename`tabledata!(x 1;x 2))}] / plain json text +html.setmodifier[`trades;{.j.j `name`data!("upd";`tablename`tabledata!(x 1;x 2))}] ``` Signals an error if `tbl` has not been registered via `addtables`. @@ -91,7 +100,7 @@ Signals an error if `tbl` has not been registered via `addtables`. html.dataformat[msgtype;msgdata] ``` -Wraps a message into a `` `name`data `` dictionary, javascript-formatting each table in `msgdata` (a list or dictionary of tables). Used by host data functions that the front end requests over the websocket, e.g. TorQ's monitor `start` call returning several tables at once. +Wraps a message into a `` `name`data `` dictionary, javascript-formatting each table in `msgdata` (a list or dictionary of tables). Useful for request/reply calls where the front end asks for several tables at once. ### evaluate @@ -99,67 +108,54 @@ Wraps a message into a `` `name`data `` dictionary, javascript-formatting each t html.evaluate[inputdict] ``` -Takes a q dictionary (already deserialised from JSON), extracts the `func` key, calls the named function with any additional keys as arguments, and returns the result. Used internally by the `.z.ws` handler — the handler does the JSON deserialisation before calling this function. - -## WebSocket handler - -The module registers a `.z.ws` handler that receives bytes from the browser, deserialises them to a q dict, calls `evaluate`, JSON-encodes the result, and sends it back. - -Browser clients subscribe by calling `sub` through the evaluate dispatch — the same mechanism used for any funcmap function. There is no separate `wssub` function; `sub` handles both websocket and IPC callers: - -```json -{"func": "sub", "arg1": "trades", "arg2": ""} -``` - -`arg1` is the table name (empty string subscribes to all tables), `arg2` is the sym filter (empty string means all syms). The caller is responsible for converting these to q symbols before calling `evaluate` — `integrationtest.q` shows the pattern. +Extracts the `func` key from a q dictionary, calls the named function with any additional keys as arguments, and returns the result. Used internally by `.z.ws` — the handler deserialises JSON before calling this. Functions in `funcmap` (`sub`, `addtables`, `pub`, `dataformat`) are resolved first; other global functions are reachable by name. -`sub` returns `(tablename; current data)` as the reply, giving the subscriber an initial snapshot to render before live updates arrive. +## Usage Example -Subscriptions are cleaned up when a connection closes via both `.z.wc` (websocket) and `.z.pc` (IPC). Any existing `.z.wc`/`.z.pc` handlers are preserved by wrapping, and the wiring happens only once across repeated `init` calls. +```q +html:use`di.html +log:use`di.log -## Logging +logdep:`info`warn`error!(log.info;log.warn;log.error) -The module logs at three points: +/ set static file directory before init +setenv[`KDBHTML;"/opt/app/html"] +html.init[enlist[`log]!enlist logdep] -- On `init`: confirms the module started and which `homedir` was set -- On `evaluate`: logs at error level when a WebSocket-invoked function fails (the error is also re-thrown to the caller) +/ register tables +html.addtables[`trades`quotes] -The `log` dict must be injected via `init`. Log functions must be dyadic (`{[ctx;msg]}`) — `ctx` is a symbol tag and `msg` is a string. kx.log instances (detected by the presence of `getlvl`/`sinks`/`fmts` keys) are wrapped automatically into the dyadic contract by prepending `string[ctx],": "` to the message. Stored on `.z.m` and called as `.z.m.log[\`info][\`di.html;"message"]`. The module uses all three levels (`info`, `warn`, `error`) internally, though only `info` is required by validation. +/ publish from a timer or upd handler +html.pub[`trades;newdata] +``` -## Example with kx.log +With kx.log, pass the log instance directly — it is normalised automatically: ```q -/ wire up logging from the kx.log module logger:use`kx.log kxlog:logger.createLog[] -logdep:`info`warn`error!( - {[m] kxlog.info[m]}; - {[m] kxlog.warn[m]}; - {[m] kxlog.error[m]}) +html.init[enlist[`log]!enlist kxlog] +``` -/ initialise html module — set KDBHTML before calling init -setenv[`KDBHTML;"/opt/app/html"] -html:use`di.html -html.init[enlist[`log]!enlist logdep] +## Running Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.html ``` -## Browser testing (development) +All exported functions are covered. The one path not unit-tested is `pub` physically delivering a message over a live WebSocket connection, which requires a real client handle. -The default `.z.ws` handler uses kdb+ binary IPC format (`-9!`/`-8!`) and is designed to work with the KX c.js WebSocket library. For development testing without c.js, use the included `modulesetup.q`: +For interactive browser testing, use the included integration test script: ```bash q di/html/integrationtest.q -p 5678 ``` -This starts a process on the given port with a plain-text JSON websocket handler. Open `test.html` in a browser (or navigate to `http://localhost:5678/test.html`) to connect and interact with the module. Run `tick[\`trades;5]` in the q session to push live data to browser subscribers. - -## Testing +This starts a process with a plain-text JSON websocket handler (no c.js required). Navigate to `http://localhost:5678/test.html` to subscribe to tables and push live data via `tick[\`trades;5]` in the q session. -```q -k4unit:use`di.k4unit -k4unit.moduletest`di.html -``` +## Notes -All exported functions are covered. The one path not unit-tested is `pub` physically -delivering a message over a live WebSocket connection, which requires a real client -handle. +- The default `.z.ws` handler sends kdb+ binary IPC (`-8!`) and requires the KX c.js library in the browser. Use `setmodifier` to switch individual tables to plain JSON text for clients without c.js. +- `.z.wc` fires on WebSocket close; `.z.pc` fires on any IPC port close. Both are wired so that `sub` works over plain IPC handles as well as WebSocket connections. +- `evaluate` resolves function names first from the module `funcmap`, then from the global namespace. Restrict which globals are reachable by controlling what is defined in the process. From 2b863c3571066df14c981207bce6e2872e9b08de Mon Sep 17 00:00:00 2001 From: Ruairi-wq2 Date: Tue, 30 Jun 2026 14:19:07 +0100 Subject: [PATCH 09/10] Document kdb-x default KDBHTML location in html.md kdb-x ships with analyst/html set as the default KDBHTML directory. Added note explaining the default, fallback, and how to override. Co-Authored-By: Claude Sonnet 4.6 --- di/html/html.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/di/html/html.md b/di/html/html.md index 09384f0a..0e282f96 100644 --- a/di/html/html.md +++ b/di/html/html.md @@ -29,7 +29,7 @@ You need two things: |---|---|---|---| | `` `log `` | dict | yes | Logging functions keyed on at minimum `` `info ``, each `{[ctx;msg]}` where `ctx` is a symbol and `msg` is a string. kx.log instances are normalised automatically. | -The HTML home directory is configured via the `KDBHTML` environment variable (not a deps key). If unset, `"html"` is used. Set it before calling `init`. +The HTML home directory is configured via the `KDBHTML` environment variable (not a deps key). kdb-x sets `KDBHTML` by default to the `analyst/html` directory in the kdb-x installation, which contains the KX Analyst UI. If `KDBHTML` is unset, `"html"` (relative to the working directory) is used as a fallback. Set or override `KDBHTML` before calling `init` to point at your own HTML files. ## Initialisation From b1be26ada66d62d61304eee2614fe744b300f9ef Mon Sep 17 00:00:00 2001 From: Ruairi-wq2 Date: Tue, 30 Jun 2026 14:45:57 +0100 Subject: [PATCH 10/10] Fix normlog comment position to match multi-line function style Co-Authored-By: Claude Sonnet 4.6 --- di/html/html.q | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/di/html/html.q b/di/html/html.q index 42bdf2bb..0955337c 100644 --- a/di/html/html.q +++ b/di/html/html.q @@ -12,9 +12,9 @@ modifier:()!(); / flag so direct .z handler wiring happens only once across repeated init calls zwired:0b; -/ normalise a log dict: if it looks like a kx.log instance (has getlvl/sinks/fmts keys), -/ wrap each level into a dyadic {[c;m]} function that embeds the context symbol into the message normlog:{[logdict] + / normalise a log dict: if it looks like a kx.log instance (has getlvl/sinks/fmts keys), + / wrap each level into a dyadic {[c;m]} function that embeds the context symbol into the message $[any `getlvl`sinks`fmts in key logdict; `info`warn`error!( {[fn;c;m] fn[string[c],": ",m]}[logdict`info;];