Feature HTML by Ruairi-wq2 · Pull Request #111 · DataIntellectTech/kdbx-modules

Ruairi-wq2 · 2026-06-30T15:20:31Z

The big issue or concern I have with this module is that, the exported functions allow for html serving over the browser, but in order to make use, the user would need to have their own html doc which calls the exported functions in the module. The components for doing so are shipped with the module, but implementation could be awkward. I include a integration script with a test.html which demonstrates the use case. I have added a brief section on building a client HTML page but am not certain if this is the correct way to go about this.

- Drop wssub (thin wrapper over sub) and end (TorQ-specific .u.end broadcast) - Add setmodifier[tbl;fn] so callers can override per-table encoding - Fix default modifier to apply jsformat — timestamps and dates now convert to JS-friendly formats out of the box rather than sending raw kdb+ values - Remove updformat (dead code; jsformat is now called directly in the modifier) - Remove readpage/readpagereplacehostport from docs and test.html (TorQ relics) - Update modulesetup.q to cast JSON numeric arg2 to long for tick[] calls - All 41 tests passing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Log functions updated to dyadic {[ctx;msg]} contract; kx.log instances normalised via normlog to wrap monadic functions into the required form - init validates log at minimum an `info key; warn/error not required - modulesetup.q renamed to integrationtest.q with dynamic port via -p flag - test.html subscription fixed to call sub via evaluate (wssub removed); snapshot now rendered into table panel on subscribe - Docs updated to reflect log contract, normlog behaviour, and sub dispatch Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Sections now follow: Features, How it works, Dependencies, Initialisation, Exported Functions, Usage Example, Running Tests, Notes. Added practical setup explanation (q process + HTML files on same port). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

kdb-x ships with analyst/html set as the default KDBHTML directory. Added note explaining the default, fallback, and how to override. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

DI-Software-Engineering · 2026-06-30T15:21:34Z

@@ -0,0 +1,182 @@
+/ di.html - websocket pub/sub and html page serving


The entire file operates in the default (root) namespace — there is no \d .di.html / \d . pair wrapping the module code. In TorQ convention every module file must open with \d .namespace and close with \d ., otherwise all symbols (subtables, subs, modifier, zwired, normlog, etc.) are injected directly into the root namespace and will collide with any other module or user code.

DI-Software-Engineering · 2026-06-30T15:31:20Z

+/ di.html - websocket pub/sub and html page serving
+
+/ list of table names registered for pub/sub
+subtables:`symbol$();


All module state and functions are defined in the root namespace (no \d .di.html / \d . namespace guard). TorQ convention requires every file to open with \d .ns and close with \d .. Without this, all names pollute the global namespace and can clash with other modules.

DI-Software-Engineering · 2026-06-30T15:31:22Z

+
+execdict:{[inputdict]
+  / extracts the func key and any additional args from a dictionary and calls the function
+  / args are passed to the function in the order the keys appear after func


execdict calls f @ 1 when the input dict has only the func key (i.e. 1=count key inputdict). This passes the integer 1 as the sole argument to f, not a call with no args. The intention is clearly a no-argument call (f[]). The @ 1 form means "apply f to 1", so neg would return -1 instead of signalling arity; but for any function that does not expect an integer arg this silently produces wrong results or a type error.

DI-Software-Engineering · 2026-06-30T15:31:24Z

+  / args are passed to the function in the order the keys appear after func
+  / checks the module funcmap first (module-local functions), then falls back to value for globals
+  if[not `func in key inputdict;'"no func in dictionary"];
+  fname:`$inputdict`func;


wshandler is defined before evaluate and funcmap, which it references. In q, function bodies are not closed over at definition time for names resolved via value/global lookup — this is fine at runtime. However funcmap is also referenced inside execdict (line 89), which is defined before funcmap on line 106. Since funcmap is a global variable assigned later, the first call to execdict before init could fail if funcmap has not yet been assigned. Ensure module load order guarantees funcmap is defined before any call path reaches execdict.

DI-Software-Engineering · 2026-06-30T15:31:25Z

+  colvals:value coldict;
+  converters:typemap type each colvals;
+  :flip k!converters@'colvals;
+  };


In del, subs[tbl;;0] projects the list of subscriber pairs to extract handles. If subs[tbl] is the empty list (), this projection will error ('type or 'rank) rather than returning an empty list. The guard if[not tbl in subtables;:()] is absent here; closehandle calls del for every table on close, so a table with zero subscribers triggers this path.

DI-Software-Engineering · 2026-06-30T15:31:27Z

+  / if handle is not found, ? returns count of list and drop has no effect
+  idx:subs[tbl;;0]?handle;
+  .z.m.subs:@[subs;tbl;_;idx];
+  };


add assigns subs via .z.m.subs but then reads value tbl directly. If tbl is not defined in the global namespace (e.g. a table registered via addtables before the global variable exists), value tbl signals an error and the subscriber has already been added to subs — leaving a dangling handle entry.

DI-Software-Engineering · 2026-06-30T15:31:28Z

+  / args are passed to the function in the order the keys appear after func
+  / checks the module funcmap first (module-local functions), then falls back to value for globals
+  if[not `func in key inputdict;'"no func in dictionary"];
+  fname:`$inputdict`func;


wshandler sends -8!.j.j[evaluate[...]]. .j.j returns a string; -8! on a string produces the IPC-serialised bytes for that string, which is the c.js binary protocol. However, if evaluate signals an error (the trap in evaluate re-throws with '), the error will propagate out of wshandler uncaught, crashing the .z.ws handler and potentially closing the connection without sending an error reply to the client.

DI-Software-Engineering · 2026-06-30T15:31:30Z

+  };
+
+/ module-local functions callable via websocket evaluate
+/ execdict checks here first before falling back to value for global lookups


.z.wc and .z.pc are assigned directly with : instead of using .dotz.set. This violates TorQ convention and will silently overwrite any handlers already set by the TorQ framework, even though the code attempts to preserve them by wrapping — the TorQ framework's own chain mechanism is bypassed.

DI-Software-Engineering · 2026-06-30T15:31:32Z

+  / safely calls execdict on the input, logging then re-throwing any errors with context
+  :@[execdict;inputdict;{[d;e] m:"failed to execute ",(-3!d)," : ",e;.z.m.log[`error][`di.html;m];'"di.html: ",m}[inputdict]];
+  };
+


.z.ws is assigned directly instead of via .dotz.set, violating TorQ convention. Any existing .z.ws handler set by the framework is silently overwritten.

DI-Software-Engineering · 2026-06-30T15:31:33Z

+  :add[tbl;syms];
+  };
+
+setmodifier:{[tbl;fn]


.z.m.log[info][di.html;...] is called but .z.m.log is only set inside init. If addtables is called before init, this line will signal 'log (undefined variable). The log call should be guarded or log should be initialised to a default no-op at module load time.

DI-Software-Engineering · 2026-06-30T16:00:11Z

@@ -0,0 +1,182 @@
+/ di.html - websocket pub/sub and html page serving


The entire file is in the root (default) namespace — no \d .di.html / \d . wrapping. Per TorQ convention every module file must declare a namespace. All symbols and functions land in the global namespace, creating collision risk and violating the module boundary.

DI-Software-Engineering · 2026-06-30T16:00:13Z

@@ -0,0 +1,182 @@
+/ di.html - websocket pub/sub and html page serving


Single-slash comments (/ ...) are used throughout. In q a single / on a line by itself opens a multi-line comment block; single-slash inline/line comments are a common source of silent comment-blocks when a blank line precedes them. TorQ convention requires // for line comments.

DI-Software-Engineering · 2026-06-30T16:00:15Z

+
+closehandle:{[handle]
+  / removes the given handle from all subscriber lists when a connection closes
+  del[;handle] each subtables;


execdict calls f @ 1 when the input dict has only the func key (1=count key inputdict). The intent is to call f with no arguments (or call it monadically on the dictionary), but f @ 1 passes the integer 1 as the argument. This will produce wrong results or a type error for any zero-argument function. It should be f[] for a niladic call, or the branch condition is wrong.

DI-Software-Engineering · 2026-06-30T16:00:17Z

+  / if already subscribed, updates the sym filter by taking union with new syms
+  / if not subscribed, appends a new (handle; syms) pair to the list
+  / returns (tablename; current data) so the subscriber can initialise their local copy
+  i:subs[tbl;;0]?.z.w;


converters:typemap type each colvals looks up a type short in typemap. For columns whose type is NOT in typemap, the lookup returns a null (::). Then converters@'colvals applies :: as a function to each element of the column, which projects the identity :: over lists — this works in practice but is fragile. More critically, type each colvals returns a list of shorts (e.g. 11h for a sym column), but typemap keys are 12 13 14 15 16 17 18 19h. If a column type short is not in the map, typemap[t] returns (::) which is then applied. This is the intended design, but colvals is value coldict which is a list of column vectors; converters@'colvals applies a potentially mixed list of functions to column vectors — this is fine for mapped types but (::)@col returns col unchanged. The logic is correct, but jstsiso8601 etc. are applied to entire vectors via @', meaning each converter must be rank-1 (accept a whole vector). Verify jstsiso8601 handles the full vector at once rather than scalar — the code does use where/bulk indexing, so this is consistent, but the type each colvals returns shorts like -12h for typed lists vs 12h for general: for a typed timestamp vector type returns -12h, NOT 12h, so the lookup typemap[-12h] will miss and the column will NOT be converted. The typemap keys are positive shorts; q typed lists have negative type codes.

DI-Software-Engineering · 2026-06-30T16:00:18Z

+  };
+
+dataformat:{[msgtype;msgdata]
+  / wraps a message into a name/data dictionary, javascript-formatting each table in msgdata


In del, subs[tbl;;0] attempts to index into the list at column 0 (the handle). If subs[tbl] is an empty list (), this will signal a rank/type error rather than returning an empty list to search. The guard if[not tbl in subtables;...] is not present here; del is called from closehandle for every table regardless of whether the handle was subscribed, so an empty subscriber list will cause a runtime error.

DI-Software-Engineering · 2026-06-30T16:00:20Z

+  :$[1=count key inputdict;f @ 1;f . args];
+  };
+
+/ websocket message handler - module-level so it carries the module context for evaluate


wshandler is defined at load time and captures its closure context. It calls evaluate which is defined later in the file. In q, function bodies are not evaluated until called, so forward references to names are resolved at call time — this is fine. However, wshandler sends the result of evaluate serialised with -8!.j.j[...]. If evaluate throws (re-throws with '), the error propagates uncaught out of .z.ws, which kdb+ will send as an error string to the client. This is probably acceptable but the behaviour differs from returning a structured error JSON response. More critically: .j.j is called on the raw q result of evaluate, which may include kdb+ types (tables, symbols, etc.) that .j.j cannot serialise cleanly — there is no error trap around the .j.j call itself.

DI-Software-Engineering · 2026-06-30T16:00:23Z

+  / initialise the module; deps must contain a log key
+  / deps: dict with `log key -> dict with at minimum `info!{[c;m]} (dyadic: ctx symbol, msg string)
+  / kx.log instances are normalised automatically via normlog
+  / wires .z.ws/.z.wc/.z.pc handlers once; .h.HOME set from KDBHTML env var (else "html")


.z.wc and .z.pc are assigned directly (:.z.wc:... / .z.pc:...) instead of using .dotz.set. TorQ convention requires .dotz.set so that multiple modules can safely compose handlers. Direct assignment silently overwrites any handler set by another module.

DI-Software-Engineering · 2026-06-30T16:00:25Z

+/ execdict checks here first before falling back to value for global lookups
+funcmap:`sub`addtables`pub`dataformat!(sub;addtables;pub;dataformat);
+
+init:{[deps]


.z.ws is assigned directly instead of using .dotz.set, violating TorQ convention and potentially overwriting a handler already registered by another module (e.g. a TorQ WebSocket gateway).

DI-Software-Engineering · 2026-06-30T16:00:27Z

+    if[count data:sel[data;s 1];
+      (neg first s) modifier[tbl]@(`upd;tbl;data)];
+    }[tbl;data] each subs tbl;
+  };


In sub, when tbl~\`` the function recurses with sub[;syms] each subtablesand returns a list of(tablename; data)pairs — one per table. The caller inaddreturns a pair(tbl; data). However the .z.wshandler doesneg[.z.w] -8!.j.j[evaluate[...]]— ifsubis called with `` `` the result is a list of pairs, not a single pair, and .j.j serialises the outer list. The browser-side `wssubTo` handler does `Array.isArray(r[0]) ? r : [r]` to handle this. This is a fragile protocol — a list-of-pairs vs a single pair relies on the JS-side type check. Not a crash bug, but the ambiguous return type is a correctness concern.

DI-Software-Engineering · 2026-06-30T16:00:28Z

+jstsfromd:{[x] "j"$86400000 * 10957 + `long$x};
+
+/ converts a list of time, second, or minute values to milliseconds since midnight
+jstsfromt:{[x] "j"$"t"$x};


jstsiso8601 uses count each d:string \date$xand thenwhere 10=count each dto find non-null entries. A null timestamp cast to date gives0Nd; string 0Ndis"" (empty string, count 0), not length 10. Non-null dates stringify to 10 chars ("2024-01-02"). So null filtering is correct. However, dd[;4 7]:" -"is a single assignment attempting to set positions 4 and 7 of each string to"-"— this requiresddto be a list of strings of equal length, which is guaranteed by thewhere 10=count eachfilter. The issue is the assignmentdd[;4 7]:" -"sets both positions to the same value: q interprets" -"as a 2-char string, assigning char" "to position 4 and"-"to position 7. The original date strings fromstringalready contain"."at those positions (e.g."2024.01.02"). The intent is to replace "."with"-", so both positions should be "-". But dd[;4 7]:"-"would assign the single char"-"to both positions.dd[;4 7]:" -"assigns" "to position 4 and"-"to position 7, resulting in"2024 01-02"— a space instead of a dash at position 4. The correct assignment isdd[;4 7]:2#"-"`.

DI-Software-Engineering · 2026-06-30T16:00:30Z

+html:use`di.html;
+
+logdep:`info`warn`error!(
+  {[c;m] -1 "[INFO] ",string[c]," ",m};


Raw output via -1 and -2 is used directly instead of routing through the log dependency or .lg.*. TorQ convention requires .lg.o/.lg.e for process output.

DI-Software-Engineering · 2026-06-30T16:00:32Z

+  tsubs:(get ` sv hns,`subs) t;
+  if[count tsubs; {[m;s] (neg s 0) m}[msg;] each tsubs];
+  };
+


p:system"p" returns a long integer; -1 "...port ",p will fail with a type error because string concatenation with , requires both sides to be strings/char lists and p is a long. It should be -1 "...port ",string p.

DI-Software-Engineering · 2026-06-30T16:00:33Z

+  / if already subscribed, updates the sym filter by taking union with new syms
+  / if not subscribed, appends a new (handle; syms) pair to the list
+  / returns (tablename; current data) so the subscriber can initialise their local copy
+  i:subs[tbl;;0]?.z.w;


type each colvals — for a typed vector (e.g. a timestamp column), type returns the negative short (e.g. -12h), not 12h. The typemap keys are 12 13 14 15 16 17 18 19h (positive shorts). Therefore typemap[-12h] is a null (::), and ALL typed columns will be left unconverted by jsformat. Only untyped (mixed/general) lists would match positive type codes, which is never the case for well-formed table columns. The fix is to use abs type each colvals when doing the typemap lookup.

DI-Software-Engineering · 2026-06-30T16:00:34Z

DIReview Summary

2 critical | 11 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

Ruairi-wq2 · 2026-07-03T08:49:58Z

Further to my comment about client side usage of the html doc, as far as I can see, TorQ only uses html common code when serving monitor UI. Heartbeat.q publishes to the monitor process when hb.subenabled is true. So essentially monitor.q is sole loader of code/common/html.q and has a packaged html doc for displaying health of registered processes.

matthewv192 and others added 10 commits June 15, 2026 11:09

Independent module using log dependency. With passing unit tests

e59a53c

Changed html to be in line with other modules

259738e

Changed in line with dexters comment about required logging dependency

5f9a3fe

Removed camel case and fixed remaining unit tests

4556fb1

Remove remaining wssub label references from test.html

20d04ed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Document kdb-x default KDBHTML location in html.md

2b863c3

kdb-x ships with analyst/html set as the default KDBHTML directory. Added note explaining the default, fallback, and how to override. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix normlog comment position to match multi-line function style

b1be26a

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

DI-Software-Engineering reviewed Jun 30, 2026

View reviewed changes

Comment thread di/html/html.q

DI-Software-Engineering reviewed Jun 30, 2026

View reviewed changes

Ruairi-wq2 changed the title ~~Extract html~~ Feature HTML Jul 1, 2026

		@@ -0,0 +1,182 @@
		/ di.html - websocket pub/sub and html page serving

Uh oh!

Conversation

Ruairi-wq2 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DI-Software-Engineering commented Jun 30, 2026

DIReview Summary

Uh oh!

Ruairi-wq2 commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Ruairi-wq2 commented Jun 30, 2026 •

edited

Loading