From 132ca554e1baedda4edb59435c86ea820fed75d8 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 16 Jun 2026 16:53:20 +0100 Subject: [PATCH 1/9] basic scafolding of feature --- di/serverselect/init.q | 4 + di/serverselect/serverselect.md | 213 ++++++++++++++++++++++++++++++++ di/serverselect/serverselect.q | 167 +++++++++++++++++++++++++ di/serverselect/test.csv | 100 +++++++++++++++ 4 files changed, 484 insertions(+) create mode 100644 di/serverselect/init.q create mode 100644 di/serverselect/serverselect.md create mode 100644 di/serverselect/serverselect.q create mode 100644 di/serverselect/test.csv diff --git a/di/serverselect/init.q b/di/serverselect/init.q new file mode 100644 index 00000000..4323911e --- /dev/null +++ b/di/serverselect/init.q @@ -0,0 +1,4 @@ +/ library for selecting backend servers from a registered pool based on servertype or attribute requirements +\l ::serverselect.q + +export:([init;addserverattr;addserver;setserveractive;getserverids;addserversfromtable;getservers]) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md new file mode 100644 index 00000000..64189143 --- /dev/null +++ b/di/serverselect/serverselect.md @@ -0,0 +1,213 @@ +# di.serverselect + +Module for registering backend servers and selecting them by servertype or attribute requirements. Extracted from the `.gw` namespace across TorQ's `gatewaylib.q` and `gateway.q`. Designed as the server-selection layer of a gateway — decoupled from query execution and connection management. + +## Usage + +```q +srvsel:use`di.serverselect + +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] + +/ register servers as they connect +srvsel.addserver[h;`rdb] +srvsel.addserverattr[h;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)] + +/ mark inactive on disconnect +srvsel.setserveractive[h;0b] + +/ select by servertype +srvsel.getserverids[`rdb`hdb] + +/ select by attribute requirements +srvsel.getserverids[enlist[`date]!enlist enlist 2024.01.01] +``` + +### Typical gateway integration + +```q +srvsel:use`di.serverselect +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] + +/ on open - register the connecting server +.z.po:{[h] srvsel.addserverattr[h;getproctype[h];getattributes[h]]} + +/ on close - mark the server inactive +.z.pc:{[h] srvsel.setserveractive[h;0b]} + +/ on query - select target servers then dispatch +serverids:srvsel.getserverids[`rdb`hdb] +``` + +### Bulk registration from a TorQ connection table + +In a TorQ stack, `.servers.SERVERS` is populated by `trackservers.q`. Pass it directly: + +```q +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] + +/ or register all process types +srvsel.addserversfromtable[`ALL; .servers.SERVERS] +``` + +## API + +### `init[deps]` + +Wire injectable dependencies. Must be called before any other function. + +| Key | Required | Type | Description | +|---|---|---|---| +| `` `log `` | yes | dict | Functions keyed `` `info`warn`error ``, each with signature `{[ctx;msg]}` | + +Errors with prefix `di.serverselect:` if `log` is missing or does not contain all three required keys. + +--- + +### `addserverattr[handle;servertype;attributes]` + +Register a server with a handle, servertype, and attribute dictionary. Sets `active:1b` on registration. + +| Parameter | Type | Description | +|---|---|---| +| `handle` | int | Connection handle | +| `servertype` | symbol | Process type (e.g. `` `rdb ``, `` `hdb ``) | +| `attributes` | dict | Key-value attribute pairs (e.g. `` `date`sym!(2024.01.01;`AAPL) ``) | + +```q +srvsel.addserverattr[5i; `hdb; `date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] +``` + +--- + +### `addserver[handle;servertype]` + +Register a server with no attributes. Convenience wrapper over `addserverattr`. + +```q +srvsel.addserver[5i; `rdb] +``` + +--- + +### `setserveractive[handle;active]` + +Mark a registered server active or inactive. Call with `0b` on disconnect, `1b` on reconnect. + +| Parameter | Type | Description | +|---|---|---| +| `handle` | int | Connection handle of the server to update | +| `active` | boolean | `1b` = active, `0b` = inactive | + +```q +srvsel.setserveractive[5i; 0b] / server disconnected +srvsel.setserveractive[5i; 1b] / server reconnected +``` + +--- + +### `getserverids[att]` + +Return server IDs matching the given servertype list or attribute requirement dictionary. + +| Parameter | Type | Description | +|---|---|---| +| `att` | symbol list or dict | Servertype list (11h) — returns IDs grouped by type. Attribute dict (99h) — returns IDs satisfying the requirements. | + +**Symbol list path** — validates that all requested types exist and are active: + +```q +srvsel.getserverids[`rdb] +srvsel.getserverids[`rdb`hdb] +``` + +**Attribute dict path** — matches servers whose attribute dictionaries satisfy the requirements. Supports two matching strategies controlled by the reserved key `` `attributetype ``: + +| `attributetype` | Behaviour | +|---|---| +| `` `cross `` (default) | Every combination of attribute values must be coverable — e.g. every date must be available for every sym | +| `` `independent `` | Each attribute value only needs to be matched by at least one server | + +The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a partial match is acceptable. If `0b`, an error is thrown when the requirements cannot be fully satisfied. + +```q +/ cross match (default): find servers covering the full date × sym cross product +srvsel.getserverids[`date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] + +/ independent match: each date and each sym just needs one server somewhere +srvsel.getserverids[`date`sym`attributetype!(2024.01.01 2024.01.02; `AAPL`MSFT; `independent)] + +/ scope to a specific servertype then apply attribute filter +srvsel.getserverids[`servertype`date!(`hdb; enlist 2024.01.01)] + +/ strict mode: error if requirements cannot be fully satisfied +srvsel.getserverids[`date`besteffort!(enlist 2024.01.01; 0b)] +``` + +--- + +### `addserversfromtable[proctypes;conntable]` + +Register servers from a connection table, skipping handles already active in the pool. + +| Parameter | Type | Description | +|---|---|---| +| `proctypes` | symbol or symbol list | Process types to register; pass `` `ALL `` to register all | +| `conntable` | table | Must have columns: `w` (int handle), `proctype` (symbol), `attributes` (dict per row) | + +```q +/ TorQ stack — pass .servers.SERVERS directly +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] + +/ custom connection table +conntab:([] w:1i 2i; proctype:`rdb`hdb; attributes:(()!(); `date!enlist 2024.01.01)) +srvsel.addserversfromtable[`ALL; conntab] +``` + +--- + +### `getservers[]` + +Return the current registered server table. + +```q +srvsel.getservers[] +``` + +Schema: + +| Column | Type | Description | +|---|---|---| +| `serverid` | int (keyed) | Unique auto-assigned server ID | +| `handle` | int | Connection handle | +| `servertype` | symbol | Process type | +| `active` | boolean | Whether the server is currently active | +| `attributes` | any | Attribute dictionary registered with the server | + +## Log dependency contract + +`di.serverselect` requires a log dependency dictionary with keys `` `info`warn`error ``: + +```q +`info`warn`error!({[ctx;msg] ...};{[ctx;msg] ...};{[ctx;msg] ...}) +``` + +`di.log` satisfies this contract: + +```q +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] +``` + +Context symbols used by `di.serverselect` in log calls: + +| Context | Level | When | +|---|---|---| +| `` `addserverattr `` | info | Server registered | +| `` `setserveractive `` | info | Server active flag changed | +| `` `addserversfromtable `` | info | Bulk registration — count of servers being added | diff --git a/di/serverselect/serverselect.q b/di/serverselect/serverselect.q new file mode 100644 index 00000000..2a2328c1 --- /dev/null +++ b/di/serverselect/serverselect.q @@ -0,0 +1,167 @@ +/ library for selecting backend servers from a registered pool based on servertype or attribute requirements + +/ registered server pool - populated by addserverattr +servers:([serverid:`u#`int$()] handle:`int$(); servertype:`symbol$(); active:`boolean$(); attributes:()); + +/ autoincrement counter for server IDs +serverid:0i; + +init:{[deps] + / wire injectable log dependency (required) + logdict:$[99h=type deps; + $[`log in key deps; + $[99h=type deps`log; deps`log; ()!()]; + ()!()]; + ()!()]; + if[not count logdict; + '"di.serverselect: log dependency is required; pass `info`warn`error functions - ", + "see di.log for a default implementation or refer to confluence documentation"; + ]; + if[not all (`info`warn`error) in key logdict; + '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key logdict); + ]; + .z.m.loginfo:logdict`info; + .z.m.logwarn:logdict`warn; + .z.m.logerr:logdict`error; + }; + +/ internal - increment and return the next unique server ID +nextserverid:{ + .z.m.serverid:serverid+1i; + :.z.m.serverid; + }; + +addserverattr:{[h;st;att] + / register a server handle with a servertype and attribute dictionary + .z.m.loginfo[`addserverattr;"registering server: handle=",(string h),", type=",string st]; + .z.m.servers:servers upsert (nextserverid[];h;st;1b;att); + }; + +addserver:{[h;st] + / register a server with no attributes; convenience wrapper over addserverattr + addserverattr[h;st;()!()]; + }; + +setserveractive:{[h;a] + / mark a registered server active (1b) or inactive (0b); called on connect and disconnect + .z.m.loginfo[`setserveractive;"marking handle=",(string h)," active=",string a]; + .z.m.servers:update active:a from servers where handle=h; + }; + +getservers:{[] + / return the current registered server table + :servers; + }; + +addserversfromtable:{[proctypes;conntable] + / register active servers from a connection table filtered by proctype + / conntable must have columns: w (int handle), proctype (symbol), attributes (dict per row) + / pass proctypes:`ALL to register all process types + activehandles:(0i;0Ni),exec handle from servers where active; + rows:select w,proctype,attributes from conntable where + ((proctype in proctypes) or proctypes~`ALL), + not w in activehandles; + .z.m.loginfo[`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; + addserverattr'[rows`w;rows`proctype;rows`attributes]; + }; + +getserverids:{[att] + / return server IDs matching a servertype list or attribute requirement dictionary + / att: symbol list of servertypes, or dict of attribute requirements (optionally keyed on `servertype) + if[99h<>type att; + if[not 11h=abs type att; + '"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; + ]; + servertype:distinct att,(); + activeservers:exec distinct servertype from servers where active; + allservers:exec distinct servertype from servers; + activeserversmsg:". available servers: ",", " sv string activeservers; + if[any null att; + '"di.serverselect: null servertype passed as argument",activeserversmsg; + ]; + if[count servertype except activeservers; + '"di.serverselect: ", + $[max not servertype in allservers; + "not valid servers: ",", " sv string servertype except allservers; + "requested servers currently inactive: ",", " sv string servertype except activeservers + ],activeserversmsg; + ]; + :(exec serverid by servertype from servers where active)[servertype]; + ]; + serverids:$[`servertype in key att; + raze getserveridstype[delete servertype from att] each (),att`servertype; + getserveridstype[att;`all]]; + if[all 0=count each serverids;'"di.serverselect: no servers match requested attributes"]; + :serverids; + }; + +/ internal - filter active servers by servertype and attribute requirements +getserveridstype:{[att;typ] + / resolve besteffort and attributetype control keys then dispatch to cross or independent matching + besteffort:1b; + attype:`cross; + svrs:$[typ=`all; + exec serverid!attributes from servers where active; + exec serverid!attributes from servers where active,servertype=typ]; + if[`besteffort in key att; + if[-1h=type att`besteffort; besteffort:att`besteffort]; + att:delete besteffort from att; + ]; + if[`attributetype in key att; + if[-11h=type att`attributetype; attype:att`attributetype]; + att:delete attributetype from att; + ]; + res:$[attype=`independent; + getserversindependent[att;svrs;besteffort]; + getserverscross[att;svrs;besteffort]]; + serverids:first value flip $[99h=type res; key res; res]; + if[all 0=count each serverids;'"di.serverselect: no servers match ",string[typ]," requested attributes"]; + :serverids; + }; + +/ internal - build a cross product table from a nested dictionary +buildcross:{(cross/){flip (enlist y)#x}[x] each key x}; + +/ internal - initial filter shared by cross and independent matching +/ drops servers missing any required attribute key, ranks survivors by coverage +getserversinitial:{[req;att] + if[0=count req; :([]serverid:enlist key att)]; + att:(where all each (key req) in/: key each att)#att; + if[not count att;'"di.serverselect: no servers report all requested attributes"]; + s:update serverid:key att from value req in'/: (key req)#/:att; + s:s idesc value min each sum each' `serverid xkey s; + s:`serverid xkey 0!(key req) xgroup s; + :s; + }; + +/ internal - find servers satisfying the cross product of all attribute requirements +/ each attribute combination must be coverable by a single server +getserverscross:{[req;att;besteffort] + if[0=count req; :([]serverid:enlist key att)]; + s:getserversinitial[req;att]; + reqcross:buildcross[req]; + / scan through each server group accumulating which cross-product rows have been covered + util:flip `remaining`found!flip ( + {[x;y;z] (y[0] except found; y[0] inter found:$[0=count y[0];y[0];buildcross x@'where each z])}[req]\ + )[(reqcross;());value s]; + if[(count last util`remaining) and not besteffort; + '"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; + ]; + s:1!(0!s) w:where not 0=count each util`found; + :(key s)!distinct each' flip each util[w]`found; + }; + +/ internal - find servers satisfying attribute requirements independently +/ each individual requirement only needs to be matched by one server +getserversindependent:{[req;att;besteffort] + if[0=count req; :([]serverid:enlist key att)]; + s:getserversinitial[req;att]; + / mask out server groups whose contribution is already covered by earlier groups + filter:(value s)¬ -1 _ (0b&(value s) enlist 0),maxs value s; + alldone:1+first where all each all each' maxs value s; + if[(null alldone) and not besteffort; + '"di.serverselect: cannot satisfy query - not all attributes can be matched"; + ]; + s:1!(0!s) w:where any each any each' filter; + :(key s)!{(key x)!(value x)@'where each y key x}[req] each value s&filter w; + }; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv new file mode 100644 index 00000000..46d7e633 --- /dev/null +++ b/di/serverselect/test.csv @@ -0,0 +1,100 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,srvsel:use`di.serverselect,1,1,load module +before,0,0,q,mylog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,no-op mock logger +before,0,0,q,"caplog:([] fn:`symbol$();ctx:`symbol$();msg:())",1,1,initialise log capture table +before,0,0,q,logcap:`info`warn`error!({[c;m] `caplog upsert(`info;c;m)};{[c;m] `caplog upsert(`warn;c;m)};{[c;m] `caplog upsert(`error;c;m)}),1,1,define capturing mock logger +before,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init with no-op logger for setup + +/ ---- init: dependency injection validation ---- +comment,,,,,,,init - dependency injection validation +fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: as deps +fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects null log dep +fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty dict +fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects dict missing log key +fail,0,0,q,srvsel.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value +fail,0,0,q,srvsel.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key +true,0,0,q,"(7#@[srvsel.init;(::);{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect + +/ ---- getservers: initial state ---- +comment,,,,,,,getservers - initial state before any addserverattr +true,0,0,q,0=count srvsel.getservers[],1,1,servers empty on fresh module load +true,0,0,q,`serverid`handle`servertype`active`attributes~cols srvsel.getservers[],1,1,servers has correct schema + +/ ---- addserverattr / addserver: registration ---- +comment,,,,,,,addserverattr and addserver - server registration +run,0,0,q,srvsel.addserver[1i;`rdb],1,1,register rdb server with no attributes +run,0,0,q,srvsel.addserver[2i;`rdb],1,1,register second rdb server +run,0,0,q,"srvsel.addserverattr[3i;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)]",1,1,register hdb server with date and sym attributes +true,0,0,q,3=count srvsel.getservers[],1,1,three servers registered +true,0,0,q,all srvsel.getservers[][`active],1,1,all registered servers are active by default +true,0,0,q,`rdb`rdb`hdb~exec servertype from srvsel.getservers[],1,1,servertypes match registered order +true,0,0,q,(1i;2i;3i)~exec handle from srvsel.getservers[],1,1,handles match registered order +true,0,0,q,1 2 3i~exec serverid from srvsel.getservers[],1,1,server IDs autoincrement from 1 + +/ ---- setserveractive: marking active/inactive ---- +comment,,,,,,,setserveractive - active flag management +run,0,0,q,srvsel.setserveractive[2i;0b],1,1,mark second rdb server inactive +true,0,0,q,not (exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is now inactive +true,0,0,q,(exec active from srvsel.getservers[] where handle=1i)[0],1,1,handle 1 remains active +run,0,0,q,srvsel.setserveractive[2i;1b],1,1,mark second rdb server active again +true,0,0,q,(exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is active again + +/ ---- getserverids: symbol list path ---- +comment,,,,,,,getserverids - symbol list (servertype) path +true,0,0,q,0 Date: Thu, 18 Jun 2026 11:35:17 +0100 Subject: [PATCH 2/9] Added more of the required functions, fixed bugs and added more tests --- di/serverselect/init.q | 3 +- di/serverselect/serverselect.md | 6 +- di/serverselect/serverselect.q | 111 +++++++++++++++++++++++--------- di/serverselect/test.csv | 101 +++++++++++++++++++---------- 4 files changed, 154 insertions(+), 67 deletions(-) diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 4323911e..5512c3c2 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -1,4 +1,3 @@ -/ library for selecting backend servers from a registered pool based on servertype or attribute requirements \l ::serverselect.q -export:([init;addserverattr;addserver;setserveractive;getserverids;addserversfromtable;getservers]) +export:([init;addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable;getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md index 64189143..e0f5f6f7 100644 --- a/di/serverselect/serverselect.md +++ b/di/serverselect/serverselect.md @@ -64,7 +64,7 @@ Wire injectable dependencies. Must be called before any other function. |---|---|---|---| | `` `log `` | yes | dict | Functions keyed `` `info`warn`error ``, each with signature `{[ctx;msg]}` | -Errors with prefix `di.serverselect:` if `log` is missing or does not contain all three required keys. +Errors with prefix `di.serverselect:` if `configs` is not a dict, `log` key is missing, or `log` value is not a dict with all three required keys. --- @@ -170,12 +170,12 @@ srvsel.addserversfromtable[`ALL; conntab] --- -### `getservers[]` +### `getserverstable[]` Return the current registered server table. ```q -srvsel.getservers[] +srvsel.getserverstable[] ``` Schema: diff --git a/di/serverselect/serverselect.q b/di/serverselect/serverselect.q index 2a2328c1..07ccf45a 100644 --- a/di/serverselect/serverselect.q +++ b/di/serverselect/serverselect.q @@ -1,28 +1,22 @@ / library for selecting backend servers from a registered pool based on servertype or attribute requirements -/ registered server pool - populated by addserverattr -servers:([serverid:`u#`int$()] handle:`int$(); servertype:`symbol$(); active:`boolean$(); attributes:()); +/ registered server pool - populated by addserverfull / addserverattr / addserver +servers:([serverid:`u#`int$()] handle:`int$(); procname:`symbol$(); servertype:`symbol$(); hpup:`symbol$(); active:`boolean$(); lastp:`timestamp$(); hits:`int$(); attributes:()); / autoincrement counter for server IDs serverid:0i; -init:{[deps] - / wire injectable log dependency (required) - logdict:$[99h=type deps; - $[`log in key deps; - $[99h=type deps`log; deps`log; ()!()]; - ()!()]; - ()!()]; - if[not count logdict; - '"di.serverselect: log dependency is required; pass `info`warn`error functions - ", - "see di.log for a default implementation or refer to confluence documentation"; - ]; - if[not all (`info`warn`error) in key logdict; - '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key logdict); - ]; - .z.m.loginfo:logdict`info; - .z.m.logwarn:logdict`warn; - .z.m.logerr:logdict`error; +init:{[configs] + / wire log dependency - required; configs must be a dict with `log key + if[99h<>type configs; + '"di.serverselect: configs must be a dict with `log key"]; + if[not `log in key configs; + '"di.serverselect: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type configs`log; + '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key configs`log; + '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key configs`log)]; + .z.m.log:configs`log; }; / internal - increment and return the next unique server ID @@ -31,24 +25,34 @@ nextserverid:{ :.z.m.serverid; }; +/ internal - update last-access timestamp and hit count for a server handle +updatestats:{[h] + .z.m.servers:update lastp:.z.p,hits:hits+1i from servers where handle=h; + }; + +addserverfull:{[h;pname;st;hp;att] + / register a server with full details: handle, procname, servertype, hpup and attribute dictionary + .z.m.log[`info][`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; + .z.m.servers:servers upsert (nextserverid[];h;pname;st;hp;1b;0Np;0i;att); + }; + addserverattr:{[h;st;att] - / register a server handle with a servertype and attribute dictionary - .z.m.loginfo[`addserverattr;"registering server: handle=",(string h),", type=",string st]; - .z.m.servers:servers upsert (nextserverid[];h;st;1b;att); + / register a server with servertype and attributes; procname and hpup default to null + addserverfull[h;`;st;`;att]; }; addserver:{[h;st] - / register a server with no attributes; convenience wrapper over addserverattr - addserverattr[h;st;()!()]; + / register a server with no attributes; procname and hpup default to null + addserverfull[h;`;st;`;()!()]; }; setserveractive:{[h;a] / mark a registered server active (1b) or inactive (0b); called on connect and disconnect - .z.m.loginfo[`setserveractive;"marking handle=",(string h)," active=",string a]; + .z.m.log[`info][`setserveractive;"marking handle=",(string h)," active=",string a]; .z.m.servers:update active:a from servers where handle=h; }; -getservers:{[] +getserverstable:{[] / return the current registered server table :servers; }; @@ -56,15 +60,64 @@ getservers:{[] addserversfromtable:{[proctypes;conntable] / register active servers from a connection table filtered by proctype / conntable must have columns: w (int handle), proctype (symbol), attributes (dict per row) + / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types activehandles:(0i;0Ni),exec handle from servers where active; - rows:select w,proctype,attributes from conntable where + rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), not w in activehandles; - .z.m.loginfo[`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; - addserverattr'[rows`w;rows`proctype;rows`attributes]; + .z.m.log[`info][`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; + pnames:$[`procname in cols rows; rows`procname; count[rows]#`]; + hpups:$[`hpup in cols rows; rows`hpup; count[rows]#`]; + addserverfull'[rows`w;pnames;rows`proctype;hpups;rows`attributes]; + }; + +attributematch:{[req;avail] + / compute match result for each key in req against what avail advertises + / returns dict of attrname!(complete_match_bool;matched_values) for each required attribute key + / keys present in req but absent in avail return (0b;()) + vals:key[req] inter key avail; + notpresent:noval!(count noval:key[req] except key avail)#enlist(0b;()); + :notpresent,vals!{($[0>type y;x~y;all x in y];(x,()) inter y,())}'[req vals;avail vals]; }; +getservers:{[nameortype;lookups;req] + / look up active servers by servertype or procname with per-attribute match scoring + / nameortype: `servertype or `procname; pass ` as lookups to return all active servers + / req: attribute requirements dict - use ()!() for no attribute filtering + / returns table with attribmatch column showing (complete_bool;matched_values) per attribute key + r:$[`~lookups; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active; + nameortype~`servertype; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,servertype in lookups; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups]; + if[0=count r;:update attribmatch:attributes from r]; + am:attributematch[req] each r`attributes; + :update attribmatch:am from r; + }; + +selector:{[servertable;selection] + / pick one row from servertable using the given strategy + / selection: `roundrobin (least recently used), `any (random), `last (most recently used) + :$[selection=`roundrobin; first `lastp xasc servertable; + selection=`any; rand servertable; + selection=`last; last `lastp xasc servertable; + '"di.serverselect: unknown selection strategy: ",string selection]; + }; + +getserverbytype:{[ptype;serverval;selection] + / return a single server attribute value for a servertype using the given selection strategy + / ptype: servertype symbol; serverval: column to return e.g. `handle or `hpup; selection: `roundrobin`any`last + r:getservers[`servertype;ptype;()!()]; + if[not count r;:()]; + r:selector[r;selection]; + updatestats[r`handle]; + :r serverval; + }; + +gethandlebytype:getserverbytype[;`handle;]; +gethpbytype:getserverbytype[;`hpup;]; + getserverids:{[att] / return server IDs matching a servertype list or attribute requirement dictionary / att: symbol list of servertypes, or dict of attribute requirements (optionally keyed on `servertype) diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 46d7e633..42e88888 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -7,37 +7,71 @@ before,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init with no-op logger f / ---- init: dependency injection validation ---- comment,,,,,,,init - dependency injection validation -fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: as deps -fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects null log dep -fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty dict -fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects dict missing log key +run,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init accepts valid log dependency +fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: (not a dict) +fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty config (missing log key) +fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects config without log key +fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects log dep that is not a dict fail,0,0,q,srvsel.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value fail,0,0,q,srvsel.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key -true,0,0,q,"(7#@[srvsel.init;(::);{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect +true,0,0,q,"(15#@[srvsel.init;enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn));{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect -/ ---- getservers: initial state ---- -comment,,,,,,,getservers - initial state before any addserverattr -true,0,0,q,0=count srvsel.getservers[],1,1,servers empty on fresh module load -true,0,0,q,`serverid`handle`servertype`active`attributes~cols srvsel.getservers[],1,1,servers has correct schema +/ ---- getserverstable: initial state ---- +comment,,,,,,,getserverstable - initial state before any addserverattr +true,0,0,q,0=count srvsel.getserverstable[],1,1,servers empty on fresh module load +true,0,0,q,`serverid`handle`procname`servertype`hpup`active`lastp`hits`attributes~cols srvsel.getserverstable[],1,1,servers has correct schema -/ ---- addserverattr / addserver: registration ---- +/ ---- addserverattr / addserver / addserverfull: registration ---- comment,,,,,,,addserverattr and addserver - server registration run,0,0,q,srvsel.addserver[1i;`rdb],1,1,register rdb server with no attributes run,0,0,q,srvsel.addserver[2i;`rdb],1,1,register second rdb server run,0,0,q,"srvsel.addserverattr[3i;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)]",1,1,register hdb server with date and sym attributes -true,0,0,q,3=count srvsel.getservers[],1,1,three servers registered -true,0,0,q,all srvsel.getservers[][`active],1,1,all registered servers are active by default -true,0,0,q,`rdb`rdb`hdb~exec servertype from srvsel.getservers[],1,1,servertypes match registered order -true,0,0,q,(1i;2i;3i)~exec handle from srvsel.getservers[],1,1,handles match registered order -true,0,0,q,1 2 3i~exec serverid from srvsel.getservers[],1,1,server IDs autoincrement from 1 +true,0,0,q,3=count srvsel.getserverstable[],1,1,three servers registered +true,0,0,q,all exec active from srvsel.getserverstable[],1,1,all registered servers are active by default +true,0,0,q,`rdb`rdb`hdb~exec servertype from srvsel.getserverstable[],1,1,servertypes match registered order +true,0,0,q,(1i;2i;3i)~exec handle from srvsel.getserverstable[],1,1,handles match registered order +true,0,0,q,1 2 3i~exec serverid from srvsel.getserverstable[],1,1,server IDs autoincrement from 1 +true,0,0,q,all null exec procname from srvsel.getserverstable[],1,1,procname is null when registered via addserverattr +true,0,0,q,all null exec hpup from srvsel.getserverstable[],1,1,hpup is null when registered via addserverattr +run,0,0,q,"srvsel.addserverfull[4i;`myproc;`hdb;`:localhost:5555;(enlist`date)!enlist 2024.12.01]",1,1,register server with full details via addserverfull +true,0,0,q,4=count srvsel.getserverstable[],1,1,fourth server registered via addserverfull +true,0,0,q,`myproc~first exec procname from srvsel.getserverstable[] where handle=4i,1,1,procname populated via addserverfull +true,0,0,q,`:localhost:5555~first exec hpup from srvsel.getserverstable[] where handle=4i,1,1,hpup populated via addserverfull / ---- setserveractive: marking active/inactive ---- comment,,,,,,,setserveractive - active flag management run,0,0,q,srvsel.setserveractive[2i;0b],1,1,mark second rdb server inactive -true,0,0,q,not (exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is now inactive -true,0,0,q,(exec active from srvsel.getservers[] where handle=1i)[0],1,1,handle 1 remains active +true,0,0,q,not (exec active from srvsel.getserverstable[] where handle=2i)[0],1,1,handle 2 is now inactive +true,0,0,q,(exec active from srvsel.getserverstable[] where handle=1i)[0],1,1,handle 1 remains active run,0,0,q,srvsel.setserveractive[2i;1b],1,1,mark second rdb server active again -true,0,0,q,(exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is active again +true,0,0,q,(exec active from srvsel.getserverstable[] where handle=2i)[0],1,1,handle 2 is active again + + +/ ---- getservers ---- +comment,,,,,,,getservers - server lookup with attribute match scoring +true,0,0,q,`serverid`procname`servertype`hpup`handle`lastp`attributes`attribmatch~cols srvsel.getservers[`;`;()!()],1,1,getservers returns table with correct columns +true,0,0,q,count[select from srvsel.getserverstable[] where active]=count srvsel.getservers[`;`;()!()],1,1,getservers with ` lookups returns all active servers +true,0,0,q,all `hdb=exec servertype from srvsel.getservers[`servertype;`hdb;()!()],1,1,getservers filters by servertype correctly +true,0,0,q,0=count srvsel.getservers[`servertype;`nonexistent;()!()],1,1,getservers returns empty table for unknown servertype +true,0,0,q,0 Date: Thu, 18 Jun 2026 14:20:06 +0100 Subject: [PATCH 3/9] adding more tests and writeup --- di/serverselect/serverselect.md | 256 ++++++++++++++++---------------- di/serverselect/test.csv | 117 +++++++++++++++ 2 files changed, 244 insertions(+), 129 deletions(-) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md index e0f5f6f7..a871ae90 100644 --- a/di/serverselect/serverselect.md +++ b/di/serverselect/serverselect.md @@ -1,144 +1,158 @@ -# di.serverselect +# Server Selection -Module for registering backend servers and selecting them by servertype or attribute requirements. Extracted from the `.gw` namespace across TorQ's `gatewaylib.q` and `gateway.q`. Designed as the server-selection layer of a gateway — decoupled from query execution and connection management. +`serverselect.q` maintains a pool of registered backend servers and selects from them by servertype or attribute requirements. Extracted from the `.gw` namespace in TorQ's `gatewaylib.q` and `gateway.q`, it provides the server-selection layer of a gateway — decoupled from query execution and connection management. -## Usage - -```q -srvsel:use`di.serverselect +--- -log:use`di.log -logdep:`info`warn`error!(log.info;log.warn;log.error) -srvsel.init[enlist[`log]!enlist logdep] +## :sparkles: Features -/ register servers as they connect -srvsel.addserver[h;`rdb] -srvsel.addserverattr[h;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)] +- Register backend servers with servertype, procname, host/port, and attribute dictionaries +- Track active/inactive state per server, updated on connect and disconnect +- Select servers using round-robin, most-recently-used, or random strategy +- Query servers by servertype list or attribute requirement dictionary +- Attribute matching supports cross-product and independent strategies with configurable best-effort mode +- Bulk registration from a TorQ-compatible connection table +- Injected log dependency — no hard module dependencies -/ mark inactive on disconnect -srvsel.setserveractive[h;0b] +--- -/ select by servertype -srvsel.getserverids[`rdb`hdb] +## :memo: Dependencies -/ select by attribute requirements -srvsel.getserverids[enlist[`date]!enlist enlist 2024.01.01] -``` +| Dependency | Required | Description | +|---|---|---| +| `log` | yes | Dict of `info`warn`error functions, each with signature `{[ctx;msg]}` | -### Typical gateway integration +Pass the log dependency via `init`: ```q -srvsel:use`di.serverselect log:use`di.log logdep:`info`warn`error!(log.info;log.warn;log.error) srvsel.init[enlist[`log]!enlist logdep] - -/ on open - register the connecting server -.z.po:{[h] srvsel.addserverattr[h;getproctype[h];getattributes[h]]} - -/ on close - mark the server inactive -.z.pc:{[h] srvsel.setserveractive[h;0b]} - -/ on query - select target servers then dispatch -serverids:srvsel.getserverids[`rdb`hdb] ``` -### Bulk registration from a TorQ connection table +--- -In a TorQ stack, `.servers.SERVERS` is populated by `trackservers.q`. Pass it directly: +## :label: Server Table Schema -```q -srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] +Servers are tracked in the `servers` keyed table (keyed on `serverid`): -/ or register all process types -srvsel.addserversfromtable[`ALL; .servers.SERVERS] -``` +| Column | Type | Description | +|---|---|---| +| `serverid` | `int` (key) | Unique auto-assigned server ID | +| `handle` | `int` | Connection handle | +| `procname` | `symbol` | Process name (null if not provided) | +| `servertype` | `symbol` | Process type e.g. `` `rdb ``, `` `hdb `` | +| `hpup` | `symbol` | Host/port symbol e.g. `` `:host:5010 `` (null if not provided) | +| `active` | `boolean` | Whether the server is currently active | +| `lastp` | `timestamp` | Last time this server was selected | +| `hits` | `int` | Number of times this server has been selected | +| `attributes` | `any` | Attribute dictionary registered with the server | -## API +--- -### `init[deps]` +## :gear: Configuration -Wire injectable dependencies. Must be called before any other function. +`init` wires the required log dependency. It must be called before any other function. -| Key | Required | Type | Description | -|---|---|---|---| -| `` `log `` | yes | dict | Functions keyed `` `info`warn`error ``, each with signature `{[ctx;msg]}` | +```q +srvsel.init[enlist[`log]!enlist logdep] +``` -Errors with prefix `di.serverselect:` if `configs` is not a dict, `log` key is missing, or `log` value is not a dict with all three required keys. +Throws with prefix `di.serverselect:` if: +- `configs` is not a dictionary +- `` `log `` key is missing +- `log` value is not a dictionary with `` `info`warn`error `` keys --- -### `addserverattr[handle;servertype;attributes]` +## :wrench: Functions -Register a server with a handle, servertype, and attribute dictionary. Sets `active:1b` on registration. +### Registration -| Parameter | Type | Description | -|---|---|---| -| `handle` | int | Connection handle | -| `servertype` | symbol | Process type (e.g. `` `rdb ``, `` `hdb ``) | -| `attributes` | dict | Key-value attribute pairs (e.g. `` `date`sym!(2024.01.01;`AAPL) ``) | +| Function | Description | +|---|---| +| `addserverfull[h;pname;st;hp;att]` | Register a server with full details: handle, procname, servertype, hpup, attributes | +| `addserverattr[h;st;att]` | Register a server with servertype and attributes; procname and hpup default to null | +| `addserver[h;st]` | Register a server with no attributes | +| `setserveractive[h;active]` | Mark a server active (`1b`) or inactive (`0b`) | +| `addserversfromtable[proctypes;conntable]` | Bulk-register from a connection table, filtered by proctype | +| `getserverstable[]` | Return the full registered server table | + +`addserversfromtable` skips handles that are already active. Pass `` `ALL `` as `proctypes` to register all process types. The `conntable` must have columns `w` (int handle), `proctype` (symbol), `attributes` (dict per row); `procname` and `hpup` are optional. ```q -srvsel.addserverattr[5i; `hdb; `date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] +/ register on connect +srvsel.addserverattr[h; getproctype[h]; getattributes[h]] + +/ mark inactive on disconnect +srvsel.setserveractive[h; 0b] + +/ bulk registration from TorQ connection table +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] ``` --- -### `addserver[handle;servertype]` - -Register a server with no attributes. Convenience wrapper over `addserverattr`. +### Query and Selection -```q -srvsel.addserver[5i; `rdb] -``` +| Function | Description | +|---|---| +| `getservers[nameortype;lookups;req]` | Return active servers matching a servertype or procname filter, with per-attribute match scoring | +| `selector[servertable;selection]` | Pick one row from a server table using a selection strategy | +| `getserverbytype[ptype;col;sel]` | Return one column value for a servertype using the given strategy; updates `lastp` and `hits` | +| `gethandlebytype[ptype;sel]` | Convenience projection of `getserverbytype` returning `handle` | +| `gethpbytype[ptype;sel]` | Convenience projection of `getserverbytype` returning `hpup` | ---- +`getservers` returns a table including an `attribmatch` column — a dictionary of `attrname!(complete_match_bool;matched_values)` per attribute key in `req`. -### `setserveractive[handle;active]` +`selector` supports three strategies: -Mark a registered server active or inactive. Call with `0b` on disconnect, `1b` on reconnect. +| Strategy | Behaviour | +|---|---| +| `` `roundrobin `` | Pick server with the oldest `lastp` (least recently used) | +| `` `any `` | Pick a random server | +| `` `last `` | Pick server with the newest `lastp` (most recently used) | -| Parameter | Type | Description | -|---|---|---| -| `handle` | int | Connection handle of the server to update | -| `active` | boolean | `1b` = active, `0b` = inactive | +All three return `()` if no matching server is found. ```q -srvsel.setserveractive[5i; 0b] / server disconnected -srvsel.setserveractive[5i; 1b] / server reconnected +/ get a handle, round-robin across rdbs +srvsel.gethandlebytype[`rdb; `roundrobin] + +/ get host/port for an hdb +srvsel.gethpbytype[`hdb; `roundrobin] ``` --- -### `getserverids[att]` +### Server ID Lookup + +```q +srvsel.getserverids[att] +``` -Return server IDs matching the given servertype list or attribute requirement dictionary. +Returns server IDs matching a servertype list or attribute requirement dictionary. Used as the primary dispatch input — pass the result to your async or sync query handler to target specific servers. -| Parameter | Type | Description | -|---|---|---| -| `att` | symbol list or dict | Servertype list (11h) — returns IDs grouped by type. Attribute dict (99h) — returns IDs satisfying the requirements. | +#### Symbol list path -**Symbol list path** — validates that all requested types exist and are active: +Pass a symbol or symbol list of servertypes. Validates that all requested types are registered and currently active. ```q srvsel.getserverids[`rdb] srvsel.getserverids[`rdb`hdb] ``` -**Attribute dict path** — matches servers whose attribute dictionaries satisfy the requirements. Supports two matching strategies controlled by the reserved key `` `attributetype ``: +Throws if any requested type is null, unregistered, or all-inactive. -| `attributetype` | Behaviour | -|---|---| -| `` `cross `` (default) | Every combination of attribute values must be coverable — e.g. every date must be available for every sym | -| `` `independent `` | Each attribute value only needs to be matched by at least one server | +#### Attribute dict path -The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a partial match is acceptable. If `0b`, an error is thrown when the requirements cannot be fully satisfied. +Pass a dictionary of attribute requirements. Servers whose attribute dictionaries satisfy the requirements are returned. ```q -/ cross match (default): find servers covering the full date × sym cross product +/ cross match (default): every date must be available for every sym srvsel.getserverids[`date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] -/ independent match: each date and each sym just needs one server somewhere +/ independent match: each date and sym just needs one server somewhere srvsel.getserverids[`date`sym`attributetype!(2024.01.01 2024.01.02; `AAPL`MSFT; `independent)] / scope to a specific servertype then apply attribute filter @@ -148,66 +162,50 @@ srvsel.getserverids[`servertype`date!(`hdb; enlist 2024.01.01)] srvsel.getserverids[`date`besteffort!(enlist 2024.01.01; 0b)] ``` ---- - -### `addserversfromtable[proctypes;conntable]` +##### Attribute matching strategies -Register servers from a connection table, skipping handles already active in the pool. - -| Parameter | Type | Description | -|---|---|---| -| `proctypes` | symbol or symbol list | Process types to register; pass `` `ALL `` to register all | -| `conntable` | table | Must have columns: `w` (int handle), `proctype` (symbol), `attributes` (dict per row) | - -```q -/ TorQ stack — pass .servers.SERVERS directly -srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] +| `` `attributetype `` | Behaviour | +|---|---| +| `` `cross `` (default) | Every combination of attribute values must be coverable by a single server | +| `` `independent `` | Each individual attribute value only needs to be matched by at least one server | -/ custom connection table -conntab:([] w:1i 2i; proctype:`rdb`hdb; attributes:(()!(); `date!enlist 2024.01.01)) -srvsel.addserversfromtable[`ALL; conntab] -``` +The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a partial match is acceptable. Set to `0b` to throw if requirements cannot be fully satisfied. --- -### `getserverstable[]` - -Return the current registered server table. +## :test_tube: Example ```q -srvsel.getserverstable[] -``` - -Schema: +/ load module +srvsel:use`di.serverselect -| Column | Type | Description | -|---|---|---| -| `serverid` | int (keyed) | Unique auto-assigned server ID | -| `handle` | int | Connection handle | -| `servertype` | symbol | Process type | -| `active` | boolean | Whether the server is currently active | -| `attributes` | any | Attribute dictionary registered with the server | +/ wire log dependency +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] -## Log dependency contract +/ register servers as they connect (.z.po handler) +.z.po:{[h] + srvsel.addserverattr[h; getproctype[h]; getattributes[h]]; + } -`di.serverselect` requires a log dependency dictionary with keys `` `info`warn`error ``: +/ mark inactive on disconnect (.z.pc handler) +.z.pc:{[h] + srvsel.setserveractive[h; 0b]; + } -```q -`info`warn`error!({[ctx;msg] ...};{[ctx;msg] ...};{[ctx;msg] ...}) -``` +/ bulk register from TorQ stack on startup +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] -`di.log` satisfies this contract: +/ get server IDs for a query by servertype +serverids:srvsel.getserverids[`rdb`hdb] -```q -log:use`di.log -logdep:`info`warn`error!(log.info;log.warn;log.error) -srvsel.init[enlist[`log]!enlist logdep] -``` +/ get server IDs by attribute requirement +serverids:srvsel.getserverids[`date`sym!(enlist 2024.01.01; enlist `AAPL)] -Context symbols used by `di.serverselect` in log calls: +/ get a handle directly (round-robin across rdbs) +h:srvsel.gethandlebytype[`rdb; `roundrobin] -| Context | Level | When | -|---|---|---| -| `` `addserverattr `` | info | Server registered | -| `` `setserveractive `` | info | Server active flag changed | -| `` `addserversfromtable `` | info | Bulk registration — count of servers being added | +/ inspect registered server pool +srvsel.getserverstable[] +``` diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 42e88888..febd74c2 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -133,3 +133,120 @@ run,0,0,q,"srvsel.addserversfromtable[`rdb;([]w:enlist 98i;proctype:enlist`rdb;a true,0,0,q,any caplog[`fn]=`info,1,1,addserversfromtable emits info log entries true,0,0,q,"any caplog[`msg] like ""*servers from connection table*""",1,1,addserversfromtable logs count of servers being registered true,0,0,q,`addserversfromtable~first (select from caplog where fn=`info)[`ctx],1,1,addserversfromtable log uses addserversfromtable context symbol + +/ reset to no-op logger for edge case tests +run,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,reset to no-op logger for edge case tests + +/ ---- addserverfull: edge cases ---- +comment,,,,,,,addserverfull - edge cases +run,0,0,q,"srvsel.addserverfull[70i;`first;`rdb;`:first:7000;()!()]",1,1,register first entry at handle 70 +run,0,0,q,"srvsel.addserverfull[70i;`second;`hdb;`:second:7070;()!()]",1,1,register second entry at same handle (keyed by serverid not handle) +true,0,0,q,2=count select from srvsel.getserverstable[] where handle=70i,1,1,addserverfull with duplicate handle creates two distinct rows (new serverid each time) +fail,0,0,q,"srvsel.addserverfull[`badhandle;`;`rdb;`;()!()]",1,1,addserverfull rejects symbol handle (type error on int column) +fail,0,0,q,"srvsel.addserverfull[71i;`;""rdb"";`;()!()]",1,1,addserverfull rejects string servertype (type error on symbol column) + +/ ---- addserver: edge cases ---- +comment,,,,,,,addserver - edge cases +true,0,0,q,(()!())~first exec attributes from srvsel.getserverstable[] where handle=1i,1,1,addserver stores empty attributes dict +true,0,0,q,null first exec procname from srvsel.getserverstable[] where handle=1i,1,1,addserver stores null procname +true,0,0,q,null first exec hpup from srvsel.getserverstable[] where handle=1i,1,1,addserver stores null hpup +fail,0,0,q,"srvsel.addserver[`badhandle;`rdb]",1,1,addserver rejects symbol handle (type error) +fail,0,0,q,"srvsel.addserver[72i;""rdb""]",1,1,addserver rejects string servertype (type error) + +/ ---- addserverattr: edge cases ---- +comment,,,,,,,addserverattr - edge cases +true,0,0,q,null first exec procname from srvsel.getserverstable[] where handle=3i,1,1,addserverattr stores null procname +true,0,0,q,null first exec hpup from srvsel.getserverstable[] where handle=3i,1,1,addserverattr stores null hpup +fail,0,0,q,"srvsel.addserverattr[`badhandle;`hdb;()!()]",1,1,addserverattr rejects symbol handle (type error) +fail,0,0,q,"srvsel.addserverattr[73i;""hdb"";()!()]",1,1,addserverattr rejects string servertype (type error) + +/ ---- setserveractive: edge cases ---- +comment,,,,,,,setserveractive - edge cases +run,0,0,q,srvsel.setserveractive[9999i;0b],1,1,setserveractive on non-existent handle is a no-op +true,0,0,q,0=count select from srvsel.getserverstable[] where handle=9999i,1,1,non-existent handle not inserted by setserveractive +fail,0,0,q,"srvsel.setserveractive[1i;`yes]",1,1,setserveractive rejects symbol active flag (type error) +fail,0,0,q,"srvsel.setserveractive[""badhandle"";0b]",1,1,setserveractive rejects string handle (type error) + +/ ---- getservers: edge cases ---- +comment,,,,,,,getservers - edge cases +run,0,0,q,srvsel.setserveractive[1i;0b],1,1,mark handle 1 inactive for getservers inactive-exclusion test +true,0,0,q,not (1i) in exec handle from srvsel.getservers[`servertype;`rdb;()!()],1,1,getservers excludes inactive servers from results +run,0,0,q,srvsel.setserveractive[1i;1b],1,1,restore handle 1 active after inactive-exclusion test +true,0,0,q,"all (exec servertype from srvsel.getservers[`servertype;`rdb`hdb;()!()]) in `rdb`hdb",1,1,getservers accepts symbol list as lookups +true,0,0,q,4i in exec handle from srvsel.getservers[`procname;`myproc;()!()],1,1,getservers procname filter returns server with matching procname + +/ ---- selector: edge cases ---- +comment,,,,,,,selector - edge cases +run,0,0,q,"singleseltab:([]handle:enlist 77i;lastp:enlist 2024.01.15D00:00)",1,1,create single-row selector table +true,0,0,q,77i~(srvsel.selector[singleseltab;`roundrobin])`handle,1,1,selector roundrobin on single row returns that row +true,0,0,q,77i~(srvsel.selector[singleseltab;`last])`handle,1,1,selector last on single row returns that row +true,0,0,q,77i~(srvsel.selector[singleseltab;`any])`handle,1,1,selector any on single row returns that row +run,0,0,q,"emptyseltab:([]handle:`int$();lastp:`timestamp$())",1,1,create empty selector table +true,0,0,q,0Ni~(srvsel.selector[emptyseltab;`roundrobin])`handle,1,1,selector roundrobin on empty table returns null handle +true,0,0,q,0Ni~(srvsel.selector[emptyseltab;`last])`handle,1,1,selector last on empty table returns null handle + +/ ---- getserverbytype: direct tests ---- +comment,,,,,,,getserverbytype - direct tests +true,0,0,q,`rdb~srvsel.getserverbytype[`rdb;`servertype;`roundrobin],1,1,getserverbytype returns correct column value for servertype +true,0,0,q,()~srvsel.getserverbytype[`nonexistent;`handle;`roundrobin],1,1,getserverbytype returns () for unknown servertype +true,0,0,q,"(srvsel.getserverbytype[`hdb;`hpup;`roundrobin]) in exec hpup from srvsel.getserverstable[] where servertype=`hdb",1,1,getserverbytype returns hpup value for hdb type + +/ ---- gethpbytype: edge cases ---- +comment,,,,,,,gethpbytype - edge cases +true,0,0,q,()~srvsel.gethpbytype[`nonexistent;`roundrobin],1,1,gethpbytype returns () for unknown servertype + +/ ---- getserverids: edge cases ---- +comment,,,,,,,getserverids - edge cases +run,0,0,q,srvsel.getserverids[`symbol$()],1,1,getserverids with empty symbol list runs without error +true,0,0,q,"0 Date: Thu, 25 Jun 2026 11:21:51 +0100 Subject: [PATCH 4/9] Migrated serverselect from dilogging to kx logging, bugfixes, added argument validation, updates to test.csv and added integeration testing --- di/serverselect/init.q | 5 + di/serverselect/integration.q | 200 ++++++++++++++++++++++++++++++++ di/serverselect/serverselect.md | 56 ++++----- di/serverselect/serverselect.q | 143 ++++++++++++++--------- di/serverselect/test.csv | 117 +++++-------------- 5 files changed, 349 insertions(+), 172 deletions(-) create mode 100644 di/serverselect/integration.q diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 5512c3c2..63dd8baa 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -1,3 +1,8 @@ \l ::serverselect.q +/ logging is an injected dependency: the start-up script that wires the modules together - +/ or the user at run time - must call init with a required `log dependency before using the +/ module. kx.log is intentionally NOT loaded here. +/ note: the injected log dict is stored in .z.m.log and called as .z.m.log[`info]["msg"] + export:([init;addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable;getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q new file mode 100644 index 00000000..f76b3b4f --- /dev/null +++ b/di/serverselect/integration.q @@ -0,0 +1,200 @@ +/ =========================================================================== +/ di.serverselect integration test +/ End-to-end narrative test driving every exported function through a +/ realistic gateway lifecycle: register -> activate/deactivate -> query -> +/ select -> bulk-register -> error handling. Complements the k4unit suite +/ in test.csv with assertions on real usage rather than per-call checks. +/ Run (QPATH must include this repo and the kx module dir): +/ QPATH=/path/to/kx/mod:/path/to/kdbx-modules q integration.q +/ Exits with a non-zero code equal to the number of failed assertions. +/ =========================================================================== + +srvsel:use`di.serverselect + +/ ---- tiny assertion harness ---- +/ note: avoid `desc`/`exp`/`log` as names - they are reserved words in KDB-X +PASS:0; FAIL:0; +lbl:{[ok] $[ok;" ok | ";" FAIL | "]}; +chk:{[nm;got;expd] + ok:got~expd; + $[ok;PASS+:1;FAIL+:1]; + m:lbl[ok],nm; + if[not ok; m:m," || got=",(.Q.s1 got)," exp=",.Q.s1 expd]; + -1 m; + }; +errtok:`$"__ERRORED__"; +chkerr:{[nm;f] + r:@[f;();{[e](errtok;e)}]; + ok:(0 null procname";1b;all null exec procname from srvsel.getserverstable[]] +chk["addserver -> null hpup";1b;all null exec hpup from srvsel.getserverstable[]] +chk["addserver -> empty attributes dict";(()!());first exec attributes from srvsel.getserverstable[] where handle=4i] +chk["addserver -> active by default";1b;all exec active from srvsel.getserverstable[]] + +/ =========================================================================== +hdr"STEP 3 addserverattr (servertype + attributes)" +srvsel.addserverattr[6i;`hdb;`date`sym!((d1;d2);`A`B)] +srvsel.addserverattr[7i;`hdb;`date`sym!((d2;d3);`B`C)] +chk["two hdb servers added";4=count srvsel.getserverstable[];1b] +chk["addserverattr stores attributes";`date`sym!((d1;d2);`A`B);first exec attributes from srvsel.getserverstable[] where handle=6i] +chk["addserverattr -> null procname";1b;null first exec procname from srvsel.getserverstable[] where handle=6i] + +/ =========================================================================== +hdr"STEP 4 addserverfull (full details)" +srvsel.addserverfull[8i;`gw1;`gateway;`:host:5000;(enlist`region)!enlist`EU] +chk["fifth server registered";5=count srvsel.getserverstable[];1b] +chk["addserverfull -> procname populated";`gw1;first exec procname from srvsel.getserverstable[] where handle=8i] +chk["addserverfull -> hpup populated";`:host:5000;first exec hpup from srvsel.getserverstable[] where handle=8i] +chk["serverid sequence is 1..5";1 2 3 4 5i;exec serverid from srvsel.getserverstable[]] +chk["handles as registered";4 5 6 7 8i;exec handle from srvsel.getserverstable[]] +chk["servertypes as registered";`rdb`rdb`hdb`hdb`gateway;exec servertype from srvsel.getserverstable[]] + +/ =========================================================================== +hdr"STEP 5 setserveractive (deactivate / reactivate)" +srvsel.setserveractive[5i;0b] +chk["handle 5 now inactive";0b;first exec active from srvsel.getserverstable[] where handle=5i] +chk["getservers excludes inactive rdb";1b;not 5i in exec handle from srvsel.getservers[`servertype;`rdb;()!()]] +srvsel.setserveractive[5i;1b] +chk["handle 5 reactivated";1b;first exec active from srvsel.getserverstable[] where handle=5i] +srvsel.setserveractive[9999i;0b] +chk["setserveractive on missing handle is a no-op";0=count select from srvsel.getserverstable[] where handle=9999i;1b] + +/ =========================================================================== +hdr"STEP 6 getservers (lookup + attribute match scoring)" +chk["lookups=` returns all active";5;count srvsel.getservers[`;`;()!()]] +chk["servertype=hdb returns 2";2;count srvsel.getservers[`servertype;`hdb;()!()]] +chk["servertype list rdb,hdb returns 4";4;count srvsel.getservers[`servertype;`rdb`hdb;()!()]] +chk["procname=gw1 returns the gateway";enlist 8i;exec handle from srvsel.getservers[`procname;`gw1;()!()]] +chk["unknown servertype returns empty";0;count srvsel.getservers[`servertype;`nope;()!()]] +chk["attribmatch column present";1b;`attribmatch in cols srvsel.getservers[`;`;()!()]] +chk["complete date match flagged for some hdb";1b;any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] + +/ =========================================================================== +hdr"STEP 7 selector (strategies on a known table)" +seltab:([]serverid:101 102 103i;handle:201 202 203i;lastp:(2024.01.03D0;2024.01.01D0;2024.01.02D0)) +chk["roundrobin picks oldest lastp";202i;(srvsel.selector[seltab;`roundrobin])`handle] +chk["last picks newest lastp";201i;(srvsel.selector[seltab;`last])`handle] +chk["any picks a row from the table";1b;((srvsel.selector[seltab;`any])`handle) in 201 202 203i] +single:([]serverid:enlist 9i;handle:enlist 99i;lastp:enlist 2024.06.01D0) +chk["single-row roundrobin returns that row";99i;(srvsel.selector[single;`roundrobin])`handle] +empt:([]serverid:`int$();handle:`int$();lastp:`timestamp$()) +chk["empty table roundrobin -> null handle";0Ni;(srvsel.selector[empt;`roundrobin])`handle] +chk["empty table any -> null handle";0Ni;(srvsel.selector[empt;`any])`handle] + +/ =========================================================================== +hdr"STEP 8 getserverbytype / gethandlebytype / gethpbytype" +chk["gethandlebytype rdb returns a live rdb handle";1b;(srvsel.gethandlebytype[`rdb;`roundrobin]) in 4 5i] +chk["gethpbytype gateway returns its hpup";`:host:5000;srvsel.gethpbytype[`gateway;`roundrobin]] +chk["getserverbytype returns requested column value";`gw1;srvsel.getserverbytype[`gateway;`procname;`roundrobin]] +chk["gethandlebytype unknown type -> ()";();srvsel.gethandlebytype[`nope;`roundrobin]] +chk["gethpbytype unknown type -> ()";();srvsel.gethpbytype[`nope;`roundrobin]] +/ round-robin rotation: with exactly 2 active rdbs, 4 LRU picks must cover both +rot:srvsel.gethandlebytype[`rdb] each 4#`roundrobin +chk["roundrobin rotates across both rdbs";4 5i;asc distinct rot] +/ selection bumps hits +hitsbefore:exec sum hits from srvsel.getserverstable[] where servertype=`gateway +srvsel.gethandlebytype[`gateway;`roundrobin] +chk["selection increments hits";1b;(hitsbefore+1)=exec sum hits from srvsel.getserverstable[] where servertype=`gateway] + +/ =========================================================================== +hdr"STEP 9 getserverids - symbol (servertype) path" +chk["single servertype rdb -> rdb serverids";1 2i;asc raze srvsel.getserverids[`rdb]] +chk["servertype list rdb,hdb -> all four";1 2 3 4i;asc raze srvsel.getserverids[`rdb`hdb]] +srvsel.setserveractive[4i;0b]; srvsel.setserveractive[5i;0b] +chkerr["all requested servertype inactive -> error";{srvsel.getserverids[`rdb]}] +srvsel.setserveractive[4i;1b]; srvsel.setserveractive[5i;1b] + +/ =========================================================================== +hdr"STEP 10 getserverids - attribute dict path" +chk["single attribute date=d1 matches hdb 6";1b;3i in raze srvsel.getserverids[enlist[`date]!enlist enlist d1]] +chk["cross (d1,d2)x(A,B) satisfied by hdb 6 alone";1b;3i in raze srvsel.getserverids[`date`sym!((d1;d2);`A`B)]] +chk["independent date(d1,d2,d3)";1b;0 hdbs 6,7";3 4i;asc raze srvsel.getserverids[`servertype`date!(`hdb;enlist d2)]] +chk["empty requirement dict -> all active serverids";1 2 3 4 5i;asc raze srvsel.getserverids[()!()]] +chkerr["besteffort=0b impossible -> error";{srvsel.getserverids[`date`besteffort!(enlist 2099.01.01;0b)]}] +chkerr["no server has requested attribute value -> error";{srvsel.getserverids[enlist[`date]!enlist enlist 2099.01.01]}] + +/ =========================================================================== +hdr"STEP 11 addserversfromtable (bulk registration)" +conntab:([]w:10 11i;proctype:`rdb`hdb;attributes:(()!();`date`sym!((d1;d2);`A`B))) +srvsel.addserversfromtable[`rdb`hdb;conntab] +chk["bulk-registered handles 10,11";2=count select from srvsel.getserverstable[] where handle in 10 11i;1b] +/ skip already-active handles (4 active, 12 new) +conntab2:([]w:4 12i;proctype:`rdb`rdb;attributes:(()!();()!())) +n0:count srvsel.getserverstable[] +srvsel.addserversfromtable[`rdb;conntab2] +chk["already-active handle 4 skipped, only 12 added";n0+1;count srvsel.getserverstable[]] +/ proctype filter: only rdb of a mixed table +conntab3:([]w:20 21i;proctype:`tickerplant`rdb;attributes:(()!();()!())) +srvsel.addserversfromtable[`rdb;conntab3] +chk["proctype filter registers only the rdb (handle 21)";1b;(0type configs; - '"di.serverselect: configs must be a dict with `log key"]; - if[not `log in key configs; +init:{[deps] + / wire injectable dependencies - must be called before any other function + / the log dependency is required; there is no fallback + / deps: a dict with a `log key; the log value is `info`warn`error!(...) - monadic + / message loggers (a kx.log instance, or a bespoke logger of the same shape) + / example: di.serverselect.init[enlist[`log]!enlist logdep] + if[99h<>type deps; + '"di.serverselect: deps must be a dict with `log key"]; + if[not `log in key deps; '"di.serverselect: log dependency is required; pass `info`warn`error functions keyed on `log"]; - if[99h<>type configs`log; + if[99h<>type deps`log; '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; - if[not all (`info`warn`error) in key configs`log; - '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key configs`log)]; - .z.m.log:configs`log; + if[not all (`info`warn`error) in key deps`log; + '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:deps`log; + }; + +raiseerror:{[msg] + / internal - log an error then signal it, so failures are observable as well as thrown + .z.m.log[`error] msg; + 'msg; }; -/ internal - increment and return the next unique server ID nextserverid:{ + / internal - increment and return the next unique server ID .z.m.serverid:serverid+1i; :.z.m.serverid; }; -/ internal - update last-access timestamp and hit count for a server handle -updatestats:{[h] - .z.m.servers:update lastp:.z.p,hits:hits+1i from servers where handle=h; +updatestats:{[sid] + / internal - update last-access timestamp and hit count for a single server + / keyed on serverid (unique) rather than handle, which may be shared by multiple servers + .z.m.servers:update lastp:.z.p,hits:hits+1i from servers where serverid=sid; }; addserverfull:{[h;pname;st;hp;att] / register a server with full details: handle, procname, servertype, hpup and attribute dictionary - .z.m.log[`info][`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; + if[not -6h=type h;raiseerror"di.serverselect: addserverfull: handle must be an int; got type ",string type h]; + if[not -11h=type st;raiseerror"di.serverselect: addserverfull: servertype must be a symbol; got type ",string type st]; + .z.m.log[`info]["addserverfull: registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; .z.m.servers:servers upsert (nextserverid[];h;pname;st;hp;1b;0Np;0i;att); }; @@ -48,7 +69,9 @@ addserver:{[h;st] setserveractive:{[h;a] / mark a registered server active (1b) or inactive (0b); called on connect and disconnect - .z.m.log[`info][`setserveractive;"marking handle=",(string h)," active=",string a]; + if[not -6h=type h;raiseerror"di.serverselect: setserveractive: handle must be an int; got type ",string type h]; + if[not -1h=type a;raiseerror"di.serverselect: setserveractive: active flag must be a boolean; got type ",string type a]; + .z.m.log[`info]["setserveractive: marking handle=",(string h)," active=",string a]; .z.m.servers:update active:a from servers where handle=h; }; @@ -62,11 +85,14 @@ addserversfromtable:{[proctypes;conntable] / conntable must have columns: w (int handle), proctype (symbol), attributes (dict per row) / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types + if[not all `w`proctype`attributes in cols conntable; + raiseerror"di.serverselect: addserversfromtable: conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable; + ]; activehandles:(0i;0Ni),exec handle from servers where active; rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), not w in activehandles; - .z.m.log[`info][`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; + .z.m.log[`info]["addserversfromtable: registering ",(string count rows)," servers from connection table"]; pnames:$[`procname in cols rows; rows`procname; count[rows]#`]; hpups:$[`hpup in cols rows; rows`hpup; count[rows]#`]; addserverfull'[rows`w;pnames;rows`proctype;hpups;rows`attributes]; @@ -90,7 +116,9 @@ getservers:{[nameortype;lookups;req] select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active; nameortype~`servertype; select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,servertype in lookups; - select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups]; + nameortype~`procname; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups; + raiseerror"di.serverselect: getservers: nameortype must be `servertype or `procname; got: ",string nameortype]; if[0=count r;:update attribmatch:attributes from r]; am:attributematch[req] each r`attributes; :update attribmatch:am from r; @@ -102,7 +130,7 @@ selector:{[servertable;selection] :$[selection=`roundrobin; first `lastp xasc servertable; selection=`any; rand servertable; selection=`last; last `lastp xasc servertable; - '"di.serverselect: unknown selection strategy: ",string selection]; + raiseerror"di.serverselect: unknown selection strategy: ",string selection]; }; getserverbytype:{[ptype;serverval;selection] @@ -111,7 +139,7 @@ getserverbytype:{[ptype;serverval;selection] r:getservers[`servertype;ptype;()!()]; if[not count r;:()]; r:selector[r;selection]; - updatestats[r`handle]; + updatestats[r`serverid]; :r serverval; }; @@ -121,36 +149,43 @@ gethpbytype:getserverbytype[;`hpup;]; getserverids:{[att] / return server IDs matching a servertype list or attribute requirement dictionary / att: symbol list of servertypes, or dict of attribute requirements (optionally keyed on `servertype) - if[99h<>type att; - if[not 11h=abs type att; - '"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; - ]; - servertype:distinct att,(); - activeservers:exec distinct servertype from servers where active; - allservers:exec distinct servertype from servers; - activeserversmsg:". available servers: ",", " sv string activeservers; - if[any null att; - '"di.serverselect: null servertype passed as argument",activeserversmsg; - ]; - if[count servertype except activeservers; - '"di.serverselect: ", - $[max not servertype in allservers; - "not valid servers: ",", " sv string servertype except allservers; - "requested servers currently inactive: ",", " sv string servertype except activeservers - ],activeserversmsg; - ]; - :(exec serverid by servertype from servers where active)[servertype]; - ]; + / dispatch the symbol-list path to getserveridsbytype and the dict path to getserveridstype + if[99h<>type att; :getserveridsbytype att]; serverids:$[`servertype in key att; raze getserveridstype[delete servertype from att] each (),att`servertype; getserveridstype[att;`all]]; - if[all 0=count each serverids;'"di.serverselect: no servers match requested attributes"]; + if[all 0=count each serverids; + raiseerror"di.serverselect: no servers match requested attributes"; + ]; :serverids; }; -/ internal - filter active servers by servertype and attribute requirements +getserveridsbytype:{[att] + / internal - resolve server IDs for a servertype symbol or symbol list + / validates each requested type is registered and currently active + if[not 11h=abs type att; + raiseerror"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; + ]; + servertype:distinct att,(); + activeservers:exec distinct servertype from servers where active; + allservers:exec distinct servertype from servers; + activeserversmsg:". available servers: ",", " sv string activeservers; + if[any null att; + raiseerror"di.serverselect: null servertype passed as argument",activeserversmsg; + ]; + if[count servertype except activeservers; + raiseerror"di.serverselect: ", + $[max not servertype in allservers; + "not valid servers: ",", " sv string servertype except allservers; + "requested servers currently inactive: ",", " sv string servertype except activeservers + ],activeserversmsg; + ]; + :(exec serverid by servertype from servers where active)[servertype]; + }; + getserveridstype:{[att;typ] - / resolve besteffort and attributetype control keys then dispatch to cross or independent matching + / internal - filter active servers by servertype and attribute requirements + / resolves besteffort and attributetype control keys then dispatches to cross or independent matching besteffort:1b; attype:`cross; svrs:$[typ=`all; @@ -168,28 +203,30 @@ getserveridstype:{[att;typ] getserversindependent[att;svrs;besteffort]; getserverscross[att;svrs;besteffort]]; serverids:first value flip $[99h=type res; key res; res]; - if[all 0=count each serverids;'"di.serverselect: no servers match ",string[typ]," requested attributes"]; + if[all 0=count each serverids; + raiseerror"di.serverselect: no servers match ",string[typ]," requested attributes"; + ]; :serverids; }; / internal - build a cross product table from a nested dictionary buildcross:{(cross/){flip (enlist y)#x}[x] each key x}; -/ internal - initial filter shared by cross and independent matching -/ drops servers missing any required attribute key, ranks survivors by coverage getserversinitial:{[req;att] + / internal - initial filter shared by cross and independent matching + / drops servers missing any required attribute key, ranks survivors by coverage if[0=count req; :([]serverid:enlist key att)]; att:(where all each (key req) in/: key each att)#att; - if[not count att;'"di.serverselect: no servers report all requested attributes"]; + if[not count att;raiseerror"di.serverselect: no servers report all requested attributes"]; s:update serverid:key att from value req in'/: (key req)#/:att; s:s idesc value min each sum each' `serverid xkey s; s:`serverid xkey 0!(key req) xgroup s; :s; }; -/ internal - find servers satisfying the cross product of all attribute requirements -/ each attribute combination must be coverable by a single server getserverscross:{[req;att;besteffort] + / internal - find servers satisfying the cross product of all attribute requirements + / each attribute combination must be coverable by a single server if[0=count req; :([]serverid:enlist key att)]; s:getserversinitial[req;att]; reqcross:buildcross[req]; @@ -198,22 +235,22 @@ getserverscross:{[req;att;besteffort] {[x;y;z] (y[0] except found; y[0] inter found:$[0=count y[0];y[0];buildcross x@'where each z])}[req]\ )[(reqcross;());value s]; if[(count last util`remaining) and not besteffort; - '"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; + raiseerror"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; ]; s:1!(0!s) w:where not 0=count each util`found; :(key s)!distinct each' flip each util[w]`found; }; -/ internal - find servers satisfying attribute requirements independently -/ each individual requirement only needs to be matched by one server getserversindependent:{[req;att;besteffort] + / internal - find servers satisfying attribute requirements independently + / each individual requirement only needs to be matched by one server if[0=count req; :([]serverid:enlist key att)]; s:getserversinitial[req;att]; / mask out server groups whose contribution is already covered by earlier groups filter:(value s)¬ -1 _ (0b&(value s) enlist 0),maxs value s; alldone:1+first where all each all each' maxs value s; if[(null alldone) and not besteffort; - '"di.serverselect: cannot satisfy query - not all attributes can be matched"; + raiseerror"di.serverselect: cannot satisfy query - not all attributes can be matched"; ]; s:1!(0!s) w:where any each any each' filter; :(key s)!{(key x)!(value x)@'where each y key x}[req] each value s&filter w; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index febd74c2..7ef42284 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -1,20 +1,7 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,srvsel:use`di.serverselect,1,1,load module -before,0,0,q,mylog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,no-op mock logger -before,0,0,q,"caplog:([] fn:`symbol$();ctx:`symbol$();msg:())",1,1,initialise log capture table -before,0,0,q,logcap:`info`warn`error!({[c;m] `caplog upsert(`info;c;m)};{[c;m] `caplog upsert(`warn;c;m)};{[c;m] `caplog upsert(`error;c;m)}),1,1,define capturing mock logger -before,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init with no-op logger for setup - -/ ---- init: dependency injection validation ---- -comment,,,,,,,init - dependency injection validation -run,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init accepts valid log dependency -fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: (not a dict) -fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty config (missing log key) -fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects config without log key -fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects log dep that is not a dict -fail,0,0,q,srvsel.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value -fail,0,0,q,srvsel.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key -true,0,0,q,"(15#@[srvsel.init;enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn));{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect +before,0,0,q,"noop:`info`warn`error!({[m]};{[m]};{[m]})",1,1,define a no-op monadic logger +before,0,0,q,srvsel.init[enlist[`log]!enlist noop],1,1,init the module with the required log dependency before use / ---- getserverstable: initial state ---- comment,,,,,,,getserverstable - initial state before any addserverattr @@ -73,6 +60,14 @@ true,0,0,q,(srvsel.gethpbytype[`hdb;`roundrobin]) in exec hpup from srvsel.getse run,0,0,q,srvsel.gethandlebytype[`rdb;`roundrobin],1,1,gethandlebytype call updates lastp and hits via updatestats true,0,0,q,"0<(exec hits from srvsel.getserverstable[] where handle in exec handle from srvsel.getserverstable[] where servertype=`rdb,active)[0]",1,1,hits incremented after gethandlebytype call +/ ---- updatestats: keyed on serverid not handle ---- +comment,,,,,,,updatestats - a selection updates only the chosen serverid even when a handle is shared +run,0,0,q,"srvsel.addserverfull[60i;`dupa;`zdb;`:h:1;()!()]",1,1,register first server on shared handle 60 (unique servertype zdb) +run,0,0,q,"srvsel.addserverfull[60i;`dupb;`zdb;`:h:2;()!()]",1,1,register second server on same handle 60 +run,0,0,q,srvsel.gethandlebytype[`zdb;`roundrobin],1,1,select one zdb server by handle +true,0,0,q,1=sum exec hits from srvsel.getserverstable[] where servertype=`zdb,1,1,exactly one hit recorded across both shared-handle servers (not two) +true,0,0,q,"1=count select from srvsel.getserverstable[] where servertype=`zdb,not null lastp",1,1,only the selected serverid had its lastp updated + / ---- getserverids: symbol list path ---- comment,,,,,,,getserverids - symbol list (servertype) path true,0,0,q,0 Date: Thu, 25 Jun 2026 16:46:34 +0100 Subject: [PATCH 5/9] updating tests --- di/serverselect/integration.q | 4 ++-- di/serverselect/serverselect.md | 22 +++++++++------------- di/serverselect/test.csv | 6 +++--- 3 files changed, 14 insertions(+), 18 deletions(-) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index f76b3b4f..4d00205c 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -185,8 +185,8 @@ chk["injected logger receives the message text";1b;any caplog[`msg] like "*regis caplog:0#caplog @[{srvsel.selector[([]handle:1 2i;lastp:2#.z.p);`bogus]};();{}] chk["injected logger captures an error message";1b;0 Date: Mon, 29 Jun 2026 11:33:54 +0100 Subject: [PATCH 6/9] adding in the standard logging block and ensuring .z.m is used correctly throughout. Fixed a couple errors --- di/serverselect/init.q | 6 ++- di/serverselect/integration.q | 12 +++--- di/serverselect/serverselect.md | 28 +++++++------- di/serverselect/serverselect.q | 67 ++++++++++++++++++++------------- di/serverselect/test.csv | 6 +-- 5 files changed, 69 insertions(+), 50 deletions(-) diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 63dd8baa..135b1e20 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -3,6 +3,8 @@ / logging is an injected dependency: the start-up script that wires the modules together - / or the user at run time - must call init with a required `log dependency before using the / module. kx.log is intentionally NOT loaded here. -/ note: the injected log dict is stored in .z.m.log and called as .z.m.log[`info]["msg"] +/ note: the injected log dict is stored in .z.m.log and called as .z.m.log[`info][`ctx;"msg"] -export:([init;addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable;getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) +export:([init; + addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable; + getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index 4d00205c..c73a8e32 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -35,7 +35,7 @@ hdr:{[t] -1""; -1"==== ",t," ===="; }; / init must be called before use (log is a required injected dependency, no default). / use a no-op logger for the behavioural steps; STEP 13 exercises init itself. -srvsel.init[enlist[`log]!enlist `info`warn`error!({[m]};{[m]};{[m]})] +srvsel.init[enlist[`log]!enlist `info`warn`error!({[c;m]};{[c;m]};{[c;m]})] d1:2024.01.01; d2:2024.01.02; d3:2024.01.03 @@ -91,7 +91,8 @@ chk["servertype list rdb,hdb returns 4";4;count srvsel.getservers[`servertype;`r chk["procname=gw1 returns the gateway";enlist 8i;exec handle from srvsel.getservers[`procname;`gw1;()!()]] chk["unknown servertype returns empty";0;count srvsel.getservers[`servertype;`nope;()!()]] chk["attribmatch column present";1b;`attribmatch in cols srvsel.getservers[`;`;()!()]] -chk["complete date match flagged for some hdb";1b;any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] +chk["complete date match flagged for some hdb";1b; + any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] / =========================================================================== hdr"STEP 7 selector (strategies on a known table)" @@ -151,7 +152,8 @@ chk["already-active handle 4 skipped, only 12 added";n0+1;count srvsel.getserver / proctype filter: only rdb of a mixed table conntab3:([]w:20 21i;proctype:`tickerplant`rdb;attributes:(()!();()!())) srvsel.addserversfromtable[`rdb;conntab3] -chk["proctype filter registers only the rdb (handle 21)";1b;(0type deps; '"di.serverselect: deps must be a dict with `log key"]; @@ -28,13 +28,26 @@ init:{[deps] '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; if[not all (`info`warn`error) in key deps`log; '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:deps`log; + .z.m.log:normlog deps`log; }; -raiseerror:{[msg] - / internal - log an error then signal it, so failures are observable as well as thrown - .z.m.log[`error] msg; - 'msg; +normlog:{[logdict] + / internal - normalise the injected logger to a binary `info`warn`error!{[c;m]} dict + / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[any `getlvl`sinks`fmts in key logdict; + `info`warn`error!( + {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); + logdict] + }; + +raiseerror:{[ctx;msg] + / internal - log an error under ctx then signal it, so failures are observable as well as thrown + .z.m.log[`error][ctx;msg]; + '"di.serverselect: ",string[ctx],": ",msg; }; nextserverid:{ @@ -51,9 +64,9 @@ updatestats:{[sid] addserverfull:{[h;pname;st;hp;att] / register a server with full details: handle, procname, servertype, hpup and attribute dictionary - if[not -6h=type h;raiseerror"di.serverselect: addserverfull: handle must be an int; got type ",string type h]; - if[not -11h=type st;raiseerror"di.serverselect: addserverfull: servertype must be a symbol; got type ",string type st]; - .z.m.log[`info]["addserverfull: registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; + if[not -6h=type h;raiseerror[`addserverfull;"handle must be an int; got type ",string type h]]; + if[not -11h=type st;raiseerror[`addserverfull;"servertype must be a symbol; got type ",string type st]]; + .z.m.log[`info][`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; .z.m.servers:servers upsert (nextserverid[];h;pname;st;hp;1b;0Np;0i;att); }; @@ -69,9 +82,9 @@ addserver:{[h;st] setserveractive:{[h;a] / mark a registered server active (1b) or inactive (0b); called on connect and disconnect - if[not -6h=type h;raiseerror"di.serverselect: setserveractive: handle must be an int; got type ",string type h]; - if[not -1h=type a;raiseerror"di.serverselect: setserveractive: active flag must be a boolean; got type ",string type a]; - .z.m.log[`info]["setserveractive: marking handle=",(string h)," active=",string a]; + if[not -6h=type h;raiseerror[`setserveractive;"handle must be an int; got type ",string type h]]; + if[not -1h=type a;raiseerror[`setserveractive;"active flag must be a boolean; got type ",string type a]]; + .z.m.log[`info][`setserveractive;"marking handle=",(string h)," active=",string a]; .z.m.servers:update active:a from servers where handle=h; }; @@ -86,13 +99,13 @@ addserversfromtable:{[proctypes;conntable] / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types if[not all `w`proctype`attributes in cols conntable; - raiseerror"di.serverselect: addserversfromtable: conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable; + raiseerror[`addserversfromtable;"conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable]; ]; activehandles:(0i;0Ni),exec handle from servers where active; rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), not w in activehandles; - .z.m.log[`info]["addserversfromtable: registering ",(string count rows)," servers from connection table"]; + .z.m.log[`info][`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; pnames:$[`procname in cols rows; rows`procname; count[rows]#`]; hpups:$[`hpup in cols rows; rows`hpup; count[rows]#`]; addserverfull'[rows`w;pnames;rows`proctype;hpups;rows`attributes]; @@ -118,7 +131,7 @@ getservers:{[nameortype;lookups;req] select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,servertype in lookups; nameortype~`procname; select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups; - raiseerror"di.serverselect: getservers: nameortype must be `servertype or `procname; got: ",string nameortype]; + raiseerror[`getservers;"nameortype must be `servertype or `procname; got: ",string nameortype]]; if[0=count r;:update attribmatch:attributes from r]; am:attributematch[req] each r`attributes; :update attribmatch:am from r; @@ -130,7 +143,7 @@ selector:{[servertable;selection] :$[selection=`roundrobin; first `lastp xasc servertable; selection=`any; rand servertable; selection=`last; last `lastp xasc servertable; - raiseerror"di.serverselect: unknown selection strategy: ",string selection]; + raiseerror[`selector;"unknown selection strategy: ",string selection]]; }; getserverbytype:{[ptype;serverval;selection] @@ -155,7 +168,7 @@ getserverids:{[att] raze getserveridstype[delete servertype from att] each (),att`servertype; getserveridstype[att;`all]]; if[all 0=count each serverids; - raiseerror"di.serverselect: no servers match requested attributes"; + raiseerror[`getserverids;"no servers match requested attributes"]; ]; :serverids; }; @@ -164,21 +177,21 @@ getserveridsbytype:{[att] / internal - resolve server IDs for a servertype symbol or symbol list / validates each requested type is registered and currently active if[not 11h=abs type att; - raiseerror"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; + raiseerror[`getserveridsbytype;"servertype must be a symbol list (11h) or attribute dict (99h)"]; ]; servertype:distinct att,(); activeservers:exec distinct servertype from servers where active; allservers:exec distinct servertype from servers; activeserversmsg:". available servers: ",", " sv string activeservers; if[any null att; - raiseerror"di.serverselect: null servertype passed as argument",activeserversmsg; + raiseerror[`getserveridsbytype;"null servertype passed as argument",activeserversmsg]; ]; if[count servertype except activeservers; - raiseerror"di.serverselect: ", + raiseerror[`getserveridsbytype; $[max not servertype in allservers; "not valid servers: ",", " sv string servertype except allservers; "requested servers currently inactive: ",", " sv string servertype except activeservers - ],activeserversmsg; + ],activeserversmsg]; ]; :(exec serverid by servertype from servers where active)[servertype]; }; @@ -204,7 +217,7 @@ getserveridstype:{[att;typ] getserverscross[att;svrs;besteffort]]; serverids:first value flip $[99h=type res; key res; res]; if[all 0=count each serverids; - raiseerror"di.serverselect: no servers match ",string[typ]," requested attributes"; + raiseerror[`getserveridstype;"no servers match ",string[typ]," requested attributes"]; ]; :serverids; }; @@ -217,7 +230,7 @@ getserversinitial:{[req;att] / drops servers missing any required attribute key, ranks survivors by coverage if[0=count req; :([]serverid:enlist key att)]; att:(where all each (key req) in/: key each att)#att; - if[not count att;raiseerror"di.serverselect: no servers report all requested attributes"]; + if[not count att;raiseerror[`getserversinitial;"no servers report all requested attributes"]]; s:update serverid:key att from value req in'/: (key req)#/:att; s:s idesc value min each sum each' `serverid xkey s; s:`serverid xkey 0!(key req) xgroup s; @@ -235,7 +248,7 @@ getserverscross:{[req;att;besteffort] {[x;y;z] (y[0] except found; y[0] inter found:$[0=count y[0];y[0];buildcross x@'where each z])}[req]\ )[(reqcross;());value s]; if[(count last util`remaining) and not besteffort; - raiseerror"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; + raiseerror[`getserverscross;"cannot satisfy query - cross product of all attributes cannot be matched"]; ]; s:1!(0!s) w:where not 0=count each util`found; :(key s)!distinct each' flip each util[w]`found; @@ -250,7 +263,7 @@ getserversindependent:{[req;att;besteffort] filter:(value s)¬ -1 _ (0b&(value s) enlist 0),maxs value s; alldone:1+first where all each all each' maxs value s; if[(null alldone) and not besteffort; - raiseerror"di.serverselect: cannot satisfy query - not all attributes can be matched"; + raiseerror[`getserversindependent;"cannot satisfy query - not all attributes can be matched"]; ]; s:1!(0!s) w:where any each any each' filter; :(key s)!{(key x)!(value x)@'where each y key x}[req] each value s&filter w; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 57116579..59128c93 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -1,6 +1,6 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,srvsel:use`di.serverselect,1,1,load module -before,0,0,q,"noop:`info`warn`error!({[m]};{[m]};{[m]})",1,1,define a no-op monadic logger +before,0,0,q,"noop:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,1,define a no-op binary logger {[c;m]} before,0,0,q,srvsel.init[enlist[`log]!enlist noop],1,1,init the module with the required log dependency before use / ---- getserverstable: initial state ---- @@ -175,7 +175,7 @@ fail,0,0,q,"srvsel.getserverids[`servertype`date!(`nonexistent;enlist 2024.01.01 / ---- init: dependency injection (required log) ---- comment,,,,,,,init - inject and validate the required log dependency run,0,0,q,"caplog:([]lvl:`symbol$();msg:())",1,1,initialise log capture table -run,0,0,q,"caplogger:`info`warn`error!({[m]`caplog upsert(`info;m)};{[m]`caplog upsert(`warn;m)};{[m]`caplog upsert(`error;m)})",1,1,define a bespoke capturing logger (info/warn/error single-arg) +run,0,0,q,"caplogger:`info`warn`error!({[c;m]`caplog upsert(`info;m)};{[c;m]`caplog upsert(`warn;m)};{[c;m]`caplog upsert(`error;m)})",1,1,define a bespoke capturing binary logger {[c;m]} run,0,0,q,srvsel.init[enlist[`log]!enlist caplogger],1,1,re-init with the bespoke capturing logger run,0,0,q,caplog:0#caplog,1,1,reset capture run,0,0,q,"srvsel.addserver[40i;`rdb]",1,1,register a server while the capturing logger is active @@ -188,6 +188,6 @@ fail,0,0,q,srvsel.init[(::)],1,1,init rejects a non-dictionary deps fail,0,0,q,srvsel.init[()!()],1,1,init rejects deps missing the log key fail,0,0,q,srvsel.init[enlist[`other]!enlist noop],1,1,init rejects deps without a log key fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects a log value that is not a dict -fail,0,0,q,"srvsel.init[enlist[`log]!enlist(`info`warn!({[m]};{[m]}))]",1,1,init rejects a log dict missing the error key +fail,0,0,q,"srvsel.init[enlist[`log]!enlist(`info`warn!({[c;m]};{[c;m]}))]",1,1,init rejects a log dict missing the error key true,0,0,q,"""di.serverselect""~15#@[srvsel.init;(::);{x}]",1,1,init error message is prefixed di.serverselect run,0,0,q,srvsel.init[enlist[`log]!enlist noop],1,1,restore the no-op logger From 5160a23c8ec147ae62ef1e433998c6a64e1a69bc Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 29 Jun 2026 12:10:58 +0100 Subject: [PATCH 7/9] fixing the serverselect.md --- di/serverselect/serverselect.md | 41 +++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md index 1a571e2f..8e8a8f8e 100644 --- a/di/serverselect/serverselect.md +++ b/di/serverselect/serverselect.md @@ -4,7 +4,7 @@ --- -## :sparkles: Features +## Features - Register backend servers with servertype, procname, host/port, and attribute dictionaries - Track active/inactive state per server, updated on connect and disconnect @@ -16,7 +16,7 @@ --- -## :gear: Initialisation & Dependencies +## Initialisation & Dependencies `init` wires the module's injected dependencies and **must be called before any other function**. The `log` dependency is **required** — there is no fallback, and the module does not load `kx.log` itself. Initialising the logging framework is the job of the start-up script that ties the modules together, or of the user at run time. @@ -46,7 +46,7 @@ srvsel.init[enlist[`log]!enlist mylog] --- -## :label: Server Table Schema +## Server Table Schema Servers are tracked in the `servers` keyed table (keyed on `serverid`): @@ -64,7 +64,7 @@ Servers are tracked in the `servers` keyed table (keyed on `serverid`): --- -## :wrench: Functions +## Functions ### Registration @@ -174,7 +174,7 @@ The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a pa --- -## :test_tube: Example +## Example ```q / load and initialise - init must be called before any other function @@ -201,3 +201,34 @@ h:srvsel.gethandlebytype[`rdb; `roundrobin] / inspect registered server pool srvsel.getserverstable[] ``` + +--- + +## Testing + +Both test suites require KDB-X (the `use` module system). Set `QPATH` so `di.*` and `kx.*` modules resolve — it must include this repository and the `kx` module directory: + +```bash +export QPATH=/path/to/kx/mod:/path/to/kdbx-modules +``` + +### Unit tests (k4unit) + +`test.csv` is a k4unit manifest covering every exported function. Run it from a q session (or script): + +```q +k4unit:use`di.k4unit; +k4unit.moduletest`di.serverselect; / prints the results table; "All tests passed" on success +``` + +The `before` rows load the module and `init` it with a no-op logger, so the tests run standalone with no external logging framework. + +### Integration test + +`integration.q` is an end-to-end narrative test that drives every exported function through a realistic gateway lifecycle (register → activate/deactivate → query → select → bulk-register → error handling) and prints a `PASS`/`FAIL` summary. Run it directly: + +```bash +QPATH=/path/to/kx/mod:/path/to/kdbx-modules q integration.q +``` + +It exits with a non-zero code equal to the number of failed assertions (`0` when all pass). From ee7a90e7193c9f51ca81255c3f7974dc4cea1a9b Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 30 Jun 2026 14:06:16 +0100 Subject: [PATCH 8/9] changing intergration tests to actually open other processes rather than pretend, updated testing to cover this and added more robust logging to main script --- di/serverselect/integration.q | 370 +++++++++++++++++++------------- di/serverselect/serverselect.md | 11 +- di/serverselect/serverselect.q | 5 + di/serverselect/test.csv | 1 + 4 files changed, 229 insertions(+), 158 deletions(-) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index c73a8e32..b3c334f4 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -1,15 +1,27 @@ / =========================================================================== -/ di.serverselect integration test -/ End-to-end narrative test driving every exported function through a -/ realistic gateway lifecycle: register -> activate/deactivate -> query -> -/ select -> bulk-register -> error handling. Complements the k4unit suite -/ in test.csv with assertions on real usage rather than per-call checks. +/ di.serverselect integration test (real processes) +/ Spawns child q listeners, opens REAL IPC handles to them, and drives every +/ exported function through a realistic gateway lifecycle - register -> +/ query -> select -> route a live query to the chosen handle -> bulk-register +/ -> disconnect -> error handling -> init. Unlike a simulated test, the +/ selection functions hand back handles that are actually queried, so the +/ assertions prove queries reach the expected backend process. +/ . +/ Self-contained: no helper files. The child backends are bare `q` listeners +/ whose identity (`whoami`) and `ping` api are injected over IPC at setup; the +/ test launches and tears them down itself. +/ . +/ Requirements: +/ - run THIS script with KDB-X q (needs `use`) +/ - a q on PATH (override with $QBIN) to launch the child listeners +/ - some free TCP ports: the test scans for free ports from a pid-derived +/ base (so concurrent runs don't collide); set $SSPORT to pin the base / Run (QPATH must include this repo and the kx module dir): -/ QPATH=/path/to/kx/mod:/path/to/kdbx-modules q integration.q +/ QPATH=/path/to/kx/mod:/path/to/kdbx-modules /q integration.q / Exits with a non-zero code equal to the number of failed assertions. / =========================================================================== -srvsel:use`di.serverselect +srvsel:use`di.serverselect; / ---- tiny assertion harness ---- / note: avoid `desc`/`exp`/`log` as names - they are reserved words in KDB-X @@ -33,169 +45,217 @@ chkerr:{[nm;f] }; hdr:{[t] -1""; -1"==== ",t," ===="; }; -/ init must be called before use (log is a required injected dependency, no default). -/ use a no-op logger for the behavioural steps; STEP 13 exercises init itself. -srvsel.init[enlist[`log]!enlist `info`warn`error!({[c;m]};{[c;m]};{[c;m]})] +d1:2024.01.01; d2:2024.01.02; d3:2024.01.03; -d1:2024.01.01; d2:2024.01.02; d3:2024.01.03 +/ ---- child-process management ---- +qbin:$[0count free; + '"could not find ",string[n]," free ports scanning up from ",string start]; + :n#free; + }; +ports:findfree[baseport;count names]; -/ =========================================================================== -hdr"STEP 2 addserver (no attributes)" -srvsel.addserver[4i;`rdb] -srvsel.addserver[5i;`rdb] -chk["two rdb servers registered";2=count srvsel.getserverstable[];1b] -chk["serverids autoincrement from 1";1 2i;exec serverid from srvsel.getserverstable[]] -chk["addserver -> null procname";1b;all null exec procname from srvsel.getserverstable[]] -chk["addserver -> null hpup";1b;all null exec hpup from srvsel.getserverstable[]] -chk["addserver -> empty attributes dict";(()!());first exec attributes from srvsel.getserverstable[] where handle=4i] -chk["addserver -> active by default";1b;all exec active from srvsel.getserverstable[]] +launch:{[port] + / start a bare q listener detached (no script needed - identity is injected over IPC) + system qbin," -p ",string[port]," -q /dev/null 2>&1 &"; + }; -/ =========================================================================== -hdr"STEP 3 addserverattr (servertype + attributes)" -srvsel.addserverattr[6i;`hdb;`date`sym!((d1;d2);`A`B)] -srvsel.addserverattr[7i;`hdb;`date`sym!((d2;d3);`B`C)] -chk["two hdb servers added";4=count srvsel.getserverstable[];1b] -chk["addserverattr stores attributes";`date`sym!((d1;d2);`A`B);first exec attributes from srvsel.getserverstable[] where handle=6i] -chk["addserverattr -> null procname";1b;null first exec procname from srvsel.getserverstable[] where handle=6i] +connect:{[port] + / open a handle, retrying for up to ~10s while the child binds and accepts + hp:`$":localhost:",string port; + step:{[hp;h] if[not null h; :h]; system"sleep 0.2"; @[hopen;(hp;500);{0Ni}]}[hp]; + :50 step/ 0Ni; + }; -/ =========================================================================== -hdr"STEP 4 addserverfull (full details)" -srvsel.addserverfull[8i;`gw1;`gateway;`:host:5000;(enlist`region)!enlist`EU] -chk["fifth server registered";5=count srvsel.getserverstable[];1b] -chk["addserverfull -> procname populated";`gw1;first exec procname from srvsel.getserverstable[] where handle=8i] -chk["addserverfull -> hpup populated";`:host:5000;first exec hpup from srvsel.getserverstable[] where handle=8i] -chk["serverid sequence is 1..5";1 2 3 4 5i;exec serverid from srvsel.getserverstable[]] -chk["handles as registered";4 5 6 7 8i;exec handle from srvsel.getserverstable[]] -chk["servertypes as registered";`rdb`rdb`hdb`hdb`gateway;exec servertype from srvsel.getserverstable[]] +teardown:{[] + / always-run cleanup. first kill by captured pid (exact); then a port-based + / fallback so children are reaped even when we never connected (no pid). + / the [-]p bracket keeps the pattern from matching this very kill command. + live:pids where not null pids; + if[count live; @[system;"kill ",(" " sv string live)," 2>/dev/null; true";{}]]; + {@[system;"pkill -f \"bin/q [-]p ",string[x],"\" 2>/dev/null; true";{}]} each ports; + @[hclose;;{}] each handles where not null handles; + }; -/ =========================================================================== -hdr"STEP 5 setserveractive (deactivate / reactivate)" -srvsel.setserveractive[5i;0b] -chk["handle 5 now inactive";0b;first exec active from srvsel.getserverstable[] where handle=5i] -chk["getservers excludes inactive rdb";1b;not 5i in exec handle from srvsel.getservers[`servertype;`rdb;()!()]] -srvsel.setserveractive[5i;1b] -chk["handle 5 reactivated";1b;first exec active from srvsel.getserverstable[] where handle=5i] -srvsel.setserveractive[9999i;0b] -chk["setserveractive on missing handle is a no-op";0=count select from srvsel.getserverstable[] where handle=9999i;1b] +/ launch the fleet, connect, capture each child's own pid (.z.i) for exact-pid teardown +-1 "selected free ports: ",.Q.s1 ports; +launch each ports; +handles:connect each ports; +pids:{$[null x; 0Ni; @[x;".z.i";{0Ni}]]} each handles; +/ guarantee the fleet is reaped on ANY exit (normal, error-abort, or exit code) +.z.exit:{teardown[]}; +if[any null handles; + -2 "failed to connect to child processes on ports: ",(" " sv string ports where null handles); + exit 3]; +/ inject each backend's identity + ping api over IPC +{[h;nm] h "whoami:`",string nm; h "ping:{whoami}";}'[handles;names]; +-1 "spawned ",(string count handles)," backend processes; real handles: ",.Q.s1 handles; -/ =========================================================================== -hdr"STEP 6 getservers (lookup + attribute match scoring)" -chk["lookups=` returns all active";5;count srvsel.getservers[`;`;()!()]] -chk["servertype=hdb returns 2";2;count srvsel.getservers[`servertype;`hdb;()!()]] -chk["servertype list rdb,hdb returns 4";4;count srvsel.getservers[`servertype;`rdb`hdb;()!()]] -chk["procname=gw1 returns the gateway";enlist 8i;exec handle from srvsel.getservers[`procname;`gw1;()!()]] -chk["unknown servertype returns empty";0;count srvsel.getservers[`servertype;`nope;()!()]] -chk["attribmatch column present";1b;`attribmatch in cols srvsel.getservers[`;`;()!()]] -chk["complete date match flagged for some hdb";1b; - any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] +/ route: select a server of type t via strategy sel, then SEND ping[] over the chosen handle +route:{[t;sel] h:srvsel.gethandlebytype[t;sel]; $[()~h; `NONE; h "ping[]"]}; +/ map serverids -> their handles (to route attribute-path selections) +sidhandles:{[sids] exec handle from srvsel.getserverstable[] where serverid in sids}; +/ the gateway's host/port for the hpup column - its real (dynamically chosen) address +gwhp:`$":localhost:",string ports 4; -/ =========================================================================== -hdr"STEP 7 selector (strategies on a known table)" -seltab:([]serverid:101 102 103i;handle:201 202 203i;lastp:(2024.01.03D0;2024.01.01D0;2024.01.02D0)) -chk["roundrobin picks oldest lastp";202i;(srvsel.selector[seltab;`roundrobin])`handle] -chk["last picks newest lastp";201i;(srvsel.selector[seltab;`last])`handle] -chk["any picks a row from the table";1b;((srvsel.selector[seltab;`any])`handle) in 201 202 203i] -single:([]serverid:enlist 9i;handle:enlist 99i;lastp:enlist 2024.06.01D0) -chk["single-row roundrobin returns that row";99i;(srvsel.selector[single;`roundrobin])`handle] -empt:([]serverid:`int$();handle:`int$();lastp:`timestamp$()) -chk["empty table roundrobin -> null handle";0Ni;(srvsel.selector[empt;`roundrobin])`handle] -chk["empty table any -> null handle";0Ni;(srvsel.selector[empt;`any])`handle] +/ test steps run at top level (a single enclosing function would exceed q's +/ per-function constant limit). chk/chkerr trap their own assertion failures; +/ .z.exit guarantees teardown regardless of how the script ends. -/ =========================================================================== -hdr"STEP 8 getserverbytype / gethandlebytype / gethpbytype" -chk["gethandlebytype rdb returns a live rdb handle";1b;(srvsel.gethandlebytype[`rdb;`roundrobin]) in 4 5i] -chk["gethpbytype gateway returns its hpup";`:host:5000;srvsel.gethpbytype[`gateway;`roundrobin]] -chk["getserverbytype returns requested column value";`gw1;srvsel.getserverbytype[`gateway;`procname;`roundrobin]] -chk["gethandlebytype unknown type -> ()";();srvsel.gethandlebytype[`nope;`roundrobin]] -chk["gethpbytype unknown type -> ()";();srvsel.gethpbytype[`nope;`roundrobin]] -/ round-robin rotation: with exactly 2 active rdbs, 4 LRU picks must cover both -rot:srvsel.gethandlebytype[`rdb] each 4#`roundrobin -chk["roundrobin rotates across both rdbs";4 5i;asc distinct rot] -/ selection bumps hits -hitsbefore:exec sum hits from srvsel.getserverstable[] where servertype=`gateway -srvsel.gethandlebytype[`gateway;`roundrobin] -chk["selection increments hits";1b;(hitsbefore+1)=exec sum hits from srvsel.getserverstable[] where servertype=`gateway] +/ init must be called before use (log is a required injected dependency, no default) +srvsel.init[enlist[`log]!enlist `info`warn`error!({[c;m]};{[c;m]};{[c;m]})]; -/ =========================================================================== -hdr"STEP 9 getserverids - symbol (servertype) path" -chk["single servertype rdb -> rdb serverids";1 2i;asc raze srvsel.getserverids[`rdb]] -chk["servertype list rdb,hdb -> all four";1 2 3 4i;asc raze srvsel.getserverids[`rdb`hdb]] -srvsel.setserveractive[4i;0b]; srvsel.setserveractive[5i;0b] -chkerr["all requested servertype inactive -> error";{srvsel.getserverids[`rdb]}] -srvsel.setserveractive[4i;1b]; srvsel.setserveractive[5i;1b] +/ ========================================================================= +hdr"STEP 1 initial empty state"; +chk["fresh module: server table empty";0=count srvsel.getserverstable[];1b]; +chk["schema columns correct";`serverid`handle`procname`servertype`hpup`active`lastp`hits`attributes;cols srvsel.getserverstable[]]; -/ =========================================================================== -hdr"STEP 10 getserverids - attribute dict path" -chk["single attribute date=d1 matches hdb 6";1b;3i in raze srvsel.getserverids[enlist[`date]!enlist enlist d1]] -chk["cross (d1,d2)x(A,B) satisfied by hdb 6 alone";1b;3i in raze srvsel.getserverids[`date`sym!((d1;d2);`A`B)]] -chk["independent date(d1,d2,d3)";1b;0 hdbs 6,7";3 4i;asc raze srvsel.getserverids[`servertype`date!(`hdb;enlist d2)]] -chk["empty requirement dict -> all active serverids";1 2 3 4 5i;asc raze srvsel.getserverids[()!()]] -chkerr["besteffort=0b impossible -> error";{srvsel.getserverids[`date`besteffort!(enlist 2099.01.01;0b)]}] -chkerr["no server has requested attribute value -> error";{srvsel.getserverids[enlist[`date]!enlist enlist 2099.01.01]}] +/ ========================================================================= +hdr"STEP 2 registration of live handles (addserver / addserverattr / addserverfull)"; +srvsel.addserver[handles 0;`rdb]; +srvsel.addserver[handles 1;`rdb]; +srvsel.addserverattr[handles 2;`hdb;`date`sym!((d1;d2);`A`B)]; +srvsel.addserverattr[handles 3;`hdb;`date`sym!((d2;d3);`B`C)]; +srvsel.addserverfull[handles 4;`gw1;`gateway;gwhp;(enlist`region)!enlist`EU]; +chk["five servers registered";5=count srvsel.getserverstable[];1b]; +chk["serverids autoincrement 1..5";1 2 3 4 5i;exec serverid from srvsel.getserverstable[]]; +chk["handles stored are the real open handles";handles til 5;exec handle from srvsel.getserverstable[]]; +chk["servertypes as registered";`rdb`rdb`hdb`hdb`gateway;exec servertype from srvsel.getserverstable[]]; +chk["addserverfull populated procname";`gw1;first exec procname from srvsel.getserverstable[] where servertype=`gateway]; +chk["addserverfull populated hpup";gwhp;first exec hpup from srvsel.getserverstable[] where servertype=`gateway]; +chk["addserver -> empty attributes";(()!());first exec attributes from srvsel.getserverstable[] where handle=handles 0]; +chk["all active by default";1b;all exec active from srvsel.getserverstable[]]; -/ =========================================================================== -hdr"STEP 11 addserversfromtable (bulk registration)" -conntab:([]w:10 11i;proctype:`rdb`hdb;attributes:(()!();`date`sym!((d1;d2);`A`B))) -srvsel.addserversfromtable[`rdb`hdb;conntab] -chk["bulk-registered handles 10,11";2=count select from srvsel.getserverstable[] where handle in 10 11i;1b] -/ skip already-active handles (4 active, 12 new) -conntab2:([]w:4 12i;proctype:`rdb`rdb;attributes:(()!();()!())) -n0:count srvsel.getserverstable[] -srvsel.addserversfromtable[`rdb;conntab2] -chk["already-active handle 4 skipped, only 12 added";n0+1;count srvsel.getserverstable[]] -/ proctype filter: only rdb of a mixed table -conntab3:([]w:20 21i;proctype:`tickerplant`rdb;attributes:(()!();()!())) -srvsel.addserversfromtable[`rdb;conntab3] -chk["proctype filter registers only the rdb (handle 21)";1b; - (0 null handle";0Ni;(srvsel.selector[empt;`roundrobin])`handle]; -/ =========================================================================== -hdr"STEP 13 init (required log dependency - kx.log or bespoke)" -caplog:([]lvl:`symbol$();msg:()) -caplogger:`info`warn`error!({[c;m]`caplog upsert(`info;m)};{[c;m]`caplog upsert(`warn;m)};{[c;m]`caplog upsert(`error;m)}) -srvsel.init[enlist[`log]!enlist caplogger] -caplog:0#caplog -srvsel.addserver[40i;`rdb] -chk["injected logger captures an info message";1b;0 ()";();srvsel.gethandlebytype[`nope;`roundrobin]]; +chk["a single hdb query is answered by an hdb";1b;(route[`hdb;`roundrobin]) in `hdb1`hdb2]; +hitsbefore:exec sum hits from srvsel.getserverstable[] where servertype=`gateway; +srvsel.gethandlebytype[`gateway;`roundrobin]; +chk["selection increments hits";1b;(hitsbefore+1)=exec sum hits from srvsel.getserverstable[] where servertype=`gateway]; + +/ ========================================================================= +hdr"STEP 6 getserverids - servertype (symbol) path"; +chk["rdb -> the two rdb serverids";1 2i;asc raze srvsel.getserverids[`rdb]]; +chk["rdb,hdb -> all four serverids";1 2 3 4i;asc raze srvsel.getserverids[`rdb`hdb]]; +srvsel.setserveractive[handles 0;0b]; srvsel.setserveractive[handles 1;0b]; +chkerr["all requested servertype inactive -> error";{srvsel.getserverids[`rdb]}]; +srvsel.setserveractive[handles 0;1b]; srvsel.setserveractive[handles 1;1b]; + +/ ========================================================================= +hdr"STEP 7 getserverids - attribute path + routing to the matched backend"; +/ hdb1 covers (d1,d2)/(A,B); hdb2 covers (d2,d3)/(B,C) +chk["date=d1 -> hdb1 only";enlist`hdb1; + distinct {x"ping[]"} each sidhandles raze srvsel.getserverids[enlist[`date]!enlist enlist d1]]; +chk["cross (d1,d2)x(A,B) -> hdb1 alone covers it";enlist`hdb1; + distinct {x"ping[]"} each sidhandles raze srvsel.getserverids[`date`sym!((d1;d2);`A`B)]]; +chk["independent over d1,d2,d3 + A,B,C -> both hdbs contribute";`hdb1`hdb2; + asc distinct {x"ping[]"} each sidhandles raze srvsel.getserverids[`date`sym`attributetype!((d1;d2;d3);`A`B`C;`independent)]]; +chk["servertype key scopes attribute filter to hdbs";3 4i;asc raze srvsel.getserverids[`servertype`date!(`hdb;enlist d2)]]; +chk["empty requirement dict -> all active serverids";1 2 3 4 5i;asc raze srvsel.getserverids[()!()]]; +chkerr["besteffort=0b impossible -> error";{srvsel.getserverids[`date`besteffort!(enlist 2099.01.01;0b)]}]; +chkerr["no server has requested attribute value -> error";{srvsel.getserverids[enlist[`date]!enlist enlist 2099.01.01]}]; + +/ ========================================================================= +hdr"STEP 8 addserversfromtable (bulk registration of live handles)"; +conntab:([]w:handles 5 6;proctype:`rdb`hdb;attributes:(()!();`date`sym!((d1;d3);`A`C))); +srvsel.addserversfromtable[`rdb`hdb;conntab]; +chk["bulk-registered handles 5,6 now present";2=count select from srvsel.getserverstable[] where handle in handles 5 6;1b]; +chk["bulk-registered rdb3 is reachable via its handle";`rdb3;(handles 5)"ping[]"]; +chk["bulk-registered rdb3 now appears in rdb serverids";1b;6i in raze srvsel.getserverids[`rdb]]; +/ skip-already-active: re-offering a live handle must not duplicate it +n0:count srvsel.getserverstable[]; +srvsel.addserversfromtable[`rdb;([]w:enlist handles 0;proctype:enlist`rdb;attributes:enlist()!())]; +chk["already-active handle skipped (no new row)";n0;count srvsel.getserverstable[]]; +/ (proctype filter, `ALL, and optional procname/hpup columns are covered in test.csv) + +/ ========================================================================= +hdr"STEP 9 disconnect: kill a backend, deactivate it, prove routing adapts"; +/ a real gateway's .z.pc handler would call setserveractive[h;0b] on the dropped handle +@[system;"kill ",string[pids 0]," 2>/dev/null; true";{}]; system"sleep 0.3"; +srvsel.setserveractive[handles 0;0b]; +/ active rdbs are now rdb2 (live) and rdb3 (live); the dead rdb1 must never be picked +postresp:{[i] route[`rdb;`roundrobin]} each til 6; +chk["killed/deactivated rdb1 is never selected";1b;not `rdb1 in postresp]; +chk["no routed query hits a dead handle (all answered)";1b;not `NONE in postresp]; +chk["the two remaining live rdbs both serve";`rdb2`rdb3;asc distinct postresp]; + +/ ========================================================================= +hdr"STEP 10 error handling (every guarded path signals + logs)"; +chkerr["addserverfull rejects non-int handle";{srvsel.addserverfull[`bad;`;`rdb;`;()!()]}]; +chkerr["addserverfull rejects non-symbol servertype";{srvsel.addserverfull[71i;`;"rdb";`;()!()]}]; +chkerr["addserver rejects bad handle";{srvsel.addserver[`bad;`rdb]}]; +chkerr["addserverattr rejects bad servertype";{srvsel.addserverattr[73i;"hdb";()!()]}]; +chkerr["setserveractive rejects non-int handle";{srvsel.setserveractive["bad";0b]}]; +chkerr["setserveractive rejects non-boolean flag";{srvsel.setserveractive[1i;`yes]}]; +chkerr["getservers rejects bad nameortype";{srvsel.getservers[`badname;`x;()!()]}]; +chkerr["selector rejects unknown strategy";{srvsel.selector[seltab;`bogus]}]; +chkerr["getserverids rejects null servertype";{srvsel.getserverids[`]}]; +chkerr["getserverids rejects unregistered servertype";{srvsel.getserverids[`nope]}]; +chkerr["getserverids rejects non-symbol non-dict arg";{srvsel.getserverids[42]}]; +chkerr["getserverids rejects nested/malformed attribute value";{srvsel.getserverids[(enlist`date)!enlist enlist d1,d2]}]; +chkerr["addserversfromtable rejects missing columns";{srvsel.addserversfromtable[`rdb;([]x:enlist 1i)]}]; + +/ ========================================================================= +hdr"STEP 11 init (required log dependency - bespoke + kx.log)"; +caplog:([]lvl:`symbol$();msg:()); +caplogger:`info`warn`error!({[c;m]`caplog upsert(`info;m)};{[c;m]`caplog upsert(`warn;m)};{[c;m]`caplog upsert(`error;m)}); +srvsel.init[enlist[`log]!enlist caplogger]; +caplog:0#caplog; +srvsel.addserver[40i;`rdb]; +chk["injected logger captures an info message";1b;0type att; :getserveridsbytype att]; + / validate attribute-requirement values are atoms or simple vectors (control keys excepted); + / a nested/general-list value (type 0h) would otherwise crash the cross matcher with a raw 'type + reqkeys:key[att] except `servertype`attributetype`besteffort; + if[count badkeys:reqkeys where 0h=type each att reqkeys; + raiseerror[`getserverids;"attribute requirement values must be atoms or simple vectors; nested/mixed for: ",", " sv string badkeys]]; serverids:$[`servertype in key att; raze getserveridstype[delete servertype from att] each (),att`servertype; getserveridstype[att;`all]]; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 59128c93..1db409d5 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -171,6 +171,7 @@ true,0,0,q,"0 Date: Fri, 3 Jul 2026 09:07:46 +0100 Subject: [PATCH 9/9] Added a fix from the automatic reviewer --- di/serverselect/integration.q | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index b3c334f4..1975277a 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -56,7 +56,10 @@ pids:`int$(); / no hardcoded ports: derive a starting point from this pid (so concurrent runs / on a shared host don't collide) - or take an explicit base from $SSPORT - then / scan upward for ports that are actually free. nothing assumes a fixed port. -inuse:{[p] @[{hclose hopen x; 1b}; (`$":localhost:",string p;200); 0b]}; +/ probe whether a localhost port is in use: a free port refuses instantly, so this short +/ timeout only bounds the wait for a filtered (silently-dropped) port - keep it small so a +/ full scan of 10*count names candidates cannot stall for many seconds +inuse:{[p] @[{hclose hopen x; 1b}; (`$":localhost:",string p;50); 0b]}; baseport:$[0/dev/null 2>&1 &"; + / start a bare q listener detached and capture its shell background pid ($!) - so every + / child can be reaped by exact pid later, whether or not we ever connect to it, and with + / no dependence on the qbin path (identity/api are injected over IPC once connected) + :"I"$first system qbin," -p ",string[port]," -q /dev/null 2>&1 & echo $!"; }; connect:{[port] @@ -81,20 +86,20 @@ connect:{[port] }; teardown:{[] - / always-run cleanup. first kill by captured pid (exact); then a port-based - / fallback so children are reaped even when we never connected (no pid). - / the [-]p bracket keeps the pattern from matching this very kill command. + / always-run cleanup. kill every launched child by its captured pid (exact). the pkill + / fallback is a last resort for the rare case a pid was never captured - keyed on our own + / unique port and -q flag (NOT the qbin path, which is configurable), so a custom qbin + / cannot break it. the [-]p bracket keeps the pattern from matching this very kill command. live:pids where not null pids; if[count live; @[system;"kill ",(" " sv string live)," 2>/dev/null; true";{}]]; - {@[system;"pkill -f \"bin/q [-]p ",string[x],"\" 2>/dev/null; true";{}]} each ports; + {@[system;"pkill -f \"[-]p ",string[x]," -q\" 2>/dev/null; true";{}]} each ports; @[hclose;;{}] each handles where not null handles; }; -/ launch the fleet, connect, capture each child's own pid (.z.i) for exact-pid teardown +/ launch the fleet (capturing each child's shell pid for exact-pid teardown), then connect -1 "selected free ports: ",.Q.s1 ports; -launch each ports; +pids:launch each ports; handles:connect each ports; -pids:{$[null x; 0Ni; @[x;".z.i";{0Ni}]]} each handles; / guarantee the fleet is reaped on ANY exit (normal, error-abort, or exit code) .z.exit:{teardown[]}; if[any null handles; @@ -212,6 +217,10 @@ chk["already-active handle skipped (no new row)";n0;count srvsel.getserverstable / ========================================================================= hdr"STEP 9 disconnect: kill a backend, deactivate it, prove routing adapts"; / a real gateway's .z.pc handler would call setserveractive[h;0b] on the dropped handle +/ guard: the kill is only a genuine test if rdb1's pid was captured. if pids 0 were null, +/ string 0Ni -> "0Ni" and the kill silently no-ops, degrading STEP 9 into a deactivation-only +/ test. fail loudly here rather than pass on a half-exercised path. +chk["rdb1 pid captured (so the kill below is real)";1b;not null pids 0]; @[system;"kill ",string[pids 0]," 2>/dev/null; true";{}]; system"sleep 0.3"; srvsel.setserveractive[handles 0;0b]; / active rdbs are now rdb2 (live) and rdb3 (live); the dead rdb1 must never be picked