diff --git a/.gitignore b/.gitignore index 1377554e..54399fca 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ *.swp +di/dbwrite/sortconfig.csv diff --git a/di/dbwrite/dbwrite.md b/di/dbwrite/dbwrite.md new file mode 100644 index 00000000..c4ab8fc6 --- /dev/null +++ b/di/dbwrite/dbwrite.md @@ -0,0 +1,208 @@ +# di.dbwrite + +Write, sort, and attribute utilities for kdb+ processes that persist data to disk (rdb, wdb, tickerlogreplay). + +A **config table** (`tabname`,`att`,`column`,`sort`) drives which columns are sorted and which attributes are applied per table. Load it once with `readcsv` (from a CSV) or `setconfig` (from an in-memory table); `sort` and `savedown` read from module state automatically. A table with no explicit entry falls back to a `default` row; if no config has been loaded, the built-in default (sort every table by `time` ascending) applies. + +--- + +## Features + +- Write an in-memory table to a date-partitioned HDB with `savedown` — enumerates syms, writes, sorts per stored config, then runs `.Q.gc[]` +- Append rows to an existing partition with `appenddown` — enumerates syms and appends; sort separately when the partition is complete +- Sort on-disk table partitions by configured columns using `xasc`, then apply kdb+ attributes (`p`,`s`,`g`,`u`) +- Config loaded once via `readcsv` (from a CSV) or `setconfig` (from a table); inspectable at any time with `getconfig` +- Sort and attribute errors are caught-and-logged (a single partition failure does not halt the run); config and write errors are raised to the caller with a `di.dbwrite:` prefix + +--- + +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | Functions `info`,`warn`,`error` — each monadic `{[msg] ...}` | + +The `log` dependency must be passed to `init`. The module throws if it is absent, `(::)`, or missing any of the three keys. The functions are monadic message loggers (`{[msg] ...}`); a [`kx.log`](https://github.com/KxSystems/logging) instance satisfies the contract. The module folds its own context into each message as a `dbwrite:` prefix, so no per-call context argument is needed: + +```q +logger:use`kx.log +loginst:logger.createLog[] +logdep:`info`warn`error!(loginst`info;loginst`warn;loginst`error) +dbwrite:use`di.dbwrite +dbwrite.init[enlist[`log]!enlist logdep] +``` + +--- + +## The config table + +| Column | Type | Description | +|---|---|---| +| `tabname` | symbol | Table name, or `` `default `` as a catch-all fallback | +| `att` | symbol | Attribute applied after sort: `p`,`s`,`g`,`u`, or empty (`` ` ``) for none | +| `column` | symbol | Column to sort and/or attribute | +| `sort` | boolean | `1b` — include in the `xasc` sort key; `0b` — attribute only | + +Build it directly and load with `setconfig`, or read it from a CSV with `readcsv`. A CSV must have exactly the four columns `tabname,att,column,sort` in **any** order (the result is normalised to canonical order); a missing, extra, or misnamed column raises a clear `di.dbwrite:` error rather than silently mis-parsing. + +``` +tabname,att,column,sort +trade,p,sym,1 +trade,,price,0 +default,,time,1 +``` + +--- + +## Functions + +| Function | Description | +|---|---| +| `init[deps]` | Wire injected dependencies; must be called first | +| `readcsv[file]` | Read a config CSV and store it in module state | +| `setconfig[t]` | Store a hand-built config table in module state | +| `getconfig[]` | Return the currently stored config (`::` if not yet set) | +| `sort[tabname;dirs]` | Sort on-disk partition(s) for a table and apply attributes per stored config | +| `savedown[dir;part;tabname;data]` | Write an in-memory table to an HDB partition, sort it, then run gc | +| `appenddown[dir;part;tabname;data]` | Append rows to an existing partition (no sort) | +| `applyattr[dloc;colname;att]` | Apply a single kdb+ attribute to an on-disk column | + +--- + +### `init[deps]` + +Wire injected dependencies. Must be called before any other function. Also resets the stored sort config to `(::)`. + +| Arg | Type | Description | +|---|---|---| +| `deps` | dict | Must contain `` `log `` → `` `info`warn`error!(infofn;warnfn;errfn) `` | + +Throws (prefixed `di.dbwrite:`) if `deps` is not a dict, `log` is missing, or the log dict lacks any required key. + +--- + +### `readcsv[file]` + +Read a config CSV and store it in module state (equivalent to calling `setconfig` with the parsed result). The stored config is used by subsequent calls to `sort` and `savedown`. + +| Parameter | Type | Description | +|---|---|---| +| `file` | symbol/hsym/string | Path to the CSV. Symbols are coerced with `hsym`; strings are converted via `hsym `$`. Errors (`di.dbwrite:`) if not a symbol or string. | + +The CSV is parsed field-by-field and fully validated before storing — it does **not** silently pad, truncate, or coerce malformed rows. A clear `di.dbwrite:` error is raised if: the header is not exactly `tabname,att,column,sort` (any order); any data row does not have exactly four fields; any `sort` value is not `0` or `1`; or any `att` value is not in `` ` `p`s`g`u ``. Column order is normalised to canonical. + +```q +dbwrite.readcsv `:config/sort.csv +dbwrite.readcsv "config/sort.csv" +``` + +--- + +### `setconfig[t]` + +Store a hand-built config table in module state. Alternative to `readcsv` when the config is constructed in-session rather than read from a file. Validates the table before storing — throws (`di.dbwrite:`) on any schema or content error. + +| Parameter | Type | Description | +|---|---|---| +| `t` | table | Config table with columns `tabname`,`att`,`column`,`sort` | + +```q +dbwrite.setconfig ([] tabname:`trade`default; att:`p`; column:`sym`time; sort:11b) +``` + +--- + +### `getconfig[]` + +Return the currently stored sort config. Returns `(::)` if `init` has been called but neither `readcsv` nor `setconfig` has been called yet. + +```q +dbwrite.getconfig[] +``` + +--- + +### `sort[tabname;dirs]` + +Sort and apply attributes to the on-disk partition(s) for one table, using the config stored by `readcsv` or `setconfig`. Falls back to the built-in default (sort by `time` ascending) if no config has been loaded. + +| Parameter | Type | Description | +|---|---|---| +| `tabname` | symbol | Table name. Errors (`di.dbwrite:`) if not a symbol. | +| `dirs` | hsym, or list of hsyms | Partition directory or directories. | + +Row lookup: the table's own rows → the `default` row → otherwise a warn is logged and `()` returned. Each partition is processed independently; a failure on one is logged and does not halt the rest. + +```q +dbwrite.sort[`trade; (`:hdb/2024.01.02/trade; `:hdb/2024.01.03/trade)] +``` + +--- + +### `savedown[dir;part;tabname;data]` + +Write an in-memory table to a date-partitioned HDB partition, sort it per the stored config, then run `.Q.gc[]`. Enumerates symbol columns against the HDB sym file before writing. Sorting and attributes are applied by `sort` *after* the write — so an attribute like `p#` is only applied once its column is correctly grouped. + +| Parameter | Type | Description | +|---|---|---| +| `dir` | hsym | HDB root directory (e.g. `` `:hdb ``). | +| `part` | date/month/int | Partition value. | +| `tabname` | symbol | Table name — determines the partition subdirectory. | +| `data` | table | In-memory table to write. | + +Throws on write failure. + +```q +dbwrite.savedown[`:hdb; 2024.01.02; `trade; data] +``` + +--- + +### `appenddown[dir;part;tabname;data]` + +Append rows to an existing on-disk partition (enumerates syms); does **not** sort. Keeping sort separate allows multiple intraday appends without re-sorting a growing partition on each call — sort once when the partition is complete. + +| Parameter | Type | Description | +|---|---|---| +| `dir` | hsym | HDB root directory. | +| `part` | date/month/int | Partition value. | +| `tabname` | symbol | Table name. | +| `data` | table | Rows to append. | + +Throws (prefixed `di.dbwrite:`) if the partition does not exist, or on write failure. + +```q +/ intraday: append each batch as it arrives +dbwrite.appenddown[`:hdb; 2024.01.02; `trade; batch] +/ end-of-day: sort once when done +dbwrite.sort[`trade; .Q.par[`:hdb; 2024.01.02; `trade]] +``` + +--- + +### `applyattr[dloc;colname;att]` + +Apply a single kdb+ attribute to one on-disk column (best-effort: logs and swallows errors so a run continues). A non-attribute `att` (the empty sentinel or any value outside `` `p`s`g`u ``) is a silent no-op. + +```q +dbwrite.applyattr[`:hdb/2024.01.02/trade; `sym; `p] +``` + +--- + +## Running tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.dbwrite +``` + +The suite injects monadic mock loggers (`{[msg] ...}`): a no-op logger, and a capturing logger that records `(level;msg)` so log behaviour can be asserted. It also wires a real `kx.log` instance through `init` to confirm the module works end-to-end against the system logger. On-disk behaviour (sort, attributes, `savedown`/`appenddown`) is exercised against real splayed partitions and cleaned up afterwards. It covers: dependency-injection validation; `readcsv` stores-and-verifies / column-order independence / header-validation failures; `setconfig` happy path and validation failures; `sort` edge cases / resolution / on-disk results / multi-dir / non-fatal partition failure; `savedown` write+sort (default and explicit config, and a table without `sym`); `appenddown` append-without-sort then explicit sort, and the non-existent-partition error; `applyattr`; the `dbwrite:`-prefixed logging contract; and the real `kx.log` integration. + +--- + +## Exported symbols + +```q +export:([init;readcsv;setconfig;getconfig;sort;applyattr;savedown;appenddown]) +``` diff --git a/di/dbwrite/dbwrite.q b/di/dbwrite/dbwrite.q new file mode 100644 index 00000000..c948236d --- /dev/null +++ b/di/dbwrite/dbwrite.q @@ -0,0 +1,225 @@ +/ dbwrite - write, sort, and attribute utilities for on-disk data +/ used by processes that persist data to disk (rdb, wdb, tickerlogreplay) + +/ attributes that may legitimately appear in a config (empty leaves a column unattributed) +validatts:``p`s`g`u; + +/ built-in fallback config - sort every table by time ascending when no config is supplied +defaultparams:([] tabname:enlist`default; att:enlist`; column:enlist`time; sort:enlist 1b); + +init:{[deps] + / wire the injected log dependency + / deps: `log!(logdict) where logdict is `info`warn`error!(infofn;warnfn;errfn) - required + / the functions are monadic {[msg]} loggers; a kx.log instance satisfies this (use`kx.log;createLog[]) + if[99h<>type deps; + '"di.dbwrite: deps must be a dict with a `log key; see kx.log for a logger"]; + if[not `log in key deps; + '"di.dbwrite: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.dbwrite: log value must be a dict of `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.dbwrite: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:deps`log; + .z.m.sortconfig:(::); + }; + +readcsv:{[file] + / read a sort-config csv and store it in .z.m.sortconfig; used by sort and savedown + / the csv must have the columns tabname,att,column,sort (in any order) + / file: hsym, bare symbol, or string path + if[10h=type file; file:hsym `$file]; + if[-11h=type file; if[not ":" = first string file; file:hsym file]]; + if[not -11h=type file; + '"di.dbwrite: readcsv file must be a symbol or string path, got type ",string type file]; + t:parsecsv @[readfile; file; readerr[file]]; + checkconfig t; + .z.m.log[`info]["dbwrite: read ",(string count t)," sort config row(s) from ",string file]; + .z.m.sortconfig:t; + }; + +setconfig:{[t] + / set .z.m.sortconfig from an in-memory table; alternative to readcsv when config is built in-session + / t must be a table with columns tabname,att,column,sort + checkconfig t; + .z.m.sortconfig:t; + }; + +getconfig:{[] + / return the current sort config stored in .z.m.sortconfig; (::) if not yet set + :.z.m.sortconfig; + }; + +/ internal - protected file read; only the i/o so a genuine read failure gets the readerr message +readfile:{[file] + / returns the raw csv lines; header validation and parsing happen in parsecsv + .z.m.log[`info]["dbwrite: reading sort config from ",string file]; + :read0 file; + }; + +/ internal - log and rethrow a csv read failure +readerr:{[file;e] + / build the message once, surface it under the dbwrite context, then rethrow it to the caller + m:"failed to read ",string[file],": ",e; + .z.m.log[`error]["dbwrite: ",m]; + 'm; + }; + +/ internal - validate the header and data rows, then parse csv lines into a config table +parsecsv:{[lines] + / parse field-by-field rather than via 0:, which silently pads/truncates/coerces malformed rows + / map columns by header name so order does not matter; runs outside the readfile i/o trap + if[0=count lines; + '"di.dbwrite: csv has no header row"]; + hdr:`$"," vs first lines; + if[not (asc distinct hdr)~`att`column`sort`tabname; + '"di.dbwrite: csv header must be exactly tabname,att,column,sort; got: ",", " sv string hdr]; + rows:"," vs/: 1 _ lines; + if[count bad:where (count each rows)<>count hdr; + '"di.dbwrite: csv data row(s) ",(", " sv string 1+bad)," must have ",(string count hdr)," fields"]; + c:hdr!$[count rows; flip rows; (count hdr)#enlist ()]; + if[not all (c`sort) in enlist each "01"; + '"di.dbwrite: the sort column must contain only 0 or 1"]; + :([] tabname:`$c`tabname; att:`$c`att; column:`$c`column; sort:"B"$c`sort); + }; + +/ internal - validate a sort-config table, signalling a clear error if it is malformed +checkconfig:{[t] + / guards every sort call so a hand-built or csv-derived table is rejected early if wrong + if[98h<>type t; + '"di.dbwrite: config must be a table with columns `tabname`att`column`sort"]; + c:cols t; + badcols:c where not c in `tabname`att`column`sort; + if[count badcols; + '"di.dbwrite: unrecognised config column(s): ",", " sv string badcols]; + missingcols:(`tabname`att`column`sort) where not (`tabname`att`column`sort) in c; + if[count missingcols; + '"di.dbwrite: missing required config column(s): ",", " sv string missingcols]; + if[any null t`tabname; + '"di.dbwrite: config tabname must not be null"]; + if[any null t`column; + '"di.dbwrite: config column must not be null"]; + if[not 1h=type t`sort; + '"di.dbwrite: the sort column must be boolean"]; + badatts:at where not (at:distinct t`att) in validatts; + if[count badatts; + '"di.dbwrite: unrecognised attribute(s) in att column: ",", " sv string badatts]; + }; + +sort:{[tabname;dirs] + / sort and apply attributes to the on-disk partition dirs for one table using .z.m.sortconfig + / both the sort column order and the att assignments (p/s/g/u) are driven by the config + / falls back to defaultparams if config has not been set via readcsv or setconfig + / tabname: symbol; dirs: hsym or list of hsyms (partition directories e.g. from .Q.par) + config:$[(::)~.z.m.sortconfig; defaultparams; .z.m.sortconfig]; + checkconfig config; + if[not -11h=type tabname; + '"di.dbwrite: tabname must be a symbol, got type ",string type tabname]; + st:string tabname; + .z.m.log[`info]["dbwrite: sorting the ",st," table"]; + sp:getsortparams[config;tabname;st]; + if[not count sp; :()]; + sortdir[sp] each distinct (),dirs; + .z.m.log[`info]["dbwrite: finished sorting the ",st," table"]; + }; + +/ internal - log a sort message then return the resolved rows +logreturn:{[lvl;msg;rows] + / keeps each branch body in getsortparams to a single statement + .z.m.log[lvl]["dbwrite: ",msg]; + :rows; + }; + +/ internal - resolve which config rows apply to a table +getsortparams:{[config;tab;st] + / tab is the table-name symbol (NOT a table); named to avoid clashing with the tabname column + / a table uses its own rows; unlisted tables fall back to the default row, else are skipped + if[count tabsp:select from config where tabname=tab; + :logreturn[`info;"sort params found for: ",st;tabsp]]; + if[count defsp:select from config where tabname=`default; + :logreturn[`info;"no sort params for: ",st,"; using defaults";defsp]]; + :logreturn[`warn;"no sort params for: ",st,"; skipping sort";0#config]; + }; + +/ internal - log a sort failure without rethrowing so remaining partitions still run +sorterr:{[sc;dl;e] + / a single partition failure should not halt the whole run + .z.m.log[`error]["dbwrite: failed to sort ",string[dl]," by ",(", " sv string sc),": ",e]; + :(); + }; + +/ internal - sort one partition directory by the given columns +sortcolumns:{[dloc;sortcols] + / split out of sortdir so the conditional body there stays a single statement + .z.m.log[`info]["dbwrite: sorting ",string[dloc]," by: ",", " sv string sortcols]; + .[xasc;(sortcols;dloc); + sorterr[sortcols;dloc]]; + }; + +/ internal - sort columns and apply attributes for a single on-disk partition directory +sortdir:{[sp;dloc] + / sort by the columns flagged sort=1b, then hand every row to applyattr (it skips non-attributes) + sortcols:exec column from sp where sort, not null column; + if[count sortcols; sortcolumns[dloc;sortcols]]; + applyattr[dloc;;]'[sp`column;sp`att]; + }; + +/ internal - log an attribute application failure without rethrowing +attrerr:{[dl;cn;at;e] + / logs failure and continues so other columns and partitions still get processed + .z.m.log[`error]["dbwrite: unable to apply ",string[at]," attr to ",string[cn]," in ",string[dl],": ",e]; + :(); + }; + +applyattr:{[dloc;colname;att] + / apply a single kdb+ attribute to an on-disk column; logs and swallows errors so a run continues + / dloc: hsym (partition directory e.g. `:hdb/2024.01.01/trade); colname: symbol; att: symbol (p|s|g|u or empty) + / skip anything that is not a real attribute - covers the empty none-sentinel and any bad value + if[not att in `p`s`g`u; :()]; + .z.m.log[`info]["dbwrite: applying ",string[att]," attr to ",string[colname]," in ",string dloc]; + .[{@[x;y;z#]}; + (dloc;colname;att); + attrerr[dloc;colname;att]]; + }; + +savedown:{[dir;part;tabname;data] + / write an in-memory table to a date-partitioned hdb partition, then sort it per .z.m.sortconfig + / dir: hdb root (hsym); part: partition value (date/month/int); tabname: symbol; data: in-memory table + / enumerates syms against the hdb sym file; sorting and attributes are driven by .z.m.sortconfig + .z.m.log[`info]["dbwrite: saving ",string[tabname]," partition ",string[part]," to ",string dir]; + path:` sv (.Q.par[dir;part;tabname];`); + path set .Q.en[dir;data]; + sort[tabname;path]; + .z.m.log[`info]["dbwrite: finished saving ",string tabname]; + gc[]; + }; + +appenddown:{[dir;part;tabname;data] + / append rows to an existing on-disk partition (enumerates syms); does not sort + / call sort separately once the partition is complete, to avoid re-sorting on every append + / dir: hdb root (hsym); part: partition value; tabname: symbol; data: in-memory table + .z.m.log[`info]["dbwrite: appending ",string[tabname]," partition ",string[part]," in ",string dir]; + path:` sv (.Q.par[dir;part;tabname];`); + if[not count @[key;path;{`$()}]; + '"di.dbwrite: appenddown partition does not exist at ",string path]; + .[path;();,;.Q.en[dir;data]]; + .z.m.log[`info]["dbwrite: finished appending ",string tabname]; + }; + +/ internal - render a memory-usage dict as a "key=val MB; ..." string +fmtmem:{[m] + / m: dict of MB values keyed by .Q.w field name + :"; " sv "=" sv' flip (string key m; (string value m),\:" MB"); + }; + +/ format current process memory stats as a loggable string +memstats:{[] + / convert .Q.w[] (bytes) to MB and render it via fmtmem + :"mem stats: ",fmtmem `long$.Q.w[]%1048576; + }; + +gc:{[] + / run .Q.gc[] and log before/after memory stats + .z.m.log[`info]["dbwrite: starting garbage collect. ",memstats[]]; + r:.Q.gc[]; + .z.m.log[`info]["dbwrite: garbage collection returned ",(string `long$r%1048576),"MB. ",memstats[]]; + }; \ No newline at end of file diff --git a/di/dbwrite/init.q b/di/dbwrite/init.q new file mode 100644 index 00000000..6bf98f85 --- /dev/null +++ b/di/dbwrite/init.q @@ -0,0 +1,6 @@ +/ dbwrite module - write, sort, and attribute utilities for on-disk data +/ used by processes that persist data to disk (rdb, wdb, tickerlogreplay) + +\l ::dbwrite.q + +export:([init;readcsv;setconfig;getconfig;sort;applyattr;savedown;appenddown]) diff --git a/di/dbwrite/test.csv b/di/dbwrite/test.csv new file mode 100644 index 00000000..90bc79a7 --- /dev/null +++ b/di/dbwrite/test.csv @@ -0,0 +1,254 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,dbwrite:use`di.dbwrite,1,1,load module +before,0,0,q,mylog:`info`warn`error!({[m]};{[m]};{[m]}),1,1,define no-op monadic mock logger (kx.log-style) +before,0,0,q,"caplog:([] fn:`symbol$();msg:())",1,1,initialise log capture table +before,0,0,q,logcap:`info`warn`error!({[m] `caplog upsert(`info;m)};{[m] `caplog upsert(`warn;m)};{[m] `caplog upsert(`error;m)}),1,1,define capturing monadic mock logger +before,0,0,q,"rmrf:{[d] if[11h=type k:key d; rmrf each ` sv/:d,/:k]; @[hdel;d;{}]}",1,1,recursive delete helper for on-disk cleanup +before,0,0,q,"`:tmp_dbw_valid.csv 0: (""tabname,att,column,sort"";""trade,p,sym,1"";""trade,,time,0"";""default,p,sym,1"")",1,1,write valid sort csv +before,0,0,q,"`:tmp_dbw_badcols.csv 0: (""tabname,attr,col,sort"";""trade,p,sym,1"")",1,1,write csv with bad column names +before,0,0,q,"`:tmp_dbw_badatt.csv 0: (""tabname,att,column,sort"";""trade,z,sym,1"")",1,1,write csv with invalid attribute value +before,0,0,q,"`:tmp_dbw_3col.csv 0: (""tabname,att,sort"";""trade,p,1"")",1,1,write csv missing a column +before,0,0,q,"`:tmp_dbw_empty.csv 0: enlist ""tabname,att,column,sort""",1,1,write header-only sort csv +before,0,0,q,"`:tmp_dbw_nodfl.csv 0: (""tabname,att,column,sort"";""trade,p,sym,1"")",1,1,write sort csv with no default row +before,0,0,q,"`:tmp_dbw_5col.csv 0: (""tabname,att,column,sort,extra"";""trade,p,sym,1,foo"")",1,1,write csv with an extra column +before,0,0,q,"`:tmp_dbw_reorder.csv 0: (""tabname,att,sort,column"";""trade,p,1,sym"";""trade,,0,time"";""default,p,1,sym"")",1,1,write valid csv with columns reordered +before,0,0,q,`:tmp_dbw_nohdr.csv 0: (),1,1,write a zero-line csv (no header) +before,0,0,q,"`:tmp_dbw_short.csv 0: (""tabname,att,column,sort"";""trade,p,sym"")",1,1,write csv with a too-short data row +before,0,0,q,"`:tmp_dbw_long.csv 0: (""tabname,att,column,sort"";""trade,p,sym,1,extra"")",1,1,write csv with a too-long data row +before,0,0,q,"`:tmp_dbw_badbool.csv 0: (""tabname,att,column,sort"";""trade,p,sym,x"")",1,1,write csv with a non-boolean sort value +before,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,init with no-op logger +before,0,0,q,cfg:([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,reusable valid config table (hand-built) +before,0,0,q,nodflcfg:([] tabname:enlist`trade; att:enlist`p; column:enlist`sym; sort:enlist 1b),1,1,reusable config with a trade row but no default + +comment,,,,,,,init - dependency injection validation (type errors / empty / null) +fail,0,0,q,dbwrite.init[(::)],1,1,init rejects :: as deps +fail,0,0,q,dbwrite.init[42],1,1,init rejects a non-dict deps value +fail,0,0,q,dbwrite.init[()!()],1,1,init rejects empty dict +fail,0,0,q,dbwrite.init[enlist[`other]!enlist mylog],1,1,init rejects dict missing log key +fail,0,0,q,dbwrite.init[enlist[`log]!enlist(::)],1,1,init rejects null log dep +fail,0,0,q,dbwrite.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value +fail,0,0,q,dbwrite.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key +true,0,0,q,"""di.dbwrite:""~11#@[{dbwrite.init[(::)]};(::);{x}]",1,1,error message is prefixed di.dbwrite +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,re-init with valid log dep + +comment,,,,,,,readcsv - stores a config from a csv into module state +run,0,0,q,dbwrite.readcsv `:tmp_dbw_valid.csv,1,1,readcsv stores config from a valid csv without error +true,0,0,q,3=count dbwrite.getconfig[],1,1,stored config has 3 rows +true,0,0,q,`tabname`att`column`sort~cols dbwrite.getconfig[],1,1,stored config has the correct column schema +true,0,0,q,(dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,stored config matches csv content +run,0,0,q,dbwrite.readcsv `:tmp_dbw_empty.csv,1,1,readcsv stores an empty config for a header-only csv +true,0,0,q,0=count dbwrite.getconfig[],1,1,empty csv produces empty stored config +run,0,0,q,dbwrite.readcsv `:tmp_dbw_reorder.csv,1,1,readcsv with reordered columns stores correct config +true,0,0,q,(dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,readcsv is column-order independent +true,0,0,q,`tabname`att`column`sort~cols dbwrite.getconfig[],1,1,readcsv normalises reordered columns to canonical order +run,0,0,q,"dbwrite.readcsv ""tmp_dbw_valid.csv""",1,1,readcsv accepts a string file path +true,0,0,q,3=count dbwrite.getconfig[],1,1,string path readcsv stores the correct number of rows +true,0,0,q,(dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,string path readcsv stores the correct config + +comment,,,,,,,readcsv - input and header validation failures +fail,0,0,q,dbwrite.readcsv `:nonexistent_file.csv,1,1,readcsv errors on a missing file +fail,0,0,q,dbwrite.readcsv 42,1,1,readcsv errors on a non-symbol non-string file argument +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_badcols.csv,1,1,readcsv rejects a csv with wrong column names +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_3col.csv,1,1,readcsv rejects a csv missing a required column +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_5col.csv,1,1,readcsv rejects a csv with an extra column +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_nohdr.csv,1,1,readcsv rejects a csv with no header row +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_short.csv,1,1,readcsv rejects a data row with too few fields +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_long.csv,1,1,readcsv rejects a data row with too many fields +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_badbool.csv,1,1,readcsv rejects a non-boolean sort value +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_badatt.csv,1,1,readcsv rejects a csv with invalid attribute values + +comment,,,,,,,setconfig - stores a hand-built table into module state +run,0,0,q,dbwrite.setconfig[cfg],1,1,setconfig accepts a valid hand-built table +true,0,0,q,(dbwrite.getconfig[])~cfg,1,1,getconfig returns the table stored by setconfig +run,0,0,q,dbwrite.setconfig[0#cfg],1,1,setconfig accepts an empty (but correctly typed) table +true,0,0,q,0=count dbwrite.getconfig[],1,1,empty table stored correctly + +comment,,,,,,,setconfig - config validation (type errors / bad content) +fail,0,0,q,dbwrite.setconfig[42],1,1,setconfig rejects a non-table value +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; bad:enlist`p; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects unrecognised config columns +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`sym)],1,1,setconfig rejects a missing required config column +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`z; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects unknown attribute values +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`sym; sort:enlist 1)],1,1,setconfig rejects a non-boolean sort column +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`; att:enlist`p; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects a null tabname in config +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`; sort:enlist 1b)],1,1,setconfig rejects a null column in config + +comment,,,,,,,sort - happy path using stored config (set via setconfig or readcsv) +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to hand-built cfg +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort with a hand-built stored config and no partitions +run,0,0,q,dbwrite.readcsv `:tmp_dbw_valid.csv,1,1,set config via readcsv +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort with a csv-loaded stored config and no partitions + +comment,,,,,,,sort - tabname type validation +fail,0,0,q,dbwrite.sort[42;enlist`:/],1,1,sort errors on a non-symbol table name +true,0,0,q,"""di.dbwrite:""~11#@[{dbwrite.sort[42;enlist`:/]};(::);{x}]",1,1,non-symbol table name gives a di.dbwrite:-prefixed error + +comment,,,,,,,sort - edge cases (empty config / null att / empty dirs / atom dir) +run,0,0,q,dbwrite.setconfig[0#cfg],1,1,set empty config +true,0,0,q,()~dbwrite.sort[`trade;enlist`:/],1,1,empty config yields no work and returns () +run,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`; column:enlist`sym; sort:enlist 0b)],1,1,set config with a null (no) attribute row +run,0,0,q,dbwrite.sort[`trade;enlist`:/],1,1,sort accepts a null (no) attribute row +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort handles an empty partition list without error +run,0,0,q,dbwrite.sort[`trade;`:/],1,1,sort accepts a single atom hsym dir not just a list + +comment,,,,,,,sort - params resolution (table-specific / default / none / default fallback) +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;enlist`:/],1,1,sort uses the table-specific params row +run,0,0,q,dbwrite.sort[`other;enlist`:/],1,1,sort falls back to the default params row +run,0,0,q,dbwrite.setconfig[nodflcfg],1,1,set config to nodflcfg (no default row) +true,0,0,q,()~dbwrite.sort[`other;enlist`:/],1,1,sort returns () when no params found and no default row +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset module state (clears sortconfig to (::)) +run,0,0,q,dbwrite.sort[`anytable;()],1,1,sort with unset sortconfig falls back to the built-in default config + +comment,,,,,,,sort - on-disk end-to-end (explicit config sorts by sym and applies p#) +run,0,0,q,`:dbw_sort_tp/.d set `sym`price,1,1,write column order file +run,0,0,q,`:dbw_sort_tp/sym set `IBM`AAPL`MSFT,1,1,write unsorted sym column +run,0,0,q,`:dbw_sort_tp/price set 200 100 300f,1,1,write price column +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg before sort +run,0,0,q,dbwrite.sort[`trade;`:dbw_sort_tp/],1,1,sort the trade partition by its config +true,0,0,q,`AAPL`IBM`MSFT~exec sym from get `:dbw_sort_tp/,1,1,sym sorted ascending on disk +true,0,0,q,`p=attr get `:dbw_sort_tp/sym,1,1,p attribute applied to sym on disk + +comment,,,,,,,sort - on-disk end-to-end (default config sorts by time) +run,0,0,q,`:dbw_def_tp/.d set `time`sym,1,1,write column order file +run,0,0,q,`:dbw_def_tp/time set 2024.01.01D09:00 2024.01.01D08:00,1,1,write unsorted time column +run,0,0,q,`:dbw_def_tp/sym set `IBM`AAPL,1,1,write sym column +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset sortconfig to (::) for default-fallback test +run,0,0,q,dbwrite.sort[`anytable;`:dbw_def_tp/],1,1,sort using the built-in default config +true,0,0,q,(asc exec time from get `:dbw_def_tp/)~exec time from get `:dbw_def_tp/,1,1,time sorted ascending by the default config + +comment,,,,,,,sort - multiple partition dirs in one call and a non-fatal partition failure +run,0,0,q,`:dbw_md1/.d set `sym`px,1,1,write md1 column order +run,0,0,q,`:dbw_md1/sym set `c`a`b,1,1,write md1 unsorted sym +run,0,0,q,`:dbw_md1/px set 1 2 3f,1,1,write md1 price +run,0,0,q,`:dbw_md2/.d set `sym`px,1,1,write md2 column order +run,0,0,q,`:dbw_md2/sym set `b`c`a,1,1,write md2 unsorted sym +run,0,0,q,`:dbw_md2/px set 1 2 3f,1,1,write md2 price +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;(`:dbw_md1/;`:dbw_md2/)],1,1,sort two partition dirs in one call +true,0,0,q,`a`b`c~exec sym from get `:dbw_md1/,1,1,first partition sorted +true,0,0,q,`a`b`c~exec sym from get `:dbw_md2/,1,1,second partition sorted +run,0,0,q,`:dbw_good/.d set `sym`px,1,1,write good-partition column order +run,0,0,q,`:dbw_good/sym set `c`a`b,1,1,write good-partition unsorted sym +run,0,0,q,`:dbw_good/px set 1 2 3f,1,1,write good-partition price +run,0,0,q,dbwrite.sort[`trade;(`:dbw_good/;`:/)],1,1,sort a good dir alongside a bad one +true,0,0,q,`a`b`c~exec sym from get `:dbw_good/,1,1,good partition still sorted despite the bad one failing + +comment,,,,,,,applyattr - applies an attribute on disk and is a no-op for non-attributes +run,0,0,q,`:dbw_attr_tp/.d set `sym`price,1,1,write column order file +run,0,0,q,`:dbw_attr_tp/sym set `IBM`AAPL`MSFT,1,1,write sym column +run,0,0,q,`:dbw_attr_tp/price set 200 100 300f,1,1,write price column +run,0,0,q,dbwrite.applyattr[`:dbw_attr_tp/;`sym;`p],1,1,apply p attr to sym +true,0,0,q,`p=attr get `:dbw_attr_tp/sym,1,1,p attribute applied to sym on disk +run,0,0,q,dbwrite.applyattr[`:dbw_attr_tp/;`price;`z],1,1,applyattr is a silent no-op for an invalid attribute +true,0,0,q,`=attr get `:dbw_attr_tp/price,1,1,invalid attribute leaves the column unattributed + +comment,,,,,,,savedown - write an in-memory table to an hdb partition then sort it +run,0,0,q,"sdtbl:([]time:2024.01.01D09:00 2024.01.01D08:00;sym:`IBM`AAPL;price:100 200f)",1,1,unsorted test table +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset sortconfig to (::) for default-config savedown +run,0,0,q,dbwrite.savedown[`:dbw_hdb;2024.01.01;`trade;sdtbl],1,1,savedown with the default config (sort by time) +run,0,0,q,"sdpath:` sv (.Q.par[`:dbw_hdb;2024.01.01;`trade];`)",1,1,resolve the partition path +true,0,0,q,2=count get sdpath,1,1,two rows written to the partition +true,0,0,q,(asc exec time from get sdpath)~exec time from get sdpath,1,1,partition sorted by time after savedown +true,0,0,q,0