Feature dbwrite by Ruairi-wq2 · Pull Request #95 · DataIntellectTech/kdbx-modules

Ruairi-wq2 · 2026-06-03T09:47:13Z

DBwrite module code, including test.csv suite. Init.q and dbwrite.q code to define logic and export it. dbwrite.md as a README.

Ruairi-wq2 · 2026-06-03T09:48:21Z

q)k4unit:use`di.k4unit
q)k4unit.moduletest[`di.dbwrite]
2026.06.03T09:37:00.789 start
2026.06.03T09:37:00.789 :/home/ruairi_conlon/TorQ/di/kdbx-modules/di/dbwrite/test.csv 60 test(s)
2026.06.03T09:37:00.803 end
Test results:
....
true   0  0     q    `AAPL`IBM~exec sym from get `:dbwrite_def_tp/                                          1      :/home/ruairi_conlon/TorQ/di/kdbx-modules/di/dbwrite/test.csv 0   0       1  1    1       1     2026.06.03T09:37:00.803
true   0  0     q    4=logcount                                                                             1      :/home/ruairi_conlon/TorQ/di/kdbx-modules/di/dbwrite/test.csv 0   0       1  1    1       1     2026.06.03T09:37:00.803
All tests passed

…ull guard - Export savedownmanipulation so consumers can register per-table pre-write functions - loadconfig warns and loads defaultparams when called with a null file rather than silently failing - sort falls back to defaultparams directly; removes defaultfile indirection from init.q - Tests: add savedownmanipulation registered function and error recovery cases, loadconfig null case, move all on-disk cleanup to after blocks - Docs updated throughout to match Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…nput crash - loadconfig: explicit symbol type check with clear error before null check - applyattr: guard against invalid att (must be in `p`s`g`u), null colname guard - sort: type check for d (must be symbol or list); empty list returns () safely - test.csv: 78 tests covering wrong types, nulls, empty inputs across all functions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ng test setup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

alowrydi · 2026-06-03T15:53:10Z

+  .z.m.loginfo[`dbwrite;"garbage collection returned ",(string `long$r%1048576),"MB. ",.z.M.memstats[]]
+  };
+
+init:{[config;deps]


config argument not actually called within this function?

jonnypress · 2026-06-18T16:53:09Z

You need to think about this a bit more and apply some judgement. When we are stripping code out of TorQ we are trying to créate a set of standalone, isolated modules. What is the purpose of this module? It is to provide a set of utilities for writing tables, including sorting them on disk. Postreplay and savedownmanipulation likely don’t have relevance in here (the are artifacts of other parts of TorQ). Can you also coordinate with Alex because there’s overlap with the sort lib, I think the sort lib should live in here.

- Add savedown[dir;part;tabname;data]: enumerates syms, applies p#sym, writes partition via .Q.par + set, then calls sort - Add upsert[dir;part;tabname;data]: appends to existing partition via functional amend, then re-sorts - Remove manipulate, savedownmanipulation, postreplay (TorQ-influenced) - Update export dict, init comment, tests, and docs accordingly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Rename upsert to appenddown (upsert is a reserved word in kdb+) - Remove auto-sort from appenddown; caller sorts explicitly when done - Fix savedown/appenddown to use trailing-slash path via ` sv (.Q.par[...];`) so xasc treats it as a splayed table directory (without slash, set writes a binary file and xasc fails with "Not a directory") - Add existence guard to appenddown: throws if partition path does not exist - Update init.q export dict, tests, and docs accordingly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… p attribute before sorting resulting in a ufail

- readcsv now stores parsed config in .z.m.sortconfig instead of returning it - add setconfig to allow consumers to provide an in-memory config table directly - sort and savedown drop their config parameter; both read .z.m.sortconfig, falling back to defaultparams - gc is called at the end of savedown; removed from exports as it is not dbwrite-specific - init initialises .z.m.sortconfig to (::) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- add getconfig export to allow consumers to inspect .z.m.sortconfig - readcsv now calls checkconfig after parsecsv so invalid att values are caught at load time - tests updated throughout: readcsv/sort/savedown call sites use new signatures; config validation tests moved to setconfig section; gc section removed; setconfig happy-path and validation tests added (159 tests, all passing) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- remove stateless config-per-call design description; module now stores config in .z.m.sortconfig - document readcsv as a store operation (not a return); add setconfig and getconfig sections - update sort and savedown signatures (config arg removed from both) - note gc is called automatically by savedown; remove gc from exported symbols list - update test suite description to match new setconfig/getconfig coverage Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…conflicts

…plyattr docs - readcsv now accepts string paths (10h) by converting via hsym `$, avoiding the manual hsym `$"..." dance; bare symbols still coerced as before - fix malformed test csv: string path test line had unescaped double quotes causing '/ parse error when k4unit loaded the suite - add three tests covering string path happy path - add type comment to applyattr (dloc/colname/att were undocumented) - clarify sort comment: explicit that att assignments from config are applied Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Ruairi-wq2 · 2026-06-24T09:45:38Z

As it stands, the sort function operates on one table & partition combo. A list of partitions can be passed.

/ end-of-day, one partition
dbwrite.sort[trade; .Q.par[:hdb; .z.d; `trade]]

Considering adding the following use case functions. Not sure if they would get used/be useful.

sortpart[dir;part] — sort all tables in one partition (e.g., today's date), no need to list tables or build paths manually
sortdb[dir] — sort everything (useful for rebuild, potentially slow)
sortdb[dir;partitions] — sort a specific set of partitions across all tables

Also probably a style choice, do we want the modules to be lightweight and functional or bulkier with a bit more usecase coverage??

Ruairi-wq2 added 2 commits June 2, 2026 16:17

feature/dbwritemodule-Initial-commit

d1b7a7e

Update dbwrite.md and dbwrite.q

dac127a

Ruairi-wq2 and others added 3 commits June 3, 2026 14:09

Update dbwrite.md: fix stale defaultparams references, add mock-loggi…

c2ac971

…ng test setup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

alowrydi reviewed Jun 3, 2026

View reviewed changes

jonnypress reviewed Jun 18, 2026

View reviewed changes

Comment thread di/dbwrite/dbwrite.q Outdated

Comment thread di/dbwrite/dbwrite.q Outdated

Ruairi-wq2 and others added 11 commits June 19, 2026 11:51

merging sort modules functionality into dbwrite

c2a2f00

updating to contain changes made by Ruairi and general bug fixes

f8cfc19

expanding on tests to remove stubbed tests, fixed a bug with applying…

c631d34

… p attribute before sorting resulting in a ufail

Changing the logging from di logging to kx logging

fc019ed

Changing the logging from di logging to kx logging + resolving merge …

abca559

…conflicts

Ruairi-wq2 changed the title ~~Feature/dbwritemodule~~ Feature dbwrite Jul 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature dbwrite#95

Feature dbwrite#95
Ruairi-wq2 wants to merge 16 commits into
mainfrom
feature/dbwritemodule

Ruairi-wq2 commented Jun 3, 2026

Uh oh!

Ruairi-wq2 commented Jun 3, 2026

Uh oh!

alowrydi Jun 3, 2026

Uh oh!

jonnypress commented Jun 18, 2026

Uh oh!

Uh oh!

Uh oh!

Ruairi-wq2 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Ruairi-wq2 commented Jun 3, 2026

Uh oh!

Ruairi-wq2 commented Jun 3, 2026

Uh oh!

alowrydi Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

jonnypress commented Jun 18, 2026

Uh oh!

Uh oh!

Uh oh!

Ruairi-wq2 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants