feat: materialized lots table for holdings tracking#792
feat: materialized lots table for holdings tracking#792triantos wants to merge 22 commits intoafadil:mainfrom
Conversation
triantos
force-pushed
the
data-model-refactor
branch
from
683d35c to
0e380d6
Compare
Introduces the `lots` table — the persistent, relational form of tax lots. Each row represents one acquisition with its own cost basis, open date, and disposal tracking. Initially empty; the holdings calculator will begin shadow-writing lot rows alongside existing JSON snapshots once the lot repository is wired in. Changes: - Migration: CREATE TABLE lots with indexes (account_id/asset_id, asset/open, account/open, open_activity_id) - schema.rs: add lots table definition and joinables to accounts/assets - crates/core/src/lots/: new LotRecord, DisposalMethod, HoldingPeriod types No behavioral changes. The existing JSON snapshot path is untouched.
After each holdings recalculation the snapshot service writes a row to the `lots` table for every open lot in every position, in parallel with the existing JSON snapshot path. The JSON snapshots remain authoritative; the lot rows exist so both representations can be compared. Any quantity mismatch between the two is logged at ERROR severity. The lot repository is opt-in via SnapshotService::with_lot_repository(), so existing callers are unaffected until they wire it in. open_activity_id is left NULL for now — transferred sub-lots use composite IDs that don't correspond to rows in the activities table.
Both apps (Tauri desktop and server) now pass a LotsRepository to SnapshotService, so lot rows are written on every holdings recalculation. For existing users, the lots table starts empty. On desktop startup, backfill_lots_if_needed() checks the row count and triggers a full holdings recalculation if the table is empty, ensuring lot rows are populated without any manual action. Adds count_open_lots() to LotRepositoryTrait to support the check.
…rite Unit tests (core::lots): - extract_lot_records: AAPL (3 lots), LQD bond ETF (2 lots), AAPL Jun 2026 \$200 call option (1 lot), mixed portfolio totalling all three - check_lot_quantity_consistency: passing case and mismatch detection Integration tests (storage-sqlite::lots): - replace_lots_for_account: inserts, replaces, and clears correctly - replace_only_affects_target_account: other accounts' rows are untouched Snapshot service test: - MockLotRepository records calls; verifies that after a full recalculation with AAPL (3 buys → 100 shares), LQD (2 buys → 150 shares), and one options position, the lot repository receives the correct quantities
…lots table Security positions now sourced from lots JOIN assets instead of snapshot JSON. contract_multiplier from asset.contract_multiplier(), currency from asset.quote_ccy. Cash balances still come from snapshot. Verified on 18-account test DB: 35 securities, quantities, cost bases, and market values all match pre-switch baseline exactly.
Replace replace_lots_for_account (wipe+reinsert) with sync_lots_for_account (upsert open, mark closed) so lot rows are never deleted. Fully consumed lots are stamped with is_closed=1, close_date, and close_activity_id sourced from the disposing activity's date and ID. - FifoReductionResult gains fully_consumed_lot_ids to surface which lots drained to zero during a reduction - HoldingsCalculator records LotClosure entries for all three reduction sites: handle_sell, handle_transfer_out, handle_adjustment (OPTION_EXPIRY) - snapshot_service clears the disposed_lots cache before each run and passes closures to sync_lots_for_account alongside the open lots - storage-sqlite implements sync_lots_for_account: row-by-row upsert (SQLite limitation) + UPDATE for each closure - New test: sync_lots_records_closures_for_full_sell verifies a fully sold lot appears in closures with the correct date/activity, not in open lots
Adds two new LotRepositoryTrait methods: - get_lots_as_of_date(account_ids, date): lots active on a given date - get_all_lots_for_account(account_id): all lots (open + closed) for memory-side date filtering Switches NetWorthService.get_net_worth to read security positions from the lots table instead of snapshot.positions. Cash balances still come from snapshots (extra state, will be removed once cash is tracked independently of snapshots).
… lots ValuationService.calculate_valuation_history now fetches all lots for the account once via get_all_lots_for_account and filters per date in memory, replacing iteration over snapshot.positions for securities. Currency, is_alternative, and contract_multiplier are still read from the snapshot's position entries (extra state carried until ValuationService gains direct asset-repo access). Cash positions are unaffected.
Changes the holdings_from_snapshot signature from (&snapshot, base_currency) to (account_id, date, base_currency). Security positions now come from get_lots_as_of_date; cash balances still fetched from the snapshot for that date (extra state carried until cash is tracked independently). Also includes individual lot detail in the returned holdings (display_lots), which was absent in the snapshot-based version.
Mirrors the Tauri backfill_lots_if_needed logic: on startup, count open lots and if zero, run a full holdings recalculation to populate the table. Runs non-blocking in a tokio::spawn so it doesn't delay server readiness. Adds lots_repository to AppState so the scheduler can check the count.
Two callsites called per-account lot queries with account_id="TOTAL",
which has no rows in the lots table, causing:
- build_live_holdings_from_snapshot: returned 0 securities, leaving
only the aggregate cash balance (-$14.8M) as live holdings
- calculate_valuation_history: investment_market_value=0, making the
TOTAL daily valuation equal to the (negative) cash balance
Add get_all_open_lots() and get_all_lots() to LotRepositoryTrait and
use them in holdings_service and valuation_service respectively when
account_id == "TOTAL", so both paths aggregate lots across all accounts.
When save_manual_snapshot is called for the latest snapshot on an account, synthesize one LotRecord per position (using snapshot date, quantity, average cost, and total cost basis from the position) and call replace_lots_for_account. This makes HOLDINGS-mode accounts visible in lot-backed holdings and valuation queries.
triantos
force-pushed
the
data-model-refactor
branch
from
0e380d6 to
3d15027
Compare
Blocking issues
Additional issues
|
Add `original_quantity` field to the snapshot Lot struct, set at creation time in add_lot, add_lot_values, and add_transferred_lots. Never modified by reduce_lots_fifo or apply_split, so it preserves the as-acquired quantity through all subsequent mutations. extract_lot_records now writes original_quantity (the acquisition amount) and remaining_quantity (the current amount) as distinct values to the lots table. Old snapshots that predate this field deserialize original_quantity as zero via serde(default); extract_lot_records falls back to remaining_quantity in that case. This is the prerequisite for historical as-of queries that replay activities forward from each lot's original_quantity anchor.
The startup lot backfill calls recalculate_holdings_snapshots(None, Full) which only processes TRANSACTIONS-mode accounts. HOLDINGS-mode accounts were permanently skipped because the lots table became non-empty after the first backfill, suppressing future runs. Add backfill_lots_for_holdings_accounts() to SnapshotServiceTrait. It iterates HOLDINGS-mode accounts and synthesizes lots from each account's latest snapshot via the existing refresh_lots_from_latest_snapshot method. Both server and Tauri schedulers now call this after the activity-replay backfill, ensuring all account types have lot rows on first launch.
… delete sync_lots_for_account now deletes lots for the account that weren't produced by the current recalculation. Orphans arise when activities are deleted (FK SET NULL) and rebuilds create new lots with new IDs. Also adds migration to change open_activity_id FK from ON DELETE SET NULL to ON DELETE CASCADE, so activity deletion immediately removes lots. The migration cleans up existing orphans during the table rebuild.
When a sell activity is deleted and lots are recalculated, the lot is re-emitted as open. The ON CONFLICT DO UPDATE clause was only updating remaining_quantity and total_cost_basis, leaving is_closed=1 and close_date set from the prior state. The lot remained invisible to open-lot queries. Add is_closed, close_date, and close_activity_id to the upsert SET clause so that reopened lots are correctly marked as open.
The lots table stores current state (remaining_quantity is mutated in place by sells). Historical as-of queries were returning current quantities for past dates — buy 10, sell 4, query before the sell returned 6 instead of 10. Add replay_lots_to_date() in lots/mod.rs: resets each lot to original_quantity, then replays Sell/TransferOut/Adjustment/Split activities in chronological FIFO order up to the requested date. 7 unit tests cover partial sell, full sell, FIFO across lots, splits, split+sell, and cross-account isolation. Inject activity_repository into HoldingsService and NetWorthService. Both get_lots_as_of_date call sites now fetch activities and replay. Also fixes the SQL bug in the storage method: .assume_not_null() on nullable close_date was dropping open lots (NULL > 'x' is NULL in SQL). Replaced with is_not_null().and(close_date.gt(date)).
The is_latest check in save_manual_snapshot used get_latest_snapshot_before_date(account_id, snapshot_date + 1) which returned the just-saved snapshot itself, making the condition always true. Historical HOLDINGS snapshots would overwrite lots from the latest snapshot. Replace with a check for any snapshots after the saved date. If none exist, this is the latest and lots should be written.
…ntly coercing to zero Multiple lot readers used unwrap_or_default() or filter_map with .ok() to parse Decimal fields from TEXT columns. Malformed values silently produced zero quantities or dropped lots from display with no signal. Replace with unwrap_or_else + log::error! at all lot parse sites in holdings_service, net_worth_service, and valuation_service. The filter_map in lot_records_to_display_lots now uses inspect_err to log before dropping.
The method counts all rows in the lots table (open and closed), not just open lots. The name was misleading after is_closed was introduced. Callers only use it to check "is the table empty" for the startup backfill guard.
The handler was rewritten to read security positions from the lots table, which silently returned an empty Vec when no data exists at the requested date. Restore the original behavior of erroring on missing data, returning 404 Not Found via ApiError::NotFound.
|
I've addressed all of the issues you raised. Note there were some corner case bugs in previous handling of lots which are now fixed. |
2 participants
Summary
lotstable that tracks individual tax lots (acquisitions) as first-class database rows, replacing the JSON-serialized snapshot approach for position trackingMotivation
The existing holdings system stores positions as serialized JSON inside snapshot blobs. This makes it impossible to query individual lots for tax reporting, forces full-history replay on every quote update, and prevents incremental position updates. The lots table solves all three: lots are queryable rows, valuations become
lots JOIN quotes, and activity changes update only affected lots.Changes
lotstable with migration,LotRecordRust model,LotRepositoryTraitget_net_worth,calculate_valuation_history,holdings_from_snapshot, andbuild_live_holdingsall read from the lots table instead of deserializing JSON snapshotsTest plan
cargo fmt --all -- --checkpassescargo clippy --workspace -- -D warningspasses (core, server, ai, storage)cargo test --workspace— 1,316 tests pass, 0 failurespnpm format:checkpasses