diff --git a/CHANGELOG.md b/CHANGELOG.md index ffb8e930..3e1c2496 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,104 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Removed +## [0.14.0] - 2026-07-12 + +### Added + +- **The canary runs against real venues (closes the `canary-roundtrip` + epic).** Venue identity oracle: our store-recorded fills == the + venue-reported fills (exact triples), per-asset balance deltas == the + fills math (fee-denomination-aware — never an assumed zero), position + flat, cost bounded. `--mode testnet` wired (Binance sandbox; Kraken + refused — no testnet); `--mode live` implemented behind the existing + `live_enabled` gate + typed confirmation. `doc/dev/09-go-live.md` names + the live canary as the road-to-1.0 #1 validation vehicle. **Proven on + real Binance testnet: 16/16 checks PASS** (real cancel accepted, venue + dedup on the duplicate client-order-id, cost 0.00511 USDT). (#223) +- **`Fill.fee_asset`** — the venue may charge the fee in a non-quote asset + (Binance charges a market buy's commission in BASE; found by the canary's + identity oracle refusing an unexplained −8E-8 BTC). Additive domain field + (`None` = quote, the historic meaning), threaded through the Binance + adapter, the store (migration) and the oracle's fills math. Also: + Binance HTTP-400 `{code,msg}` rejections now map to the same domain + errors as in-band bodies (they escaped as transport errors); + `BrokerConfig.symbols` threads the per-symbol trade-history scope Binance + requires for `fills()`. (#223) +- **`trading-bot canary`** — the platform self-test as a one-liner (paper by + default): self-contained factory engine funded with `--budget`, quantity + sized to the venue's REAL minimums (public-endpoint resolver + real last + price as the mark), refusal before any order when the exact implied cost + exceeds `--max-cost`, evidence table (PASS/FAIL, expected vs observed) + and exit code. `--mode testnet|live` reserved for the venue leaf. (#222) +- **The canary scenario** (`application/canary.py`) — a deterministic + platform self-test: cancel probe (far-off resting limit → real cancel, + persisted), client-order-id idempotency probe, then a sequential market + round-trip, every step recorded as expected-vs-observed evidence. The + **exact paper oracle** asserts realised PnL == −Σ fees, flat position, + store-persisted terminal orders and balance deltas, Decimal-exact. + Supporting seams: `paper_starting_balances` config (factory-funded + simulators) and a strict-paper marketability rule (a passive-side limit + with an injected mark now RESTS instead of filling at its price — + permissive/markless behaviour unchanged). (#221) +- `/api/orders` rows carry `reject_reason` — the UI's expanded order detail + rendered it defensively since #216; the API now serializes it (additive; + contract sweep updated). (#219) +- **Timezone label + honest empty states (closes the `dashboard-tables-ux` + epic).** Every page footer says which timezone its times render in; the + Orders page's filtered-to-zero tables say "No orders/fills match the + filters." instead of masquerading as an empty book. (#218) +- **Honest bar timing.** "last bar X ago → next in Y" chip on the strategies + roster (replacing the "Next bar" column) and the strategy-detail header — + both derived from `last_asof_ts` + span (server truth), never a fake + countdown (`—` when nothing was evaluated). The epoch-aligned + `nextBarCloseMs` guess is retired. (#217) +- **Orders tables show what executed; fills live under their order.** All + three surfaces gain Filled % (exact fraction on hover), Avg fill and + Value (filled × avg fill, else qty × limit — the tooltip says which); + clicking an order (detail + Orders pages) expands its fills, ids, + limit/stop, Σ fees and reject reason. The strategy detail's standalone + "Recent fills" table is **removed** (its data lives in the expansions); + the Orders page's audit fills view gains Order and Value columns. (#216) +- **Positions tables read asset-first, with a value.** Overview + strategy + detail: Asset | Qty | Avg entry | Price *(as-of)* | Value | Unrealised | + Realised (net) — values prefer the configured display currency, marks + always show their freshness (relative as-of, absolute + source in the + tooltip). Click a row for the native pair, cumulative fees, the + gross-vs-fees realised breakdown and the mark provenance: the shared + **expandable-row helper** lands in `base.html` (keyboard-accessible, + expanded state survives the SSE/poll rebuilds). (#215) +- **`GET /api/balances` + `last_asof_ts` documentation + the epic contract + sweep (closes the `api-completeness` epic).** Per running unit, the + broker's balances as exact Decimal strings (stopped units absent; a broker + error degrades to an `error` row, HTTP 200 — poll-safe); the seam for the + future positions↔balances cross-check and the canary live oracle. One + consolidated additive-only contract test now pins the exact field sets of + all six frozen endpoints before the road-to-1.0 API freeze. (#214) +- **Display currency** — `display_currency` (global default), per-exchange + `display_currency_overrides` and static `conversion_rates` in `AppConfig`; + server-side converted `*_display` money fields (pure + `application/display_ccy.py`) on position rows + (`value_display`/`unrealised_display`), strategy rows + (`total_value_display`/`unrealised_display`) and KPI rows + (`realised_pnl_display`/`fees_paid_display`), each tagged with the resolved + `display_currency`. Missing rate → `null`, never a guessed conversion; + native-quote fields untouched. (#212) +- **Position rows carry mark, value, unrealised and fee currency.** + `/api/positions` rows gain `mark` / `mark_asof_ts` / `mark_source` + (`bar_close` from the mark cache, `last_fill` fallback, never untagged), + `value` (mark × |qty|), `unrealised` (sign-correct for shorts) and + `fee_ccy` — and the strategy aggregate now uses the SAME mark policy, so + the roster total equals the row sum (verified Decimal-exact on both live + books' copies against real dccd frames). All additive; existing fields + contract-regression-tested. (#211) +- **Per-engine mark cache** (`application/mark_cache.py`) — the portfolio + runner publishes each rebalance's last dccd bar closes per symbol + (`Mark(price, asof_ms, source="bar_close")`, exact Decimal) into + `Engine.mark_cache`, so the API layer can serve marks with their as-of + timestamp without any I/O. Single-instrument `StrategyRunner` is a + documented follow-up seam. Verified against the real dccd store: cache == + independently-read frame closes, exact price and asof. (#209) + ## [0.13.0] - 2026-07-11 ### Added diff --git a/doc/dev/03-decisions.md b/doc/dev/03-decisions.md index c47df4ab..2830c082 100644 --- a/doc/dev/03-decisions.md +++ b/doc/dev/03-decisions.md @@ -6,6 +6,88 @@ rejected approaches as tombstones. --- +### 2026-07-12 The venue oracle is an accounting identity; fees carry their asset (PR #223) [accepted] +- **Choice**: on a real venue the canary asserts the **identity** — our + fills == venue-reported fills, per-asset balance deltas == the fills math + — never a precomputed PnL. Supporting discoveries hardened the domain: + `Fill.fee_asset` (Binance charges market-buy commissions in BASE; `None` + keeps the historic quote meaning), Binance HTTP-400 rejections mapped to + domain errors, `BrokerConfig.symbols` for Binance's per-symbol + trade-history scope, and a 15 % venue probe offset (the venue's + `PERCENT_PRICE_BY_SIDE` band rejects 50 %). A settle loop attributes the + venue's async executions to our client-order-ids and re-emits them on the + bus — venue truth flows through the ordinary fill plumbing. +- **Why**: spread and drift make absolute expectations flaky exactly where + correctness matters most; the identity is exact on any venue. The two + intermediate testnet failures WERE the canary working — each became a + domain fix with offline tests before the final green run. PnL folding + still values fees at face value when fee_asset ≠ quote (documented + approximation); the balance identity itself is exact. +- **Rejected alternatives**: absolute-PnL venue oracle (nondeterministic); + ignoring sub-1e-7 balance dust (it was the fee-denomination bug, not + dust); adapter-specific oracle code (the broker port suffices — the + oracle stays venue-agnostic). + +### 2026-07-12 The canary: two oracles, probes before money, resting limits in strict paper (PR #221) [accepted] +- **Choice**: the canary runs probes FIRST (cancel, idempotency — both free) + and only then risks the round-trip; paper mode gets an EXACT oracle + (PnL == −Σ fees, flat, store-persisted terminals, balance deltas), venue + modes will get the identity oracle (leaf 03) — never a precomputed + absolute PnL. Sequential legs, never simultaneous (self-trade + prevention). To make the cancel probe implementable, strict paper gains a + marketability rule: a passive-side limit with an injected mark RESTS until + cancelled; permissive/markless behaviour is byte-identical (regression + tests pin it). +- **Why**: probes cost nothing and validate the two scariest paths (real + cancel = the kill-switch; venue-side client-order-id dedup = the + idempotency invariant) before any money moves. Paper is deterministic so + exactness is free; venues are not, so correctness must be an accounting + identity, not a price prediction. The simulator's fill-everything-at-limit + model made "a resting order" inexpressible — a simulator that cannot rest + an order cannot rehearse a cancel. +- **Rejected alternatives**: simultaneous buy+sell (STP nondeterminism); + skipping probes in paper (the paper canary is the per-release regression — + it must cover the same steps the live one will); relaxing the + marketability rule beyond strict+mark (would silently change every + existing permissive test and the daemon's mark-less maker legs). + +### 2026-07-11 One display numeraire, converted server-side, never guessed (PR #212) [accepted] +- **Choice**: `convert()` always targets the single global + `display_currency` (exact Decimal, identity when the source quote already + matches, `conversion_rates` lookup otherwise, `None` on a missing rate — + never guessed). The per-exchange override is a display **label** on the + row (`resolve_currency`), numerically truthful only when paired with an + identity-equivalent rate — exactly the stablecoin case it exists for + (binance→USDT with `USDT: 1`). +- **Why**: server-side conversion gives every consumer the same figures (no + per-client drift); a single numeraire keeps cross-exchange aggregation + meaningful; refusing to guess a rate keeps a missing `EUR` declaration + visibly `null` instead of silently wrong — money display errors are worse + than money display gaps. +- **Rejected alternatives**: client-side conversion from an exposed rates + config (N consumers, N roundings, N bugs); per-exchange numeraires + (aggregates across exchanges become meaningless); defaulting unknown + rates to 1 (silently wrong for anything non-stable). + +### 2026-07-11 Marks are bar closes with a mandatory as-of, one policy at every altitude (PR #211) [accepted] +- **Choice**: v1 mark = the last dccd bar close the runner saw (published to + a per-engine cache at rebalance time), serialized ONLY together with + `mark_asof_ts` and `mark_source` (`bar_close` | `last_fill` fallback). + Both the per-position rows and the strategy aggregate (`_unrealised_of`) + read the same `_mark_of` helper — one mark policy at every altitude. The + API path does zero I/O; a live ticker is post-1.0. +- **Why**: the bar close is the price the strategy actually evaluated on — + fresher and more honest than the previous last-own-fill mark for + daily-rebalance books (which only moved when the unit itself traded). A + mark without its timestamp invites mistaking a day-old close for a live + price — the UI can only render "as of", never bare. Splitting policies + between rows and aggregate would let the roster disagree with its own + detail rows. +- **Rejected alternatives**: last-own-fill as primary (stale for days on + low-churn books); fetching a ticker on the API path (network I/O per + request, and a freshness the engine's own decisions don't have); marks + without `mark_source` (the fallback would masquerade as fresh data). + ### 2026-07-11 Venue minimums shape quantities upstream; the Order keeps the bare Instrument (PR #204) [accepted] - **Choice**: the round-up-or-skip policy (`order_prep.prepare_leg`) runs at leg preparation with the resolved venue spec — but the routed `Order` diff --git a/doc/dev/06-status.md b/doc/dev/06-status.md index 13e9db54..984f2e30 100644 --- a/doc/dev/06-status.md +++ b/doc/dev/06-status.md @@ -146,6 +146,35 @@ and `CHANGELOG.md` for what shipped. simulator rejects like the venue. Verified on the live books' copies with real specs: the audit's whole dust rebalance (14 legs) skips; Kraken round-ups land exactly on `ordermin`; two-pass convergence proven. +- **`api-completeness` (2026-07-12, PRs #209–#214)**: the API says what the + engine knows — per-engine mark cache (bar closes published at rebalance), + position rows with `mark`/`mark_asof_ts`/`mark_source`/`value`/ + `unrealised`/`fee_ccy` under ONE mark policy (aggregate == Σ rows, + Decimal-exact on the real books), server-side display currency + (never-guess-a-rate), `GET /api/balances`, `last_asof_ts` documented, and + an additive-only contract sweep across the six frozen endpoints — ready + for the road-to-1.0 #5 freeze. Known seam: paper broker balances are empty + across restarts (the simulator's internal ledger is not persisted; + `/api/balances` relays it honestly) — the canary/guardrail funding hook. +- **`dashboard-tables-ux` (2026-07-12, PRs #215–#218)**: the dashboard renders + the full 2026-07-11 UX design — asset-first positions with price (as-of) / + value / unrealised, shared expandable rows (SSE-surviving), orders with + Filled % / Avg fill / Value and click-to-expand fills (standalone fills + table retired from the detail page), honest last→next bar chip, timezone + label, filtered-vs-empty states. The `reject_reason` follow-up shipped in + #219. +- **`canary-roundtrip` (2026-07-12, PRs #221–#223)**: `trading-bot canary` — + the deterministic platform self-test. Probes first (real cancel = the + kill-switch path; duplicate client-order-id = the idempotency invariant), + then a sequential round-trip; paper oracle exact (PnL == −Σ fees, flat, + persisted terminals, balance deltas), venue oracle = the accounting + identity (fills/balances == venue-reported, fee-denomination-aware) + + bounded cost. **Proven on real Binance testnet: 16/16 PASS** (cost + 0.00511 USDT). Two real findings hardened the domain: `Fill.fee_asset` + (Binance charges market-buy fees in BASE) and Binance HTTP-400 → domain + error mapping. `09-go-live.md` names the live canary as the road-to-1.0 + #1 validation vehicle. Cadence: paper per release, testnet at will, live + once per venue at go-live. ## Pending diff --git a/doc/dev/07-roadmap.md b/doc/dev/07-roadmap.md index 9c29f24f..23eaefec 100644 --- a/doc/dev/07-roadmap.md +++ b/doc/dev/07-roadmap.md @@ -97,48 +97,33 @@ order — none started yet, each is a `/pick-task` candidate: 200-row tail + server timestamps on snapshot endpoints, then the public contract is declared stable — 0.x-style breaking removals end at 1.0. -## UI consistency & accounting integrity (2026-07-11) +## UI consistency & accounting integrity (2026-07-11) — SHIPPED A dashboard-driven audit (position ≠ Σ orders on `alloc1-binance`) uncovered two -engine bugs and a set of API/UX gaps. Shipped 2026-07-11 (plan trees archived): -the `paper-integrity` epic (durable paper ids, order↔fill lifecycle sync, -persisted orphan-closes — PRs #190–#194, v0.12.0; restart the daemon to heal -the live books), the `accounting-guardrail` epic (pure invariant checker, -startup + 60 s-TTL wiring with SSE alerts, per-strategy health surface incl. -the kill-switch fold — the *status* half of road-to-1.0 #3 — and the dashboard -health pill; PRs #197–#201), and the `venue-minimums` epic (keyless cached -spec resolver, round-up-or-skip order prep with the spot-sell cap, strict -factory paper — PRs #203–#206; sub-minimum orders are structurally -impossible on the portfolio path). In dependency order: - -4. [ ] **`api-completeness` — API exposes what the app layer knows.** Per- - position mark price (v1 = last dccd bar close, always with its `asof` ts; - live ticker is post-1.0), value + unrealised PnL per position; display - currency (global default + per-exchange override, static conversion rates); - `/api/balances`; `fee_ccy`; `last_asof_ts` on every surface. Must land - before the API contract freeze (road-to-1.0 #5). -5. [ ] **`dashboard-tables-ux` — tables redesign.** Positions keyed by asset - (not pair) with value/mark/unrealised as primary columns; expandable rows - for secondary detail (fills under their order, ids, fee breakdown — - expanded state survives the SSE re-render); fills demoted to order detail - (flat audit view stays on the Orders page); "last bar → next bar" timing - chip; timezone affordance. -6. [ ] **`canary-roundtrip` — deterministic self-test strategy.** A minimal - round-trip canary run as its own strategy unit with its own tiny ledger - (~10 USDT): **sequential** market buy *x* then sell *x* (never simultaneous - — self-trade prevention would make it non-deterministic), plus two free - probes: a far-off-limit **cancel** (the real kill-switch path) and a - client-order-id **idempotency** re-submit. Two oracles: **paper** = exact - Decimal equality against the precomputed expectation (PnL = −2·fees, flat - position, terminal orders — CI-runnable, per release); **live** = exact - *accounting identity* (our fills/balances == venue-reported) + **bounded** - total cost (≤ 1–2 EUR) — never a precomputed absolute PnL (spread/drift are - not deterministic). Cadence: paper per release; testnet at will; live once - per venue at go-live + after any broker-adapter change. Depends on #1 - (order lifecycle) + #2 (health surface for the pass/fail report) + #3 - (venue minimums; Binance BNB-fee discount must be off or modeled). The live - canary is the validation vehicle for **Road to v1.0.0 #1** (real-key - enablement: venue idempotency, real cancel, balance reconciliation). +engine bugs and a set of API/UX gaps. **All six epics shipped 2026-07-11/12** +(plan trees archived; details in `06-status.md`, decisions in `03-decisions.md`): + +1. `paper-integrity` — durable paper ids, order↔fill lifecycle sync, persisted + orphan-closes (PRs #190–#194, **v0.12.0**). +2. `accounting-guardrail` — pure invariant checker, startup + 60 s-TTL wiring + with SSE alerts, per-strategy health surface incl. the kill-switch fold + (the *status* half of road-to-1.0 #3), dashboard health pill + (PRs #197–#201, **v0.13.0**). +3. `venue-minimums` — keyless cached spec resolver, round-up-or-skip order + prep with the spot-sell cap, strict factory paper (PRs #203–#206, + **v0.13.0**). +4. `api-completeness` — mark cache + bar-close marks with mandatory as-of, + value/unrealised/`fee_ccy` per position, server-side display currency, + `GET /api/balances`, additive-only contract sweep — ready for the + road-to-1.0 #5 freeze (PRs #209–#214). +5. `dashboard-tables-ux` — asset-first positions, shared expandable rows, + orders with Filled %/Value + fills as expanded detail, honest bar-timing + chip, timezone + filtered empty states (PRs #215–#219). +6. `canary-roundtrip` — deterministic self-test (`trading-bot canary`): + probes + round-trip, exact paper oracle, venue identity oracle **proven + 16/16 on real Binance testnet**; the live canary is the named validation + vehicle for road-to-1.0 #1 (PRs #221–#223; found `Fill.fee_asset` and the + Binance HTTP-400 mapping along the way). ## Not gating 1.0 (post-1.0 candidates) diff --git a/doc/dev/09-go-live.md b/doc/dev/09-go-live.md index ac9be47c..b1c4a863 100644 --- a/doc/dev/09-go-live.md +++ b/doc/dev/09-go-live.md @@ -79,12 +79,12 @@ missing any one refuses with a non-zero exit / a raised ruff check trading_bot/ mypy trading_bot/ ``` -5. **Validate against a real-key sandbox.** This is the one remaining - prerequisite and is **not done in-repo**: before any real order, exercise the - private endpoints (AddOrder / OpenOrders / balances / fills) against a real - Kraken key (ideally a low-limit, throwaway key) and confirm the - venue-reported state matches what the engine requested — see *Proven vs - pending*. **Do not skip this.** +5. **Validate against the real venue — run the canary.** Before any real order, + exercise the private endpoints (place / cancel / open orders / balances / + fills) and confirm the venue-reported state matches what the engine + requested. The **canary** (`trading-bot canary`) is the named in-repo + vehicle for exactly this — see *The canary — the validation vehicle* under + *Proven vs pending*. **Do not skip this.** Only after all five does a live `run` proceed to wire the live adapter. @@ -133,6 +133,42 @@ The left column is what the offline suite demonstrates today. The right column i the one remaining bridge to live — it requires a real key and is **out of scope for this repository's automated tests** (which never hit a real venue). +### The canary — the validation vehicle + +The pending column is exactly what the **canary** +([`application/canary.py`](../../trading_bot/application/canary.py), the +`trading-bot canary` command) proves against a real venue, in one gated run — a +minimum-size sequential round-trip plus two free probes, with an +expectation-vs-observation evidence table and a non-zero exit on the first +violation: + +```bash +trading-bot canary --exchange binance --mode testnet # sandbox: fake money, real API +trading-bot canary --exchange --mode live -c cfg.yaml # the go-live act (all gates + typed ack) +``` + +- **Venue idempotency** — the probe re-submits the *same* client-order-id and + asserts, from **venue reads** (`open_orders` / `fills`), that no duplicate + order was created. +- **Real cancel (the kill path)** — the probe places a far-below-market limit + and cancels it **on the venue**: a real cancel accepted, no fill, no balance + move, `CANCELLED` persisted. +- **Balance reconciliation** — the venue **identity oracle**: our recorded + fills == the venue-reported fills (`fills()` re-fetch; ids/qtys/prices, + exact), the venue's per-asset balance deltas == what our fills imply (the + fills math), position flat, and the total cost (the venue's quote delta) + bounded by `--max-cost`. Never a precomputed absolute PnL — a real venue's + spread/drift are not deterministic. + +`--mode testnet` runs the factory's sandbox path (hard-pinned testnet URL, +`BINANCE_TESTNET_*` keys, no `live_enabled` needed — Kraken has no spot testnet +and is refused). `--mode live` is the operator's **go-live act** for road-to-1.0 +#1: it demands the full gate stack above (`--config` with `live_enabled: true`, +credentials, all risk limits) **plus a typed confirmation** (the dashboard's +go-live phrase), and only then trades the smallest venue-legal size. Cadence: +paper per release, testnet at will, live once per venue at go-live and after +any adapter change. + --- ## Running LS1 (the first real portfolio strategy) diff --git a/doc/dev/plans/api-completeness/00-plan.md b/doc/dev/plans/api-completeness/00-plan.md deleted file mode 100644 index e0234e31..00000000 --- a/doc/dev/plans/api-completeness/00-plan.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -plan: api-completeness -kind: global -status: planning -roadmap: "4. [ ] **`api-completeness` — API exposes what the app layer knows.** (UI consistency & accounting integrity, 2026-07-11)" -release_on_done: false ---- - -# api-completeness — the API says what the engine knows - -## Goal - -The 2026-07-11 audit's gap analysis: the app layer computes per-instrument -marks and unrealised PnL, knows balances, fee currencies and bar timestamps — -and the API serializes almost none of it, so the dashboard cannot show a -position's value or current price. Close the server-side gaps (UI consumption -is epic E). Must land before the road-to-1.0 API contract freeze (#5). - -## Pinned decisions (maintainer, 2026-07-11 — implement, do not re-decide) - -- **Mark policy v1**: mark = **the last dccd bar close** — the same data the - strategy evaluates on. Every mark serialized WITH `mark_asof_ts` and - `mark_source: "bar_close" | "last_fill"` (fallback = last own-fill price - when no bar data is reachable). Live ticker is post-1.0. The UI must never - be able to mistake a stale mark for a live price — the timestamp is - non-optional. -- **Display currency**: global default + per-exchange override + static - `conversion_rates` config; conversion happens **server-side** (one truth - for every consumer): rows/aggregates gain `value_display` + - `display_currency` fields alongside the native-quote figures (which stay - untouched — contract). -- **API path does NO fresh I/O**: marks come from a cache the runner updates - at its own cadence (`_latest_closes` at portfolio_runner.py:481 sees every - frame); balances are read from the broker with the same care the existing - snapshot endpoints use. -- **Public contract**: only ADD fields/endpoints; every leaf carries a - contract-regression test (the accounting-guardrail leaf-03 pattern). -- All money = exact `str(Decimal)`. - -## Scouted anchors - -- `supervisor._mark_map(fills)` (supervisor.py:1335) — today's last-fill - mark; `_unrealised_of` consumes it; stays as the fallback. -- `portfolio_runner._latest_closes(frames)` (:838, used at :481) — the - per-symbol last close + the frame's asof, computed every rebalance: the - publish point for the mark cache. -- `_position_row_dict` (app.py) — serializes 8 fields today; `PositionRow` - (supervisor.py) is the carrier to extend. -- `Broker.balances()` — on every adapter incl. paper. -- `last_asof_ts` — already on `/api/strategies` rows. - -## Decomposition - -1. `01-mark-cache` — the runner publishes `(close, asof_ms)` per symbol into - a per-engine mark cache; supervisor reads it with the last-fill fallback. -2. `02-position-row-marks` — `PositionRow` + `/api/positions` rows gain - `mark`, `mark_asof_ts`, `mark_source`, `value` (mark × |net_qty|), - `unrealised`, `fee_ccy`; `_unrealised_of` upgraded to prefer the cache. -3. `03-display-currency` — config (`display_currency`, per-exchange map, - `conversion_rates`) + server-side converted `value_display`/ - `display_currency` on position rows and strategy aggregates. -4. `04-balances-and-asof` — `GET /api/balances`; `last_asof_ts` surfaced on - the strategy-detail payloads; full contract-regression sweep. - -## Leaf checklist - -- [ ] 01 mark-cache — feat/mark-cache — medium -- [ ] 02 position-row-marks — feat/position-row-marks — medium -- [ ] 03 display-currency — feat/display-currency — medium -- [ ] 04 balances-and-asof — feat/api-balances — medium - -## Dependencies - -Serial: 01 → 02 → 03 → 04 (02 consumes 01's cache; 03 converts 02's values; -04 touches the same `app.py` as 03 — no parallel pairs). - -## Done criteria - -- `/api/positions` rows over the real books show a mark from the real dccd - store with its asof, a value, an unrealised figure and a fee currency; - `mark_source` honest (`bar_close` when the feed data exists, `last_fill` - otherwise). -- `/api/balances` returns the paper balances for the running units. -- Cross-quote strategy aggregates carry a converted `value_display` in the - configured display currency. -- Every pre-existing API field byte-identical (contract sweep green). -- Roadmap line removed by the last leaf. diff --git a/doc/dev/plans/api-completeness/01-mark-cache.md b/doc/dev/plans/api-completeness/01-mark-cache.md deleted file mode 100644 index 4d481c2a..00000000 --- a/doc/dev/plans/api-completeness/01-mark-cache.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -plan: api-completeness/01-mark-cache -kind: leaf -status: planned -complexity: medium -depends: [] -parallel: false -branch: feat/mark-cache -pr: "" ---- - -# 01 — the engine's mark cache (bar closes, published by the runner) - -## Goal - -The engine should always know "the last close the strategy saw, and when it -was" per symbol — without any I/O on the API path. New -`trading_bot/application/mark_cache.py`: - -- `@dataclass(frozen=True, slots=True) class Mark`: `price: Money`, - `asof_ms: int`, `source: Literal["bar_close", "last_fill"]`. -- `class MarkCache`: `update(symbol: Symbol, price: Money, asof_ms: int) -> - None` (source is always `bar_close` on this path) and - `get(symbol) -> Mark | None`, `all() -> dict[Symbol, Mark]`. Plain dict, - updated from the runner's async loop, read from the API thread-context — - scout how other runner→supervisor state is shared (e.g. `last_eval_ts`) - and mirror the same discipline (the codebase's existing answer to this - sharing is the authority; do not invent new locking). - -Wiring: - -- `Engine` (service_factory.py) gains a `mark_cache: MarkCache` field - (default_factory, like `spec_resolver`). -- `portfolio_runner`: right after `prices = self._latest_closes(frames)` - (portfolio_runner.py:481), publish every `(symbol, close, asof_ms)` into - the engine's cache. The asof: the frame's last bar timestamp — read how - `_latest_closes`/the frames carry time (the runner already computes the - rebalance `asof_ms`; use the same value it stamps on signals). -- `StrategyRunner` (single-instrument): publish its instrument's latest - close the same way IF the seam is equally direct; if its loop differs - materially, note the follow-up in the module docstring instead of forcing - it (the dashboard units are portfolio units). -- The supervisor does NOT consume it yet (leaf 02). - -## Files to change - -- `trading_bot/application/mark_cache.py` — **new**. -- `trading_bot/application/service_factory.py` — `Engine.mark_cache`. -- `trading_bot/application/portfolio_runner.py` — the publish call (+ the - constructor/threading if the runner doesn't already hold the engine or the - cache — mirror how `spec_resolver` was threaded in #204). -- `trading_bot/application/run_app.py` — threading if needed (same pattern). -- `trading_bot/tests/application/test_mark_cache.py` — **new**. -- The portfolio-runner test module — one integration test. - -## Steps - -1. Read the `spec_resolver` threading (#204's diff) and `_latest_closes`; - mirror both. -2. Implement + wire + tests; all three gates green - (`~/.pyenv/versions/trading_bot_env/bin/python -m pytest`, `-m ruff check - trading_bot/`, `-m ruff format --check trading_bot/`). - -## Tests - -- `test_mark_cache_update_get_all` (Decimal-exact, latest write wins). -- Runner integration: after one rebalance tick over the existing test - harness, the engine's cache holds every traded symbol's close with the - tick's asof; a second tick updates in place. -- `test_engine_carries_mark_cache` (factory default). - -## Verification on real data - -Drive one rebalance-shaped pass over a scratch copy of -`var/dashboard/alloc1-binance.sqlite` with the REAL dccd frames (the -daemon's manifest `configs/dashboard.yaml` points at the store under -`~/data/arthurserver` — read-only): after the tick, dump the cache and -report 3 sample symbols' `(price, asof_ms→ISO, source)`; each price must -equal the last close visible in the dccd frame for that symbol and the asof -must be the frame's last bar time. Originals under `var/` and port 8000 -untouched. - -## Closeout - -- CHANGELOG `[Unreleased] > Added`: "per-engine mark cache — the runner - publishes each rebalance's last closes (`bar_close` marks with their asof) - for the API layer to read without I/O (#XX)." -- ADR: fold into leaf 02's entry (one mark-policy decision for the epic) - unless a sharing/locking nuance emerged. -- Tick leaf 01 in `00-plan.md`; archive per `/finish-task`. diff --git a/doc/dev/plans/api-completeness/02-position-row-marks.md b/doc/dev/plans/api-completeness/02-position-row-marks.md deleted file mode 100644 index 2b1b5223..00000000 --- a/doc/dev/plans/api-completeness/02-position-row-marks.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -plan: api-completeness/02-position-row-marks -kind: leaf -status: planned -complexity: medium -depends: [01] -parallel: false -branch: feat/position-row-marks -pr: "" ---- - -# 02 — position rows carry mark, value, unrealised and fee currency - -## Goal - -`/api/positions` rows finally answer "what is this position worth, at what -price, as of when": extend `PositionRow` (supervisor.py) and -`_position_row_dict` (app.py) with — all ADDITIVE, existing fields untouched: - -- `mark`, `mark_asof_ts`, `mark_source`: from the engine's `MarkCache` - (leaf 01) via `Symbol`; fallback when the cache has no entry: the - existing last-own-fill mark (`_mark_map`) with `mark_source="last_fill"` - and the fill's ts as `mark_asof_ts`; if neither exists → all three `null`. -- `value`: `mark × |net_qty|` (exact Decimal string; `null` when no mark). -- `unrealised`: `(mark − avg_entry_price) × net_qty` (sign-correct for - shorts — mirror `_unrealised_of`'s arithmetic; `null` when no mark or no - entry price). -- `fee_ccy`: the instrument's quote (`symbol.quote`). - -Also upgrade the strategy-aggregate unrealised (`_unrealised_of`, -supervisor.py:~1930) to prefer the mark cache over the last-fill map (same -fallback order as the rows) so the roster's `total_value`/`unrealised` and -the per-position figures agree — one mark policy everywhere. - -## Files to change - -- `trading_bot/application/supervisor.py` — `PositionRow` fields + - `positions()` fill-in + `_unrealised_of` cache preference. -- `trading_bot/interfaces/api/app.py` — `_position_row_dict` pass-through. -- Supervisor + dashboard test modules — tests below. - -## Steps - -1. Read `PositionRow`/`positions()`/`_unrealised_of` and the - field-addition convention from the strategy-capital epic; implement. -2. All three gates green. - -## Tests - -- `test_position_row_mark_from_cache` (bar_close mark + value + unrealised, - Decimal-exact, short position sign checked). -- `test_position_row_falls_back_to_last_fill` (no cache entry → last-fill - mark, `mark_source="last_fill"`). -- `test_position_row_no_mark_is_null` (flat/unknown → nulls, no crash). -- `test_fee_ccy_is_quote`. -- `test_aggregate_and_row_agree` (strategy `unrealised` == Σ row unrealised - under one mark policy). -- Contract regression: every pre-existing `/api/positions` field unchanged. - -## Verification on real data - -Real supervisor start over scratch copies of BOTH live books (the daemon's -real manifest sources, read-only; dccd store under `~/data/arthurserver`), -TestClient GET `/api/positions`: report 3 real rows per book with -mark/asof/source/value/unrealised — marks must be `bar_close` with a recent -asof when the dccd data covers the symbol, and the strategy aggregate must -equal the row sum. Originals and port 8000 untouched. - -## Closeout - -- CHANGELOG `[Unreleased] > Added`: "`/api/positions` rows carry `mark` / - `mark_asof_ts` / `mark_source` / `value` / `unrealised` / `fee_ccy`; the - strategy aggregate now uses the same bar-close mark policy (#XX)." -- ADR: the mark policy (bar_close v1 + tagged fallback, timestamp - non-optional; live ticker post-1.0). -- Tick leaf 02 in `00-plan.md`; archive per `/finish-task`. diff --git a/doc/dev/plans/api-completeness/03-display-currency.md b/doc/dev/plans/api-completeness/03-display-currency.md deleted file mode 100644 index 984e875b..00000000 --- a/doc/dev/plans/api-completeness/03-display-currency.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -plan: api-completeness/03-display-currency -kind: leaf -status: planned -complexity: medium -depends: [02] -parallel: false -branch: feat/display-currency -pr: "" ---- - -# 03 — display currency: global default, per-exchange override, static rates - -## Goal - -One configured display currency so cross-quote figures can be read (and -later aggregated) in a single unit — converted **server-side** so every -consumer agrees (pinned decision). All additive. - -Config (`trading_bot/application/config.py`, app/daemon level — same -altitude as `paper_strict`): - -- `display_currency: str = "USD"` (canonical asset code); -- `display_currency_overrides: dict[str, str] = {}` (exchange → currency, - e.g. `{"binance": "USDT"}`); -- `conversion_rates: dict[str, Decimal] = {}` — rate of ONE unit of a quote - currency IN the display currency (e.g. `{"USDT": "1", "EUR": "1.08"}`); - identity rate implied for the display currency itself. Validate: positive - Decimals via the money-field pattern. - -Serialization (additive): - -- position rows (`_position_row_dict`): `display_currency` (resolved for the - row's exchange) + `value_display` / `unrealised_display` (converted from - the row's quote via the rates; `null` when no rate is declared for that - quote — NEVER guess a rate); -- strategy rows (`_status_dict`) and `/api/kpi` aggregate rows: same pair on - the money aggregates (`total_value_display` etc. — pick names mirroring - the existing fields + `_display` suffix, consistently); -- a small pure helper `application/display_ccy.py` (`resolve_currency - (exchange, config)`, `convert(amount, from_ccy, config) -> Money | None`) - so supervisor/app share one conversion — no float, exact Decimal, `None` - on missing rate. - -## Files to change - -- `trading_bot/application/config.py` — the three fields. -- `trading_bot/application/display_ccy.py` — **new** pure helper. -- `trading_bot/interfaces/api/app.py` — row/aggregate serialization. -- `trading_bot/tests/application/test_display_ccy.py` — **new**; dashboard - test module — API-shape tests. - -## Steps - -1. Read the config validation patterns + where `_position_row_dict` / - `_status_dict` / `_kpi_row_dict` live; implement helper → config → - serialization. -2. All three gates green. - -## Tests - -- Pure: identity conversion; declared-rate conversion (Decimal-exact); - missing rate → `None`; override resolution (binance→USDT, default - elsewhere); invalid rate rejected at config parse. -- API: rows carry `value_display`/`display_currency`; missing rate → - `null` displays with native fields untouched; contract regression on all - pre-existing fields. - -## Verification on real data - -Real supervisor over the two book copies with -`display_currency="USD"`, `overrides={"binance":"USDT"}`, -`rates={"USDT":"1"}`: `/api/positions` — Binance rows display in USDT -(identity vs native), Kraken rows in USD; declare `rates={}` and verify the -displays go `null` while native values stay; report sample JSON rows. -Originals and port 8000 untouched. - -## Closeout - -- CHANGELOG `[Unreleased] > Added`: "display currency (global + - per-exchange override + static `conversion_rates`) — server-side converted - `*_display` money fields on position/strategy/KPI rows; missing rate → - null, never a guessed conversion (#XX)." -- ADR: server-side conversion + never-guess-a-rate policy. -- Tick leaf 03 in `00-plan.md`; archive per `/finish-task`. diff --git a/doc/dev/plans/api-completeness/04-balances-and-asof.md b/doc/dev/plans/api-completeness/04-balances-and-asof.md deleted file mode 100644 index 678d9681..00000000 --- a/doc/dev/plans/api-completeness/04-balances-and-asof.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -plan: api-completeness/04-balances-and-asof -kind: leaf -status: planned -complexity: medium -depends: [03] -parallel: false -branch: feat/api-balances -pr: "" ---- - -# 04 — /api/balances, last_asof surfacing, contract sweep - -## Goal - -Close the epic's remaining gaps: - -1. **`GET /api/balances`** (auth-gated like the other read endpoints; - optional `?strategy=` filter mirroring `/api/positions`): per RUNNING - unit, the broker's `balances()` serialized as - `{strategy, exchange, mode, balances: {asset: "exact-decimal"}}` rows - (grouped shape mirroring the other collection endpoints). `balances()` - is async on the port — check how other handlers await broker/supervisor - async calls and mirror it; a stopped unit contributes nothing; a broker - error degrades to an empty row with an `error` string field, never a 500 - (the endpoint must be safe to poll). - Note in the docstring: this is the prerequisite seam for the - positions↔balances cross-check (accounting-guardrail extension) and the - canary-roundtrip live oracle (roadmap #6). -2. **`last_asof_ts` surfacing**: verify it is present on every payload the - detail page consumes (`/api/strategies` has it; check the capital/pnl - payloads need nothing extra) and add it where the audit found it missing - — the goal is that epic E's "last bar → next bar" chip needs NO further - server change. Document in the endpoint docstrings what the field means - (as-of of the last completed evaluation). -3. **Contract-regression sweep**: one test module pass asserting the exact - pre-epic shapes of `/api/strategies`, `/api/positions`, `/api/fills`, - `/api/orders`, `/api/kpi`, `/api/health` still hold field-for-field - (additive-only proof for the whole epic, before the API freeze). - -## Files to change - -- `trading_bot/interfaces/api/app.py` — the endpoint + docstrings. -- `trading_bot/application/supervisor.py` — a `balances()` snapshot method - if the handler shouldn't reach into units directly (mirror how - `positions()` wraps unit access). -- Dashboard/control-api test modules — tests below. - -## Steps - -1. Read the collection-endpoint patterns (`/api/positions` grouping, auth, - read_only) and the supervisor snapshot wrappers; implement. -2. All three gates green. - -## Tests - -- `/api/balances`: running paper unit → its simulator balances (exact - Decimal strings); stopped unit absent; `?strategy=` filter; broker error - → error row, HTTP 200; auth + read-only behaviour identical to peers. -- `last_asof_ts` presence assertions on the consuming payloads. -- The epic-wide contract sweep (exact pre-epic shapes, additive-only). - -## Verification on real data - -Real supervisor over the two book copies, TestClient: -`GET /api/balances` → report the actual paper balances rows per unit (the -simulator's seeded quote balances net of the books' fills); confirm -`last_asof_ts` on `/api/strategies` matches the dccd store's last bar for -each unit. Originals and port 8000 untouched. - -## Closeout - -- CHANGELOG `[Unreleased] > Added`: "`GET /api/balances` (per running unit, - poll-safe) + `last_asof_ts` surfaced for the bar-timing chip; epic-wide - additive-only contract sweep (#XX)." -- `doc/dev/06-status.md`: epic shipped note. -- Last leaf: remove the `api-completeness` roadmap line, set the global - `00-plan.md` done, archive the tree per `/finish-task`. diff --git a/pyproject.toml b/pyproject.toml index f5c22e78..bc9dee93 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "trading_bot" -version = "0.13.0" +version = "0.14.0" description = "Execution & orchestration layer of the trading triptych — live strategies, order routing, risk. Hexagonal, async-first." readme = "README.md" license = { text = "MIT" } diff --git a/trading_bot/application/__init__.py b/trading_bot/application/__init__.py index f8b104ff..16e3a46d 100644 --- a/trading_bot/application/__init__.py +++ b/trading_bot/application/__init__.py @@ -102,6 +102,17 @@ collected without aborting the other legs (a :class:`~trading_bot.application.portfolio_runner.RebalanceResult` reports submitted-vs-failed). +* canary — :func:`~trading_bot.application.canary.run_canary` (with + :func:`~trading_bot.application.canary.paper_oracle`, + :func:`~trading_bot.application.canary.venue_oracle` and the + :class:`~trading_bot.application.canary.CanaryReport` / + :class:`~trading_bot.application.canary.CanaryCheck` evidence types), the + deterministic self-test: a sequential round-trip plus free cancel/idempotency + probes over a **dedicated**, explicitly-funded engine, every expectation vs + observation recorded exactly — paper's oracle is exact Decimal equality + (realised PnL == −Σ fees, flat, balances moved by fees only); the venue + oracle is the accounting identity (our fills/balances == venue-reported) + plus a bounded cost. * run_app — :func:`~trading_bot.application.run_app.run_app` (and :func:`~trading_bot.application.run_app.build_runners`), the **triptych entrypoint**: one :class:`~trading_bot.application.config.AppConfig` → @@ -117,6 +128,13 @@ from __future__ import annotations +from trading_bot.application.canary import ( + CanaryCheck, + CanaryReport, + paper_oracle, + run_canary, + venue_oracle, +) from trading_bot.application.capital_service import CapitalPolicy, CapitalService from trading_bot.application.config import ( AppConfig, @@ -244,6 +262,12 @@ # wiring "Engine", "build_engine", + # canary (deterministic self-test) + "run_canary", + "paper_oracle", + "venue_oracle", + "CanaryReport", + "CanaryCheck", # entrypoint "run_app", "build_runners", diff --git a/trading_bot/application/canary.py b/trading_bot/application/canary.py new file mode 100644 index 00000000..c655f8f7 --- /dev/null +++ b/trading_bot/application/canary.py @@ -0,0 +1,1220 @@ +"""The canary — a deterministic self-test the platform runs on itself. + +The canary is the operator's proof that a wired engine actually round-trips +money correctly on a venue mode, **before** any strategy is trusted with it: +a tiny **sequential round-trip** (market buy *x* → confirm the fill → market +sell *x* → confirm → flat) preceded by two **free probes**, with an **oracle +computed in advance**. Every step's expectation and observation is recorded as +an exact string in a :class:`CanaryReport` — the evidence table — and the run +fails on the first violated expectation while still snapshotting and reporting +everything it measured. + +Design (pinned in ``doc/dev/plans/canary-roundtrip/00-plan.md``) +---------------------------------------------------------------- +* **Sequential, never simultaneous.** The buy and the sell are placed one + after the other, each confirmed before the next order is placed. A crossed + simultaneous pair would trip venue self-trade prevention and make the + outcome non-deterministic; a sequential round-trip is deterministic on every + venue mode. +* **Probes before money.** Both probes are *free* (they can never spend): + they run **before** the round-trip so the expensive step is only reached on + an engine whose kill path (cancel) and idempotency guarantee have just been + demonstrated live. + + - **Cancel probe** — place a limit buy far below the market + (``probe_offset_pct`` percent below the mark, default 50%), cancel it, and + assert: no fill, no balance move, ``CANCELLED`` persisted in the store. + This exercises the real kill-switch path end to end. + - **Idempotency probe** — re-submit an order carrying the **same** + client-order-id and assert the engine deduped it: the original tracked + order is returned, no second venue order, no new fill, no second store + row. This is the invariant that makes a retry safe with real money. +* **Two oracles.** The *paper* oracle (:func:`paper_oracle`, this module) is + **EXACT** — Decimal equality, no tolerance — because the simulator is fully + deterministic. The *venue* oracle (:func:`venue_oracle`, this module) can + never precompute an absolute PnL (spread and drift are not deterministic); + it asserts the **accounting identity** (our recorded fills == the + venue-reported fills re-fetched through ``broker.fills(...)``, the + venue-reported balance deltas == the deltas *our fills imply*, flat) plus a + **bounded total cost** (the quote balance delta, capped by ``max_cost``). + Keeping the two oracles distinct is what lets paper be exact without + pretending a live venue is. +* **Venue legs settle by polling the broker port.** A real (REST) venue fills + a market order asynchronously and reports the execution only through + :meth:`~trading_bot.brokers.base.Broker.fills` — and it keys those fills by + its **own** order reference, not our client-order-id (Binance ``myTrades`` + carries the venue ``orderId``). After submitting a market leg the canary + therefore polls ``broker.fills(...)``, attributes every **new** fill on the + leg's instrument and side to the leg (the canary engine is dedicated, and + the round-trip is sequential, so within the run's own fill-id window that + attribution is unambiguous), and re-emits each one on the engine bus with + the leg's client-order-id — so the ordinary fill plumbing (fill sync → + tracker/performance/store) applies venue truth exactly as it would a + simulator fill. On paper the broker fills synchronously and the settle loop + never runs. No adapter-specific code: only the broker port is read. +* **Sizing / cost philosophy.** The canary trades the **smallest venue-legal + size**: the caller sizes ``qty`` from the venue's real minimums + (``min_qty`` / ``min_notional`` via the instrument-spec resolver) so the + only money at risk is the fee on a minimum-size round-trip (paper: exactly + ``2 × fee``; venue: bounded by the configured ``max_cost``). The cancel + probe never risks even that — but its far-below price shrinks its notional, + so its quantity is scaled **up** just enough to stay venue-legal at the + probe price (a resting order's size costs nothing). +* **A dedicated engine only.** The canary trades — it must be impossible to + point it at a shared book by accident. It therefore refuses an engine + without its own store (``build_engine`` attaches a store only for an + explicit ``db_path``; there is no default), and the caller funds the engine + explicitly (``AppConfig.paper_starting_balances`` on paper). Engine + construction is the **caller's** job (the CLI leaf); this module drives a + ready engine through its own seams (``router.submit`` / ``router.cancel``, + ``broker.balances``, ``tracker``, ``perf``, ``store``) and never constructs + or mutates wiring. + +Paper exactness assumption (documented, relied on by :func:`paper_oracle`) +-------------------------------------------------------------------------- +The paper simulator fills **both** market legs at the **same injected mark**, +so the buy and sell notionals cancel exactly and only the fees remain: +realised PnL is exactly ``-(sum of fees)``, the quote balance delta is exactly +``-(sum of fees)`` and the base balance delta is exactly ``0``. On any real +venue none of this holds (spread, drift) — which is precisely why the venue +oracle asserts identities and bounds instead. +""" + +from __future__ import annotations + +# Built-in +import asyncio +import time +import uuid +from dataclasses import dataclass, field, replace +from decimal import ROUND_UP, localcontext +from typing import TYPE_CHECKING + +# Local +from trading_bot.application.events import FillEvent +from trading_bot.domain.errors import ( + BrokerError, + MissingOrder, + OrderError, + RiskLimitBreached, +) +from trading_bot.domain.money import Money, money +from trading_bot.domain.order import Order, OrderSide, OrderStatus, OrderType + +if TYPE_CHECKING: + from collections.abc import Callable + + from trading_bot.application.service_factory import Engine + from trading_bot.domain.fill import Fill + from trading_bot.domain.instrument import Instrument + +__all__ = [ + "CanaryCheck", + "CanaryReport", + "paper_oracle", + "run_canary", + "size_for_minimums", + "venue_oracle", +] + +#: Percent denominator for ``probe_offset_pct``. +_HUNDRED: Money = money("100") + +#: Default cancel-probe offset: the probe limit buy is placed this many percent +#: below the mark (50% — far enough that no realistic tick ever crosses it). +_DEFAULT_PROBE_OFFSET_PCT: Money = money("50") + +#: Precision (significant digits) for the probe-qty scaling division — matches +#: the domain's pinned ``decimal128`` precision (see ``order.AVG_PRICE_PRECISION``). +_PROBE_QTY_PRECISION: int = 34 + +#: The errors a canary order operation may legitimately surface: they become a +#: failed check (the canary reports, it does not crash), never an escape. +_ORDER_ERRORS = (BrokerError, OrderError, MissingOrder, RiskLimitBreached) + +#: How long a market leg may take to settle on a real venue (seconds): the poll +#: loop that re-fetches ``broker.fills(...)`` and pumps the leg's executions +#: onto the bus gives up after this long — the leg's check then fails with +#: exactly what was observed. Paper fills synchronously and never waits. +_VENUE_SETTLE_TIMEOUT_S: float = 20.0 + +#: Delay between two settle polls of ``broker.fills(...)`` (seconds). +_VENUE_SETTLE_POLL_S: float = 0.5 + +#: The order statuses a submitted market leg may still settle from (a venue +#: fill can only apply to a live order — mirrors ``OrderFillSync``'s rule). +_LEG_SETTLING: frozenset[OrderStatus] = frozenset( + {OrderStatus.OPEN, OrderStatus.PARTIALLY_FILLED} +) + + +@dataclass(frozen=True, slots=True) +class CanaryCheck: + """One expectation-vs-observation line of the canary's evidence table. + + Immutable: a check is a recorded fact about one scenario step. ``passed`` + is computed on the underlying **values** (exact ``Decimal`` / enum + comparisons), never on the rendered strings — two numerically equal + decimals with different exponents (``-0.10`` vs ``-0.100``) still pass. + + Parameters + ---------- + name : str + The check's stable identifier (e.g. ``"cancel_probe.resting"``). + expected : str + The expectation, rendered as an exact human-readable string. + observed : str + What was actually measured, rendered the same way. + passed : bool + Whether the observation met the expectation (value comparison). + + """ + + name: str + expected: str + observed: str + passed: bool + + +@dataclass(slots=True) +class CanaryReport: + """The canary run's full evidence: ordered checks plus run context. + + Built incrementally by :func:`run_canary`; the context fields (exchange, + mode, instrument, qty, client-order-ids, balance snapshots) carry enough + for :meth:`to_text` to print a self-contained evidence table and for + :func:`paper_oracle` to compute its exact assertions. + + Parameters + ---------- + exchange : str + The broker's venue key (``"paper"``, ``"binance"``, ...). + mode : str + The run's execution mode label (``"paper"`` / ``"testnet"`` / + ``"live"``) — the engine's configured mode, or the caller's + ``mode_label`` override (a testnet engine is ``mode: live`` + + ``testnet: true`` config-side; the report says ``"testnet"``). + instrument : Instrument + The instrument the canary traded. + qty : Decimal + The round-trip quantity (base units) requested per leg. + probe_cid, buy_cid, sell_cid : str + The client-order-ids the run minted for its three orders. + checks : list of CanaryCheck + The ordered evidence lines, in scenario order (oracle checks last). + cost : Decimal or None + The run's total realised cost, ``-(realised PnL)`` — positive means + the canary paid (fees/spread). ``None`` until the run completes its + final snapshot. + initial_balances, final_balances : dict of str to Decimal + The broker-reported balance snapshots taken before the first order and + after the last (exact ``Decimal``, keyed by canonical asset code). + venue_fills : list of Fill + The **venue-reported** fills this run produced: the final + ``broker.fills(...)`` re-fetch, minus every fill id the venue already + reported before the first order. Exactly as the venue rendered them + (ids, quantities, prices — the venue's own order reference, not our + client-order-ids). :func:`venue_oracle` compares our recorded fills + against these; empty until the run's final snapshot. + + """ + + exchange: str + mode: str + instrument: Instrument + qty: Money + probe_cid: str = "" + buy_cid: str = "" + sell_cid: str = "" + checks: list[CanaryCheck] = field(default_factory=list) + cost: Money | None = None + initial_balances: dict[str, Money] = field(default_factory=dict) + final_balances: dict[str, Money] = field(default_factory=dict) + venue_fills: list[Fill] = field(default_factory=list) + + @property + def passed(self) -> bool: + """Whether the run passed: at least one check, and every check green.""" + return bool(self.checks) and all(check.passed for check in self.checks) + + def to_text(self) -> str: + """Render the evidence table as stable, exact text (one check per line). + + The output is deterministic for a deterministic run: it contains no + timestamps and no minted ids — only the run context, each check's + exact expected/observed strings, the total cost and the verdict. + + Returns + ------- + str + The multi-line evidence table. + + """ + lines = [ + f"canary — exchange={self.exchange} mode={self.mode} " + f"instrument={self.instrument} qty={self.qty}" + ] + lines.extend( + f"[{'PASS' if check.passed else 'FAIL'}] {check.name}: " + f"expected={check.expected} | observed={check.observed}" + for check in self.checks + ) + lines.append(f"cost: {'n/a' if self.cost is None else self.cost}") + lines.append(f"result: {'PASS' if self.passed else 'FAIL'}") + return "\n".join(lines) + + +def _fmt_balances(balances: dict[str, Money]) -> str: + """Render a balances mapping as a stable ``ASSET=amount`` line (sorted).""" + if not balances: + return "(empty)" + return " ".join(f"{asset}={amount}" for asset, amount in sorted(balances.items())) + + +def _probe_qty(instrument: Instrument, qty: Money, probe_price: Money) -> Money: + """The cancel-probe quantity: ``qty``, scaled up to stay legal at the probe price. + + The probe is priced far below the mark, which shrinks its notional — a + quantity that is venue-legal for the round-trip at the mark can fall below + ``min_notional`` at the probe price and be rejected before it ever rests. + A resting order's size costs nothing (it never fills), so the probe scales + its quantity **up** to the smallest lot-aligned amount whose notional at + ``probe_price`` clears the venue minimum (and to ``min_qty`` if that is + higher). With no ``min_notional`` on the spec (or one already satisfied), + ``qty`` is used unchanged. + + Parameters + ---------- + instrument : Instrument + The (spec-carrying) instrument the probe trades. + qty : Decimal + The round-trip quantity the caller sized at the mark. + probe_price : Decimal + The probe's (already quantized) far-below limit price. + + Returns + ------- + Decimal + The probe quantity: ``qty``, or the smallest venue-legal scale-up. + + """ + min_notional = instrument.min_notional + if min_notional is None or qty * probe_price >= min_notional: + required = qty + else: + # Divide under an explicit context rounding UP so the scaled quantity's + # notional can never land a hair below the minimum. + with localcontext() as ctx: + ctx.prec = _PROBE_QTY_PRECISION + ctx.rounding = ROUND_UP + required = +(min_notional / probe_price) + step = instrument.qty_step + if step is not None: + # Lot-align upward: snap down, then add one lot if that undershot. + snapped = instrument.quantize_qty(required) + if snapped < required: + snapped += step + required = snapped + if instrument.min_qty is not None and required < instrument.min_qty: + required = instrument.min_qty + return required + + +def size_for_minimums(instrument: Instrument, price: Money) -> Money: + """The smallest venue-legal round-trip quantity for ``instrument`` at ``price``. + + Implements the canary's sizing philosophy (module docstring): trade the + smallest venue-legal size, sized **from scratch** against the venue's real + minimums (``min_qty`` / ``min_notional``, e.g. via the instrument-spec + resolver, ``application/instrument_specs.py``) rather than a caller-chosen + quantity. Unlike :func:`_probe_qty` — which *scales up* an already-legal + round-trip quantity for a *different* (probe) price — this computes the + minimum directly: the smallest lot-aligned quantity whose notional at + ``price`` clears ``min_notional``, raised to ``min_qty`` if that is higher. + + Parameters + ---------- + instrument : Instrument + The (spec-carrying) instrument to size. Must carry at least one of + ``min_qty`` / ``min_notional`` — see Raises. + price : Decimal + The reference price (the mark) the notional is computed against. Must + be strictly positive. + + Returns + ------- + Decimal + The smallest venue-legal quantity. + + Raises + ------ + ValueError + If ``instrument`` carries neither ``min_qty`` nor ``min_notional`` (an + unresolved / bare spec — e.g. a degraded resolver fetch, or an + exchange the resolver does not dispatch): sizing "the smallest legal + amount" with no real venue minimum to size against would be guessing, + which the caller must refuse instead of this function silently doing + so. Also raised if ``price`` is not strictly positive. + + """ + if instrument.min_qty is None and instrument.min_notional is None: + raise ValueError( + f"{instrument} carries no venue minimums (min_qty/min_notional): " + "refusing to guess a venue-legal quantity" + ) + price = money(price) + if price <= 0: + raise ValueError(f"price must be strictly positive, got {price}") + + min_notional = instrument.min_notional + min_qty = instrument.min_qty + if min_notional is None: + # min_qty is guaranteed set here (guarded above) — the lot minimum + # alone is the smallest legal size with no notional floor to clear. + assert min_qty is not None + return min_qty + + # Divide under an explicit context rounding UP so the sized quantity's + # notional can never land a hair below the minimum (mirrors _probe_qty). + with localcontext() as ctx: + ctx.prec = _PROBE_QTY_PRECISION + ctx.rounding = ROUND_UP + required = +(min_notional / price) + step = instrument.qty_step + if step is not None: + # Lot-align upward: snap down, then add one lot if that undershot. + snapped = instrument.quantize_qty(required) + if snapped < required: + snapped += step + required = snapped + if min_qty is not None and required < min_qty: + required = min_qty + return required + + +def paper_oracle( + engine: Engine, + report: CanaryReport, + *, + expected_fill_count: int = 2, +) -> list[CanaryCheck]: + """The EXACT paper oracle: Decimal-equality assertions over the whole run. + + Computable in advance because the simulator is deterministic and fills + **both round-trip legs at the same injected mark** (see the module + docstring's exactness assumption): the two notionals cancel exactly and + only fees remain. The checks, all exact (no tolerance): + + * realised PnL == ``-(sum of fees)`` — fees summed over the **store's** + persisted fills (fills are the sole PnL truth; the store is their + persistence), cross-checking the live performance service against what + was actually recorded; + * the position is flat (``net_qty == 0``); + * both round-trip orders are ``FILLED`` with ``filled_qty == qty`` **in + the store** (not just in memory); + * the quote balance delta is exactly ``-(sum of fees)`` and the base + balance delta is exactly ``0`` (same-mark assumption); + * the round-trip produced exactly ``expected_fill_count`` fills. + + Parameters + ---------- + engine : Engine + The canary's dedicated engine (store, tracker, performance service). + report : CanaryReport + The completed scenario report — supplies the balance snapshots, the + round-trip client-order-ids and the quantity. + expected_fill_count : int, optional + How many fills the round-trip must have produced. Defaults to ``2`` + (one per leg — the factory paper broker's ``"immediate"`` fill model); + a caller running a chunked fill model passes its own figure. + + Returns + ------- + list of CanaryCheck + The oracle's checks, in the order above. + + Raises + ------ + ValueError + If ``engine`` has no store (the oracle's assertions are store-based; + a canary engine always carries its own dedicated store). + + """ + store = engine.store + if store is None: + raise ValueError( + "paper_oracle requires an engine with its own store: the exact " + "assertions are checked against persisted rows, not memory" + ) + store.flush() + checks: list[CanaryCheck] = [] + + # 1. Realised PnL == -(sum of fees), fees from the persisted fills. + fills = store.fills() + total_fees = sum((fill.fee for fill in fills), money("0")) + realised = engine.perf.realised_pnl() + checks.append( + CanaryCheck( + name="oracle.realised_pnl_is_minus_fees", + expected=f"realised_pnl={-total_fees}", + observed=f"realised_pnl={realised}", + passed=realised == -total_fees, + ) + ) + + # 2. Flat position (a never-touched instrument counts as flat). + position = engine.tracker.position(report.instrument) + net_qty = position.net_qty if position is not None else money("0") + checks.append( + CanaryCheck( + name="oracle.position_flat", + expected="net_qty=0", + observed=f"net_qty={net_qty}", + passed=net_qty == 0, + ) + ) + + # 3./4. Both round-trip orders FILLED with filled_qty == qty IN THE STORE. + for name, cid in ( + ("oracle.buy_filled_in_store", report.buy_cid), + ("oracle.sell_filled_in_store", report.sell_cid), + ): + row = store.get_order(cid) + checks.append( + CanaryCheck( + name=name, + expected=f"status=filled filled_qty={report.qty}", + observed=( + "store row missing" + if row is None + else f"status={row.status.value} filled_qty={row.filled_qty}" + ), + passed=( + row is not None + and row.status is OrderStatus.FILLED + and row.filled_qty == report.qty + ), + ) + ) + + # 5. Balance deltas: quote moved by exactly -(sum of fees); base by 0. + quote = report.instrument.symbol.quote + base = report.instrument.symbol.base + zero = money("0") + quote_delta = report.final_balances.get(quote, zero) - report.initial_balances.get( + quote, zero + ) + base_delta = report.final_balances.get(base, zero) - report.initial_balances.get( + base, zero + ) + checks.append( + CanaryCheck( + name="oracle.quote_delta_is_minus_fees", + expected=f"quote_delta={-total_fees}", + observed=f"quote_delta={quote_delta}", + passed=quote_delta == -total_fees, + ) + ) + checks.append( + CanaryCheck( + name="oracle.base_delta_zero", + expected="base_delta=0", + observed=f"base_delta={base_delta}", + passed=base_delta == 0, + ) + ) + + # 6. Fill count: the round-trip produced exactly the expected executions. + roundtrip_cids = {report.buy_cid, report.sell_cid} + fill_count = sum(1 for fill in fills if fill.client_order_id in roundtrip_cids) + checks.append( + CanaryCheck( + name="oracle.fill_count", + expected=f"fills={expected_fill_count}", + observed=f"fills={fill_count}", + passed=fill_count == expected_fill_count, + ) + ) + return checks + + +def _fmt_fill_lines(triples: list[tuple[str, Money, Money]]) -> str: + """Render sorted ``(fill_id, qty, price)`` triples as one exact, stable line.""" + if not triples: + return "(none)" + return " ".join( + f"[id={fid} qty={qty} price={price}]" for fid, qty, price in triples + ) + + +def venue_oracle( + max_cost: Money, +) -> Callable[[Engine, CanaryReport], list[CanaryCheck]]: + """Build the venue **identity** oracle for a real-venue (testnet/live) run. + + A real venue's absolute PnL is not precomputable (spread and drift are not + deterministic), so — unlike :func:`paper_oracle` — this oracle asserts the + **accounting identity** plus a **bounded cost**, all on exact ``Decimal`` + values (see the module docstring's "Two oracles"): + + * ``oracle.fills_match_venue`` — OUR recorded fills (the store's persisted + fills for the two round-trip client-order-ids) == the **venue-reported** + fills (:attr:`CanaryReport.venue_fills`, the run's final + ``broker.fills(...)`` re-fetch minus the pre-run baseline), compared as + sorted ``(fill_id, qty, price)`` triples — ids, quantities and prices, + exact. Client-order-ids are deliberately **not** compared: a venue keys + its fill reports by its own order reference (Binance ``myTrades`` + carries the venue ``orderId``), and the run's settle step is what + attributed each venue fill to its leg. + * ``oracle.position_flat`` — the tracker's net position is ``0``. + * ``oracle.base_delta_matches_fills`` / ``oracle.quote_delta_matches_fills`` + — the venue-reported balance deltas (final − initial snapshots, per + asset) equal the deltas **our recorded fills imply** — the fills math, + never a hardcoded zero, **fee-denomination-aware** (each fill's fee is + charged against its :attr:`~trading_bot.domain.fill.Fill.fee_asset`; + ``None`` means quote). Base: ``Σ buy qty − Σ sell qty − Σ base-fees`` + (Binance charges a market buy's commission in the base asset — observed + live on the testnet as exactly this base dust). Quote: ``Σ sell notional + − Σ buy notional − Σ quote-fees``. If the venue reports base dust after + the round trip, the check reports it exactly and compares it against the + fills math, not against zero — mirroring reconcile's rule that the + broker's fills are the truth balances must be explained by. + * ``oracle.fee_delta_matches_fills[ASSET]`` — one extra check per fee + denominated in an asset that is **neither** base nor quote (e.g. a BNB + fee discount): that asset's venue-reported delta must equal ``−Σ`` of + those fees. Emitted only when such fees exist, so the common runs keep + their fixed five checks. + * ``oracle.cost_bounded`` — the run's total cost, ``-(venue quote balance + delta)``, is at most ``max_cost``; reported exactly. Fees charged in + other assets are visible in their own identity lines above (converting + them to quote would need a price the run does not have — never + fabricated). + + Parameters + ---------- + max_cost : Decimal + The cost bound (quote units): the most the whole round-trip may cost + (fees + spread + drift), as measured by the venue's own quote balance + delta. + + Returns + ------- + callable + An ``oracle(engine, report) -> list[CanaryCheck]`` suitable for + :func:`run_canary`'s ``oracle`` parameter. + + Raises + ------ + ValueError + (From the returned oracle.) If ``engine`` has no store — our recorded + fills are read from the persisted rows, exactly like + :func:`paper_oracle`. + + """ + max_cost = money(max_cost) + + def _venue_oracle(engine: Engine, report: CanaryReport) -> list[CanaryCheck]: + store = engine.store + if store is None: + raise ValueError( + "venue_oracle requires an engine with its own store: our " + "recorded fills are read from persisted rows, not memory" + ) + store.flush() + checks: list[CanaryCheck] = [] + zero = money("0") + + # OUR recorded fills: the store's persisted rows for the two legs. + roundtrip_cids = {report.buy_cid, report.sell_cid} + ours = [ + fill for fill in store.fills() if fill.client_order_id in roundtrip_cids + ] + + # 1. Identity: our recorded fills == the venue-reported fills, compared + # as sorted (fill_id, qty, price) triples — exact Decimal equality. + ours_triples = sorted((f.fill_id, f.qty, f.price) for f in ours) + venue_triples = sorted((f.fill_id, f.qty, f.price) for f in report.venue_fills) + checks.append( + CanaryCheck( + name="oracle.fills_match_venue", + expected=f"venue fills: {_fmt_fill_lines(venue_triples)}", + observed=f"our fills: {_fmt_fill_lines(ours_triples)}", + passed=ours_triples == venue_triples, + ) + ) + + # 2. Flat position (a never-touched instrument counts as flat). + position = engine.tracker.position(report.instrument) + net_qty = position.net_qty if position is not None else zero + checks.append( + CanaryCheck( + name="oracle.position_flat", + expected="net_qty=0", + observed=f"net_qty={net_qty}", + passed=net_qty == 0, + ) + ) + + # 3./4. Balance deltas: the venue's reported movement must equal what + # our fills imply — the fills math, never a hardcoded zero. Each + # fee is charged against the asset it is denominated in (a fill's + # fee_asset; None means quote — Binance charges a market buy's + # commission in the BASE asset, observed live on the testnet). + base = report.instrument.symbol.base + quote = report.instrument.symbol.quote + base_delta = report.final_balances.get( + base, zero + ) - report.initial_balances.get(base, zero) + quote_delta = report.final_balances.get( + quote, zero + ) - report.initial_balances.get(quote, zero) + bought = sum((f.qty for f in ours if f.side is OrderSide.BUY), zero) + sold = sum((f.qty for f in ours if f.side is OrderSide.SELL), zero) + buy_notional = sum( + (f.qty * f.price for f in ours if f.side is OrderSide.BUY), zero + ) + sell_notional = sum( + (f.qty * f.price for f in ours if f.side is OrderSide.SELL), zero + ) + fees_by_asset: dict[str, Money] = {} + for fill in ours: + asset = fill.fee_asset if fill.fee_asset is not None else quote + fees_by_asset[asset] = fees_by_asset.get(asset, zero) + fill.fee + expected_base = bought - sold - fees_by_asset.get(base, zero) + expected_quote = sell_notional - buy_notional - fees_by_asset.get(quote, zero) + checks.append( + CanaryCheck( + name="oracle.base_delta_matches_fills", + expected=f"base_delta={expected_base}", + observed=f"base_delta={base_delta}", + passed=base_delta == expected_base, + ) + ) + checks.append( + CanaryCheck( + name="oracle.quote_delta_matches_fills", + expected=f"quote_delta={expected_quote}", + observed=f"quote_delta={quote_delta}", + passed=quote_delta == expected_quote, + ) + ) + # One extra identity line per fee denominated in a third asset (e.g. a + # BNB fee discount): its venue delta must equal -(those fees) exactly. + for asset in sorted(fees_by_asset): + if asset in (base, quote): + continue + asset_fees = fees_by_asset[asset] + asset_delta = report.final_balances.get( + asset, zero + ) - report.initial_balances.get(asset, zero) + checks.append( + CanaryCheck( + name=f"oracle.fee_delta_matches_fills[{asset}]", + expected=f"{asset}_delta={-asset_fees}", + observed=f"{asset}_delta={asset_delta}", + passed=asset_delta == -asset_fees, + ) + ) + + # 5. Bounded cost: what the round-trip actually cost, by the venue's + # own quote balance delta — never a precomputed absolute PnL. + cost = -quote_delta + checks.append( + CanaryCheck( + name="oracle.cost_bounded", + expected=f"cost<={max_cost}", + observed=f"cost={cost}", + passed=cost <= max_cost, + ) + ) + return checks + + return _venue_oracle + + +async def run_canary( + engine: Engine, + instrument: Instrument, + *, + qty: Money, + probe_offset_pct: Money = _DEFAULT_PROBE_OFFSET_PCT, + run_id: str | None = None, + oracle: Callable[[Engine, CanaryReport], list[CanaryCheck]] | None = paper_oracle, + mode_label: str | None = None, +) -> CanaryReport: + """Run the canary scenario on a ready, dedicated ``engine``. + + Drives the engine's **own** seams, in the pinned order: balance snapshot → + **cancel probe** (far-below limit buy, cancel, assert no fill / no balance + move / ``CANCELLED`` persisted) → **idempotency probe** (re-submit the same + client-order-id, assert the engine deduped) → **round-trip** (market buy + ``qty``, confirm in tracker+store; market sell ``qty``, confirm; assert + flat) → final balance snapshot → the ``oracle``'s exact checks. Every step + appends :class:`CanaryCheck`\\ s; **a failed check stops placing further + orders** (and best-effort cancels anything the run left live) but the + final snapshot, cost and report still happen — the run fails loudly with + everything it measured. + + Engine construction (and funding, and sizing ``qty`` from the venue + minimums) is the caller's job — see the module docstring. + + Parameters + ---------- + engine : Engine + The dedicated canary engine. Must carry its **own** store (persistence + checks are part of the scenario), a broker with a mark for + ``instrument`` (:meth:`~trading_bot.brokers.base.Broker.ticker`), and + explicit funding for the round-trip. + instrument : Instrument + The (spec-carrying) instrument to trade. + qty : Decimal + The round-trip quantity per leg (base units). Must be strictly + positive, lot-aligned and venue-legal at the mark — the smallest + venue-legal size, by the sizing philosophy. + probe_offset_pct : Decimal, optional + How far below the mark the cancel probe is priced, in percent. + Defaults to ``50``. Must be strictly inside ``(0, 100)``. + run_id : str, optional + The token embedded in the run's client-order-ids + (``canary-{run_id}-probe/buy/sell``). Defaults to a fresh random + token, so re-running over the same engine/store never collides ids. + oracle : callable, optional + The mode's oracle, called as ``oracle(engine, report)`` **only when + every scenario check passed** (an aborted run's oracle expectations + were never measured) and its checks appended to the report. Defaults + to :func:`paper_oracle`; a real-venue (testnet/live) run passes + :func:`venue_oracle`'s built oracle instead. ``None`` skips the oracle. + mode_label : str, optional + The mode the report should carry (``report.mode``). Defaults to the + engine's ``config.mode`` — pass ``"testnet"`` for a testnet engine, + whose config is ``mode: live`` + a ``testnet: true`` broker (the + factory's sandbox path), so the evidence says what actually ran. + + Returns + ------- + CanaryReport + The full evidence: ordered checks, run context, balance snapshots and + total realised cost. ``report.passed`` is the verdict. + + Raises + ------ + ValueError + If ``engine`` has no store, ``qty`` is not strictly positive, or + ``probe_offset_pct`` is outside ``(0, 100)`` — caller wiring errors, + refused before any order is placed. + BrokerError + If the broker has no mark price for ``instrument`` (the pre-trade + ticker read fails) — nothing has been placed yet. + + """ + if engine.store is None: + raise ValueError( + "the canary requires an engine with its own dedicated store " + "(build_engine(..., db_path=...)): persistence checks are part of " + "the scenario, and a shared/absent store must be impossible" + ) + qty = money(qty) + if qty <= 0: + raise ValueError(f"canary qty must be strictly positive, got {qty}") + probe_offset_pct = money(probe_offset_pct) + if not (0 < probe_offset_pct < _HUNDRED): + raise ValueError( + f"probe_offset_pct must be in (0, 100), got {probe_offset_pct}" + ) + + broker = engine.broker + router = engine.router + store = engine.store + token = run_id if run_id is not None else uuid.uuid4().hex[:8] + report = CanaryReport( + exchange=broker.name, + mode=mode_label if mode_label is not None else engine.config.mode, + instrument=instrument, + qty=qty, + probe_cid=f"canary-{token}-probe", + buy_cid=f"canary-{token}-buy", + sell_cid=f"canary-{token}-sell", + ) + + # --- initial snapshot (before any order) --------------------------------- # + mark = await broker.ticker(instrument) + report.initial_balances = dict(await broker.balances()) + # One venue-truth fill snapshot before any order. Its size baselines the + # cancel probe's no-fill check; its id set is the run's fill baseline — + # everything the venue reports beyond it is this run's doing (the market + # legs' settle attribution, and the final ``report.venue_fills`` re-fetch). + initial_fills = await broker.fills() + fill_ids_before: frozenset[str] = frozenset(f.fill_id for f in initial_fills) + attributed: set[str] = set(fill_ids_before) + ok = True + + # --- cancel probe: the free kill-path rehearsal --------------------------- # + probe_price = instrument.quantize_price( + mark * (_HUNDRED - probe_offset_pct) / _HUNDRED + ) + if probe_price <= 0: + raise ValueError( + f"cancel-probe price quantized to {probe_price} (mark {mark}, " + f"offset {probe_offset_pct}%): not a placeable limit price" + ) + probe_qty = _probe_qty(instrument, qty, probe_price) + fills_before = len(initial_fills) + tracked_probe: Order | None = None + try: + tracked_probe = await router.submit( + Order( + client_order_id=report.probe_cid, + instrument=instrument, + side=OrderSide.BUY, + qty=probe_qty, + type=OrderType.LIMIT, + limit_price=probe_price, + ) + ) + except _ORDER_ERRORS as exc: + report.checks.append( + CanaryCheck( + name="cancel_probe.resting", + expected=f"status=open filled_qty=0 venue_fills={fills_before}", + observed=f"error: {exc}", + passed=False, + ) + ) + ok = False + if tracked_probe is not None: + fills_now = len(await broker.fills()) + resting = ( + tracked_probe.status is OrderStatus.OPEN + and tracked_probe.filled_qty == 0 + and fills_now == fills_before + ) + report.checks.append( + CanaryCheck( + name="cancel_probe.resting", + expected=f"status=open filled_qty=0 venue_fills={fills_before}", + observed=( + f"status={tracked_probe.status.value} " + f"filled_qty={tracked_probe.filled_qty} venue_fills={fills_now}" + ), + passed=resting, + ) + ) + ok = ok and resting + + if ok: + try: + cancelled = await router.cancel(report.probe_cid) + cancel_ok = ( + cancelled.status is OrderStatus.CANCELLED and cancelled.filled_qty == 0 + ) + observed = ( + f"status={cancelled.status.value} filled_qty={cancelled.filled_qty}" + ) + except _ORDER_ERRORS as exc: + cancel_ok = False + observed = f"error: {exc}" + report.checks.append( + CanaryCheck( + name="cancel_probe.cancelled", + expected="status=cancelled filled_qty=0", + observed=observed, + passed=cancel_ok, + ) + ) + ok = ok and cancel_ok + + if ok: + balances_now = dict(await broker.balances()) + unmoved = balances_now == report.initial_balances + report.checks.append( + CanaryCheck( + name="cancel_probe.balances_unmoved", + expected=_fmt_balances(report.initial_balances), + observed=_fmt_balances(balances_now), + passed=unmoved, + ) + ) + ok = ok and unmoved + + if ok: + store.flush() + row = store.get_order(report.probe_cid) + persisted = ( + row is not None + and row.status is OrderStatus.CANCELLED + and row.filled_qty == 0 + ) + report.checks.append( + CanaryCheck( + name="cancel_probe.cancelled_persisted", + expected="store status=cancelled filled_qty=0", + observed=( + "store row missing" + if row is None + else f"store status={row.status.value} filled_qty={row.filled_qty}" + ), + passed=persisted, + ) + ) + ok = ok and persisted + + # --- idempotency probe: a duplicate client-order-id must dedup ------------ # + if ok: + open_before = len(await broker.open_orders()) + fills_before = len(await broker.fills()) + store.flush() + rows_before = len(store.orders()) + expected = ( + f"original order returned venue_open={open_before} " + f"venue_fills={fills_before} store_orders={rows_before}" + ) + try: + # A genuine duplicate submission: a NEW Order object carrying the + # probe's client-order-id, exactly as a retry would re-build it. + returned: Order | None = await router.submit( + Order( + client_order_id=report.probe_cid, + instrument=instrument, + side=OrderSide.BUY, + qty=probe_qty, + type=OrderType.LIMIT, + limit_price=probe_price, + ) + ) + error: Exception | None = None + except _ORDER_ERRORS as exc: + returned, error = None, exc + open_after = len(await broker.open_orders()) + fills_after = len(await broker.fills()) + store.flush() + rows_after = len(store.orders()) + deduped = ( + error is None + and returned is tracked_probe + and open_after == open_before + and fills_after == fills_before + and rows_after == rows_before + ) + if error is not None: + observed = f"error: {error}" + else: + identity = ( + "original order returned" + if returned is tracked_probe + else "a different order returned" + ) + observed = ( + f"{identity} venue_open={open_after} " + f"venue_fills={fills_after} store_orders={rows_after}" + ) + report.checks.append( + CanaryCheck( + name="idempotency.deduped", + expected=expected, + observed=observed, + passed=deduped, + ) + ) + ok = ok and deduped + + # --- round-trip: market buy qty, confirm; market sell qty, confirm; flat -- # + if ok: + ok = await _market_leg( + engine, report, report.buy_cid, OrderSide.BUY, attributed + ) + if ok: + position = engine.tracker.position(instrument) + net_qty = position.net_qty if position is not None else money("0") + in_tracker = net_qty == qty + report.checks.append( + CanaryCheck( + name="roundtrip.buy_in_tracker", + expected=f"net_qty={qty}", + observed=f"net_qty={net_qty}", + passed=in_tracker, + ) + ) + ok = ok and in_tracker + if ok: + ok = await _market_leg( + engine, report, report.sell_cid, OrderSide.SELL, attributed + ) + if ok: + position = engine.tracker.position(instrument) + net_qty = position.net_qty if position is not None else money("0") + flat = net_qty == 0 + report.checks.append( + CanaryCheck( + name="roundtrip.flat", + expected="net_qty=0", + observed=f"net_qty={net_qty}", + passed=flat, + ) + ) + ok = ok and flat + + # --- final snapshot + verdict (always, even on an aborted run) ------------ # + if not ok: + # Best-effort cleanup: never leave a canary order live on an abort. + # (A reducing action, not a placement — the stop-placing rule holds.) + for cid in (report.probe_cid, report.buy_cid, report.sell_cid): + live = router.get(cid) + if live is not None and not live.is_terminal: + try: + await router.cancel(cid) + except _ORDER_ERRORS: + pass # the report already fails; cleanup is best-effort + store.flush() + report.final_balances = dict(await broker.balances()) + # Re-fetch the venue's fills one last time: everything beyond the pre-run + # baseline is this run's venue-reported truth (the venue oracle's side of + # the identity). On paper these are simply the round-trip's own two fills. + final_fills = await broker.fills() + report.venue_fills = [ + fill for fill in final_fills if fill.fill_id not in fill_ids_before + ] + report.cost = -engine.perf.realised_pnl() + if ok and oracle is not None: + report.checks.extend(oracle(engine, report)) + return report + + +async def _settle_market_leg( + engine: Engine, + report: CanaryReport, + tracked: Order, + side: OrderSide, + attributed: set[str], +) -> None: + """Wait for a real venue to report the leg's executions; pump them onto the bus. + + A REST venue fills a market order asynchronously and reports the execution + only through :meth:`~trading_bot.brokers.base.Broker.fills` — keyed by its + **own** order reference, not our client-order-id (see the module + docstring). This loop polls the broker port, attributes every **new** fill + (id not in ``attributed``) on the leg's instrument and side to the leg, + and re-emits each one on the engine bus with the leg's client-order-id and + spec-carrying instrument — so the ordinary fill plumbing (fill sync → + tracker / performance / store) applies venue truth exactly as it would a + simulator fill. Returns once ``tracked`` leaves its live statuses (the + fill sync drove it to ``FILLED``) or after :data:`_VENUE_SETTLE_TIMEOUT_S` + — the caller's checks then record exactly what was (not) observed. + + Parameters + ---------- + engine : Engine + The canary engine (broker port + bus). + report : CanaryReport + The run's report — supplies the instrument the fills must match. + tracked : Order + The just-submitted, still-live tracked market order. + side : OrderSide + The leg's side; a venue fill on the other side is never attributed. + attributed : set of str + Every fill id already accounted for (the pre-run baseline plus fills + attributed by earlier legs). Mutated in place as fills are attributed. + + """ + deadline = time.monotonic() + _VENUE_SETTLE_TIMEOUT_S + while tracked.status in _LEG_SETTLING: + for fill in await engine.broker.fills(): + if fill.fill_id in attributed: + continue + if fill.instrument.symbol != report.instrument.symbol: + continue + if fill.side is not side: + continue + attributed.add(fill.fill_id) + # Attribute the venue fill to the leg: same id/qty/price/fee/ts, + # our client-order-id and spec-carrying instrument — the exact + # correlation this sequential, dedicated run just established. + engine.bus.emit( + FillEvent( + replace( + fill, + client_order_id=tracked.client_order_id, + instrument=report.instrument, + ) + ) + ) + if tracked.status not in _LEG_SETTLING or time.monotonic() >= deadline: + break + await asyncio.sleep(_VENUE_SETTLE_POLL_S) + + +async def _market_leg( + engine: Engine, + report: CanaryReport, + cid: str, + side: OrderSide, + attributed: set[str], +) -> bool: + """Place one market leg of the round-trip and confirm it filled + persisted. + + Submits a MARKET order for ``report.qty`` on ``side`` under ``cid``, lets a + real venue's executions settle (:func:`_settle_market_leg` — a no-op on + paper, whose broker fills synchronously), then appends two checks: the + tracked order is ``FILLED`` with ``filled_qty == qty``, and the **store** + row agrees. Returns whether both held (the caller stops placing on + ``False``). A submission error becomes a failed check, never an exception + out of the scenario. + """ + leg = "buy" if side is OrderSide.BUY else "sell" + tracked: Order | None = None + try: + tracked = await engine.router.submit( + Order( + client_order_id=cid, + instrument=report.instrument, + side=side, + qty=report.qty, + type=OrderType.MARKET, + ) + ) + except _ORDER_ERRORS as exc: + report.checks.append( + CanaryCheck( + name=f"roundtrip.{leg}_filled", + expected=f"status=filled filled_qty={report.qty}", + observed=f"error: {exc}", + passed=False, + ) + ) + return False + if tracked.status in _LEG_SETTLING: + # A real venue reports the market execution asynchronously via + # broker.fills(); paper never gets here (it filled synchronously). + await _settle_market_leg(engine, report, tracked, side, attributed) + filled = tracked.status is OrderStatus.FILLED and tracked.filled_qty == report.qty + report.checks.append( + CanaryCheck( + name=f"roundtrip.{leg}_filled", + expected=f"status=filled filled_qty={report.qty}", + observed=f"status={tracked.status.value} filled_qty={tracked.filled_qty}", + passed=filled, + ) + ) + if not filled: + return False + store = engine.store + assert store is not None # guarded at run_canary entry + store.flush() + row = store.get_order(cid) + in_store = ( + row is not None + and row.status is OrderStatus.FILLED + and row.filled_qty == report.qty + ) + report.checks.append( + CanaryCheck( + name=f"roundtrip.{leg}_in_store", + expected=f"store status=filled filled_qty={report.qty}", + observed=( + "store row missing" + if row is None + else f"store status={row.status.value} filled_qty={row.filled_qty}" + ), + passed=in_store, + ) + ) + return in_store diff --git a/trading_bot/application/config.py b/trading_bot/application/config.py index 88c64164..53743853 100644 --- a/trading_bot/application/config.py +++ b/trading_bot/application/config.py @@ -42,6 +42,15 @@ :class:`~trading_bot.brokers.paper.PaperBroker` construction is unaffected — its own constructor default stays permissive (``strict=False``) for existing callers/tests. +* **One display currency, converted server-side.** ``display_currency`` + (default ``"USD"``) is the single unit the API's ``*_display`` money fields + are converted into (:mod:`trading_bot.application.display_ccy`) so every + consumer reads the same converted figure — never re-derived per client. + ``display_currency_overrides`` relabels specific exchanges' rows (e.g. + ``{"binance": "USDT"}``); ``conversion_rates`` declares one unit of a quote + currency's worth in ``display_currency`` (the display currency itself is an + implied identity — no entry needed). A quote with no declared rate converts + to ``None`` (JSON ``null``) rather than a guessed figure. This module is the only place the application layer reads YAML; everything downstream consumes a validated :class:`AppConfig`. @@ -91,12 +100,23 @@ class BrokerConfig(BaseModel): endpoint — it cannot reach mainnet — so it does **not** require the ``live_enabled`` opt-in (it still needs testnet credentials). Ignored in paper mode (the simulator). A venue with no testnet (``"kraken"``) raises. + symbols : list of str, optional + The pairs this venue's **per-symbol** private endpoints are queried + for. Binance has no account-wide trade history (``myTrades`` is + per-symbol), so its adapter serves + :meth:`~trading_bot.brokers.base.Broker.fills` only for an explicit + symbol set — without one, ``fills()`` (and so reconciliation and the + canary's venue oracle) raises. Entries are parsed by the factory with + the venue's own parser (``"BTC/USDT"``, ``"BTCUSDT"``, ...); an + unparseable entry refuses at build time. Ignored by venues with + account-wide fills (Kraken) and by paper mode. Empty by default. """ name: str exchange: str testnet: bool = False + symbols: list[str] = Field(default_factory=list) @field_validator("name", "exchange") @classmethod @@ -106,6 +126,15 @@ def _non_empty(cls, v: str) -> str: raise ValueError("must be a non-empty string") return v + @field_validator("symbols") + @classmethod + def _non_empty_symbols(cls, v: list[str]) -> list[str]: + """Reject blank ``symbols`` entries (whitespace-only too).""" + for entry in v: + if not entry or not entry.strip(): + raise ValueError("symbols entries must be non-empty strings") + return v + class DataSourceConfig(BaseModel): """Where a strategy's bars come from — a dccd OHLC dataset. @@ -684,6 +713,34 @@ class AppConfig(BaseModel): everything regardless of venue minimums). Ignored outside paper mode; direct :class:`~trading_bot.brokers.paper.PaperBroker` construction is unaffected — its constructor default stays ``strict=False``. + paper_starting_balances : dict of str to Decimal, optional + Initial free balances (canonical asset code -> amount) the factory-built + :class:`~trading_bot.brokers.paper.PaperBroker` starts from — the + explicit-funding seam (paper brokers start unfunded by default; the + simulator never gates on funding, balances may go negative). Parsed + exactly from a YAML scalar (``str``/``int``) without touching ``float``. + Ignored outside paper mode. Empty by default (unfunded, the historical + behaviour). + display_currency : str, optional + The single currency the API's ``*_display`` money fields are converted + into (:mod:`trading_bot.application.display_ccy`) — the "one truth for + every consumer" numeraire. Defaults to ``"USD"``. Must be non-empty. + display_currency_overrides : dict of str to str, optional + Per-exchange relabelling of the ``display_currency`` **field** shown on + that exchange's rows (e.g. ``{"binance": "USDT"}`` — Binance trades + USDT-quoted). Does not change what currency ``*_display`` amounts are + *converted into* (still the single global ``display_currency`` above); + keep an override numerically truthful by pairing it with an identity + (``"1"``) ``conversion_rates`` entry for that currency. Empty by + default (every exchange labelled with the global default). + conversion_rates : dict of str to Decimal, optional + One unit of a quote currency's worth in ``display_currency`` (e.g. + ``{"USDT": "1", "EUR": "1.08"}``); the display currency itself needs no + entry (an implied identity rate). Parsed exactly from a YAML scalar + (``str``/``int``) without touching ``float``; every declared rate must + be **strictly positive**. A quote with no declared rate converts to + ``None`` (JSON ``null``) rather than a guessed figure — never a + fabricated conversion. Empty by default (no cross-quote conversion). brokers : list of BrokerConfig, optional The brokers to wire up. Empty by default. strategies : list of StrategyConfig, optional @@ -719,7 +776,11 @@ class AppConfig(BaseModel): mode: Literal["paper", "live"] = "paper" live_enabled: bool = False paper_strict: bool = True + paper_starting_balances: dict[str, Decimal] = Field(default_factory=dict) starting_capital: Decimal = Field(default_factory=lambda: money("100000")) + display_currency: str = "USD" + display_currency_overrides: dict[str, str] = Field(default_factory=dict) + conversion_rates: dict[str, Decimal] = Field(default_factory=dict) brokers: list[BrokerConfig] = Field(default_factory=list) strategies: list[StrategyConfig] = Field(default_factory=list) portfolios: list[PortfolioStrategyConfig] = Field(default_factory=list) @@ -727,6 +788,25 @@ class AppConfig(BaseModel): storage: StorageConfig = Field(default_factory=StorageConfig) ui: UIConfig = Field(default_factory=UIConfig) + @field_validator("display_currency") + @classmethod + def _non_empty_display_currency(cls, v: str) -> str: + """Reject a blank ``display_currency`` (whitespace-only too).""" + if not v or not v.strip(): + raise ValueError("display_currency must be a non-empty string") + return v + + @field_validator("conversion_rates") + @classmethod + def _positive_rates(cls, v: dict[str, Decimal]) -> dict[str, Decimal]: + """Reject a non-positive declared ``conversion_rates`` entry (zero too).""" + for ccy, rate in v.items(): + if rate <= 0: + raise ValueError( + f"conversion_rates[{ccy!r}] must be positive, got {rate}" + ) + return v + @field_validator("starting_capital") @classmethod def _positive_capital(cls, v: Decimal) -> Decimal: diff --git a/trading_bot/application/display_ccy.py b/trading_bot/application/display_ccy.py new file mode 100644 index 00000000..aaa16efe --- /dev/null +++ b/trading_bot/application/display_ccy.py @@ -0,0 +1,131 @@ +"""Display-currency resolution + conversion — one converted truth per amount. + +The api-completeness epic's leaf 03 (pinned decision, ``doc/dev/plans/ +api-completeness/00-plan.md``): the engine trades in whatever quote currency +each instrument declares (Binance ``USDT``, Kraken ``USD``, ...), but a +dashboard reading *across* venues needs a single unit to read totals in. This +module is the one place that answers "what currency, and what number" — the +API's row/aggregate serializers (:mod:`trading_bot.interfaces.api.app`) both +call through here so every consumer agrees (never two different converted +figures for the same amount). + +Two independent questions, two functions +----------------------------------------- +* :func:`resolve_currency` — **the label**: which currency a given exchange's + rows are *presented* under. The global :attr:`~trading_bot.application. + config.AppConfig.display_currency` (e.g. ``"USD"``), unless + :attr:`~trading_bot.application.config.AppConfig.display_currency_overrides` + names a different currency for that specific exchange (e.g. + ``{"binance": "USDT"}`` — Binance trades USDT-quoted, so its rows are + labelled ``"USDT"`` rather than converted-and-labelled ``"USD"``). +* :func:`convert` — **the number**: every ``*_display`` amount is expressed in + the *one* global ``display_currency`` (the single numeraire the "one truth + for every consumer" decision calls for) — never in a row's overridden label. + ``conversion_rates`` declares, per source quote currency, how many units of + the global ``display_currency`` one unit of that quote is worth; the + ``display_currency`` itself needs no entry (identity, rate ``1`` implied). + + This means an override is only numerically inert when its named currency is + *itself* pegged 1:1 to the global ``display_currency`` (the common case: a + dollar-stablecoin quote against a ``"USD"`` global default, declared with an + explicit identity rate, e.g. ``conversion_rates={"USDT": "1"}``) — the label + reads ``"USDT"`` and the converted figure is numerically unchanged from the + native one. Overriding a row's label to a currency whose rate is *not* ``1`` + would still convert the underlying amount into the global + ``display_currency`` — the label and the number would then disagree, so + operators should only override a label alongside a rate that keeps it + truthful. + +Never guess a rate (pinned) +---------------------------- +:func:`convert` returns ``None`` — never a fabricated number — whenever the +source currency is unknown (a mixed-quote aggregate, ``quote is None``) or has +no declared rate. The API renders that as JSON ``null``, alongside the +untouched native figure, rather than silently omitting the conversion or +inventing one. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from trading_bot.domain.money import Money, mul + +if TYPE_CHECKING: + from trading_bot.application.config import AppConfig + +__all__ = ["resolve_currency", "convert"] + + +def resolve_currency(exchange: str | None, config: AppConfig) -> str: + """The currency an exchange's rows are labelled under (the ``display_currency`` field). + + Parameters + ---------- + exchange : str or None + The row's venue key (e.g. ``"binance"``). ``None`` for an aggregate row + that spans every exchange (e.g. a KPI ``level="total"`` row) — there is + no single exchange to look up an override for, so it falls back to the + global default exactly like an unlisted exchange would. + config : AppConfig + The running config; consulted for + :attr:`~trading_bot.application.config.AppConfig.display_currency_overrides` + and the :attr:`~trading_bot.application.config.AppConfig.display_currency` + fallback. + + Returns + ------- + str + ``config.display_currency_overrides[exchange]`` when set, else + ``config.display_currency``. + + """ + if exchange is None: + return config.display_currency + return config.display_currency_overrides.get(exchange, config.display_currency) + + +def convert( + amount: Money | None, from_ccy: str | None, config: AppConfig +) -> Money | None: + """Convert ``amount`` (in ``from_ccy``) into the global ``display_currency``. + + Exact :class:`~decimal.Decimal` arithmetic throughout — never ``float``. + Never guesses: an unconvertible amount renders ``None`` (JSON ``null``), + not a fabricated figure. + + Parameters + ---------- + amount : Money or None + The native-quote figure to convert. ``None`` (e.g. an unmarked + position's ``value``/``unrealised``) passes through as ``None``. + from_ccy : str or None + The currency ``amount`` is denominated in (a position row's + ``fee_ccy``, a strategy/KPI row's ``quote``). ``None`` when the source + is unknown or ambiguous (a KPI/strategy aggregate folding units that + mix quote currencies — the dashboard's "mixed") — never guessed, so + conversion is refused. + config : AppConfig + The running config; consulted for + :attr:`~trading_bot.application.config.AppConfig.display_currency` (the + conversion target, and the implied-identity source) and + :attr:`~trading_bot.application.config.AppConfig.conversion_rates` (one + unit of ``from_ccy`` expressed in ``display_currency``). + + Returns + ------- + Money or None + ``amount`` unchanged when ``from_ccy`` already *is* the display + currency (identity, no rate needed); ``amount * rate`` when a rate is + declared for ``from_ccy``; ``None`` when ``amount`` or ``from_ccy`` is + ``None``, or no rate is declared for ``from_ccy``. + + """ + if amount is None or from_ccy is None: + return None + if from_ccy == config.display_currency: + return amount + rate = config.conversion_rates.get(from_ccy) + if rate is None: + return None + return mul(amount, rate) diff --git a/trading_bot/application/mark_cache.py b/trading_bot/application/mark_cache.py new file mode 100644 index 00000000..987758ea --- /dev/null +++ b/trading_bot/application/mark_cache.py @@ -0,0 +1,119 @@ +"""The engine's mark cache — the last close each runner saw, with its as-of. + +Part of the ``api-completeness`` epic's pinned mark policy (see +``doc/dev/plans/api-completeness/00-plan.md``): a mark is **the last dccd bar +close** the strategy evaluated on — never a fresh fetch. The API layer must +serialize a position's current mark without any I/O of its own, so every +rebalance publishes its per-symbol closes here, in-process, for the API +thread-context to read back later (leaf 02 wires the read side; this leaf only +publishes). + +Sharing discipline +------------------- +One :class:`MarkCache` per :class:`~trading_bot.application.service_factory +.Engine` (so, under the supervisor, one per managed unit — mirroring +:attr:`~trading_bot.application.service_factory.Engine.spec_resolver`). It is +written from the runner's async loop (:meth:`~trading_bot.application +.portfolio_runner.PortfolioRunner.rebalance`) and read from elsewhere in the +same process — exactly the sharing shape the runner's own ``last_eval_ms`` / +``last_asof_ms`` already have (see +:attr:`~trading_bot.application.portfolio_runner.PortfolioRunner.last_asof_ms`, +read directly by +:meth:`~trading_bot.application.supervisor.StrategySupervisor._status_of`): +a plain attribute — here, a plain ``dict`` — with **no lock**. This is safe for +the same reason: the daemon runs everything (the scheduler's ticks, the API +handlers) cooperatively on one asyncio event loop, so a read never interleaves +with a write mid-mutation. Do not add locking here without first revisiting +that discipline for the whole engine. +""" + +from __future__ import annotations + +# Built-in +from dataclasses import dataclass +from typing import Literal + +# Local +from trading_bot.domain.instrument import Symbol +from trading_bot.domain.money import Money + +__all__ = ["Mark", "MarkCache"] + + +@dataclass(frozen=True, slots=True) +class Mark: + """One symbol's last-known price, timestamped and sourced. + + Attributes + ---------- + price : Money + The exact (``Decimal``) price — a bar close on this path. + asof_ms : int + Milliseconds since the Unix epoch (UTC) of the data the price came + from — **not** wall-clock time the cache was written. The UI must + never mistake a stale mark for a live price, so this timestamp is + always carried alongside the price (never optional). + source : {"bar_close", "last_fill"} + How the mark was derived. Always ``"bar_close"`` on the publish path + in this module (:meth:`MarkCache.update`); ``"last_fill"`` is the + read-side fallback a consumer applies when no bar-close mark exists + yet (leaf 02), never written here. + + """ + + price: Money + asof_ms: int + source: Literal["bar_close", "last_fill"] + + +class MarkCache: + """Per-engine, latest-write-wins cache of every symbol's last mark. + + One instance per :class:`~trading_bot.application.service_factory.Engine` + (default-constructed there, like + :class:`~trading_bot.application.instrument_specs.InstrumentSpecResolver`). + A plain ``dict`` keyed by :class:`~trading_bot.domain.instrument.Symbol` — + see the module docstring for why no lock is needed. + """ + + def __init__(self) -> None: + self._marks: dict[Symbol, Mark] = {} + + def update(self, symbol: Symbol, price: Money, asof_ms: int) -> None: + """Publish ``symbol``'s latest close, overwriting any prior mark. + + Parameters + ---------- + symbol : Symbol + The traded pair this close belongs to. + price : Money + The exact (``Decimal``) close. + asof_ms : int + The as-of (ms since epoch, UTC) of the bar this close came from — + the same value the caller stamps on its signals for this tick. + + Notes + ----- + The source is always ``"bar_close"``: this is the runner's publish + path (the *only* writer of this cache); the ``"last_fill"`` fallback + is a read-side concern for a consumer with no cached mark yet. + + """ + self._marks[symbol] = Mark(price=price, asof_ms=asof_ms, source="bar_close") + + def get(self, symbol: Symbol) -> Mark | None: + """The symbol's latest published mark, or ``None`` if never published.""" + return self._marks.get(symbol) + + def all(self) -> dict[Symbol, Mark]: + """A snapshot of every symbol's latest mark. + + Returns + ------- + dict of Symbol to Mark + A fresh copy of the cache's map (the :class:`Mark` values are + frozen and shared) — mutating the returned dict never affects the + cache. + + """ + return dict(self._marks) diff --git a/trading_bot/application/portfolio_runner.py b/trading_bot/application/portfolio_runner.py index bbba3c38..9a705085 100644 --- a/trading_bot/application/portfolio_runner.py +++ b/trading_bot/application/portfolio_runner.py @@ -133,6 +133,7 @@ import polars as pl from trading_bot.application.instrument_specs import InstrumentSpecResolver + from trading_bot.application.mark_cache import MarkCache from trading_bot.application.order_router import OrderRouter from trading_bot.application.portfolio import PortfolioStrategy from trading_bot.application.position_tracker import PositionTracker @@ -296,6 +297,16 @@ class PortfolioRunner: the binding minimum, in ``(0, 1]`` (see :class:`~trading_bot.application.config.PortfolioStrategyConfig`). Defaults to ``0.5``. Only consulted when ``spec_resolver`` is given. + mark_cache : MarkCache or None, optional + The engine's per-symbol mark cache (see + :class:`~trading_bot.application.mark_cache.MarkCache`, + :attr:`~trading_bot.application.service_factory.Engine.mark_cache`). + When given, every rebalance publishes each traded symbol's latest + close and this tick's as-of into it — the same values the tick sizes + against and stamps on its signals — so the API layer can read a + strategy's current mark without any I/O of its own (see + ``doc/dev/plans/api-completeness/00-plan.md``). ``None`` (default) + publishes nothing (the legacy path, byte-for-byte unchanged). Examples -------- @@ -318,6 +329,7 @@ def __init__( spec_resolver: InstrumentSpecResolver | None = None, exchange: str | None = None, min_order_ratio: Money = money("0.5"), + mark_cache: MarkCache | None = None, ) -> None: if spec_resolver is not None and (exchange is None or not exchange.strip()): raise ValueError( @@ -340,6 +352,7 @@ def __init__( self._spec_resolver = spec_resolver self._exchange = exchange self._min_order_ratio = min_order_ratio + self._mark_cache = mark_cache # One degraded-resolver warning per runner (= unit) lifetime — see the # module docstring's venue-minimum section. self._spec_degraded_warned = False @@ -479,6 +492,13 @@ async def rebalance(self, frames: Mapping[Symbol, pl.DataFrame]) -> RebalanceRes asof = self._derive_asof_ms(frames) prices = self._latest_closes(frames) + if self._mark_cache is not None: + # Publish this tick's marks before anything else touches ``prices``: + # the same closes the sizing below reads and the same ``asof`` this + # tick's signals are stamped with (see MarkCache's module docstring + # for the sharing discipline — a plain dict, latest-write-wins). + for symbol, close in prices.items(): + self._mark_cache.update(symbol, close, asof) weights = self._strategy.signal_fn(asof, frames) # Universe-complete: cover every coin, defaulting an omitted one to a diff --git a/trading_bot/application/run_app.py b/trading_bot/application/run_app.py index 7795ddf7..eefbdd4d 100644 --- a/trading_bot/application/run_app.py +++ b/trading_bot/application/run_app.py @@ -618,7 +618,10 @@ def build_portfolio_runners( :class:`~trading_bot.application.instrument_specs.InstrumentSpecResolver` is threaded in (with the portfolio's ``venue`` and ``min_order_ratio``) so every rebalance leg is prepared against the venue's real minimums - (:func:`~trading_bot.application.order_prep.prepare_leg`). + (:func:`~trading_bot.application.order_prep.prepare_leg`). The engine's + :class:`~trading_bot.application.mark_cache.MarkCache` is threaded in too, + so every rebalance publishes its per-symbol closes for the API layer to + read without any I/O of its own. The dccd ``client`` is threaded into every :class:`PortfolioFeed` so the build is offline-testable. A daily portfolio reading a 1-minute store should inject @@ -716,6 +719,11 @@ def build_portfolio_runners( spec_resolver=engine.spec_resolver, exchange=portfolio_cfg.venue, min_order_ratio=money(portfolio_cfg.min_order_ratio), + # Mark-cache publish: the engine's per-symbol cache (one per unit, + # like spec_resolver above) so every rebalance's closes are readable + # by the API layer with no fresh I/O — see + # `trading_bot.application.mark_cache`. + mark_cache=engine.mark_cache, ) runners.append(runner) return runners diff --git a/trading_bot/application/service_factory.py b/trading_bot/application/service_factory.py index 5ea81ef6..0c430757 100644 --- a/trading_bot/application/service_factory.py +++ b/trading_bot/application/service_factory.py @@ -58,6 +58,7 @@ from trading_bot.application.config import AppConfig, BrokerConfig from trading_bot.application.events import EventBus from trading_bot.application.instrument_specs import InstrumentSpecResolver +from trading_bot.application.mark_cache import MarkCache from trading_bot.application.order_fill_sync import OrderFillSync from trading_bot.application.order_router import OrderRouter from trading_bot.application.performance_service import PerformanceService @@ -68,6 +69,11 @@ from trading_bot.brokers.kraken import KrakenBroker from trading_bot.brokers.paper import PaperBroker from trading_bot.domain.errors import BrokerError, LiveTradingNotEnabled +from trading_bot.domain.instrument import ( + Symbol, + parse_binance_symbol, + parse_kraken_pair, +) from trading_bot.domain.money import Money from trading_bot.storage.sqlite_store import SqliteStore @@ -149,6 +155,13 @@ class Engine: exactly as long as the engine does. Threaded into the portfolio runners (:func:`~trading_bot.application.run_app.build_portfolio_runners`) so every rebalance leg is prepared against the venue's real minimums. + mark_cache : MarkCache + The engine's per-symbol mark cache — one per engine (so, under the + supervisor, one per unit), living exactly as long as the engine does. + The portfolio runner publishes every rebalance's per-symbol closes here + (:func:`~trading_bot.application.run_app.build_portfolio_runners`); the + API layer will read it (leaf 02) without any I/O of its own — see + :mod:`~trading_bot.application.mark_cache`. """ @@ -166,6 +179,7 @@ class Engine: spec_resolver: InstrumentSpecResolver = field( default_factory=InstrumentSpecResolver ) + mark_cache: MarkCache = field(default_factory=MarkCache) def build_engine( @@ -265,6 +279,11 @@ def build_engine( # die with the engine. Constructed here, the single wiring point, and # threaded to the portfolio runners by build_portfolio_runners. spec_resolver=InstrumentSpecResolver(), + # One mark cache per engine (= per supervised unit), same lifetime + # discipline as spec_resolver above: constructed here and threaded to + # the portfolio runners by build_portfolio_runners, which publish every + # rebalance's per-symbol closes into it. + mark_cache=MarkCache(), ) @@ -361,6 +380,10 @@ def _build_broker(config: AppConfig, bus: EventBus) -> Broker: event_bus=bus, clock=lambda: int(time.time() * 1000), strict=config.paper_strict, + # Explicit funding seam (paper simulators start unfunded): thread + # `paper_starting_balances` so a factory-built paper engine — e.g. + # the canary's — can be funded declaratively. Empty by default. + starting_balances=config.paper_starting_balances, ) # Testnet path: a venue's sandbox (paper money on the real testnet venue). The @@ -370,7 +393,7 @@ def _build_broker(config: AppConfig, bus: EventBus) -> Broker: # broker is what opts in; a venue with no testnet raises. first: BrokerConfig | None = config.brokers[0] if config.brokers else None if first is not None and first.testnet: - broker = _build_testnet_venue(venue) + broker = _build_testnet_venue(venue, symbols=_broker_symbols(first, venue)) if not broker.has_credentials: raise BrokerError( f"testnet for venue {venue!r} requires credentials; set the " @@ -394,7 +417,7 @@ def _build_broker(config: AppConfig, bus: EventBus) -> Broker: # Opt-in is set: build the real adapter, but only if it can actually trade. # Never silently downgrade to paper. if venue in _LIVE_VENUES: - broker = _build_live_venue(venue) + broker = _build_live_venue(venue, symbols=_broker_symbols(first, venue)) if not broker.has_credentials: raise BrokerError( f"live mode requires credentials for venue {venue!r}; " @@ -462,8 +485,39 @@ def _selected_venue(config: AppConfig) -> str: return first.exchange.lower() -def _build_live_venue(venue: str) -> _LiveBroker: - """Construct the live adapter for ``venue`` (reads credentials from env).""" +def _broker_symbols(first: BrokerConfig | None, venue: str) -> tuple[Symbol, ...]: + """Parse the selected broker's declared ``symbols`` into canonical pairs. + + ``BrokerConfig.symbols`` entries are free-form venue pair strings + (``"BTC/USDT"``, ``"BTCUSDT"``, a Kraken legacy pair, ...); each is parsed + with the venue's own parser so the adapter receives canonical + :class:`~trading_bot.domain.instrument.Symbol` objects. An unparseable + entry refuses at build time with a clear :class:`BrokerError` — a broker + whose per-symbol endpoints would silently query nothing must not build. + """ + if first is None or not first.symbols: + return () + parse = parse_kraken_pair if venue == "kraken" else parse_binance_symbol + parsed: list[Symbol] = [] + for raw in first.symbols: + try: + parsed.append(parse(raw)) + except ValueError as exc: + raise BrokerError( + f"broker {first.name!r}: cannot parse symbols entry {raw!r} " + f"for venue {venue!r}: {exc}" + ) from exc + return tuple(parsed) + + +def _build_live_venue(venue: str, *, symbols: tuple[Symbol, ...] = ()) -> _LiveBroker: + """Construct the live adapter for ``venue`` (reads credentials from env). + + ``symbols`` (from ``BrokerConfig.symbols``, parsed by + :func:`_broker_symbols`) is threaded to venues whose fills are + **per-symbol** (Binance ``myTrades``); Kraken's trade history is + account-wide, so its adapter takes none. + """ if venue == "kraken": # KrakenBroker reads KRAKEN_API_KEY / KRAKEN_API_SECRET from the # environment; ``has_credentials`` reports whether both are present. @@ -472,7 +526,7 @@ def _build_live_venue(venue: str) -> _LiveBroker: # BinanceBroker reads BINANCE_API_KEY / BINANCE_API_SECRET (and the # optional BINANCE_API_BASE testnet toggle) from the environment; # ``has_credentials`` reports whether both key + secret are present. - return BinanceBroker() + return BinanceBroker(symbols=symbols) # Unreachable: callers gate on ``_LIVE_VENUES`` first. Defensive only. raise BrokerError(f"no live adapter for venue {venue!r}") @@ -496,7 +550,9 @@ def _binance_testnet_credentials() -> tuple[str, str]: return key, secret -def _build_testnet_venue(venue: str) -> _LiveBroker: +def _build_testnet_venue( + venue: str, *, symbols: tuple[Symbol, ...] = () +) -> _LiveBroker: """Construct a venue's **testnet** adapter, hard-pinned to its sandbox URL. Only venues in :data:`_TESTNET_VENUES` have a testnet. The base URL is forced @@ -504,8 +560,9 @@ def _build_testnet_venue(venue: str) -> _LiveBroker: env value is overridden) — the adapter can therefore never reach mainnet, which is why the caller skips the ``live_enabled`` opt-in for it. Credentials are the venue's **testnet** keys (``BINANCE_TESTNET_*``, falling back to ``BINANCE_*``); - see :func:`_binance_testnet_credentials`. A venue with no testnet (e.g. Kraken, - which has no public spot sandbox) raises. + see :func:`_binance_testnet_credentials`. ``symbols`` is threaded exactly as in + :func:`_build_live_venue` (Binance fills are per-symbol). A venue with no + testnet (e.g. Kraken, which has no public spot sandbox) raises. """ if venue == "binance": # Hard-pin the testnet base URL (explicit arg overrides the env default), @@ -513,7 +570,12 @@ def _build_testnet_venue(venue: str) -> _LiveBroker: # feed it the *testnet* credentials (not the mainnet key, which testnet # rejects with -2015). key, secret = _binance_testnet_credentials() - return BinanceBroker(api_key=key, api_secret=secret, base_url=TESTNET_API_BASE) + return BinanceBroker( + api_key=key, + api_secret=secret, + base_url=TESTNET_API_BASE, + symbols=symbols, + ) raise BrokerError( f"venue {venue!r} has no testnet/sandbox; testnet is available for " f"{sorted(_TESTNET_VENUES)!r} only (Kraken has no public spot testnet — " diff --git a/trading_bot/application/strategy_runner.py b/trading_bot/application/strategy_runner.py index 427d008a..899ea33f 100644 --- a/trading_bot/application/strategy_runner.py +++ b/trading_bot/application/strategy_runner.py @@ -68,6 +68,22 @@ This module lives in the application layer: it imports the pure domain and the sibling use-cases, holds money as :class:`~decimal.Decimal` end to end, and performs no I/O of its own (the router/broker do). + +Follow-up: no mark-cache publish yet (carried into the ADR) +------------------------------------------------------------- +The ``api-completeness`` epic's mark cache (see +``doc/dev/plans/api-completeness/00-plan.md``, +:mod:`~trading_bot.application.mark_cache`) is published by +:meth:`~trading_bot.application.portfolio_runner.PortfolioRunner.rebalance` +right where it already reads every coin's latest close. This runner has no +equally direct seam: :meth:`step` only reads a bar's close conditionally (when +a ``capital_provider`` is wired, inside :meth:`_reference_qty`) and never +computes this step's as-of itself — that derivation happens afterwards, in +:meth:`step_latest`, over the same ``bars`` :meth:`step` already consumed. +Forcing a publish here would mean re-deriving both values redundantly on every +call, including the plain backtest :meth:`run` path. Left as a follow-up +(today's dashboard units are portfolio units, per the epic's plan) rather than +threading a mismatched seam. """ from __future__ import annotations diff --git a/trading_bot/application/supervisor.py b/trading_bot/application/supervisor.py index 2627c5c2..13a9c56f 100644 --- a/trading_bot/application/supervisor.py +++ b/trading_bot/application/supervisor.py @@ -44,6 +44,7 @@ from trading_bot.application.accounting import Violation, check_book from trading_bot.application.capital_service import CapitalService from trading_bot.application.events import EventBus, LogEvent +from trading_bot.application.mark_cache import Mark from trading_bot.application.pnl_series import by_mode, equity_series from trading_bot.application.reconcile import reconcile from trading_bot.application.run_app import build_portfolio_runners, build_runners @@ -54,6 +55,7 @@ contributed_capital, ) from trading_bot.domain.errors import ( + BrokerError, ConfigError, LiveCapitalOpsDeferred, LiveTradingNotEnabled, @@ -86,6 +88,7 @@ from trading_bot.storage.sqlite_store import StoredFill __all__ = [ + "BalanceRow", "FillRow", "KpiLevel", "KpiRow", @@ -251,6 +254,28 @@ class PositionRow: The position's realised PnL, net of fees (exact). fees_paid : Money The position's cumulative fees (exact). + mark : Money or None + The instrument's current mark under the pinned policy (see + :meth:`StrategySupervisor._mark_of`): the engine's + :class:`~trading_bot.application.mark_cache.MarkCache` bar close when one + exists, else the last-known own-fill price (``mark_source`` says which). + ``None`` when neither exists. + mark_asof_ts : int or None + Epoch ms the mark's data is *of* — the dccd bar's asof for a + ``"bar_close"`` mark, the fill's ``ts`` for a ``"last_fill"`` one. Always + carried alongside a non-``None`` ``mark`` (never a stale price with no + timestamp); ``None`` iff ``mark`` is ``None``. + mark_source : {"bar_close", "last_fill"} or None + How ``mark`` was derived; ``None`` iff ``mark`` is ``None``. + value : Money or None + ``mark * |net_qty|`` — the exposure's current worth in quote terms. + ``None`` when ``mark`` is ``None``. + unrealised : Money or None + ``(mark - avg_entry_price) * net_qty`` — sign-correct for a short (mirrors + :meth:`StrategySupervisor._unrealised_of`'s arithmetic). ``None`` when + ``mark`` or ``avg_entry_price`` is ``None``. + fee_ccy : str + The instrument's quote asset (fees are charged in quote terms). """ @@ -262,6 +287,12 @@ class PositionRow: avg_entry_price: Money | None realised_pnl: Money fees_paid: Money + mark: Money | None = None + mark_asof_ts: int | None = None + mark_source: Literal["bar_close", "last_fill"] | None = None + value: Money | None = None + unrealised: Money | None = None + fee_ccy: str = "" @dataclass(frozen=True, slots=True) @@ -330,6 +361,48 @@ class FillRow: fill: Fill +@dataclass(frozen=True, slots=True) +class BalanceRow: + """One running unit's broker-reported free balances, tagged strategy + venue. + + The supervisor-level view of :meth:`~trading_bot.brokers.base.Broker.balances` + for a single unit — the venue's *own* view of what it holds, as opposed to the + locally-tracked :class:`PositionRow` exposure. This is the prerequisite seam + for the positions<->balances cross-check (an accounting-guardrail extension: + comparing the tracker's net exposure against what the broker actually reports) + and the canary-roundtrip live oracle (roadmap #6, a deterministic round-trip + strategy that proves the whole chain against a real venue) — both need the + broker's own balances, not just what the engine believes it holds. + + Attributes + ---------- + strategy : str + The managed unit this balance snapshot belongs to. + exchange : str + The venue the unit runs on. + mode : StrategyMode + The unit's deployment mode (``"paper"``, ``"testnet"`` or ``"live"``) — + which broker instance ``balances`` came from. + balances : dict of str to Money + Free balance per canonical asset code (exact :class:`~decimal.Decimal`), + as reported by :meth:`~trading_bot.brokers.base.Broker.balances`. Empty + when ``error`` is set (the broker call failed). + error : str or None + ``None`` on a successful fetch. Set to the broker's error message when + :meth:`~trading_bot.brokers.base.Broker.balances` raised + :class:`~trading_bot.domain.errors.BrokerError` — the row still surfaces + (never a 500) so the dashboard can poll safely through a transient venue + outage. + + """ + + strategy: str + exchange: str + mode: StrategyMode + balances: dict[str, Money] + error: str | None = None + + @dataclass(frozen=True, slots=True) class KpiRow: """A realised-PnL / fees / ratios view at one aggregation level. @@ -1339,6 +1412,33 @@ def _mark_map(fills: list[Fill]) -> dict[Instrument, Money]: marks[fill.instrument] = fill.price return marks + @staticmethod + def _mark_of(unit: _Unit, instrument: Instrument, fills: list[Fill]) -> Mark | None: + """``instrument``'s current mark, under the pinned policy (api-completeness). + + Prefers the running unit's :class:`~trading_bot.application.mark_cache. + MarkCache` (the last dccd bar close the strategy evaluated on — see + ``doc/dev/plans/api-completeness/00-plan.md``); when the cache holds + nothing for the instrument's :class:`~trading_bot.domain.instrument.Symbol` + (e.g. a symbol the runner has not ticked since restart), falls back to the + instrument's last-known price in ``fills`` (``mark_source="last_fill"``, + timestamped with that fill's ``ts``). ``None`` when neither exists. This is + the one mark policy shared by :meth:`positions` (the per-row mark) and + :meth:`_unrealised_of` (the strategy-aggregate mark) so the two always + agree. + """ + if unit.running and unit.engine is not None: + cached = unit.engine.mark_cache.get(instrument.symbol) + if cached is not None: + return cached + last_fill: Fill | None = None + for fill in fills: + if fill.instrument == instrument: + last_fill = fill + if last_fill is None: + return None + return Mark(price=last_fill.price, asof_ms=last_fill.ts, source="last_fill") + @staticmethod def _committed_value(unit: _Unit, marks: dict[Instrument, Money]) -> Money: """Capital committed to the running unit's open positions (``Σ |net_qty| × mark``). @@ -1580,6 +1680,13 @@ def positions(self) -> list[PositionRow]: in-memory read (money exact :class:`~decimal.Decimal`); stopped units and flat books contribute nothing. Empty when no unit is running. + Each row is also marked (:meth:`_mark_of`'s pinned policy: the engine's + bar-close :class:`~trading_bot.application.mark_cache.MarkCache`, else the + last own-fill price), carrying its ``value`` (``mark * |net_qty|``) and + ``unrealised`` (``(mark - avg_entry_price) * net_qty``) — both ``None`` + when unmarked, mirroring :meth:`_unrealised_of`'s arithmetic so the + strategy aggregate always equals the row sum. + Returns ------- list of PositionRow @@ -1590,8 +1697,17 @@ def positions(self) -> list[PositionRow]: rows: list[PositionRow] = [] for unit in self._running_units(): assert unit.engine is not None + stored = self._stored_fills_of(unit) + mode_fills = by_mode(stored).get(unit.mode, []) positions = unit.engine.tracker.all_positions() for position in positions.values(): + mark = self._mark_of(unit, position.instrument, mode_fills) + value = mark.price * abs(position.net_qty) if mark is not None else None + unrealised = ( + (mark.price - position.avg_entry_price) * position.net_qty + if mark is not None and position.avg_entry_price is not None + else None + ) rows.append( PositionRow( strategy=unit.name, @@ -1602,10 +1718,69 @@ def positions(self) -> list[PositionRow]: avg_entry_price=position.avg_entry_price, realised_pnl=position.realised_pnl, fees_paid=position.fees_paid, + mark=mark.price if mark is not None else None, + mark_asof_ts=mark.asof_ms if mark is not None else None, + mark_source=mark.source if mark is not None else None, + value=value, + unrealised=unrealised, + fee_ccy=position.instrument.symbol.quote, ) ) return rows + async def balances(self) -> list[BalanceRow]: + """Every running unit's broker-reported free balances, tagged strategy + venue. + + Across every **running** unit, awaits + :meth:`~trading_bot.brokers.base.Broker.balances` on its own engine's + broker (the paper simulator's seeded book, or the real venue adapter on + testnet/live) and wraps the result in a :class:`BalanceRow`. A stopped + unit contributes nothing (mirroring :meth:`positions`); a broker call that + raises :class:`~trading_bot.domain.errors.BrokerError` degrades that one + unit's row to an empty ``balances`` dict plus its ``error`` message rather + than propagating — a transient venue outage must never break the whole + collection, and the caller (the dashboard's poll loop) needs a row to show + per unit either way. + + This is the prerequisite seam for the positions<->balances cross-check + (an accounting-guardrail extension comparing the tracker's net exposure + against what the venue itself reports) and the canary-roundtrip live + oracle (roadmap #6) — both read the broker's own view through this method, + never the locally-tracked positions. + + Returns + ------- + list of BalanceRow + One row per running unit, in unit order. Empty when no unit is + running. + + """ + rows: list[BalanceRow] = [] + for unit in self._running_units(): + assert unit.engine is not None + try: + balances = await unit.engine.broker.balances() + except BrokerError as exc: + rows.append( + BalanceRow( + strategy=unit.name, + exchange=unit.exchange, + mode=unit.mode, + balances={}, + error=str(exc), + ) + ) + continue + rows.append( + BalanceRow( + strategy=unit.name, + exchange=unit.exchange, + mode=unit.mode, + balances=balances, + ) + ) + return rows + def open_orders(self) -> list[OrderRow]: """Every unit's **open** (non-terminal) orders, tagged strategy + exchange. @@ -2169,25 +2344,26 @@ def _unrealised_of(unit: _Unit, mode: str, fills: list[Fill]) -> Money | None: """Best-effort mark-to-market of the running unit's open book, in ``mode``. The running engine's tracker holds the net open positions; each is marked - against the **last-known fill price** for its instrument in ``mode``'s - stream (``(mark - avg_entry) * net_qty``) — a fully in-memory, - non-network last-known mark (continuous MTM history is out of scope). The - book is meaningful only for the unit's **current** mode (the tracker was - built for it), so a non-current mode marks to ``None``. ``None`` too when - the unit is stopped, flat, or has no priced instrument. + via :meth:`_mark_of` — the engine's bar-close + :class:`~trading_bot.application.mark_cache.MarkCache` preferred, else the + **last-known fill price** for its instrument in ``mode``'s stream + (``(mark - avg_entry) * net_qty``) — the same one mark policy + :meth:`positions` rows use, so the roster's aggregate always equals the + row sum. The book is meaningful only for the unit's **current** mode (the + tracker was built for it), so a non-current mode marks to ``None``. + ``None`` too when the unit is stopped, flat, or has no priced instrument. """ - if not unit.running or unit.engine is None or mode != unit.mode or not fills: + if not unit.running or unit.engine is None or mode != unit.mode: return None - last_price = StrategySupervisor._mark_map(fills) unrealised: Money = _ZERO marked = False for position in unit.engine.tracker.all_positions().values(): if position.is_flat or position.avg_entry_price is None: continue - mark = last_price.get(position.instrument) + mark = StrategySupervisor._mark_of(unit, position.instrument, fills) if mark is None: continue - unrealised += (mark - position.avg_entry_price) * position.net_qty + unrealised += (mark.price - position.avg_entry_price) * position.net_qty marked = True return unrealised if marked else None diff --git a/trading_bot/brokers/binance.py b/trading_bot/brokers/binance.py index 4521fe89..1529742f 100644 --- a/trading_bot/brokers/binance.py +++ b/trading_bot/brokers/binance.py @@ -75,6 +75,7 @@ import hashlib import hmac +import json import os import re import time @@ -98,7 +99,7 @@ ) from trading_bot.domain.money import Money, money from trading_bot.domain.order import Order, OrderSide, OrderType -from trading_bot.transport.http import AsyncHTTPClient +from trading_bot.transport.http import AsyncHTTPClient, HTTPError from trading_bot.transport.ratelimit import RateLimiter if TYPE_CHECKING: @@ -216,6 +217,51 @@ def _map_binance_error(code: int, msg: str, *, context: str) -> BrokerError: return BrokerError(detail) +def _map_http_error(exc: HTTPError, *, context: str) -> BrokerError: + """Map a **definitive** venue rejection carried by a transport error. + + Binance delivers request rejections as an HTTP **4xx** whose JSON body + carries the same ``{"code", "msg"}`` shape as its in-band (200) errors — + so on the real venue a rejection (a filter failure, a duplicate id, an + insufficient balance) surfaces from the transport as + :class:`~trading_bot.transport.http.HTTPError`, never as a parsed payload + reaching :meth:`BinanceBroker._raise_on_error`. This helper recovers the + body and reuses :func:`_map_binance_error`, so venue rejections become the + same specific domain errors either way; a body that is not the Binance + error shape degrades to a generic :class:`BrokerError` carrying the + transport's (secret-redacted) message. + + Only **definitive** failures reach here: the transport raises the distinct + :class:`~trading_bot.transport.http.AmbiguousRequestError` for + unknown-outcome failures (a 5xx / drop on a non-retryable POST), and that + type propagates unchanged — the reconcile-don't-retry discipline is + untouched. + + Parameters + ---------- + exc : HTTPError + The transport error (carries ``status`` and the raw response body). + context : str + The endpoint name, for the error message. + + Returns + ------- + BrokerError + The most specific mapped domain error. + + """ + error: tuple[int, str] | None = None + if exc.body: + try: + error = BinanceBroker._find_error(json.loads(exc.body)) + except ValueError: + error = None + if error is not None: + code, msg = error + return _map_binance_error(code, msg, context=context) + return BrokerError(f"Binance {context}: {exc}") + + def _sign(query: str, secret: str) -> str: """Compute Binance's HMAC-SHA256 ``signature`` for a signed request. @@ -390,6 +436,15 @@ def is_testnet(self) -> bool: """Whether this adapter is pinned to Binance's spot **testnet**.""" return self._base_url == TESTNET_API_BASE + @property + def symbols(self) -> tuple[Symbol, ...]: + """The symbol set :meth:`fills` queries (``myTrades`` is per-symbol). + + Read-only introspection — empty means :meth:`fills` refuses (Binance + has no account-wide trade history to fall back to). + """ + return self._symbols + def _require_credentials(self) -> None: """Raise :class:`BrokerError` if a private call lacks credentials.""" if not self.has_credentials: @@ -466,12 +521,20 @@ def _weight_for(endpoint: str) -> int: return _ENDPOINT_WEIGHTS.get(endpoint, _DEFAULT_ENDPOINT_WEIGHT) async def _public_get(self, endpoint: str, params: Mapping[str, Any]) -> Any: - """GET a public endpoint and return its parsed JSON (or raise).""" + """GET a public endpoint and return its parsed JSON (or raise). + + A definitive HTTP-level rejection (Binance sends its ``{"code","msg"}`` + errors with a 4xx status) is mapped to the same specific domain errors + as an in-band error body — see :func:`_map_http_error`. + """ url = f"{self._base_url}{_API_PREFIX}/{endpoint}" async with self._http as client: - payload = await client.get( - url, params=dict(params), weight=self._weight_for(endpoint) - ) + try: + payload = await client.get( + url, params=dict(params), weight=self._weight_for(endpoint) + ) + except HTTPError as exc: + raise _map_http_error(exc, context=endpoint) from exc return self._raise_on_error(payload, context=endpoint) async def _signed_request( @@ -518,13 +581,21 @@ async def _signed_request( headers = {"X-MBX-APIKEY": self._api_key} async with self._http as client: - payload = await client.request( - method, - url, - headers=headers, - retry=retry, - weight=self._weight_for(endpoint), - ) + try: + payload = await client.request( + method, + url, + headers=headers, + retry=retry, + weight=self._weight_for(endpoint), + ) + except HTTPError as exc: + # A definitive venue rejection (Binance sends its + # ``{"code","msg"}`` errors with a 4xx status): map it to the + # same specific domain errors as an in-band error body. The + # ambiguous-failure path (AmbiguousRequestError) is a distinct + # type and propagates unchanged — never blind-retried here. + raise _map_http_error(exc, context=endpoint) from exc return self._raise_on_error(payload, context=endpoint) # --- public endpoints -------------------------------------------------- # @@ -927,8 +998,19 @@ async def fills(self, since_ms: int | None = None) -> list[Fill]: @staticmethod def _build_fill(symbol: Symbol, info: Mapping[str, Any]) -> Fill: - """Build a domain :class:`Fill` from a Binance ``myTrades`` entry.""" + """Build a domain :class:`Fill` from a Binance ``myTrades`` entry. + + Binance denominates the commission in ``commissionAsset`` — the + **base** asset on a market buy (observed live: a BTC/USDT buy is + charged in BTC), the quote on a sell, or BNB under the fee discount — + so the asset is carried onto :attr:`~trading_bot.domain.fill.Fill. + fee_asset` (normalised; ``None`` when Binance omits it) rather than + silently re-labelled as quote. Venue-truth accounting (reconciliation, + the canary's identity oracle) needs the real denomination to explain + the venue's per-asset balance movement exactly. + """ side = OrderSide.BUY if info.get("isBuyer") else OrderSide.SELL + commission_asset = info.get("commissionAsset") return Fill( fill_id=str(info.get("id")), client_order_id=str(info.get("orderId")), @@ -938,6 +1020,7 @@ def _build_fill(symbol: Symbol, info: Mapping[str, Any]) -> Fill: price=money(str(info.get("price", "0"))), fee=money(str(info.get("commission", "0"))), ts=int(info.get("time", 0)), + fee_asset=(normalise(str(commission_asset)) if commission_asset else None), ) # --- composite venue id ------------------------------------------------- # diff --git a/trading_bot/brokers/paper.py b/trading_bot/brokers/paper.py index 92cf888a..d5237b8b 100644 --- a/trading_bot/brokers/paper.py +++ b/trading_bot/brokers/paper.py @@ -60,6 +60,15 @@ ``limit_price`` (LIMIT, or a priced BEST_LIMIT) or, for a MARKET order, at the injected mark price for its instrument. Exactly one :class:`Fill` is produced and ``open_orders`` is left empty. + + Exception (``strict`` mode only): a limit order priced on the **passive side + of a known mark** — a BUY strictly below, or a SELL strictly above, the + instrument's injected mark — **rests** on the book instead (no fill, no + balance move; it stays live in ``open_orders`` until cancelled), exactly as a + real venue would. With **no** mark injected for the instrument the historical + fill-at-limit shortcut is kept (the self-contained pattern the runners and + tests rely on), and the permissive (non-strict) model always fills at the + limit as before. * ``"partial"`` — the order is filled in ``partial_chunks`` equal slices (the last slice absorbs any rounding remainder so the slices sum **exactly** to the filled quantity). By default the whole quantity is consumed (the order closes @@ -209,6 +218,13 @@ class PaperBroker(Broker): venue id without creating a second paper order, mirroring the venue- side dedup the idempotency invariant relies on (Binance ``-2010``). A retried client-order-id never duplicates a paper order. + * **Marketability** — a limit order priced on the passive side of a + *known* mark (a BUY strictly below it, a SELL strictly above it) + **rests** open with no fill and no balance move, exactly like a real + venue book, until cancelled. Only applies when a mark is injected for + the order's instrument: with no mark the historical fill-at-limit + shortcut is kept, so self-contained (mark-less) strict setups behave + exactly as before. id_token : str, optional The per-instance **lifetime token** embedded in every minted id (see the @@ -448,6 +464,16 @@ async def place_order(self, order: Order) -> str: armed = self._armed_ratio self._armed_ratio = None + # Strict marketability: a limit order on the passive side of a *known* + # mark rests on the book (no fill, no balance move) exactly like a real + # venue, staying live until cancelled. With no mark injected the + # historical fill-at-limit shortcut is kept (see the module docstring's + # Fill model), and the permissive model never rests. + if self._strict and self._rests_at_placement(record): + self._venue_id_by_cid[order.client_order_id] = venue_order_id + self._open[venue_order_id] = record + return venue_order_id + price = self._execution_price_for(order, limit_price) fill_qty = self._placement_fill_qty(qty, armed) for slice_qty in self._slice(fill_qty): @@ -464,6 +490,26 @@ async def place_order(self, order: Order) -> str: self._open[venue_order_id] = record return venue_order_id + def _rests_at_placement(self, record: _OpenOrder) -> bool: + """Whether a strict-mode order rests (fills nothing) at placement. + + True only for a limit-priced order whose price sits strictly on the + **passive** side of a *known* mark for its instrument: a BUY strictly + below the mark, or a SELL strictly above it — the book would not cross, + so a real venue would rest it. A market / unpriced order never rests + (it always crosses), and an instrument with **no injected mark** keeps + the historical fill-at-limit shortcut (returns ``False``), so + self-contained mark-less setups are unaffected. + """ + if record.limit_price is None: + return False # MARKET / unpriced BEST_LIMIT: always crosses. + mark = self._prices.get(record.instrument) + if mark is None: + return False # No mark: keep the fill-at-limit shortcut. + if record.side is OrderSide.BUY: + return record.limit_price < mark + return record.limit_price > mark + def _execution_price_for(self, order: Order, limit_price: Money | None) -> Money: """Resolve the price ``order`` fills at, given its resolved limit price. diff --git a/trading_bot/domain/fill.py b/trading_bot/domain/fill.py index 1a23d381..ef9f15e1 100644 --- a/trading_bot/domain/fill.py +++ b/trading_bot/domain/fill.py @@ -63,12 +63,26 @@ class Fill: The execution price (in quote units per base unit). Must be strictly positive. fee : Decimal - The fee charged for this execution, in quote units. Must be - non-negative (zero is allowed for rebated / fee-free fills). + The fee charged for this execution, denominated in ``fee_asset`` + (the quote currency when ``fee_asset`` is ``None`` — the historic + assumption). Must be non-negative (zero is allowed for rebated / + fee-free fills). ts : int Execution timestamp as **milliseconds since the Unix epoch (UTC)**. Must be non-negative. Fills are folded in caller-supplied order; ``ts`` is carried for record-keeping and tie-breaking, not used to re-sort. + fee_asset : str or None, optional + The canonical asset code the ``fee`` is denominated in. ``None`` + (default) means the instrument's **quote** currency — the historic + assumption, and what the simulator and Kraken report. Binance charges + a market *buy*'s commission in the **base** asset (``commissionAsset``) + — carrying the denomination is what lets venue-truth accounting (the + canary's identity oracle) explain the venue's per-asset balance + movement exactly. NOTE: the PnL fold (:class:`~trading_bot.domain. + position.Position`) still treats ``fee`` as quote units at face value — + a documented approximation for non-quote fees (an exact conversion + needs a fee-asset price), acceptable because such fees are commission + dust; the balance identity above is exact regardless. Examples -------- @@ -96,6 +110,7 @@ class Fill: price: Money fee: Money ts: int + fee_asset: str | None = None def __post_init__(self) -> None: """Validate construction invariants (ids non-empty, amounts in range). @@ -137,6 +152,12 @@ def __post_init__(self) -> None: raise OrderError( self.client_order_id, f"fill ts must be non-negative, got {self.ts}" ) + if self.fee_asset is not None and not self.fee_asset.strip(): + raise OrderError( + self.client_order_id, + "fill fee_asset must be a non-empty asset code when given " + "(None means the quote currency)", + ) @property def signed_qty(self) -> Money: diff --git a/trading_bot/interfaces/api/app.py b/trading_bot/interfaces/api/app.py index 66e7f77d..8c4775d6 100644 --- a/trading_bot/interfaces/api/app.py +++ b/trading_bot/interfaces/api/app.py @@ -84,6 +84,7 @@ from pydantic import BaseModel, Field import trading_bot +from trading_bot.application.display_ccy import convert, resolve_currency from trading_bot.application.events import ( Event, FillEvent, @@ -94,10 +95,12 @@ if TYPE_CHECKING: from trading_bot.application.config import ( + AppConfig, PortfolioStrategyConfig, StrategyConfig, ) from trading_bot.application.supervisor import ( + BalanceRow, FillRow, KpiRow, OrderRow, @@ -203,6 +206,7 @@ def _order_dict(order: Order) -> dict[str, Any]: "status": order.status.value, "filled_qty": _money_str(order.filled_qty), "avg_fill_price": _money_str(order.avg_fill_price), + "reject_reason": order.reject_reason, } @@ -277,12 +281,20 @@ def _epoch_ms() -> int: # --------------------------------------------------------------------------- -def _position_row_dict(row: PositionRow) -> dict[str, Any]: +def _position_row_dict(row: PositionRow, config: AppConfig) -> dict[str, Any]: """Render a supervisor :class:`PositionRow` as a JSON-ready dict. The per-instrument exposure of one running unit, tagged with its ``strategy`` and ``exchange`` (the dashboard's group-by keys) and its ``base`` asset (the group-by-crypto key). Money fields are exact :class:`~decimal.Decimal` strings. + ``mark_asof_ts`` is already an integer epoch ms (or ``None``), like + ``last_asof_ts`` on ``/api/strategies`` — no stringification needed. + + ``display_currency`` (:func:`~trading_bot.application.display_ccy. + resolve_currency`, resolved for the row's ``exchange``) + ``value_display`` + / ``unrealised_display`` (:func:`~trading_bot.application.display_ccy.convert`, + from the row's ``fee_ccy`` quote) are additive — ``None`` (JSON ``null``) + when no rate is declared for that quote, never a guessed conversion. """ return { "strategy": row.strategy, @@ -293,6 +305,15 @@ def _position_row_dict(row: PositionRow) -> dict[str, Any]: "avg_entry_price": _money_str(row.avg_entry_price), "realised_pnl": _money_str(row.realised_pnl), "fees_paid": _money_str(row.fees_paid), + "mark": _money_str(row.mark), + "mark_asof_ts": row.mark_asof_ts, + "mark_source": row.mark_source, + "value": _money_str(row.value), + "unrealised": _money_str(row.unrealised), + "fee_ccy": row.fee_ccy, + "display_currency": resolve_currency(row.exchange, config), + "value_display": _money_str(convert(row.value, row.fee_ccy, config)), + "unrealised_display": _money_str(convert(row.unrealised, row.fee_ccy, config)), } @@ -324,7 +345,27 @@ def _fill_row_dict(row: FillRow) -> dict[str, Any]: } -def _kpi_row_dict(row: KpiRow) -> dict[str, Any]: +def _balance_row_dict(row: BalanceRow) -> dict[str, Any]: + """Render a supervisor :class:`BalanceRow` as a JSON-ready dict. + + ``balances`` is a per-asset mapping to exact Decimal strings (never a plain + ``str(dict)`` — each amount is stringified individually so no value ever + round-trips through ``float``). ``error`` is ``None`` (JSON ``null``) on a + successful fetch, or the broker's error message when its ``balances()`` call + raised — the row still renders (an empty ``balances`` dict), never a 500. + """ + return { + "strategy": row.strategy, + "exchange": row.exchange, + "mode": row.mode, + "balances": { + asset: _money_str(amount) for asset, amount in row.balances.items() + }, + "error": row.error, + } + + +def _kpi_row_dict(row: KpiRow, config: AppConfig) -> dict[str, Any]: """Render a supervisor :class:`KpiRow` as a JSON-ready dict. Money (``realised_pnl`` / ``fees_paid``) as exact Decimal strings; the ratios @@ -332,6 +373,13 @@ def _kpi_row_dict(row: KpiRow) -> dict[str, Any]: ``level="strategy"`` and JSON ``null`` at the aggregate levels (no combined curve yet); ``quote`` is the row's quote currency, or ``null`` when the row folds units that mix quote currencies (the UI renders "mixed"). + + ``display_currency`` (resolved for the row's ``exchange``, ``None`` at + ``level="total"``) + ``realised_pnl_display`` / ``fees_paid_display`` + (converted from ``quote`` — the row's only money aggregates, mirroring + :func:`_position_row_dict`'s pair) are additive — ``None`` (JSON ``null``) + when no rate is declared, or ``quote`` itself is ``None`` (mixed), never a + guessed conversion. """ return { "level": row.level, @@ -345,6 +393,11 @@ def _kpi_row_dict(row: KpiRow) -> dict[str, Any]: "sortino": _finite_or_none(row.sortino), "calmar": _finite_or_none(row.calmar), "max_drawdown": _finite_or_none(row.max_drawdown), + "display_currency": resolve_currency(row.exchange, config), + "realised_pnl_display": _money_str( + convert(row.realised_pnl, row.quote, config) + ), + "fees_paid_display": _money_str(convert(row.fees_paid, row.quote, config)), } @@ -1105,12 +1158,19 @@ def _worst_health(healths: Iterable[str]) -> str: return worst -def _status_dict(status: StrategyStatus) -> dict[str, Any]: +def _status_dict(status: StrategyStatus, config: AppConfig) -> dict[str, Any]: """Render a :class:`StrategyStatus` for JSON (money as exact Decimal string). ``last_eval_ts`` / ``last_asof_ts`` are already integer epoch ms (or ``None``) on the domain side — no stringification needed, unlike the money fields. + + ``display_currency`` (resolved for the unit's ``exchange``) + + ``total_value_display`` / ``unrealised_display`` (converted from the + unit's ``quote``, mirroring :func:`_position_row_dict`'s pair) are + additive — ``None`` (JSON ``null``) when no rate is declared for that + quote (or the quote is ``None`` — a mixed-quote portfolio), never a + guessed conversion. """ return { "name": status.name, @@ -1133,6 +1193,13 @@ def _status_dict(status: StrategyStatus) -> dict[str, Any]: "capital_policy": status.capital_policy, "health": status.health, "health_detail": list(status.health_detail), + "display_currency": resolve_currency(status.exchange, config), + "total_value_display": _money_str( + convert(status.total_value, status.quote, config) + ), + "unrealised_display": _money_str( + convert(status.unrealised, status.quote, config) + ), } @@ -1183,28 +1250,40 @@ def _guard_write() -> None: @app.get("/api/strategies") async def strategies(request: Request) -> list[dict[str, Any]]: - """List every managed strategy with its exchange / mode / running / PnL.""" - return [_status_dict(s) for s in _sup(request).status()] + """List every managed strategy with its exchange / mode / running / PnL. + + ``last_asof_ts`` is the as-of (epoch ms) of the last **completed** + evaluation — the latest bar time the strategy actually computed on, as + opposed to ``last_eval_ts`` (the wall-clock of the last *attempted* + tick). ``None`` before the first completed evaluation. This is the field + the future "last bar → next bar" timing chip (dashboard-tables-ux) reads; + it needs no further server change. + """ + sup = _sup(request) + config = sup.manifest() + return [_status_dict(s, config) for s in sup.status()] @app.post("/api/strategies/{name}/start") async def start_strategy(name: str, request: Request) -> dict[str, Any]: _guard_write() + sup = _sup(request) try: - await _sup(request).start(name) + await sup.start(name) except ConfigError as exc: raise HTTPException(status_code=404, detail=str(exc)) from exc except (BrokerError, LiveTradingNotEnabled) as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc - return {"ok": True, "status": _status_dict(_sup(request).status(name)[0])} + return {"ok": True, "status": _status_dict(sup.status(name)[0], sup.manifest())} @app.post("/api/strategies/{name}/stop") async def stop_strategy(name: str, request: Request) -> dict[str, Any]: _guard_write() + sup = _sup(request) try: - await _sup(request).stop(name) + await sup.stop(name) except ConfigError as exc: raise HTTPException(status_code=404, detail=str(exc)) from exc - return {"ok": True, "status": _status_dict(_sup(request).status(name)[0])} + return {"ok": True, "status": _status_dict(sup.status(name)[0], sup.manifest())} @app.post("/api/strategies/{name}/mode") async def set_strategy_mode( @@ -1231,8 +1310,9 @@ async def set_strategy_mode( f"{_LIVE_ACK_PHRASE!r} in the request body's 'ack' field" ), ) + sup = _sup(request) try: - await _sup(request).set_mode( + await sup.set_mode( name, body.mode, # type: ignore[arg-type] confirm_live=body.confirm, @@ -1244,7 +1324,7 @@ async def set_strategy_mode( raise HTTPException(status_code=400, detail=str(exc)) from exc except (BrokerError, LiveTradingNotEnabled) as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc - return {"ok": True, "status": _status_dict(_sup(request).status(name)[0])} + return {"ok": True, "status": _status_dict(sup.status(name)[0], sup.manifest())} def create_control_app( @@ -1819,9 +1899,38 @@ async def positions(request: Request, group_by: str | None = None) -> Any: status_code=422, detail=f"unknown group_by {group_by!r}; expected one of {_GROUP_BY_KEYS}", ) - rows = [_position_row_dict(row) for row in _sup(request).positions()] + sup = _sup(request) + config = sup.manifest() + rows = [_position_row_dict(row, config) for row in sup.positions()] return _grouped(rows, group_by) + # -- Balances (broker-reported free balances, one row per running unit) -- # + + @app.get("/api/balances") + async def balances( + request: Request, strategy: str | None = None + ) -> list[dict[str, Any]]: + """Broker-reported free balances, one row per running unit. + + A read (safe under ``read_only``). Each row is ``{strategy, exchange, + mode, balances: {asset: "exact-decimal"}, error}`` — the venue's own view + of what it holds (see :meth:`~trading_bot.application.supervisor. + StrategySupervisor.balances`), as opposed to the locally-tracked + exposure ``/api/positions`` reports. A stopped unit contributes nothing. + A broker error degrades that unit's row to an empty ``balances`` dict + plus a non-``null`` ``error`` string — **HTTP 200**, never a 500, so the + dashboard can poll this endpoint safely through a transient venue outage. + ``?strategy=`` filters to one unit (mirroring ``/api/orders``'s / + ``/api/fills``'s filter). Money is rendered as exact Decimal strings. + + This is the prerequisite seam for the positions<->balances cross-check + (an accounting-guardrail extension) and the canary-roundtrip live oracle + (roadmap #6): both need the venue's own reported balances, not just the + engine's locally-tracked positions. + """ + rows = [_balance_row_dict(row) for row in await _sup(request).balances()] + return _filtered(rows, crypto=None, exchange=None, strategy=strategy) + # -- Orders (open + history, aggregated, groupable + filterable) --------- # @app.get("/api/orders") @@ -1909,9 +2018,11 @@ async def kpi(request: Request, level: str = "strategy") -> list[dict[str, Any]] status_code=422, detail=f"unknown level {level!r}; expected one of {_KPI_LEVELS}", ) + sup = _sup(request) + config = sup.manifest() return [ - _kpi_row_dict(row) - for row in _sup(request).kpi(level) # type: ignore[arg-type] + _kpi_row_dict(row, config) + for row in sup.kpi(level) # type: ignore[arg-type] ] # -- PnL series (per-mode realised-PnL / equity curve over time) --------- # @@ -1928,6 +2039,10 @@ async def pnl(request: Request, strategy: str, mode: str = "all") -> dict[str, A (default ``all``) filters to a single mode. Money as exact Decimal strings; ``ts_ms`` integer. An unknown ``strategy`` is a 404; a strategy with no fills is an empty series (200, not an error). + + Carries no ``last_asof_ts`` of its own — each series point already + carries its own ``ts_ms``, so there is no separate "as of" to surface + (unlike ``/api/strategies``, a single evaluation-cadence snapshot). """ from trading_bot.domain.errors import ConfigError @@ -2118,7 +2233,7 @@ async def create_strategy( ), ) _persist(request) - return {"ok": True, "status": _status_dict(sup.status(name)[0])} + return {"ok": True, "status": _status_dict(sup.status(name)[0], sup.manifest())} @app.delete("/api/strategies/{name}") async def delete_strategy(name: str, request: Request) -> dict[str, Any]: @@ -2147,6 +2262,11 @@ async def capital(name: str, request: Request) -> dict[str, Any]: ``{allocation, contributed, realised, unrealised, total_value, withdrawable, policy, events}`` — money as exact Decimal strings; an unknown unit is a 404. + + Carries no ``last_asof_ts``: each ledger ``event`` already carries its + own ``ts`` (when the deposit/withdrawal happened), and this breakdown is + not tied to a strategy evaluation cadence the way ``/api/strategies``' / + ``/api/positions``' marks are. """ from trading_bot.domain.errors import ConfigError diff --git a/trading_bot/interfaces/cli/_render.py b/trading_bot/interfaces/cli/_render.py index 2db6b970..9734e5bf 100644 --- a/trading_bot/interfaces/cli/_render.py +++ b/trading_bot/interfaces/cli/_render.py @@ -28,6 +28,7 @@ from rich.table import Table if TYPE_CHECKING: + from trading_bot.application.canary import CanaryCheck from trading_bot.application.performance_service import PerformanceService from trading_bot.domain.instrument import Instrument from trading_bot.domain.order import Order @@ -39,6 +40,7 @@ "positions_table", "open_orders_table", "kpi_table", + "canary_table", ] @@ -218,3 +220,37 @@ def kpi_table(perf: PerformanceService, *, title: str = "Performance (KPI)") -> table.add_row("Max drawdown", fmt_ratio(perf.max_drawdown())) table.add_row("Calmar", fmt_ratio(perf.calmar())) return table + + +def canary_table(checks: list[CanaryCheck], *, title: str = "Canary evidence") -> Table: + """Build a :class:`rich.table.Table` of the canary's evidence, one row per check. + + Columns: status (``PASS``/``FAIL``, coloured), check name, expected, + observed — the same exact expected/observed strings + :meth:`~trading_bot.application.canary.CanaryReport.to_text` renders as one + flat line, tabulated instead. An empty list yields a header-only table. + + Parameters + ---------- + checks : list of CanaryCheck + The completed (or aborted) canary run's ordered checks + (:attr:`~trading_bot.application.canary.CanaryReport.checks`). + title : str, optional + The table title. Default ``"Canary evidence"``. + + Returns + ------- + rich.table.Table + The rendered evidence table. + + """ + table = Table(title=title) + table.add_column("Status") + table.add_column("Check") + table.add_column("Expected") + table.add_column("Observed") + + for check in checks: + status = "[green]PASS[/green]" if check.passed else "[red]FAIL[/red]" + table.add_row(status, check.name, check.expected, check.observed) + return table diff --git a/trading_bot/interfaces/cli/main.py b/trading_bot/interfaces/cli/main.py index 6883547d..42a52f69 100644 --- a/trading_bot/interfaces/cli/main.py +++ b/trading_bot/interfaces/cli/main.py @@ -52,6 +52,7 @@ import pathlib import signal import sys +import tempfile from collections.abc import Callable from decimal import Decimal from typing import TYPE_CHECKING, Any @@ -61,8 +62,15 @@ from rich.console import Console from trading_bot import __version__ -from trading_bot.application.config import AppConfig, StrategyConfig +from trading_bot.application.canary import ( + CanaryReport, + run_canary, + size_for_minimums, + venue_oracle, +) +from trading_bot.application.config import AppConfig, BrokerConfig, StrategyConfig from trading_bot.application.data_feed import BARS_SCHEMA, InMemoryFeed +from trading_bot.application.instrument_specs import InstrumentSpecResolver from trading_bot.application.performance_service import PerformanceService from trading_bot.application.run_app import run_app from trading_bot.application.service_factory import Engine, build_engine @@ -72,7 +80,14 @@ ma_crossover_signal, ) from trading_bot.application.strategy_runner import StrategyRunner -from trading_bot.domain.instrument import Instrument, parse_kraken_pair +from trading_bot.brokers.binance import BinanceBroker +from trading_bot.brokers.kraken import KrakenBroker +from trading_bot.brokers.paper import PaperBroker +from trading_bot.domain.instrument import ( + Instrument, + parse_binance_symbol, + parse_kraken_pair, +) from trading_bot.domain.money import Money, from_float, money from trading_bot.domain.order import Order, OrderSide, OrderType from trading_bot.interfaces.cli import _render @@ -97,6 +112,12 @@ #: The go-live runbook the ``--live`` refusals point the user at. _RUNBOOK = "doc/dev/09-go-live.md" +#: The exact phrase the operator must type to run a **live** (real-money) canary +#: — the CLI's typed confirmation, mirroring the dashboard's server-side go-live +#: acknowledgement (:data:`trading_bot.interfaces.api.app._LIVE_ACK_PHRASE`, +#: kept equal by a test; not imported here so the CLI never pulls FastAPI in). +_LIVE_ACK_PHRASE = "I UNDERSTAND" + #: The host values treated as loopback (local-only). Binding any *other* host makes #: the dashboard reachable off the box, so the serve paths that expose the control #: surface require a token there. Kept in sync with the config-layer guard @@ -581,6 +602,530 @@ def _resolve_kpi_capital( return money(str(Decimal(str(_KPI_DEFAULT_CAPITAL)))) +# --- canary ------------------------------------------------------------------ # + +#: The paper simulator's fee rate, in basis points — matches +#: :class:`~trading_bot.brokers.paper.PaperBroker`'s own default +#: (``fee_bps=money("10")``); ``build_engine`` never overrides it (no +#: ``AppConfig`` knob exists for it), so this is exactly the rate a +#: factory-built canary engine pays. Used only to state the pre-trade cost +#: bound in ``--max-cost``'s help text and the refusal message — the run +#: itself measures the real fee from the fills, never trusts this constant. +_CANARY_PAPER_FEE_BPS: Money = money("10") + +#: Basis-point denominator (``fee = notional * fee_bps / 10000``), matching +#: :mod:`trading_bot.brokers.paper`'s own fee formula. +_CANARY_BPS_DENOMINATOR: Money = money("10000") + +#: Exchanges the resolver can fetch a real spec/price for (public +#: ``instrument()`` + ``ticker()``, both keyless) — the same two the +#: :class:`~trading_bot.application.instrument_specs.InstrumentSpecResolver` +#: dispatches. Anything else has no real minimum to size against, so the +#: canary refuses rather than guess one. +_CANARY_EXCHANGES = ("kraken", "binance") + +#: Default cancel-probe offset on **paper** (percent below the mark): maximally +#: far-off, exactly reproducible — the simulator has no price band. +_PAPER_PROBE_OFFSET_PCT: Money = money("50") + +#: Default cancel-probe offset on a **real venue** (testnet/live). Real venues +#: reject a resting limit too far from the mark — Binance spot's +#: PERCENT_PRICE_BY_SIDE filter bounds a buy limit to roughly 20% below the +#: 5-minute weighted average (a 50%-below probe is rejected with ``-1013``, +#: observed on the real testnet). 15% below sits inside the band with margin +#: and still never fills within a seconds-long canary run. +_VENUE_PROBE_OFFSET_PCT: Money = money("15") + + +def _canary_price_source(exchange: str) -> KrakenBroker | BinanceBroker: + """A fresh, keyless adapter for ``exchange``'s public endpoints only. + + The same construction :class:`InstrumentSpecResolver` uses internally + (``KrakenBroker()`` / ``BinanceBroker()``, no credentials) — built here so + the CLI can also read the public last price (:meth:`~trading_bot.brokers. + base.Broker.ticker`) off the *same* adapter instance the resolver fetches + the venue minimums from: one HTTP client, two public reads. + """ + if exchange == "kraken": + return KrakenBroker() + if exchange == "binance": + return BinanceBroker() + raise typer.BadParameter( + f"canary --exchange {exchange!r} has no public spec/price source " + f"(supported: {', '.join(_CANARY_EXCHANGES)}) — refusing to guess a " + "venue-legal size or a mark" + ) + + +async def _resolve_canary_sizing( + exchange: str, symbol: str +) -> tuple[Instrument, Money, Money]: + """Resolve the real ``(instrument, mark, qty)`` a canary round-trip needs. + + ``instrument`` carries the venue's real minimums (:class:`InstrumentSpecResolver`, + the public ``instrument()``/``exchangeInfo``-shaped endpoint); ``mark`` is + the venue's real last price (the public ``ticker()`` endpoint); ``qty`` is + the smallest venue-legal quantity at that mark + (:func:`~trading_bot.application.canary.size_for_minimums`). Both public + reads go through the *same* keyless adapter instance + (:func:`_canary_price_source`). + + Raises + ------ + ValueError + If the resolver's fetch degraded to a bare, unquantized instrument + (see :class:`InstrumentSpecResolver`'s ``degraded`` set) — the canary + never guesses a size from an unresolved spec. + """ + exchange = exchange.lower() + source = _canary_price_source(exchange) + resolver = InstrumentSpecResolver( + kraken=source if exchange == "kraken" else None, + binance=source if exchange == "binance" else None, + ) + parsed_symbol = ( + parse_kraken_pair(symbol) + if exchange == "kraken" + else parse_binance_symbol(symbol) + ) + instrument = await resolver.resolve(exchange, parsed_symbol) + if exchange in resolver.degraded: + raise ValueError( + f"the instrument-spec resolver could not fetch {exchange}'s real " + f"minimums for {parsed_symbol} (endpoint degraded); refusing to " + "guess a venue-legal size — retry, or check connectivity" + ) + mark = await source.ticker(instrument) + qty = size_for_minimums(instrument, mark) + return instrument, mark, qty + + +async def _run_paper_canary( + *, + exchange: str, + symbol: str, + budget: Money, + max_cost: Money, + probe_offset_pct: Money, + db_path: pathlib.Path | None, +) -> CanaryReport: + """Build the canary's own dedicated paper engine and run the scenario. + + Sizes ``qty`` from the real venue minimums (:func:`_resolve_canary_sizing`), + refuses **before building anything** if the implied round-trip cost exceeds + ``max_cost``, then builds a fresh, strict-paper :class:`Engine` funded with + ``budget`` quote units (:attr:`~trading_bot.application.config.AppConfig. + paper_starting_balances`) — always its **own** store (``--db``, or a scratch + temp store discarded when the run ends; ``run_canary`` refuses a store-less + engine, so a shared/absent store is impossible), injects the resolved mark + and hands the ready engine to :func:`~trading_bot.application.canary.run_canary`. + """ + instrument, mark, qty = await _resolve_canary_sizing(exchange, symbol) + + # The paper round-trip's cost is EXACT (both legs fill at the same + # injected mark, see canary.py's module docstring): 2 legs x qty x mark x + # fee_bps/10000. Refuse before any engine is even built if that would + # exceed the caller's bound. + implied_cost = 2 * qty * mark * _CANARY_PAPER_FEE_BPS / _CANARY_BPS_DENOMINATOR + if implied_cost > max_cost: + raise ValueError( + f"refusing to start: the round-trip's implied cost {implied_cost} " + f"(2 x qty x mark x fee_bps/10000 = 2 x {qty} x {mark} x " + f"{_CANARY_PAPER_FEE_BPS}/{_CANARY_BPS_DENOMINATOR}) exceeds " + f"--max-cost {max_cost}; raise --max-cost or pick a cheaper --symbol" + ) + + config = AppConfig( + paper_strict=True, + paper_starting_balances={instrument.symbol.quote: Decimal(str(budget))}, + ) + + async def _build_and_run(store_path: pathlib.Path) -> CanaryReport: + engine = build_engine(config, db_path=store_path) + broker = engine.broker + assert isinstance(broker, PaperBroker) + broker.set_price(instrument, mark) + try: + return await run_canary( + engine, instrument, qty=qty, probe_offset_pct=probe_offset_pct + ) + finally: + if engine.store is not None: + engine.store.close() + + if db_path is not None: + return await _build_and_run(db_path) + # No --db: a scratch temp store (run_canary requires one — see the + # docstring above), removed with the directory once the run ends. + with tempfile.TemporaryDirectory(prefix="trading-bot-canary-") as tmp_dir: + return await _build_and_run(pathlib.Path(tmp_dir) / "canary.sqlite") + + +def _confirm_live_canary(config_path: pathlib.Path | None) -> AppConfig: + """Gate ``canary --mode live``: config opt-in first, then a typed confirmation. + + Live is real money, so the canary re-uses the EXISTING gates rather than + minting its own: the operator's ``--config`` must carry the off-by-default + ``live_enabled: true`` opt-in (the same gate ``run --live`` and the factory + enforce — the canary never sets it itself), and the operator must then + **type** the exact acknowledgement phrase (:data:`_LIVE_ACK_PHRASE` — the + dashboard's go-live confirmation pattern, enforced verbatim). Credentials + and the mandatory risk limits are re-checked one layer down by + :func:`~trading_bot.application.service_factory.build_engine`. Any missing + gate refuses with a non-zero exit **before an engine is ever built** — no + order placed. + + Returns + ------- + AppConfig + The operator's loaded config (the venue canary derives its engine + config from it — see :func:`_canary_venue_config`). + + """ + if config_path is None: + _console.print( + "[red]refusing to run canary --mode live[/red] without --config: " + "the live gates (live_enabled, risk limits, the venue's broker " + f"entry) live in the operator's config. See {_RUNBOOK}. " + "(no order was placed)" + ) + raise typer.Exit(code=1) + config = AppConfig.from_yaml(config_path) + if not config.live_enabled: + _console.print( + "[red]refusing to run canary --mode live[/red]: live is off by " + f"default. Set live_enabled: true in the config and read {_RUNBOOK} " + "first (no order was placed)." + ) + raise typer.Exit(code=1) + ack = typer.prompt( + "The LIVE canary trades REAL money on the real venue (a minimum-size " + f'round-trip + a real cancel). Type "{_LIVE_ACK_PHRASE}" to proceed', + default="", + show_default=False, + ) + if ack != _LIVE_ACK_PHRASE: + _console.print( + "[red]refusing to run canary --mode live[/red] without the exact " + f"typed acknowledgement {_LIVE_ACK_PHRASE!r} (no order was placed). " + f"See {_RUNBOOK}." + ) + raise typer.Exit(code=1) + return config + + +def _canary_venue_config( + exchange: str, symbol: str, mode: str, live_base: AppConfig | None +) -> AppConfig: + """Build the engine config for a **venue** (testnet/live) canary run. + + * ``testnet`` — a self-contained config: ``mode: live`` plus one broker + with ``testnet: true``, exactly the factory's sandbox path (hard-pinned + testnet URL, ``BINANCE_TESTNET_*`` credentials, **no** ``live_enabled`` + needed — the adapter structurally cannot reach mainnet). A venue with no + testnet (Kraken) is refused by the factory with a clear error. + * ``live`` — derived from the operator's own config (``live_base``, loaded + by :func:`_confirm_live_canary`): only the venue's matching broker + entries are kept (forced off testnet), strategies/portfolios are cleared + (the canary trades nothing but its own round-trip), and ``live_enabled`` + / ``risk`` are carried **unchanged** so the factory's existing gates do + their job — the canary never weakens a gate. + + In both cases the canary's ``symbol`` is added to the broker's ``symbols`` + so per-symbol fill endpoints (Binance ``myTrades``) serve + ``broker.fills(...)`` — the venue oracle's re-fetch channel. + """ + if mode == "testnet": + return AppConfig( + mode="live", + brokers=[ + BrokerConfig( + name=f"canary-{exchange}-testnet", + exchange=exchange, + testnet=True, + symbols=[symbol], + ) + ], + ) + assert live_base is not None # gated by _confirm_live_canary + matching = [ + broker for broker in live_base.brokers if broker.exchange.lower() == exchange + ] + if not matching: + raise ValueError( + f"canary --mode live: no broker for exchange {exchange!r} in the " + f"config (have {[b.exchange for b in live_base.brokers]!r}); add a " + "brokers entry for that venue" + ) + brokers = [ + broker.model_copy( + update={ + "testnet": False, + "symbols": sorted({*broker.symbols, symbol}), + } + ) + for broker in matching + ] + return live_base.model_copy( + update={ + "mode": "live", + "brokers": brokers, + "strategies": [], + "portfolios": [], + } + ) + + +async def _resolve_venue_sizing( + engine: Engine, exchange: str, symbol: str +) -> tuple[Instrument, Money, Money]: + """Resolve ``(instrument, mark, qty)`` against the venue the engine trades. + + The venue-run twin of :func:`_resolve_canary_sizing`, with one crucial + difference: the spec and the mark are read off the **engine's own broker** + (the testnet-pinned or live adapter), not a fresh mainnet public adapter — + a sandbox's filters and prices can differ from production, and the sizing + must be legal on the venue that will actually see the orders. + + Raises + ------ + ValueError + If the resolver's fetch degraded to a bare, unquantized instrument — + the canary never guesses a size from an unresolved spec. + """ + broker = engine.broker + resolver = InstrumentSpecResolver( + kraken=broker if isinstance(broker, KrakenBroker) else None, + binance=broker if isinstance(broker, BinanceBroker) else None, + ) + parsed_symbol = ( + parse_kraken_pair(symbol) + if exchange == "kraken" + else parse_binance_symbol(symbol) + ) + instrument = await resolver.resolve(exchange, parsed_symbol) + if exchange in resolver.degraded: + raise ValueError( + f"the instrument-spec resolver could not fetch {exchange}'s real " + f"minimums for {parsed_symbol} (endpoint degraded); refusing to " + "guess a venue-legal size — retry, or check connectivity" + ) + mark = await broker.ticker(instrument) + qty = size_for_minimums(instrument, mark) + return instrument, mark, qty + + +async def _run_venue_canary( + *, + exchange: str, + symbol: str, + mode: str, + max_cost: Money, + probe_offset_pct: Money, + db_path: pathlib.Path | None, + live_base: AppConfig | None = None, +) -> CanaryReport: + """Build a dedicated **venue** (testnet/live) engine and run the canary on it. + + Builds the engine through the factory (:func:`build_engine` — the single + wiring point, whose testnet/live gates all apply: sandbox availability, + credentials, ``live_enabled``, mandatory live risk limits), sizes the + round-trip against the venue's **own** spec and last price + (:func:`_resolve_venue_sizing`), and runs the scenario with the venue + **identity** oracle (:func:`~trading_bot.application.canary.venue_oracle`, + bounded by ``max_cost``) instead of the exact paper oracle. Always the + canary's own store (``--db``, or a scratch temp store) — never a shared + book. The account's real balances fund the run; ``--budget`` plays no role + here. + """ + exchange = exchange.lower() + if exchange not in _CANARY_EXCHANGES: + raise ValueError( + f"canary --exchange {exchange!r} has no venue adapter " + f"(supported: {', '.join(_CANARY_EXCHANGES)})" + ) + config = _canary_venue_config(exchange, symbol, mode, live_base) + + async def _build_and_run(store_path: pathlib.Path) -> CanaryReport: + # The factory raises before any I/O when a gate is missing (no testnet + # for the venue, no credentials, live not enabled, risk limits unset). + engine = build_engine(config, db_path=store_path) + try: + if engine.store is not None: + # Tag the evidence trail with the real deployment context so a + # persisted (--db) canary store reads truthfully — testnet and + # live are different money and must never commingle with paper. + engine.store.set_context(mode=mode, venue=exchange) + instrument, _mark, qty = await _resolve_venue_sizing( + engine, exchange, symbol + ) + return await run_canary( + engine, + instrument, + qty=qty, + probe_offset_pct=probe_offset_pct, + oracle=venue_oracle(max_cost), + mode_label=mode, + ) + finally: + if engine.store is not None: + engine.store.close() + + if db_path is not None: + return await _build_and_run(db_path) + with tempfile.TemporaryDirectory(prefix="trading-bot-canary-") as tmp_dir: + return await _build_and_run(pathlib.Path(tmp_dir) / "canary.sqlite") + + +@app.command() +def canary( + exchange: str = typer.Option( + "binance", + "--exchange", + help="Venue to size the round-trip against (public endpoints only: " + "min_qty/min_notional via the instrument-spec resolver, last price " + "via the public ticker endpoint). Supported: kraken, binance.", + ), + symbol: str = typer.Option( + "BTC/USDT", "--symbol", help="Canonical pair to trade (BASE/QUOTE)." + ), + mode: str = typer.Option( + "paper", + "--mode", + help="paper (default, self-contained, no venue/no key) | testnet " + "(the venue's sandbox — fake money, real API; needs the venue's " + "testnet credentials, e.g. BINANCE_TESTNET_API_KEY/_SECRET; no " + "live_enabled needed — the adapter is hard-pinned to the sandbox) | " + "live (REAL money — requires --config with live_enabled: true, " + "credentials, all risk limits set, and a typed confirmation).", + ), + budget: float = typer.Option( + 100.0, + "--budget", + help="Quote-currency balance to fund the canary's own dedicated " + "paper engine with (AppConfig.paper_starting_balances) — always a " + "fresh engine, never a shared book. Paper only: on testnet/live the " + "account's real balances fund the run.", + ), + max_cost: float = typer.Option( + 2.0, + "--max-cost", + help="The round-trip's cost bound, in quote units. Paper: refuse to " + "start (before any order) if the exact implied cost " + "(2 x qty x mark x fee_bps/10000, at the simulator's default 10bps " + "fee) exceeds it. Testnet/live: the venue oracle's bounded-cost " + "check — the venue-reported quote balance delta must not cost more " + "than this.", + ), + probe_offset_pct: float | None = typer.Option( + None, + "--probe-offset-pct", + help="How far below the mark (percent) the free cancel probe is " + "priced. Default: 50 on paper; 15 on testnet/live — real venues " + "reject a limit too far off the mark (Binance's " + "PERCENT_PRICE_BY_SIDE band is roughly +/-20%), and 15% below still " + "never fills within a seconds-long run.", + ), + db_path: pathlib.Path | None = typer.Option( + None, + "--db", + help="Persist the canary's evidence trail to this SqliteStore path. " + "Defaults to a scratch temp store, discarded when the run ends.", + ), + config_path: pathlib.Path | None = typer.Option( + None, + "--config", + "-c", + help="Operator AppConfig YAML — required for --mode live (its " + "live_enabled opt-in, risk limits and the venue's brokers entry gate " + "the run; the canary never sets live_enabled itself). Unused by " + "paper/testnet.", + ), +) -> None: + """Run the platform's deterministic self-test and print the evidence table. + + A one-liner that makes the canary (:mod:`trading_bot.application.canary`) + runnable without wiring anything by hand: it builds the canary's own + **dedicated** engine (never a shared book), sizes the round-trip to the + venue's real smallest legal quantity, runs the sequential round-trip plus + the two free probes, and prints one line per check (PASS/FAIL, name, + expected, observed) followed by the total cost and the verdict. Exits + ``0`` when every check passed, ``1`` otherwise — so a CI job or a release + checklist can gate on it directly. + + Three modes share the one scenario, with the mode's oracle: + + * ``--mode paper`` (default) — a fresh strict-paper engine funded with + ``--budget``; the **exact** paper oracle. No venue, no key. + * ``--mode testnet`` — the venue's sandbox (fake money, real API): the + factory's testnet path (hard-pinned sandbox URL, testnet credentials), + the venue **identity** oracle (our fills/balances == venue-reported, + cost bounded by ``--max-cost``). Kraken has no public spot testnet and + is refused with a clear error. + * ``--mode live`` — REAL money, the operator's go-live act: gated by the + existing ``live_enabled`` config opt-in (``--config``), venue + credentials, the mandatory live risk limits, **and** a typed + confirmation (the dashboard's go-live pattern). Same identity oracle. + """ + if mode not in ("paper", "testnet", "live"): + raise typer.BadParameter( + f"--mode must be one of paper, testnet, live; got {mode!r}" + ) + live_base: AppConfig | None = None + if mode == "live": + # Every gate is checked (and the confirmation typed) BEFORE anything + # is built or sized — a refused live canary never touches the venue. + live_base = _confirm_live_canary(config_path) + + # The probe offset's default is per-mode: 50% below on paper (maximally + # far-off, exactly reproducible), 15% below on a real venue — inside the + # venue's limit-price band (see the option help), still never filling. + if probe_offset_pct is not None: + offset = money(str(Decimal(str(probe_offset_pct)))) + else: + offset = _PAPER_PROBE_OFFSET_PCT if mode == "paper" else _VENUE_PROBE_OFFSET_PCT + + try: + if mode == "paper": + report = asyncio.run( + _run_paper_canary( + exchange=exchange, + symbol=symbol, + budget=money(str(Decimal(str(budget)))), + max_cost=money(str(Decimal(str(max_cost)))), + probe_offset_pct=offset, + db_path=db_path, + ) + ) + else: + report = asyncio.run( + _run_venue_canary( + exchange=exchange, + symbol=symbol, + mode=mode, + max_cost=money(str(Decimal(str(max_cost)))), + probe_offset_pct=offset, + db_path=db_path, + live_base=live_base, + ) + ) + except Exception as exc: # noqa: BLE001 - surface any refusal/build failure cleanly + _console.print(f"[red]refusing to run canary:[/red] {exc}") + raise typer.Exit(code=1) from exc + + _console.print( + f"canary — exchange={report.exchange} mode={report.mode} " + f"instrument={report.instrument} qty={_render.fmt_money(report.qty)}" + ) + _console.print(_render.canary_table(report.checks)) + _console.print( + f"cost: {'n/a' if report.cost is None else _render.fmt_money(report.cost)}" + ) + verdict = "[green]PASS[/green]" if report.passed else "[red]FAIL[/red]" + _console.print(f"result: {verdict}") + raise typer.Exit(code=0 if report.passed else 1) + + # --- serve ----------------------------------------------------------------- # diff --git a/trading_bot/interfaces/ui/templates/base.html b/trading_bot/interfaces/ui/templates/base.html index d63845c4..3ca4a117 100644 --- a/trading_bot/interfaces/ui/templates/base.html +++ b/trading_bot/interfaces/ui/templates/base.html @@ -118,6 +118,36 @@ .num { text-align:right; font-family:var(--font-mono); } .empty td { color:var(--muted); text-align:center; padding:1rem; } .side-buy { color:var(--ok); } .side-sell { color:var(--err); } + /* Expandable table rows (leaf 01, dashboard-tables-ux) — the shared + click-a-row-for-detail pattern (positions here; orders/fills reuse it in + leaf 02). A disclosure chevron on the first cell rotates open; the row + itself is the click/keyboard target, so no dedicated toggle control eats + column width. */ + tr.expandable { cursor:pointer; } + tr.expandable:hover td { background:var(--panel-hi); } + tr.expandable:focus-visible { outline:2px solid var(--accent); outline-offset:-2px; } + tr.expandable td:first-child { position:relative; padding-left:1.3rem; } + tr.expandable td:first-child::before { + content:'\25b8'; position:absolute; left:.3rem; top:.5rem; color:var(--muted); + font-size:.7em; transition:transform .12s; } + tr.expandable.is-expanded td:first-child::before { transform:rotate(90deg); } + /* The detail panel itself — one full-width under the owner row, laid + out as a responsive grid of label/value pairs (a
), reusing the + card's muted-label language rather than inventing new tokens. */ + tr.detail-row td { background:var(--panel-hi); padding:.8rem 1.1rem; } + .detail-grid { display:grid; grid-template-columns:repeat(auto-fit, minmax(160px, 1fr)); + gap:.5rem 1.4rem; margin:0; } + .detail-grid .detail-item { margin:0; } + .detail-grid dt { color:var(--muted); font-size:.72rem; text-transform:uppercase; + letter-spacing:.08em; margin:0; } + .detail-grid dd { margin:.15rem 0 0; font-family:var(--font-mono); font-size:.86rem; } + /* An order's expanded detail (leaf 02) nests its fills as a small table — + scope out the detail-cell padding/background the descendant + `tr.detail-row td` rule above would otherwise stamp on the nested cells. */ + tr.detail-row .order-fills { margin-top:.7rem; } + tr.detail-row .order-fills th, tr.detail-row .order-fills td { + background:none; padding:.3rem .5rem; } + tr.detail-row .order-fills-none { margin:.6rem 0 0; font-size:.85rem; } /* Strategy-name -> detail-page link, shared across the roster, Overview and Orders name cells so every place a strategy is named is deep-linkable. */ a.strat-link { color:var(--accent); text-decoration:none; } @@ -345,6 +375,17 @@ }); } + // A bare "YYYY-MM-DD" local date (no time) — used for as-of DATA dates + // (the Last-eval column's "data through 2026-07-04", the bar-timing + // chip's tooltip below), distinct from fmtDate's evaluation-INSTANT + // timestamp (which carries a time-of-day). Shared here (leaf 03, + // dashboard-tables-ux) so every page can format a data-through date, not + // just strategies.html. + function fmtDateOnly(ms) { + if (ms == null) return '—'; + return new Date(ms).toLocaleDateString('en-CA'); // en-CA -> YYYY-MM-DD + } + // Order-status -> badge CSS suffix (the .badge-status-* rules above) — one // lowercase-status map shared by the Orders page and Overview's open-orders // table so both surfaces colour-code statuses identically. Covers every @@ -389,6 +430,141 @@ escapeHtml(s.health) + ''; } + // barTimingChipHtml(s) -> the "last bar X ago -> next in Y" chip (leaf 03, + // dashboard-tables-ux), replacing the epoch-aligned "Next bar" guess (the + // now-retired epoch-aligned-boundary helper the roster used to call — it + // had no relation to the real cadence). "last" is `last_asof_ts` — server + // truth, the as-of of the last bar the runner actually evaluated; "next" + // is the honest derivation `last_asof_ts + span * 1000`, fed into the SAME + // `data-countdown-ts` / `tbTime.countdown()` live-refresh mechanism the + // health chip already uses — no new timer machinery. The tooltip carries + // both absolute instants plus "data through " (the Last-eval + // column's wording, kept verbatim). `last_asof_ts == null` (never + // evaluated) -> a muted "—" with an explanatory tooltip, NEVER a fake + // countdown. A unit whose span is missing/zero shows "last" only — there + // is no honest "next" to derive without a cadence. + function barTimingChipHtml(s) { + if (!s || s.last_asof_ts == null) { + return '—'; + } + const last = s.last_asof_ts; + const lastText = 'last bar ' + escapeHtml(fmtRelative(last)); + if (!s.span || s.span <= 0) { + const title = 'last bar ' + fmtDate(last, { seconds: true }) + + ' — data through ' + fmtDateOnly(last); + return '' + lastText + ''; + } + const next = last + s.span * 1000; + const cadence = tbTime.humanSpan(s.span); + const title = 'last bar ' + fmtDate(last, { seconds: true }) + + ' · next bar ' + fmtDate(next, { seconds: true }) + + ' — data through ' + fmtDateOnly(last) + (cadence ? ' (' + cadence + ' bar close)' : ''); + return '' + lastText + ' → next in ' + + '' + escapeHtml(tbTime.countdown(next)) + + ''; + } + + // --- Order-execution cell builders (leaf 02, dashboard-tables-ux) ------ // + // Shared by the three order surfaces (Overview open-orders, Orders page, + // strategy detail "Recent orders") so all render execution identically. + // Money/qty arrive as EXACT Decimal strings; the arithmetic here (a + // ratio, a product, a fee sum) is Number()-based and DISPLAY ONLY — the + // exact operand strings ride the title tooltip, never a rounded result. + + // orderFilledPctCell(o) -> the "Filled %" cell: filled_qty / qty as a + // percentage, the exact fraction in the tooltip. "—" when qty is missing, + // zero or unparseable (a percentage of nothing is a lie, not a zero). + function orderFilledPctCell(o) { + const qtyN = Number(o.qty); + const filledN = Number(o.filled_qty); + if (o.qty == null || o.filled_qty == null || + !isFinite(qtyN) || qtyN === 0 || !isFinite(filledN)) { + return '—'; + } + return tbFmt.pctCell(filledN / qtyN, { title: o.filled_qty + ' / ' + o.qty }); + } + + // orderValueCell(o) -> the "Value" cell: what the order is actually worth + // — filled_qty × avg_fill_price once anything filled, else + // qty × limit_price while the order is priced but unfilled, else "—" (an + // unfilled market order has no honest value yet). The tooltip names WHICH + // product is shown, with the exact Decimal factors. + function orderValueCell(o) { + const unit = tbFmt.splitInstrument(o.instrument); + const filledN = Number(o.filled_qty); + const avgN = Number(o.avg_fill_price); + if (o.filled_qty != null && o.avg_fill_price != null && + isFinite(filledN) && filledN > 0 && isFinite(avgN)) { + return tbFmt.moneyCell(filledN * avgN, { quote: unit.quote, + title: 'filled × avg fill: ' + o.filled_qty + ' × ' + o.avg_fill_price }); + } + const qtyN = Number(o.qty); + const limitN = Number(o.limit_price); + if (o.qty != null && o.limit_price != null && isFinite(qtyN) && isFinite(limitN)) { + return tbFmt.moneyCell(qtyN * limitN, { quote: unit.quote, + title: 'qty × limit price (nothing filled): ' + o.qty + ' × ' + o.limit_price }); + } + return '—'; + } + + // orderDetailHtml(o, fills) -> the expanded detail panel for one order + // (the tbExpander pattern below): client/venue ids, limit/stop prices, + // the fill count + Σ fees, reject_reason when the payload carries one, + // then THAT order's fills — client-joined by client_order_id from the + // fills payload the page already fetched (never a per-expansion request), + // oldest-first (execution order, unlike the most-recent-first flat + // tables). Used by the Orders page + the strategy detail; the Overview's + // open-orders table shows the execution columns without the expansion + // (it does not fetch /api/fills — see overview.html). + function orderDetailHtml(o, fills) { + const unit = tbFmt.splitInstrument(o.instrument); + fills = fills || []; + // Σ fees across this order's fills — display-only Number arithmetic; + // each fill's exact fee string stays one hover away per row below. + let feeSum = 0; + let haveFee = false; + fills.forEach(function (f) { + const n = Number(f.fee); + if (isFinite(n)) { feeSum += n; haveFee = true; } + }); + let html = '
' + + '
Client order id
' + + escapeHtml(o.client_order_id) + '
' + + '
Venue order id
' + + (o.venue_order_id ? escapeHtml(o.venue_order_id) : '—') + + '
' + + '
Limit price
' + + tbFmt.priceCell(o.limit_price, { quote: unit.quote }) + '
' + + '
Stop price
' + + tbFmt.priceCell(o.stop_price, { quote: unit.quote }) + '
' + + '
Fills
' + fills.length + '
' + + '
Σ fees
' + + (haveFee ? tbFmt.moneyCell(feeSum, { quote: unit.quote }) : '—') + + '
' + + (o.reject_reason + ? '
Reject reason
' + + escapeHtml(o.reject_reason) + '
' + : '') + + '
'; + if (!fills.length) { + html += '

No fills for this order.

'; + } else { + html += '' + + '' + + '' + + '' + fills.map(function (f) { + return '' + + '' + + '' + + '' + + '' + + ''; + }).join('') + '
TimeQtyPriceFee
' + + escapeHtml(fmtDate(f.ts, { seconds: true })) + '' + tbFmt.qtyCell(f.qty, { base: unit.base }) + '' + tbFmt.priceCell(f.price, { quote: unit.quote }) + '' + tbFmt.moneyCell(f.fee, { quote: unit.quote }) + '
'; + } + return html; + } + // tbRefreshGuard(key, tableEl, payload) -> 'render' | 'skip-stamp' | 'skip-silent' // Shared by the Strategies/Overview/Orders polling refreshes so a table // never visibly rebuilds when nothing changed, and never rebuilds out from @@ -413,10 +589,99 @@ return unchanged ? 'skip-stamp' : 'render'; } - // tbTime — shared cadence helpers reused by base.html's health chip and by - // strategies.html's Cadence/Next-bar columns. Plain global (like TB_READ_ONLY - // above), not window-attached — every page's inline script runs in the same - // classic-script global scope as this one, after this block has executed. + // tbExpander(tbodyEl, lookup) -> { expanded, rowHtml(key, cellsHtml, colspan) } + // The shared expandable-row helper (leaf 01, dashboard-tables-ux; positions + // here, orders/fills reuse it in leaf 02) — the pinned UX pattern is "click a + // row -> a detail panel opens beneath it" (not a hover popin, not a separate + // page). Every table this touches is FULLY REBUILT on each SSE/poll refresh + // (a page's `refreshAll()` replaces `tableEl.innerHTML` wholesale), so "is + // this row open" cannot live on the DOM node between refreshes — it lives in + // `expanded`, a `Set` of row keys created once per table and never reset; + // `rowHtml()` re-checks it on every rebuild so an open row re-appears + // expanded immediately (no flash of collapsed-then-reopened), and the + // click/keyboard handlers below toggle it in place without waiting for the + // next rebuild. + // + // tbodyEl — the persistent element itself (only its innerHTML + // changes on refresh; the listeners wired here on the element + // survive that, since delegation reads `event.target` fresh). + // lookup — (key) -> the detail panel's inner HTML for that key (a + // pre-escaped string), or a falsy value when there is nothing to + // show (e.g. the row disappeared from the latest payload). + // Called fresh on every open/rebuild, never cached, so the + // detail always reflects the last fetched data. + // + // `rowHtml(key, cellsHtml, colspan)` builds one owner `` (focusable, + // `role="button"`, `aria-expanded`) wrapping the caller's `...` cells, + // followed by the detail `` when `key` is already expanded. + // `colspan` must equal the number of visible columns in `cellsHtml` (the + // detail panel spans the full row). + function tbExpander(tbodyEl, lookup) { + const expanded = new Set(); + + function detailRowHtml(key, colspan) { + const inner = lookup(key); + if (!inner) return ''; + return '' + + '' + inner + ''; + } + + function rowHtml(key, cellsHtml, colspan) { + const isOpen = expanded.has(key); + return '' + + cellsHtml + '' + (isOpen ? detailRowHtml(key, colspan) : ''); + } + + // Toggle one row in place — no table rebuild needed for a click/Enter. + function toggle(row) { + const key = row.dataset.key; + const colspan = row.dataset.colspan; + if (expanded.has(key)) { + expanded.delete(key); + row.classList.remove('is-expanded'); + row.setAttribute('aria-expanded', 'false'); + const next = row.nextElementSibling; + if (next && next.classList.contains('detail-row')) next.remove(); + } else { + expanded.add(key); + row.classList.add('is-expanded'); + row.setAttribute('aria-expanded', 'true'); + const html = detailRowHtml(key, colspan) || + ''; + row.insertAdjacentHTML('afterend', html); + } + } + + // Event delegation on the (persistent) tbody — wired once, keeps working + // after innerHTML rebuilds. Never hijacks a click/keypress on a link, + // button or form control nested in the row (e.g. a strategy-link cell). + function isInnerControl(target) { + return !!target.closest('a, button, input, select, textarea, label'); + } + tbodyEl.addEventListener('click', function (ev) { + if (isInnerControl(ev.target)) return; + const row = ev.target.closest('tr.expandable'); + if (row && tbodyEl.contains(row)) toggle(row); + }); + tbodyEl.addEventListener('keydown', function (ev) { + if (ev.key !== 'Enter' && ev.key !== ' ') return; + if (isInnerControl(ev.target)) return; + const row = ev.target.closest('tr.expandable'); + if (!row || !tbodyEl.contains(row)) return; + ev.preventDefault(); // Space must not also scroll the page + toggle(row); + }); + + return { expanded: expanded, rowHtml: rowHtml }; + } + + // tbTime — shared cadence helpers reused by base.html's health chip and + // bar-timing chip, and by strategies.html's Cadence column. Plain global + // (like TB_READ_ONLY above), not window-attached — every page's inline + // script runs in the same classic-script global scope as this one, after + // this block has executed. const tbTime = { // countdown(tsMs) -> a compact, stable-width duration string counting down // to an epoch-ms timestamp: "42s", "12m05s", "1h05m", "1d02h". Clamps to @@ -449,23 +714,15 @@ if (seconds % 60 === 0) return (seconds / 60) + 'm'; return seconds + 's'; }, - // nextBarCloseMs(spanSeconds) -> the next epoch-aligned bar boundary, in - // epoch ms (UTC): ceil(now_s / span) * span * 1000. Null for a missing/ - // zero/negative span. - nextBarCloseMs: function (spanSeconds) { - if (spanSeconds == null || spanSeconds <= 0) return null; - const nowS = Date.now() / 1000; - return Math.ceil(nowS / spanSeconds) * spanSeconds * 1000; - }, }; // Re-read every element carrying a `data-countdown-ts` attribute and // re-render its text via tbTime.countdown() — one shared 1s interval driving - // every countdown on the page (health chip, Overview's next-tick chip, - // strategies.html's Next-bar column) instead of a timer per element. A - // table rebuild (e.g. strategies.html's 3s poll) simply re-stamps the - // attribute on the new elements; this loop just re-reads whatever is - // currently in the DOM. + // every countdown on the page (health chip, Overview's next-tick chip, the + // bar-timing chip's "next in Y") instead of a timer per element. A table + // rebuild (e.g. strategies.html's 3s poll) simply re-stamps the attribute + // on the new elements; this loop just re-reads whatever is currently in + // the DOM. function tickCountdowns() { document.querySelectorAll('[data-countdown-ts]').forEach(function (el) { const ts = Number(el.getAttribute('data-countdown-ts')); @@ -564,7 +821,7 @@ setInterval(refreshHealth, 10000); // The one shared 1s countdown tick for the whole page (health chip, and // any per-page data-countdown-ts element, e.g. the Overview summary - // strip's next-tick chip or strategies.html's Next-bar column). + // strip's next-tick chip or the bar-timing chip's "next in Y"). tickCountdowns(); setInterval(tickCountdowns, 1000); }); diff --git a/trading_bot/interfaces/ui/templates/orders.html b/trading_bot/interfaces/ui/templates/orders.html index fec96d43..bc7b6957 100644 --- a/trading_bot/interfaces/ui/templates/orders.html +++ b/trading_bot/interfaces/ui/templates/orders.html @@ -25,7 +25,8 @@

Filter

- +

Orders (open + recent)

@@ -42,13 +43,14 @@

Orders (open + recent)

Side Type Qty - Filled + Filled % Avg fill + Value Status - Loading… + Loading…
@@ -58,7 +60,10 @@

Orders (open + recent)

- +

Fills (history)

@@ -72,14 +77,16 @@

Fills (history)

Strategy Exchange Instrument + Order Side Qty Price + Value Fee - Loading… + Loading…
@@ -152,15 +159,62 @@

Fills (history)

} // --- Orders (open + recent history) ------------------------------------ // + // Blueprint (dashboard-tables-ux leaf 02): Time | Strategy | Exchange | + // Instrument | Side | Type | Qty | Filled % | Avg fill | Value | Status — + // the Strategy/Exchange tags stay (this is the cross-strategy audit view, + // like the Fills table below); the Filled %/Value cells come from + // base.html's shared builders. A row expands (the shared tbExpander) into + // that order's fills + ids/limit/stop/Σ fees, joined client-side by + // client_order_id from the fills payload loadFills() below already fetches + // (the same filtered/capped window the Fills table shows). + + // Latest fetched order rows, keyed by client_order_id — read fresh by the + // expander's detail lookup on every open/rebuild. + var _ordersByKey = Object.create(null); + // client_order_id -> that order's fills, oldest-first (execution order). + var _fillsByOrder = Object.create(null); + + // The shared expandable-row helper (base.html) — one instance for the + // orders table; its expanded-keys Set survives every SSE/poll rebuild. + var orderExpander = tbExpander(document.getElementById('orders-body'), function (key) { + var o = _ordersByKey[key]; + return o ? orderDetailHtml(o, _fillsByOrder[key] || []) : ''; + }); + + function orderRowHtml(o) { + var sideCls = o.side === 'buy' ? 'side-buy' : 'side-sell'; + var unit = tbFmt.splitInstrument(o.instrument); + // Absolute time with seconds, visible; relative ("3m ago") on hover — + // mirrors the Fills table's Time column exactly. + var cells = + '' + escapeHtml(fmtDate(o.ts, { seconds: true })) + '' + + '' + stratLink(o.strategy) + '' + + '' + escapeHtml(o.exchange) + '' + + '' + escapeHtml(o.instrument) + '' + + '' + escapeHtml(o.side) + '' + + '' + escapeHtml(o.type) + '' + + '' + tbFmt.qtyCell(o.qty, { base: unit.base }) + '' + + '' + orderFilledPctCell(o) + '' + + '' + tbFmt.priceCell(o.avg_fill_price, { quote: unit.quote }) + '' + + '' + orderValueCell(o) + '' + + '' + statusBadge(o.status) + ''; + return orderExpander.rowHtml(o.client_order_id, cells, 11); + } + async function loadOrders() { var rows; try { rows = await api('GET', '/api/orders' + filterQuery('history=true')); } catch (e) { document.getElementById('orders-body').innerHTML = - '' + escapeHtml(e.message) + ''; + '' + escapeHtml(e.message) + ''; return []; } + // Re-index the expander's lookup before rendering, so a click that fires + // mid-render always sees current data (the positions tables' pattern). + var byKey = Object.create(null); + rows.forEach(function (o) { byKey[o.client_order_id] = o; }); + _ordersByKey = byKey; var body = document.getElementById('orders-body'); // No-flicker refresh — see tbRefreshGuard's doc in base.html. var guard = tbRefreshGuard('orders', body, rows); @@ -169,43 +223,56 @@

Fills (history)

toggleCapCaption('orders-cap', rows); if (guard === 'skip-stamp') return rows; if (!rows.length) { - body.innerHTML = 'No orders.'; + body.innerHTML = 'No orders.'; return rows; } // Most recent first (the API returns oldest-first, mirroring the Fills // table below). var ordered = rows.slice().reverse(); - body.innerHTML = ordered.map(function (o) { - var sideCls = o.side === 'buy' ? 'side-buy' : 'side-sell'; - var unit = tbFmt.splitInstrument(o.instrument); - // Absolute time with seconds, visible; relative ("3m ago") on hover — - // mirrors the Fills table's Time column exactly. - return '' + - '' + escapeHtml(fmtDate(o.ts, { seconds: true })) + '' + - '' + stratLink(o.strategy) + '' + - '' + escapeHtml(o.exchange) + '' + - '' + escapeHtml(o.instrument) + '' + - '' + escapeHtml(o.side) + '' + - '' + escapeHtml(o.type) + '' + - '' + tbFmt.qtyCell(o.qty, { base: unit.base }) + '' + - '' + tbFmt.qtyCell(o.filled_qty, { base: unit.base }) + '' + - '' + tbFmt.priceCell(o.avg_fill_price, { quote: unit.quote }) + '' + - '' + statusBadge(o.status) + '' + - ''; - }).join(''); + body.innerHTML = ordered.map(orderRowHtml).join(''); return rows; } - // --- Fills (confirmed execution history) ------------------------------- // + // --- Fills (confirmed execution history — the flat audit view) ---------- // + + // orderIdCell(id) -> a fill's owning client_order_id, truncated to ~12 + // chars for the column width; the full id rides the title tooltip (leaf 02 + // — the tie back to the order the expansions above key on). + function orderIdCell(id) { + if (!id) return '—'; + var s = String(id); + var short = s.length > 12 ? s.slice(0, 12) + '…' : s; + return '' + escapeHtml(short) + ''; + } + + // fillValueCell(f) -> Value = qty × price (display-only Number arithmetic; + // the exact Decimal factors ride the tooltip). "—" when either is missing. + function fillValueCell(f, unit) { + var qtyN = Number(f.qty); + var priceN = Number(f.price); + if (f.qty == null || f.price == null || !isFinite(qtyN) || !isFinite(priceN)) { + return '—'; + } + return tbFmt.moneyCell(qtyN * priceN, { quote: unit.quote, + title: 'qty × price: ' + f.qty + ' × ' + f.price }); + } + async function loadFills() { var rows; try { rows = await api('GET', '/api/fills' + filterQuery()); } catch (e) { document.getElementById('fills-body').innerHTML = - '' + escapeHtml(e.message) + ''; + '' + escapeHtml(e.message) + ''; return []; } + // Re-index the orders expansions' join source before rendering — the same + // filtered/capped window this table shows. + var byOrder = Object.create(null); + rows.forEach(function (f) { + (byOrder[f.client_order_id] = byOrder[f.client_order_id] || []).push(f); + }); + _fillsByOrder = byOrder; var body = document.getElementById('fills-body'); // No-flicker refresh — see tbRefreshGuard's doc in base.html. var guard = tbRefreshGuard('fills', body, rows); @@ -214,7 +281,7 @@

Fills (history)

toggleCapCaption('fills-cap', rows); if (guard === 'skip-stamp') return rows; if (!rows.length) { - body.innerHTML = 'No fills.'; + body.innerHTML = 'No fills.'; return rows; } // Most recent first (the API returns oldest-first execution order). @@ -228,9 +295,11 @@

Fills (history)

'' + stratLink(f.strategy) + '' + '' + escapeHtml(f.exchange) + '' + '' + escapeHtml(f.instrument) + '' + + '' + orderIdCell(f.client_order_id) + '' + '' + escapeHtml(f.side) + '' + '' + tbFmt.qtyCell(f.qty, { base: unit.base }) + '' + '' + tbFmt.priceCell(f.price, { quote: unit.quote }) + '' + + '' + fillValueCell(f, unit) + '' + '' + tbFmt.moneyCell(f.fee, { quote: unit.quote }) + '' + ''; }).join(''); @@ -249,9 +318,11 @@

Fills (history)

// Load both tables, then refresh the filter option lists from what came back. // The option lists are always rebuilt from the UNFILTERED universe so narrowing // one dimension never drops the others' choices — fetch a bare pass for options. + // Fills load BEFORE orders: the orders rebuild re-renders any open expanded + // detail from the fills join index, so the index must be current first. async function refreshAll() { - var orders = await loadOrders(); var fills = await loadFills(); + var orders = await loadOrders(); // Fetch the unfiltered universe (a light extra read) purely to populate the // selects with every value, not just the currently-filtered subset. try { @@ -290,5 +361,7 @@

Fills (history)

.filter { display:flex; flex-direction:column; gap:.3rem; } .ctl-label { font-size:.78rem; color:var(--muted); text-transform:uppercase; letter-spacing:.09em; } + /* A fill's truncated owning-order id (full id in the tooltip). */ + .order-id { font-family:var(--font-mono); font-size:.82em; } {% endblock %} diff --git a/trading_bot/interfaces/ui/templates/overview.html b/trading_bot/interfaces/ui/templates/overview.html index 8b4c88e9..bbc522d7 100644 --- a/trading_bot/interfaces/ui/templates/overview.html +++ b/trading_bot/interfaces/ui/templates/overview.html @@ -72,15 +72,17 @@

Positions

Exchange - Instrument - Net qty + Asset + Qty Avg entry - Realised PnL - Fees + Price (as-of) + Value + Unrealised + Realised (net) - Loading… + Loading… @@ -103,12 +105,14 @@

Open orders

Side Type Qty - Filled + Filled % + Avg fill + Value Status - Loading… + Loading… @@ -276,29 +280,100 @@

Open orders

// The Strategy column is redundant when grouping by strategy (the group // header already names it) — dropped from both the header and the row cells, // with the rebuilt per grouping mode so colspans stay consistent. + // Blueprint (dashboard-tables-ux leaf 01): positions read asset-first — Asset + // (the base, not the pair) | Qty | Avg entry | Price (as-of, the mark) | + // Value | Unrealised | Realised (net) — plus Strategy/Exchange here on the + // overview variant. A row expands (the shared tbExpander in base.html) into + // the native pair, cumulative fees and the realised gross/fees breakdown. function positionsTheadHtml(includeStrategy) { return '' + (includeStrategy ? 'Strategy' : '') + 'Exchange' + - 'Instrument' + - 'Net qty' + + 'Asset' + + 'Qty' + 'Avg entry' + - 'Realised PnL' + - 'Fees' + + 'Price (as-of)' + + 'Value' + + 'Unrealised' + + 'Realised (net)' + ''; } + + // The Price cell: the mark + a small relative "as of" stamp beside it + // (tooltip: absolute time + mark_source, the freshness-never-lies rule); '—' + // when there is no mark to show (a silent, not-null-columns miss). + function positionMarkCell(r) { + const unit = tbFmt.splitInstrument(r.instrument); + if (r.mark == null) return tbFmt.priceCell(null); + const priceHtml = tbFmt.priceCell(r.mark, { quote: unit.quote }); + if (r.mark_asof_ts == null) return priceHtml; + const title = fmtDate(r.mark_asof_ts, { seconds: true }) + + (r.mark_source ? ' · ' + r.mark_source : ''); + return priceHtml + ' (' + + escapeHtml(fmtRelative(r.mark_asof_ts)) + ')'; + } + + // Value/Unrealised: prefer value_display/unrealised_display + display_currency + // when non-null, else the native amount + the row's fee_ccy (the native + // quote); '—' when both are null — moneyCell already renders that. + function positionMoneyCell(r, nativeKey, displayKey) { + if (r[displayKey] != null && r.display_currency) { + return tbFmt.moneyCell(r[displayKey], { quote: r.display_currency }); + } + return tbFmt.moneyCell(r[nativeKey], { quote: r.fee_ccy }); + } + + // A stable key for one position row — strategy + exchange + instrument, + // since the same instrument can appear once per strategy (positions are + // never merged across strategies/exchanges). + function positionKey(r) { + return r.strategy + '|' + r.exchange + '|' + r.instrument; + } + + // Expanded detail: the native instrument pair, cumulative fees, the realised + // gross-vs-fees breakdown (gross = realised_pnl + fees_paid, computed + // client-side — display only, never fed back into a computation), and the + // mark's source + absolute as-of. + function positionDetailHtml(r) { + const gross = Number(r.realised_pnl) + Number(r.fees_paid); + const asofAbs = r.mark_asof_ts != null ? fmtDate(r.mark_asof_ts, { seconds: true }) : null; + return '
' + + '
Native pair
' + escapeHtml(r.instrument) + '
' + + '
Fees paid
' + tbFmt.moneyCell(r.fees_paid, { quote: r.fee_ccy }) + '
' + + '
Realised gross
' + tbFmt.moneyCell(gross, { quote: r.fee_ccy }) + '
' + + '
Realised net (gross − fees)
' + tbFmt.moneyCell(r.realised_pnl, { quote: r.fee_ccy }) + '
' + + '
Mark source
' + (r.mark_source ? escapeHtml(r.mark_source) : '—') + '
' + + '
Mark as of
' + (asofAbs ? escapeHtml(asofAbs) : '—') + '
' + + '
'; + } + + // Latest fetched rows, keyed by positionKey() — the expander's detail lookup + // reads this fresh on every open/rebuild rather than closing over a stale + // row, so a re-render never shows yesterday's fees_paid under today's key. + let _positionsByKey = Object.create(null); + + // The shared expandable-row helper (base.html) — one instance for this + // page's positions table, created once so its expanded-keys Set survives + // every SSE/poll rebuild of #positions-body. + const posExpander = tbExpander(document.getElementById('positions-body'), function (key) { + const r = _positionsByKey[key]; + return r ? positionDetailHtml(r) : ''; + }); + function positionRowHtml(r, includeStrategy) { const unit = tbFmt.splitInstrument(r.instrument); // {base, quote} - return '' + + const cells = (includeStrategy ? '' + stratLink(r.strategy) + '' : '') + '' + escapeHtml(r.exchange) + '' + - '' + escapeHtml(r.instrument) + '' + + '' + escapeHtml(r.base) + '' + '' + tbFmt.qtyCell(r.net_qty, { base: unit.base }) + '' + '' + tbFmt.priceCell(r.avg_entry_price, { quote: unit.quote }) + '' + - '' + - tbFmt.moneyCell(r.realised_pnl, { quote: unit.quote }) + '' + - '' + tbFmt.moneyCell(r.fees_paid, { quote: unit.quote }) + '' + - ''; + '' + positionMarkCell(r) + '' + + '' + positionMoneyCell(r, 'value', 'value_display') + '' + + '' + positionMoneyCell(r, 'unrealised', 'unrealised_display') + '' + + '' + tbFmt.moneyCell(r.realised_pnl, { quote: r.fee_ccy }) + ''; + const colspan = includeStrategy ? 9 : 8; + return posExpander.rowHtml(positionKey(r), cells, colspan); } function groupHeaderHtml(name, count, colspan) { return '' + @@ -307,7 +382,7 @@

Open orders

} async function loadPositions() { const includeStrategy = POS_GROUP !== 'strategy'; - const colspan = includeStrategy ? 7 : 6; + const colspan = includeStrategy ? 9 : 8; document.getElementById('positions-thead').innerHTML = positionsTheadHtml(includeStrategy); let data; const url = POS_GROUP @@ -320,6 +395,12 @@

Open orders

'' + escapeHtml(e.message) + ''; return; } + // Re-index the lookup the expander's detail-panel builder reads — before + // any rendering, so a click that fires mid-render always sees current data. + const flatRows = POS_GROUP ? data.flatMap(function (g) { return g.rows; }) : data; + const byKey = Object.create(null); + flatRows.forEach(function (r) { byKey[positionKey(r)] = r; }); + _positionsByKey = byKey; const body = document.getElementById('positions-body'); // No-flicker refresh — keyed by grouping, since switching POS_GROUP always // renders (a different shape) while an unchanged poll at the same grouping @@ -350,13 +431,21 @@

Open orders

} // --- Open orders ------------------------------------------------------- // + // Blueprint (dashboard-tables-ux leaf 02): the execution columns — Filled % + // | Avg fill | Value — via base.html's shared cell builders, so this table + // reads execution exactly like the Orders page / strategy detail. NO row + // expansion here (unlike those two): the expanded panel's content is the + // order's fills, and this page does not fetch /api/fills — adding that + // fetch to every 10s poll + SSE refresh solely for an optional affordance + // is disproportionate, and a fills-less panel would fork the shared + // orderDetailHtml builder. The Orders page (one click away) expands fully. async function loadOrders() { let rows; try { rows = await api('GET', '/api/orders'); } catch (e) { document.getElementById('orders-body').innerHTML = - '' + escapeHtml(e.message) + ''; + '' + escapeHtml(e.message) + ''; document.getElementById('sum-open-orders').textContent = '—'; return; } @@ -369,7 +458,7 @@

Open orders

stampUpdated('overview-orders-updated'); if (guard === 'skip-stamp') return; if (!rows.length) { - body.innerHTML = 'No open orders.'; + body.innerHTML = 'No open orders.'; return; } body.innerHTML = rows.map(function (o) { @@ -385,7 +474,9 @@

Open orders

'' + escapeHtml(o.side) + '' + '' + escapeHtml(o.type) + '' + '' + tbFmt.qtyCell(o.qty, { base: unit.base }) + '' + - '' + tbFmt.qtyCell(o.filled_qty, { base: unit.base }) + '' + + '' + orderFilledPctCell(o) + '' + + '' + tbFmt.priceCell(o.avg_fill_price, { quote: unit.quote }) + '' + + '' + orderValueCell(o) + '' + '' + statusBadge(o.status) + '' + ''; }).join(''); diff --git a/trading_bot/interfaces/ui/templates/strategies.html b/trading_bot/interfaces/ui/templates/strategies.html index da294734..6309cd10 100644 --- a/trading_bot/interfaces/ui/templates/strategies.html +++ b/trading_bot/interfaces/ui/templates/strategies.html @@ -33,7 +33,7 @@

Strategies

Mode Status Cadence - Next bar + Bar timing Last eval Realised PnL Total value @@ -85,26 +85,6 @@

Strategies

return label == null ? '—' : escapeHtml(label) + ' bars'; } - // Next bar — a live countdown (hooked into base.html's shared - // tickCountdowns()) to the next epoch-aligned bar boundary, with the - // absolute local datetime in `title`. Muted "—" for a stopped unit or a - // missing/zero span — there is nothing to count down to. - function nextBarCell(s) { - if (!s.running) return '—'; - const ts = tbTime.nextBarCloseMs(s.span); - if (ts == null) return '—'; - return '' + - escapeHtml(tbTime.countdown(ts)) + ''; - } - - // A bare "YYYY-MM-DD" local date (no time) — for the as-of DATA date in the - // Last-eval tooltip ("data through 2026-07-04"), distinct from fmtDate's - // evaluation-INSTANT timestamp (which does carry a time-of-day). - function fmtDateOnly(ms) { - if (ms == null) return '—'; - return new Date(ms).toLocaleDateString('en-CA'); // en-CA -> YYYY-MM-DD - } - // Last eval — a compact relative time since the runner's last tick attempt // (`last_eval_ts`, e.g. "3m ago"), with BOTH the absolute evaluation instant // and the as-of DATA date (`last_asof_ts` — the latest bar the runner @@ -148,7 +128,7 @@

Strategies

'' + escapeHtml(s.mode) + '' + '' + status + (health ? ' ' + health : '') + '' + '' + cadenceCell(s) + '' + - '' + nextBarCell(s) + '' + + '' + barTimingChipHtml(s) + '' + '' + lastEvalCell(s) + '' + '' + pnl + '' + '' + totalValue + '' + diff --git a/trading_bot/interfaces/ui/templates/strategy_detail.html b/trading_bot/interfaces/ui/templates/strategy_detail.html index 2a03f488..7bac34ff 100644 --- a/trading_bot/interfaces/ui/templates/strategy_detail.html +++ b/trading_bot/interfaces/ui/templates/strategy_detail.html @@ -19,6 +19,7 @@

{{ strategy_name }}

+

Loading…

← All strategies

@@ -139,22 +140,25 @@

Positions

- - - + + - - + + + + - +
ExchangeInstrumentNet qtyAssetQty Avg entryRealised PnLFeesPrice (as-of)ValueUnrealisedRealised (net)
Loading…
Loading…
- +

Recent orders

@@ -169,38 +173,14 @@

Recent orders

Side Type Qty - Filled + Filled % Avg fill + Value Status - Loading… - - -
-
- - -
-
-

Recent fills

- -
-
- - - - - - - - - - - - - +
TimeInstrumentSideQtyPriceFee
Loading…
Loading…
@@ -340,6 +320,9 @@

Adjust capital

document.getElementById('detail-run-pill').innerHTML = s.running ? 'running' : 'stopped'; + // Bar-timing chip (leaf 03, dashboard-tables-ux) — "last bar X ago -> next + // in Y", the same honest last_asof_ts-derived chip as the roster. + document.getElementById('detail-bar-timing').innerHTML = barTimingChipHtml(s); // Sub-line: kind · exchange · cadence. const cadence = tbTime.humanSpan(s.span); const bits = [escapeHtml(s.kind), escapeHtml(s.exchange)]; @@ -664,18 +647,74 @@

Adjust capital

} // --- Positions (this strategy's bucket of ?group_by=strategy) ------------ // + // Blueprint (dashboard-tables-ux leaf 01): asset-first — Asset (the base, not + // the pair) | Qty | Avg entry | Price (as-of, the mark) | Value | Unrealised + // | Realised (net); this page is already scoped to one strategy/exchange, so + // (unlike the overview variant) no Strategy/Exchange column here. A row + // expands (the shared tbExpander in base.html) into the native pair, + // cumulative fees and the realised gross/fees breakdown. + + // The Price cell: the mark + a small relative "as of" stamp beside it + // (tooltip: absolute time + mark_source); '—' when there is no mark. + function positionMarkCell(r) { + const unit = tbFmt.splitInstrument(r.instrument); + if (r.mark == null) return tbFmt.priceCell(null); + const priceHtml = tbFmt.priceCell(r.mark, { quote: unit.quote }); + if (r.mark_asof_ts == null) return priceHtml; + const title = fmtDate(r.mark_asof_ts, { seconds: true }) + + (r.mark_source ? ' · ' + r.mark_source : ''); + return priceHtml + ' (' + + escapeHtml(fmtRelative(r.mark_asof_ts)) + ')'; + } + + // Value/Unrealised: prefer value_display/unrealised_display + display_currency + // when non-null, else the native amount + the row's fee_ccy; '—' when null. + function positionMoneyCell(r, nativeKey, displayKey) { + if (r[displayKey] != null && r.display_currency) { + return tbFmt.moneyCell(r[displayKey], { quote: r.display_currency }); + } + return tbFmt.moneyCell(r[nativeKey], { quote: r.fee_ccy }); + } + + // Expanded detail: native instrument pair, cumulative fees, the realised + // gross-vs-fees breakdown (gross = realised_pnl + fees_paid, computed + // client-side — display only), mark source + absolute as-of. + function positionDetailHtml(r) { + const gross = Number(r.realised_pnl) + Number(r.fees_paid); + const asofAbs = r.mark_asof_ts != null ? fmtDate(r.mark_asof_ts, { seconds: true }) : null; + return '
' + + '
Native pair
' + escapeHtml(r.instrument) + '
' + + '
Fees paid
' + tbFmt.moneyCell(r.fees_paid, { quote: r.fee_ccy }) + '
' + + '
Realised gross
' + tbFmt.moneyCell(gross, { quote: r.fee_ccy }) + '
' + + '
Realised net (gross − fees)
' + tbFmt.moneyCell(r.realised_pnl, { quote: r.fee_ccy }) + '
' + + '
Mark source
' + (r.mark_source ? escapeHtml(r.mark_source) : '—') + '
' + + '
Mark as of
' + (asofAbs ? escapeHtml(asofAbs) : '—') + '
' + + '
'; + } + + // Latest fetched rows, keyed by instrument (one-per-instrument within a + // strategy) — read fresh by the expander's detail lookup on every open. + let _positionsByInstrument = Object.create(null); + + // The shared expandable-row helper (base.html) — one instance for this + // page's positions table; its expanded-keys Set survives every SSE/poll + // rebuild of #positions-body. + const posExpander = tbExpander(document.getElementById('positions-body'), function (key) { + const r = _positionsByInstrument[key]; + return r ? positionDetailHtml(r) : ''; + }); function positionRowHtml(r) { const unit = tbFmt.splitInstrument(r.instrument); - return '' + - '' + escapeHtml(r.exchange) + '' + - '' + escapeHtml(r.instrument) + '' + + const cells = + '' + escapeHtml(r.base) + '' + '' + tbFmt.qtyCell(r.net_qty, { base: unit.base }) + '' + '' + tbFmt.priceCell(r.avg_entry_price, { quote: unit.quote }) + '' + - '' + - tbFmt.moneyCell(r.realised_pnl, { quote: unit.quote }) + '' + - '' + tbFmt.moneyCell(r.fees_paid, { quote: unit.quote }) + '' + - ''; + '' + positionMarkCell(r) + '' + + '' + positionMoneyCell(r, 'value', 'value_display') + '' + + '' + positionMoneyCell(r, 'unrealised', 'unrealised_display') + '' + + '' + tbFmt.moneyCell(r.realised_pnl, { quote: r.fee_ccy }) + ''; + return posExpander.rowHtml(r.instrument, cells, 7); } async function loadPositions() { @@ -684,24 +723,64 @@

Adjust capital

data = await api('GET', '/api/positions?group_by=strategy'); } catch (e) { document.getElementById('positions-body').innerHTML = - '' + escapeHtml(e.message) + ''; + '' + escapeHtml(e.message) + ''; return; } const g = (data || []).find(function (x) { return x.group === STRATEGY; }); const rows = g ? g.rows : []; + // Re-index before rendering — see the overview page's identical comment. + const byInstrument = Object.create(null); + rows.forEach(function (r) { byInstrument[r.instrument] = r; }); + _positionsByInstrument = byInstrument; const body = document.getElementById('positions-body'); const guard = tbRefreshGuard('detail-positions', body, rows); if (guard === 'skip-silent') return; stampUpdated('positions-updated'); if (guard === 'skip-stamp') return; if (!rows.length) { - body.innerHTML = 'No open positions.'; + body.innerHTML = 'No open positions.'; return; } body.innerHTML = rows.map(positionRowHtml).join(''); } // --- Recent orders (this strategy's history) ----------------------------- // + // Blueprint (dashboard-tables-ux leaf 02): Time | Instrument | Side | Type | + // Qty | Filled % | Avg fill | Value | Status — the Filled %/Value cells come + // from base.html's shared builders. A row expands (the shared tbExpander) + // into that order's fills + ids/limit/stop/Σ fees, joined client-side by + // client_order_id from the fills payload loadFills() below already fetches. + + // Latest fetched order rows, keyed by client_order_id — read fresh by the + // expander's detail lookup on every open/rebuild. + let _ordersByKey = Object.create(null); + // client_order_id -> that order's fills, oldest-first (execution order) — + // the client-side join feeding each expanded panel. + let _fillsByOrder = Object.create(null); + + // The shared expandable-row helper (base.html) — one instance for this + // page's orders table; its expanded-keys Set survives every SSE/poll + // rebuild of #orders-body. + const orderExpander = tbExpander(document.getElementById('orders-body'), function (key) { + const o = _ordersByKey[key]; + return o ? orderDetailHtml(o, _fillsByOrder[key] || []) : ''; + }); + + function orderRowHtml(o) { + const sideCls = o.side === 'buy' ? 'side-buy' : 'side-sell'; + const unit = tbFmt.splitInstrument(o.instrument); + const cells = + '' + escapeHtml(fmtDate(o.ts, { seconds: true })) + '' + + '' + escapeHtml(o.instrument) + '' + + '' + escapeHtml(o.side) + '' + + '' + escapeHtml(o.type) + '' + + '' + tbFmt.qtyCell(o.qty, { base: unit.base }) + '' + + '' + orderFilledPctCell(o) + '' + + '' + tbFmt.priceCell(o.avg_fill_price, { quote: unit.quote }) + '' + + '' + orderValueCell(o) + '' + + '' + statusBadge(o.status) + ''; + return orderExpander.rowHtml(o.client_order_id, cells, 9); + } async function loadOrders() { let rows; @@ -709,69 +788,45 @@

Adjust capital

rows = await api('GET', '/api/orders?history=true&strategy=' + encodeURIComponent(STRATEGY)); } catch (e) { document.getElementById('orders-body').innerHTML = - '' + escapeHtml(e.message) + ''; + '' + escapeHtml(e.message) + ''; return; } + // Re-index the expander's lookup before rendering, so a click that fires + // mid-render always sees current data (the positions tables' pattern). + const byKey = Object.create(null); + rows.forEach(function (o) { byKey[o.client_order_id] = o; }); + _ordersByKey = byKey; const body = document.getElementById('orders-body'); const guard = tbRefreshGuard('detail-orders', body, rows); if (guard === 'skip-silent') return; stampUpdated('orders-updated'); if (guard === 'skip-stamp') return; if (!rows.length) { - body.innerHTML = 'No orders.'; + body.innerHTML = 'No orders.'; return; } // Most recent first (the API returns oldest-first). const ordered = rows.slice().reverse(); - body.innerHTML = ordered.map(function (o) { - const sideCls = o.side === 'buy' ? 'side-buy' : 'side-sell'; - const unit = tbFmt.splitInstrument(o.instrument); - return '' + - '' + escapeHtml(fmtDate(o.ts, { seconds: true })) + '' + - '' + escapeHtml(o.instrument) + '' + - '' + escapeHtml(o.side) + '' + - '' + escapeHtml(o.type) + '' + - '' + tbFmt.qtyCell(o.qty, { base: unit.base }) + '' + - '' + tbFmt.qtyCell(o.filled_qty, { base: unit.base }) + '' + - '' + tbFmt.priceCell(o.avg_fill_price, { quote: unit.quote }) + '' + - '' + statusBadge(o.status) + '' + - ''; - }).join(''); + body.innerHTML = ordered.map(orderRowHtml).join(''); } - // --- Recent fills (this strategy's confirmed executions) ----------------- // + // --- Fills (the join source for the orders' expanded detail) ------------- // + // The page's standalone fills table retired in leaf 02 — fills now live + // under their order (the expansions above). The fetch itself stays: it is + // the client_order_id join source, refreshed on the same cadence. async function loadFills() { let rows; try { rows = await api('GET', '/api/fills?strategy=' + encodeURIComponent(STRATEGY)); } catch (e) { - document.getElementById('fills-body').innerHTML = - '' + escapeHtml(e.message) + ''; - return; - } - const body = document.getElementById('fills-body'); - const guard = tbRefreshGuard('detail-fills', body, rows); - if (guard === 'skip-silent') return; - stampUpdated('fills-updated'); - if (guard === 'skip-stamp') return; - if (!rows.length) { - body.innerHTML = 'No fills.'; - return; + return; // keep the last good join — the orders table reports its own errors } - const ordered = rows.slice().reverse(); - body.innerHTML = ordered.map(function (f) { - const sideCls = f.side === 'buy' ? 'side-buy' : 'side-sell'; - const unit = tbFmt.splitInstrument(f.instrument); - return '' + - '' + escapeHtml(fmtDate(f.ts, { seconds: true })) + '' + - '' + escapeHtml(f.instrument) + '' + - '' + escapeHtml(f.side) + '' + - '' + tbFmt.qtyCell(f.qty, { base: unit.base }) + '' + - '' + tbFmt.priceCell(f.price, { quote: unit.quote }) + '' + - '' + tbFmt.moneyCell(f.fee, { quote: unit.quote }) + '' + - ''; - }).join(''); + const byOrder = Object.create(null); + rows.forEach(function (f) { + (byOrder[f.client_order_id] = byOrder[f.client_order_id] || []).push(f); + }); + _fillsByOrder = byOrder; } // --- Control actions (mode switch, start/stop, remove) ------------------- // @@ -963,8 +1018,9 @@

Adjust capital

// labels want it on first paint). loadStatus().then(function () { loadPnl(); loadCapital(); }); loadPositions(); - loadOrders(); - loadFills(); + // Fills before orders: the orders rebuild re-renders any open expanded + // detail from the fills join index, so the index must be current first. + loadFills().then(loadOrders); } document.addEventListener('DOMContentLoaded', function () { diff --git a/trading_bot/storage/sqlite_store.py b/trading_bot/storage/sqlite_store.py index eb8d7a08..efec5a5d 100644 --- a/trading_bot/storage/sqlite_store.py +++ b/trading_bot/storage/sqlite_store.py @@ -123,6 +123,10 @@ qty TEXT NOT NULL, price TEXT NOT NULL, fee TEXT NOT NULL, + -- The asset ``fee`` is denominated in; NULL means the quote currency (the + -- historic assumption — pre-migration rows backfill to NULL and read back + -- unchanged). Binance charges a market buy's commission in the BASE asset. + fee_asset TEXT, ts INTEGER NOT NULL, mode TEXT NOT NULL DEFAULT 'paper', venue TEXT NOT NULL DEFAULT '', @@ -447,8 +451,8 @@ def _record_fill_tagged(self, fill: Fill, mode: str, venue: str) -> None: """ INSERT OR IGNORE INTO fills ( fill_id, client_order_id, instrument, side, qty, price, - fee, ts, mode, venue - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + fee, fee_asset, ts, mode, venue + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( fill.fill_id, @@ -458,6 +462,7 @@ def _record_fill_tagged(self, fill: Fill, mode: str, venue: str) -> None: str(fill.qty), str(fill.price), str(fill.fee), + fill.fee_asset, fill.ts, mode, venue, @@ -899,7 +904,13 @@ def _row_to_order(row: sqlite3.Row) -> Order: def _row_to_fill(row: sqlite3.Row) -> Fill: - """Rebuild a :class:`Fill` from a stored ``fills`` row (exact Decimal).""" + """Rebuild a :class:`Fill` from a stored ``fills`` row (exact Decimal). + + ``fee_asset`` reads back ``None`` for a pre-migration (backfilled-NULL) + row — the quote-denominated historic assumption, exactly what those rows + meant when written. + """ + fee_asset = row["fee_asset"] if "fee_asset" in row.keys() else None return Fill( fill_id=str(row["fill_id"]), client_order_id=str(row["client_order_id"]), @@ -909,6 +920,7 @@ def _row_to_fill(row: sqlite3.Row) -> Fill: price=money(str(row["price"])), fee=money(str(row["fee"])), ts=int(row["ts"]), + fee_asset=None if fee_asset is None else str(fee_asset), ) @@ -941,16 +953,17 @@ def _row_to_capital_event(row: sqlite3.Row) -> CapitalEvent: def _migrate_fills_tags(conn: sqlite3.Connection) -> None: - """Add the ``mode`` / ``venue`` columns to a pre-existing ``fills`` table. + """Add the ``mode`` / ``venue`` / ``fee_asset`` columns to an old ``fills`` table. A lightweight, idempotent forward migration: ``CREATE TABLE IF NOT EXISTS`` (in :data:`_SCHEMA`) already gives a *fresh* database the tagged columns, but a database created before this leaf has the old ``fills`` shape. This inspects the live columns and ``ALTER TABLE ... ADD COLUMN`` for whichever tag is missing — SQLite backfills every existing row with the column ``DEFAULT`` - (``mode="paper"`` / ``venue=""``), so no row is lost or corrupted and the - money columns are untouched. A no-op once both columns exist (a fresh DB, or a - second open of a migrated one). + (``mode="paper"`` / ``venue=""``; ``fee_asset`` is nullable and backfills to + ``NULL`` — the quote-denominated historic meaning of those rows), so no row is + lost or corrupted and the money columns are untouched. A no-op once every + column exists (a fresh DB, or a second open of a migrated one). """ columns = {row["name"] for row in conn.execute("PRAGMA table_info(fills)")} if "mode" not in columns: @@ -959,6 +972,8 @@ def _migrate_fills_tags(conn: sqlite3.Connection) -> None: ) if "venue" not in columns: conn.execute("ALTER TABLE fills ADD COLUMN venue TEXT NOT NULL DEFAULT ''") + if "fee_asset" not in columns: + conn.execute("ALTER TABLE fills ADD COLUMN fee_asset TEXT") #: The composite primary key the current ``fills`` table carries — a fill's @@ -1006,6 +1021,7 @@ def _migrate_fills_pk(conn: sqlite3.Connection) -> None: qty TEXT NOT NULL, price TEXT NOT NULL, fee TEXT NOT NULL, + fee_asset TEXT, ts INTEGER NOT NULL, mode TEXT NOT NULL DEFAULT 'paper', venue TEXT NOT NULL DEFAULT '', @@ -1013,10 +1029,10 @@ def _migrate_fills_pk(conn: sqlite3.Connection) -> None: ); INSERT OR IGNORE INTO fills_new ( fill_id, client_order_id, instrument, side, qty, price, - fee, ts, mode, venue + fee, fee_asset, ts, mode, venue ) SELECT fill_id, client_order_id, instrument, side, qty, price, - fee, ts, mode, venue + fee, fee_asset, ts, mode, venue FROM fills ORDER BY rowid; DROP TABLE fills; diff --git a/trading_bot/tests/application/test_canary.py b/trading_bot/tests/application/test_canary.py new file mode 100644 index 00000000..8725d37b --- /dev/null +++ b/trading_bot/tests/application/test_canary.py @@ -0,0 +1,420 @@ +"""Tests for the canary scenario and its exact paper oracle. + +The canary (:mod:`trading_bot.application.canary`) is the platform's +deterministic self-test: a sequential round-trip plus two free probes over a +dedicated, explicitly-funded engine, with every expectation vs observation +recorded exactly. These tests drive the **real** engine assembly both ways the +leaf allows — the factory (:func:`~trading_bot.application.service_factory. +build_engine`, funded via ``AppConfig.paper_starting_balances``) and a +hand-wired :class:`~trading_bot.application.service_factory.Engine` (the +deliberately-broken variants) — and assert: + +* the full scenario is green end to end on a fresh strict paper engine, with + the **exact** oracle: realised PnL == −Σ fees, flat, balances moved by the + fees only, both round-trip rows ``FILLED`` in the store; +* a broker that fills the far-off cancel probe is **detected** and the run + stops placing (no round-trip orders ever reach the venue); +* a broken idempotency path (router + venue dedup both disabled) is detected + the same way; +* the oracle catches a seeded imbalance (a fill dropped from the store); +* the printed evidence table is stable and exact. + +Async tests run un-decorated (``asyncio_mode = "auto"``). +""" + +from __future__ import annotations + +# Built-in +import pathlib +import sqlite3 +from decimal import Decimal + +# Third-party +import pytest + +# Local +from trading_bot.application import ( + AppConfig, + CanaryReport, + EventBus, + OrderRouter, + PerformanceService, + PositionTracker, + RiskConfig, + RiskManager, + build_engine, + paper_oracle, + run_canary, +) +from trading_bot.application.order_fill_sync import OrderFillSync +from trading_bot.application.service_factory import Engine +from trading_bot.brokers.paper import PaperBroker +from trading_bot.domain import Instrument, Order, OrderStatus, Symbol, money +from trading_bot.storage.sqlite_store import SqliteStore + +#: A spec-carrying instrument shaped like the real Binance BTC/USDT venue spec +#: (1e-5 lot, 5-quote-unit min-notional), so strict paper quantizes and gates +#: exactly like the live venue would. +INSTRUMENT = Instrument( + Symbol("BTC", "USDT"), + price_precision=2, + qty_precision=5, + min_qty=money("0.00001"), + min_notional=money("5"), +) +MARK = money("50000") +QTY = money("0.001") + + +def _factory_engine(tmp_path: pathlib.Path, *, strict: bool = True) -> Engine: + """A factory-built, funded, strict-paper canary engine with a mark injected.""" + config = AppConfig( + paper_strict=strict, + paper_starting_balances={"USDT": Decimal("1000")}, + ) + engine = build_engine(config, db_path=tmp_path / "canary.sqlite") + broker = engine.broker + assert isinstance(broker, PaperBroker) + broker.set_price(INSTRUMENT, MARK) + return engine + + +def _hand_engine( + tmp_path: pathlib.Path, + *, + broker_cls: type[PaperBroker] = PaperBroker, + router_cls: type[OrderRouter] = OrderRouter, +) -> Engine: + """A hand-wired engine (the factory's exact shape) with injectable classes. + + Mirrors :func:`~trading_bot.application.service_factory.build_engine`'s + wiring order — broker on the bus, tracker/perf subscribed, risk-gated + router, fill sync before the store attaches — but lets a test swap in a + deliberately broken broker/router subclass. Deterministic ids + (``id_token``) and the broker's default deterministic clock. + """ + bus = EventBus() + broker = broker_cls( + prices={INSTRUMENT: MARK}, + starting_balances={"USDT": money("1000")}, + strict=True, + event_bus=bus, + id_token="canarytest", + ) + tracker = PositionTracker(event_bus=bus) + perf = PerformanceService(event_bus=bus) + risk = RiskManager( + RiskConfig(), + position_tracker=tracker, + daily_pnl_provider=perf.realised_pnl_since, + ) + router = router_cls(broker, bus, risk_manager=risk) + fill_sync = OrderFillSync(router, bus) + store = SqliteStore(tmp_path / "canary.sqlite") + store.attach(bus) + return Engine( + config=AppConfig(), + bus=bus, + broker=broker, + router=router, + fill_sync=fill_sync, + tracker=tracker, + perf=perf, + risk=risk, + store=store, + ) + + +def _names(report: CanaryReport) -> list[str]: + """The report's check names, in order.""" + return [check.name for check in report.checks] + + +# --- the full scenario, green end to end ------------------------------------ # + + +async def test_full_scenario_paper_engine_all_green(tmp_path: pathlib.Path) -> None: + """The whole canary passes on a fresh, funded, strict factory engine. + + Every check green, in the pinned order (probes before money), and the + exact paper accounting holds: realised PnL == −Σ fees == −0.10, cost == + 2 × the per-leg fee, quote balance down by exactly the fees, base flat. + """ + engine = _factory_engine(tmp_path) + + report = await run_canary(engine, INSTRUMENT, qty=QTY) + + assert report.passed, report.to_text() + for check in report.checks: + assert check.passed, f"{check.name}: {check.expected} != {check.observed}" + assert _names(report) == [ + "cancel_probe.resting", + "cancel_probe.cancelled", + "cancel_probe.balances_unmoved", + "cancel_probe.cancelled_persisted", + "idempotency.deduped", + "roundtrip.buy_filled", + "roundtrip.buy_in_store", + "roundtrip.buy_in_tracker", + "roundtrip.sell_filled", + "roundtrip.sell_in_store", + "roundtrip.flat", + "oracle.realised_pnl_is_minus_fees", + "oracle.position_flat", + "oracle.buy_filled_in_store", + "oracle.sell_filled_in_store", + "oracle.quote_delta_is_minus_fees", + "oracle.base_delta_zero", + "oracle.fill_count", + ] + + # Exact accounting: fee per leg = 50000 * 0.001 * 10bps = 0.05. + assert engine.perf.fees_paid() == money("0.1") + assert engine.perf.realised_pnl() == money("-0.1") + assert report.cost == money("0.1") # == 2 x the per-leg fee + # Balances moved by the fees only (quote), and the base is exactly flat. + assert report.initial_balances == {"USDT": money("1000")} + assert report.final_balances["USDT"] == money("999.9") + assert report.final_balances["BTC"] == money("0") + + # The store agrees: probe CANCELLED, both round-trip rows FILLED. + store = engine.store + assert store is not None + assert store.get_order(report.probe_cid).status is OrderStatus.CANCELLED + assert store.get_order(report.buy_cid).status is OrderStatus.FILLED + assert store.get_order(report.sell_cid).status is OrderStatus.FILLED + store.close() + + +async def test_probe_qty_scales_up_to_stay_venue_legal( + tmp_path: pathlib.Path, +) -> None: + """A round-trip qty legal at the mark but sub-notional at the probe price. + + ``0.0001 * 50000 = 5`` clears the venue minimum at the mark, but at the + 50%-below probe price (``25000``) the same quantity is worth only ``2.5`` + — a strict venue would reject the probe. The canary scales the probe's + quantity up to the smallest venue-legal size at the probe price + (``5 / 25000 = 0.0002``) and the run stays green. + """ + engine = _factory_engine(tmp_path) + + report = await run_canary(engine, INSTRUMENT, qty=money("0.0001")) + + assert report.passed, report.to_text() + store = engine.store + assert store is not None + probe_row = store.get_order(report.probe_cid) + assert probe_row is not None + assert probe_row.qty == money("0.0002") + assert probe_row.status is OrderStatus.CANCELLED + store.close() + + +# --- caller wiring errors are refused before any order ----------------------- # + + +async def test_canary_refuses_engine_without_store() -> None: + """No dedicated store, no canary: the run is refused before any order. + + ``build_engine`` attaches a store only for an explicit ``db_path`` (no + default), so a store-less engine is exactly the "pointed at nothing of its + own" misuse the canary must make impossible. + """ + engine = build_engine(AppConfig()) + with pytest.raises(ValueError, match="dedicated store"): + await run_canary(engine, INSTRUMENT, qty=QTY) + + +async def test_canary_validates_qty_and_offset(tmp_path: pathlib.Path) -> None: + """A non-positive qty or an out-of-range probe offset is refused early.""" + engine = _factory_engine(tmp_path) + with pytest.raises(ValueError, match="strictly positive"): + await run_canary(engine, INSTRUMENT, qty=money("0")) + with pytest.raises(ValueError, match="probe_offset_pct"): + await run_canary(engine, INSTRUMENT, qty=QTY, probe_offset_pct=money("100")) + # Nothing was placed by the refused runs. + assert await engine.broker.fills() == [] + assert engine.store is not None + engine.store.close() + + +# --- failure detection: cancel probe ----------------------------------------- # + + +async def test_cancel_probe_failure_detected_and_run_stops( + tmp_path: pathlib.Path, +) -> None: + """A broker that fills the far-off probe order fails the check — and the + run places nothing further. + + The permissive (non-strict) simulator is exactly such a broker: it fills + every limit order at its limit price regardless of the mark. The canary + must catch it on the very first check and never reach the round-trip. + """ + engine = _factory_engine(tmp_path, strict=False) + + report = await run_canary(engine, INSTRUMENT, qty=QTY) + + assert not report.passed + assert _names(report) == ["cancel_probe.resting"] + assert not report.checks[0].passed + assert "status=filled" in report.checks[0].observed + + # No further orders were placed: one venue fill (the mis-filled probe), + # one store row, and the round-trip ids never reached router or store. + broker = engine.broker + assert isinstance(broker, PaperBroker) + assert len(await broker.fills()) == 1 + store = engine.store + assert store is not None + store.flush() + assert len(store.orders()) == 1 + assert engine.router.get(report.buy_cid) is None + assert engine.router.get(report.sell_cid) is None + # It still snapshotted and reported what it measured. + assert report.final_balances != {} + assert report.cost is not None + store.close() + + +# --- failure detection: idempotency probe ------------------------------------ # + + +class _NoDedupRouter(OrderRouter): + """A deliberately broken router: forgets an id right before re-submitting it.""" + + async def submit(self, order: Order) -> Order: + self._orders.pop(order.client_order_id, None) + return await super().submit(order) + + +class _NoDedupBroker(PaperBroker): + """A deliberately broken strict broker: forgets its venue-side dedup too.""" + + async def place_order(self, order: Order) -> str: + self._venue_id_by_cid.pop(order.client_order_id, None) + return await super().place_order(order) + + +async def test_idempotency_failure_detected_and_run_stops( + tmp_path: pathlib.Path, +) -> None: + """A duplicate client-order-id that creates a second live order is caught. + + With both dedup layers disabled (router map and strict venue-side dedup), + the probe's re-submission opens a second resting venue order — the check + must fail on the identity *and* the venue open-order count, and the + round-trip must never run. + """ + engine = _hand_engine( + tmp_path, broker_cls=_NoDedupBroker, router_cls=_NoDedupRouter + ) + + report = await run_canary(engine, INSTRUMENT, qty=QTY) + + assert not report.passed + names = _names(report) + assert names[-1] == "idempotency.deduped" + assert all(name.startswith("cancel_probe.") for name in names[:-1]) + dedup_check = report.checks[-1] + assert not dedup_check.passed + assert "a different order returned" in dedup_check.observed + assert "venue_open=1" in dedup_check.observed + + # The round-trip never ran: no fills at all, no buy/sell rows. + broker = engine.broker + assert isinstance(broker, PaperBroker) + assert await broker.fills() == [] + assert engine.router.get(report.buy_cid) is None + assert engine.router.get(report.sell_cid) is None + # Best-effort cleanup cancelled the duplicate: nothing left resting. + assert await broker.open_orders() == [] + assert engine.store is not None + engine.store.close() + + +# --- the oracle catches a seeded imbalance ------------------------------------ # + + +async def test_oracle_catches_seeded_imbalance(tmp_path: pathlib.Path) -> None: + """Dropping one persisted fill turns the oracle red. + + After a green run, delete one round-trip fill row from the store (the kind + of imbalance a lost execution would leave): the fill-count check and the + realised-PnL-vs-store-fees cross-check must both fail. + """ + engine = _factory_engine(tmp_path) + report = await run_canary(engine, INSTRUMENT, qty=QTY) + assert report.passed + + store = engine.store + assert store is not None + store.flush() + conn = sqlite3.connect(tmp_path / "canary.sqlite") + try: + fill_id = conn.execute("SELECT fill_id FROM fills LIMIT 1").fetchone()[0] + conn.execute("DELETE FROM fills WHERE fill_id = ?", (fill_id,)) + conn.commit() + finally: + conn.close() + + checks = {check.name: check for check in paper_oracle(engine, report)} + assert not checks["oracle.fill_count"].passed + assert checks["oracle.fill_count"].observed == "fills=1" + assert not checks["oracle.realised_pnl_is_minus_fees"].passed + store.close() + + +# --- report printing: stable, exact strings ----------------------------------- # + + +async def test_report_printing_stable_and_exact(tmp_path: pathlib.Path) -> None: + """Two identical deterministic runs print the identical evidence table. + + A fixed ``id_token``, the simulator's deterministic clock and a fixed + ``run_id`` make the whole run reproducible; the printed table carries no + timestamps or minted ids, so it must match to the character — and selected + lines are pinned exactly. + """ + texts = [] + for sub in ("a", "b"): + subdir = tmp_path / sub + subdir.mkdir() + engine = _hand_engine(subdir) + report = await run_canary(engine, INSTRUMENT, qty=QTY, run_id="print") + assert report.passed, report.to_text() + texts.append(report.to_text()) + assert engine.store is not None + engine.store.close() + + assert texts[0] == texts[1] + lines = texts[0].splitlines() + assert lines[0] == ( + "canary — exchange=paper mode=paper instrument=BTC/USDT qty=0.001" + ) + assert lines[1] == ( + "[PASS] cancel_probe.resting: " + "expected=status=open filled_qty=0 venue_fills=0 | " + "observed=status=open filled_qty=0 venue_fills=0" + ) + assert lines[5] == ( + "[PASS] idempotency.deduped: " + "expected=original order returned venue_open=0 venue_fills=0 " + "store_orders=1 | " + "observed=original order returned venue_open=0 venue_fills=0 " + "store_orders=1" + ) + # Strict paper quantizes 0.001 to the 1e-5 lot (0.00100): numerically equal + # values pass even when the rendered exponents differ — exact, not fuzzy. + assert lines[6] == ( + "[PASS] roundtrip.buy_filled: " + "expected=status=filled filled_qty=0.001 | " + "observed=status=filled filled_qty=0.00100" + ) + assert lines[12] == ( + "[PASS] oracle.realised_pnl_is_minus_fees: " + "expected=realised_pnl=-0.10000 | observed=realised_pnl=-0.10000" + ) + assert lines[-2] == "cost: 0.10000" + assert lines[-1] == "result: PASS" diff --git a/trading_bot/tests/application/test_canary_venue.py b/trading_bot/tests/application/test_canary_venue.py new file mode 100644 index 00000000..aad2a11b --- /dev/null +++ b/trading_bot/tests/application/test_canary_venue.py @@ -0,0 +1,638 @@ +"""Tests for the canary's **venue** path: the identity oracle + the settle loop. + +The venue oracle (:func:`trading_bot.application.canary.venue_oracle`) asserts +the accounting identity — our recorded fills == the venue-reported fills, the +venue's balance deltas == what our fills imply (the fills math, never a +hardcoded zero), flat — plus a bounded cost. A real (REST) venue also fills +market orders *asynchronously* and keys its fill reports by its **own** order +reference, so :func:`~trading_bot.application.canary.run_canary` settles each +market leg by polling ``broker.fills(...)`` and attributing new fills to the +leg (see the module docstring's "Venue legs settle by polling"). + +These tests drive both against :class:`_RestVenueBroker`, a deliberately +REST-shaped fake: market orders execute **venue-side only** (recorded in its +``fills()`` view, balances mutated — nothing emitted on the bus), limit orders +rest, and every fill's ``client_order_id`` is the venue's own numeric order id +(never ours) — exactly the Binance ``myTrades`` shape the settle loop must +handle. Covered: + +* the whole scenario is green end to end over the REST venue: legs settle by + poll, the store's fills carry OUR client-order-ids while + ``report.venue_fills`` carries the venue's, and every identity/cost check + holds; +* every oracle branch turns red on the seeded violation it guards: a dropped + store fill (fills mismatch), a tampered quote balance (quote delta), venue + base dust our fills do not explain (base delta — expected rendered from the + fills math), a tiny ``max_cost`` (cost bound); +* the base-delta expectation really is the **fills math, not zero**: a venue + over-delivery within the domain fill tolerance leaves base dust that the + check *passes* because our recorded fills imply exactly that dust; +* a venue that never reports the market execution times the settle loop out — + the leg's check fails with what was observed and the abort cleanup cancels + the resting order; +* opt-in (``-m network``): the REAL Binance testnet run — the CLI's testnet + path end to end with the ``BINANCE_TESTNET_*`` keys, fake money, real API. + +Async tests run un-decorated (``asyncio_mode = "auto"``). +""" + +from __future__ import annotations + +# Built-in +import os +import pathlib +import sqlite3 +from dataclasses import replace + +# Third-party +import pytest + +# Local +from trading_bot.application import ( + AppConfig, + CanaryReport, + EventBus, + OrderRouter, + PerformanceService, + PositionTracker, + RiskConfig, + RiskManager, + run_canary, + venue_oracle, +) +from trading_bot.application import canary as canary_mod +from trading_bot.application.events import FillEvent +from trading_bot.application.order_fill_sync import OrderFillSync +from trading_bot.application.service_factory import Engine +from trading_bot.brokers.base import Capability +from trading_bot.domain import Instrument, Order, OrderStatus, Symbol, money +from trading_bot.domain.fill import Fill +from trading_bot.domain.money import Money +from trading_bot.domain.order import OrderSide, OrderType +from trading_bot.storage.sqlite_store import SqliteStore + +#: The same Binance-BTC/USDT-shaped venue spec the paper canary tests use. +INSTRUMENT = Instrument( + Symbol("BTC", "USDT"), + price_precision=2, + qty_precision=5, + min_qty=money("0.00001"), + min_notional=money("5"), +) +MARK = money("50000") +QTY = money("0.001") +MAX_COST = money("2") + +#: Per-leg venue fee at the fake's default 10 bps: 0.001 * 50000 * 0.001 = 0.05. +LEG_FEE = money("0.05") + + +class _RestVenueBroker: + """A REST-shaped fake venue: async fills, venue-keyed fill reports. + + Market orders execute immediately **venue-side**: the execution lands in + the ``fills()`` view and moves the balances, but nothing is emitted on any + bus — exactly like a real REST venue, which only *reports* executions when + polled. Limit orders rest until cancelled. Every fill's + ``client_order_id`` is the venue's own numeric order id (the Binance + ``myTrades`` shape), never the caller's — the canary's settle attribution + must not rely on client-order-id matching. + + Parameters + ---------- + balances : dict of str to Decimal + Starting free balances. + fee_bps : Decimal, optional + Fee rate in basis points, charged in **quote** on each fill. + fill_market : bool, optional + ``False`` makes market orders rest instead of executing — the + "venue never reports the execution" failure mode. + buy_overfill : Decimal, optional + Extra base quantity a market BUY delivers beyond the requested size + (venue lot rounding) — the within-tolerance dust case. Balances move + by the *delivered* quantity. + fee_in_base_on_buys : bool, optional + Charge a market BUY's commission in the **base** asset (taken out of + the delivered quantity), exactly like the real Binance venue — the + base-dust identity case observed live on the testnet. Sells stay + quote-fee'd. + + """ + + name = "restvenue" + + def __init__( + self, + *, + balances: dict[str, Money], + fee_bps: Money = money("10"), + fill_market: bool = True, + buy_overfill: Money = money("0"), + fee_in_base_on_buys: bool = False, + ) -> None: + self._balances: dict[str, Money] = { + asset: money(amount) for asset, amount in balances.items() + } + self._fee_bps = money(fee_bps) + self._fill_market = fill_market + self._buy_overfill = money(buy_overfill) + self._fee_in_base_on_buys = fee_in_base_on_buys + self._prices: dict[Symbol, Money] = {INSTRUMENT.symbol: MARK} + # venue_order_id -> the resting order's rebuild ingredients. + self._open: dict[str, Order] = {} + self._fills: list[Fill] = [] + self._seq = 0 + + def capabilities(self) -> set[Capability]: + return { + Capability.PLACE_ORDER, + Capability.CANCEL, + Capability.OPEN_ORDERS, + Capability.BALANCES, + Capability.FILLS, + Capability.TICKER, + } + + async def ticker(self, instrument: Instrument) -> Money: + return self._prices[instrument.symbol] + + async def balances(self) -> dict[str, Money]: + return { + asset: amount for asset, amount in self._balances.items() if amount != 0 + } + + async def open_orders(self) -> list[Order]: + return list(self._open.values()) + + async def fills(self, since_ms: int | None = None) -> list[Fill]: + return list(self._fills) + + async def place_order(self, order: Order) -> str: + self._seq += 1 + venue_num = str(1000 + self._seq) + venue_id = f"BTCUSDT:{venue_num}" + if order.type is OrderType.MARKET and self._fill_market: + price = self._prices[order.instrument.symbol] + base = order.instrument.symbol.base + quote = order.instrument.symbol.quote + zero = money("0") + qty = order.qty + if order.side is OrderSide.BUY: + qty += self._buy_overfill + base_fee = order.side is OrderSide.BUY and self._fee_in_base_on_buys + # Base-denominated buy commission (the real Binance shape) is a + # cut of the delivered base; otherwise the fee is quote on the + # traded notional. + fee = ( + qty * self._fee_bps / money("10000") + if base_fee + else qty * price * self._fee_bps / money("10000") + ) + self._fills.append( + Fill( + fill_id=f"V{self._seq}", + # The venue's own order reference — NOT the caller's cid. + client_order_id=venue_num, + # Bare instrument, like a real adapter's fill rebuild. + instrument=Instrument(order.instrument.symbol), + side=order.side, + qty=qty, + price=price, + fee=fee, + ts=self._seq, + fee_asset=base if base_fee else None, + ) + ) + if order.side is OrderSide.BUY: + self._balances[quote] = self._balances.get(quote, zero) - ( + qty * price + (zero if base_fee else fee) + ) + self._balances[base] = self._balances.get(base, zero) + ( + qty - (fee if base_fee else zero) + ) + else: + self._balances[base] = self._balances.get(base, zero) - qty + self._balances[quote] = ( + self._balances.get(quote, zero) + qty * price - fee + ) + else: + self._open[venue_id] = order + return venue_id + + async def cancel_order(self, venue_order_id: str) -> None: + self._open.pop(venue_order_id, None) + + +def _venue_engine(tmp_path: pathlib.Path, broker: _RestVenueBroker) -> Engine: + """A hand-wired engine (the factory's exact shape) over the REST fake. + + Mirrors ``build_engine``'s wiring order — tracker/perf subscribed, a + risk-gated router, the fill sync before the store attaches — so the settle + loop's re-emitted fills flow through the very plumbing a real venue run + uses. + """ + bus = EventBus() + tracker = PositionTracker(event_bus=bus) + perf = PerformanceService(event_bus=bus) + risk = RiskManager( + RiskConfig(), + position_tracker=tracker, + daily_pnl_provider=perf.realised_pnl_since, + ) + router = OrderRouter(broker, bus, risk_manager=risk) + fill_sync = OrderFillSync(router, bus) + store = SqliteStore(tmp_path / "canary.sqlite") + store.attach(bus) + return Engine( + config=AppConfig(), + bus=bus, + broker=broker, + router=router, + fill_sync=fill_sync, + tracker=tracker, + perf=perf, + risk=risk, + store=store, + ) + + +async def _green_run( + tmp_path: pathlib.Path, +) -> tuple[Engine, CanaryReport]: + """One full, green venue-canary run over the REST fake (asserted green).""" + engine = _venue_engine(tmp_path, _RestVenueBroker(balances={"USDT": money("1000")})) + report = await run_canary( + engine, + INSTRUMENT, + qty=QTY, + oracle=venue_oracle(MAX_COST), + mode_label="testnet", + ) + assert report.passed, report.to_text() + return engine, report + + +# --- the full scenario, green end to end over a REST-shaped venue ------------ # + + +async def test_venue_scenario_identity_holds(tmp_path: pathlib.Path) -> None: + """The whole canary passes over a REST venue: settle-by-poll + identity oracle. + + The venue reports fills asynchronously and keys them by its own order id; + the run must still confirm both legs, persist them under OUR ids, and the + identity oracle must hold with the exact venue-reported ids/qtys/prices. + """ + engine, report = await _green_run(tmp_path) + + assert report.mode == "testnet" + assert [check.name for check in report.checks] == [ + "cancel_probe.resting", + "cancel_probe.cancelled", + "cancel_probe.balances_unmoved", + "cancel_probe.cancelled_persisted", + "idempotency.deduped", + "roundtrip.buy_filled", + "roundtrip.buy_in_store", + "roundtrip.buy_in_tracker", + "roundtrip.sell_filled", + "roundtrip.sell_in_store", + "roundtrip.flat", + "oracle.fills_match_venue", + "oracle.position_flat", + "oracle.base_delta_matches_fills", + "oracle.quote_delta_matches_fills", + "oracle.cost_bounded", + ] + + # The venue reported the two round-trip fills under ITS OWN references... + assert len(report.venue_fills) == 2 + assert {fill.client_order_id for fill in report.venue_fills} == {"1002", "1003"} + # ...while the store recorded the same executions under OUR leg ids — the + # settle loop's attribution — with identical fill ids. + store = engine.store + assert store is not None + store.flush() + ours = [ + fill + for fill in store.fills() + if fill.client_order_id in {report.buy_cid, report.sell_cid} + ] + assert {fill.fill_id for fill in ours} == { + fill.fill_id for fill in report.venue_fills + } + assert store.get_order(report.buy_cid).status is OrderStatus.FILLED + assert store.get_order(report.sell_cid).status is OrderStatus.FILLED + + # Exact venue accounting: both legs at the same mark, so the cost is the + # two fees; balances moved by exactly what the fills imply. + assert report.cost == 2 * LEG_FEE + assert report.initial_balances == {"USDT": money("1000")} + assert report.final_balances == {"USDT": money("999.9")} + store.close() + + +async def test_venue_scenario_green_with_base_denominated_buy_fee( + tmp_path: pathlib.Path, +) -> None: + """The identity holds when the venue charges the BUY fee in the base asset. + + Exactly the shape the real Binance testnet exhibits (a BTC/USDT market buy + is commissioned in BTC): the venue's base balance ends with a fee-sized + dust the fills math must explain via the fill's ``fee_asset`` — expected + from the fills, not from zero — while the position stays flat (fees never + fold into the position quantity). + """ + broker = _RestVenueBroker( + balances={"USDT": money("1000")}, fee_in_base_on_buys=True + ) + engine = _venue_engine(tmp_path, broker) + + report = await run_canary( + engine, + INSTRUMENT, + qty=QTY, + oracle=venue_oracle(MAX_COST), + mode_label="testnet", + ) + + assert report.passed, report.to_text() + checks = {c.name: c for c in report.checks} + # Buy fee: 0.001 * 10bps = 0.000001 BTC, taken from the delivered base. + base_check = checks["oracle.base_delta_matches_fills"] + assert base_check.expected == "base_delta=-0.000001" + assert base_check.observed == "base_delta=-0.000001" + # Quote moved by the sell fee only (the buy fee never touched quote). + quote_check = checks["oracle.quote_delta_matches_fills"] + assert quote_check.expected == "quote_delta=-0.050" + assert report.cost is not None + if engine.store is not None: + engine.store.close() + + +# --- every oracle branch turns red on its seeded violation ------------------- # + + +async def test_oracle_detects_fills_mismatch(tmp_path: pathlib.Path) -> None: + """Dropping one persisted fill breaks the fills identity, exactly reported.""" + engine, report = await _green_run(tmp_path) + store = engine.store + assert store is not None + store.flush() + conn = sqlite3.connect(tmp_path / "canary.sqlite") + try: + fill_id = conn.execute("SELECT fill_id FROM fills LIMIT 1").fetchone()[0] + conn.execute("DELETE FROM fills WHERE fill_id = ?", (fill_id,)) + conn.commit() + finally: + conn.close() + + checks = {c.name: c for c in venue_oracle(MAX_COST)(engine, report)} + + assert not checks["oracle.fills_match_venue"].passed + # The venue side still lists both executions; ours only one. + assert checks["oracle.fills_match_venue"].expected.count("[id=") == 2 + assert checks["oracle.fills_match_venue"].observed.count("[id=") == 1 + store.close() + + +async def test_oracle_detects_quote_balance_mismatch( + tmp_path: pathlib.Path, +) -> None: + """A venue quote delta our fills do not explain fails the quote identity.""" + engine, report = await _green_run(tmp_path) + + report.final_balances["USDT"] = report.final_balances["USDT"] + money("0.01") + checks = {c.name: c for c in venue_oracle(MAX_COST)(engine, report)} + + quote_check = checks["oracle.quote_delta_matches_fills"] + assert not quote_check.passed + assert quote_check.expected == "quote_delta=-0.100" + assert quote_check.observed == "quote_delta=-0.090" + if engine.store is not None: + engine.store.close() + + +async def test_oracle_reports_unexplained_base_dust_exactly( + tmp_path: pathlib.Path, +) -> None: + """Venue base dust our fills do NOT imply fails — against the fills math. + + The expected side is computed from our recorded fills (buy qty − sell qty + == 0 here), never assumed: the venue reporting a stray base remainder (a + base-denominated fee, say) is an exact, reported identity violation. + """ + engine, report = await _green_run(tmp_path) + + report.final_balances["BTC"] = money("-0.00000001") + checks = {c.name: c for c in venue_oracle(MAX_COST)(engine, report)} + + base_check = checks["oracle.base_delta_matches_fills"] + assert not base_check.passed + assert base_check.expected == "base_delta=0.000" # the fills math, exact + assert base_check.observed == "base_delta=-1E-8" + if engine.store is not None: + engine.store.close() + + +async def test_oracle_cost_bound_breached(tmp_path: pathlib.Path) -> None: + """A run whose venue-measured cost exceeds ``max_cost`` fails the bound.""" + engine, report = await _green_run(tmp_path) + + checks = {c.name: c for c in venue_oracle(money("0.05"))(engine, report)} + + cost_check = checks["oracle.cost_bounded"] + assert not cost_check.passed + assert cost_check.expected == "cost<=0.05" + assert cost_check.observed == "cost=0.100" # 2 legs x 0.05 fee + if engine.store is not None: + engine.store.close() + + +# --- base dust: the expectation is the fills math, not zero ------------------ # + + +async def test_scenario_reports_venue_overfill_dust_and_stops( + tmp_path: pathlib.Path, +) -> None: + """A venue over-delivery within fill tolerance is reported exactly, and the + run stops before selling. + + The buy leg reads FILLED (the domain clamps within-tolerance over-fill + dust), but the tracker folds the *delivered* quantity — so the in-tracker + check records the venue's real behaviour exactly and the scenario refuses + to continue on a book that does not hold what it requested. + """ + engine = _venue_engine( + tmp_path, + _RestVenueBroker( + balances={"USDT": money("1000")}, + # 1e-8 over 0.001: excess fraction 1e-5, far inside the 0.1% + # domain fill tolerance — dust, not a material over-fill. + buy_overfill=money("0.00000001"), + ), + ) + report = await run_canary( + engine, INSTRUMENT, qty=QTY, oracle=None, mode_label="testnet" + ) + + assert not report.passed + by_name = {c.name: c for c in report.checks} + assert by_name["roundtrip.buy_filled"].passed + assert not by_name["roundtrip.buy_in_tracker"].passed + assert "0.00100001" in by_name["roundtrip.buy_in_tracker"].observed + # The sell was never placed: the run stopped on the dusted book. + assert engine.router.get(report.sell_cid) is None + if engine.store is not None: + engine.store.close() + + +async def test_base_dust_explained_by_fills_math_passes( + tmp_path: pathlib.Path, +) -> None: + """Venue base dust that our fills imply PASSES the base-delta identity. + + The pinned "compare against the fills math, not against zero" case, over + directly-seeded evidence of a *completed* round trip whose recorded fills + imply a base remainder (the buy delivered a within-tolerance hair more + than the sell returned): the venue balances reporting exactly that dust + must pass the base-delta check, whose expectation is computed from the + fills — a hardcoded-zero oracle would wrongly fail it. + """ + engine = _venue_engine(tmp_path, _RestVenueBroker(balances={"USDT": money("1000")})) + buy_qty = money("0.00100001") + sell_qty = money("0.001") + dust = buy_qty - sell_qty + report = CanaryReport( + exchange="restvenue", + mode="testnet", + instrument=INSTRUMENT, + qty=QTY, + probe_cid="canary-dust-probe", + buy_cid="canary-dust-buy", + sell_cid="canary-dust-sell", + ) + ours = [ + Fill( + fill_id="V1", + client_order_id=report.buy_cid, + instrument=INSTRUMENT, + side=OrderSide.BUY, + qty=buy_qty, + price=MARK, + fee=money("0"), + ts=1, + ), + Fill( + fill_id="V2", + client_order_id=report.sell_cid, + instrument=INSTRUMENT, + side=OrderSide.SELL, + qty=sell_qty, + price=MARK, + fee=money("0"), + ts=2, + ), + ] + for fill in ours: + engine.bus.emit(FillEvent(fill)) # store + tracker record our fills + report.initial_balances = {"USDT": money("1000")} + # The venue reports exactly what those fills imply: the base dust left + # over, and the quote moved by the notional difference. + report.final_balances = { + "USDT": money("1000") - dust * MARK, + "BTC": dust, + } + # Venue-side, the same executions under the venue's own references. + report.venue_fills = [ + replace(ours[0], client_order_id="9001"), + replace(ours[1], client_order_id="9002"), + ] + + checks = {c.name: c for c in venue_oracle(MAX_COST)(engine, report)} + + base_check = checks["oracle.base_delta_matches_fills"] + assert dust != 0 + assert base_check.passed + assert base_check.expected == f"base_delta={dust}" # the fills math, not 0 + assert base_check.observed == f"base_delta={dust}" + assert checks["oracle.fills_match_venue"].passed + assert checks["oracle.quote_delta_matches_fills"].passed + # The dusted book is still reported honestly where it IS a violation: + # the position is not flat. + assert not checks["oracle.position_flat"].passed + if engine.store is not None: + engine.store.close() + + +# --- settle timeout: a venue that never reports the execution ---------------- # + + +async def test_settle_timeout_fails_leg_with_evidence( + tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A market order the venue never reports filled times out and fails the leg. + + The settle loop gives up after the timeout; the leg's check records the + still-open order exactly, the run stops placing, and the abort cleanup + cancels the resting order on the venue. + """ + monkeypatch.setattr(canary_mod, "_VENUE_SETTLE_TIMEOUT_S", 0.05) + monkeypatch.setattr(canary_mod, "_VENUE_SETTLE_POLL_S", 0.01) + broker = _RestVenueBroker(balances={"USDT": money("1000")}, fill_market=False) + engine = _venue_engine(tmp_path, broker) + + report = await run_canary( + engine, INSTRUMENT, qty=QTY, oracle=venue_oracle(MAX_COST) + ) + + assert not report.passed + assert report.checks[-1].name == "roundtrip.buy_filled" + assert not report.checks[-1].passed + assert "status=open filled_qty=0" in report.checks[-1].observed + # The sell never ran, and the cleanup cancelled the resting buy venue-side. + assert engine.router.get(report.sell_cid) is None + assert await broker.open_orders() == [] + if engine.store is not None: + engine.store.close() + + +# --- the REAL Binance testnet run (opt-in: ``-m network``, key-gated) -------- # + + +@pytest.mark.network +async def test_real_binance_testnet_canary() -> None: + """The canary against the REAL Binance spot testnet — fake money, real API. + + Skips without testnet credentials (``BINANCE_TESTNET_API_KEY`` / + ``BINANCE_TESTNET_API_SECRET``, falling back to ``BINANCE_API_KEY`` / + ``BINANCE_API_SECRET`` — testnet.binance.vision rejects a mainnet key with + ``-2015``). Runs the CLI's own testnet path — the factory-built, + testnet-pinned engine, venue-side sizing, the full scenario, the identity + oracle — and asserts the whole evidence table is green. This is the + road-to-1.0 #1 validation vehicle: a real cancel, venue-checked + idempotency, and balance reconciliation against venue-reported fills. + """ + from trading_bot.interfaces.cli.main import _run_venue_canary + + key = os.environ.get("BINANCE_TESTNET_API_KEY") or os.environ.get("BINANCE_API_KEY") + secret = os.environ.get("BINANCE_TESTNET_API_SECRET") or os.environ.get( + "BINANCE_API_SECRET" + ) + if not key or not secret: + pytest.skip("no Binance testnet credentials in the environment") + + report = await _run_venue_canary( + exchange="binance", + symbol="BTC/USDT", + mode="testnet", + max_cost=MAX_COST, + # The CLI's venue default: inside Binance's PERCENT_PRICE_BY_SIDE + # price band (a 50%-below probe is rejected with -1013 on the venue). + probe_offset_pct=money("15"), + db_path=None, + ) + + assert report.exchange == "binance" + assert report.mode == "testnet" + assert report.passed, report.to_text() + assert len(report.venue_fills) >= 2 # both legs venue-confirmed diff --git a/trading_bot/tests/application/test_display_ccy.py b/trading_bot/tests/application/test_display_ccy.py new file mode 100644 index 00000000..88af182b --- /dev/null +++ b/trading_bot/tests/application/test_display_ccy.py @@ -0,0 +1,485 @@ +"""Tests for display-currency resolution + conversion (api-completeness leaf 03). + +Two layers, mirroring :mod:`trading_bot.application.display_ccy`'s own split: + +* **Pure** — :func:`~trading_bot.application.display_ccy.resolve_currency` / + :func:`~trading_bot.application.display_ccy.convert` directly, plus the three + new :class:`~trading_bot.application.config.AppConfig` fields' validation. +* **API shape** — a real :class:`~trading_bot.application.supervisor. + StrategySupervisor` (two venues: Kraken ``USD``-quoted, Binance + ``USDT``-quoted — the real book layout) driven through + :func:`~trading_bot.interfaces.api.create_dashboard_app` with a + :class:`fastapi.testclient.TestClient`: proves ``/api/positions``, + ``/api/strategies`` and ``/api/kpi`` rows carry the new ``display_currency`` + / ``*_display`` fields (additive) with every pre-existing field + byte-identical (contract regression, the leaf-02 pattern). +""" + +from __future__ import annotations + +# Built-in +import asyncio + +# Third-party +import pytest +from fastapi.testclient import TestClient +from pydantic import ValidationError + +# Local +from trading_bot.application.config import AppConfig +from trading_bot.application.display_ccy import convert, resolve_currency +from trading_bot.application.events import FillEvent +from trading_bot.application.supervisor import StrategySupervisor +from trading_bot.domain.fill import Fill +from trading_bot.domain.instrument import Instrument, Symbol +from trading_bot.domain.money import money +from trading_bot.domain.order import OrderSide +from trading_bot.interfaces.api import create_dashboard_app +from trading_bot.tests.application.test_supervisor import ( + _dccd_ohlc, + _FakeDccdClient, + _trend, +) + +# --------------------------------------------------------------------------- +# Pure: resolve_currency + convert +# --------------------------------------------------------------------------- + + +def _config(**overrides: object) -> AppConfig: + """A minimal paper :class:`AppConfig` with the given display-currency fields.""" + return AppConfig.model_validate({"mode": "paper", **overrides}) + + +def test_convert_identity_for_the_display_currency_itself() -> None: + """``from_ccy == display_currency`` never needs a declared rate.""" + cfg = _config(display_currency="USD") + assert convert(money("42.5"), "USD", cfg) == money("42.5") + + +def test_convert_declared_rate_is_decimal_exact() -> None: + """A declared rate converts exactly — no binary-float error.""" + cfg = _config(display_currency="USD", conversion_rates={"EUR": "1.08"}) + assert convert(money("100"), "EUR", cfg) == money("108.00") + + +def test_convert_missing_rate_is_none_never_guessed() -> None: + """No declared rate for the source currency → ``None``, not a fabricated figure.""" + cfg = _config(display_currency="USD") + assert convert(money("100"), "EUR", cfg) is None + + +def test_convert_none_amount_passes_through() -> None: + """An unmarked/unknown amount (``None``) stays ``None`` (nothing to convert).""" + cfg = _config(display_currency="USD", conversion_rates={"EUR": "1.08"}) + assert convert(None, "EUR", cfg) is None + + +def test_convert_none_from_ccy_is_none_never_guessed() -> None: + """A mixed-quote aggregate (``quote is None``) never guesses a source currency.""" + cfg = _config(display_currency="USD", conversion_rates={"USD": "1"}) + assert convert(money("100"), None, cfg) is None + + +def test_resolve_currency_override_for_named_exchange() -> None: + """An overridden exchange's rows are labelled with the override, not the default.""" + cfg = _config( + display_currency="USD", display_currency_overrides={"binance": "USDT"} + ) + assert resolve_currency("binance", cfg) == "USDT" + + +def test_resolve_currency_default_for_an_unlisted_exchange() -> None: + """An exchange with no override falls back to the global ``display_currency``.""" + cfg = _config( + display_currency="USD", display_currency_overrides={"binance": "USDT"} + ) + assert resolve_currency("kraken", cfg) == "USD" + + +def test_resolve_currency_none_exchange_falls_back_to_default() -> None: + """No single exchange (a KPI ``level="total"`` row) resolves like an unlisted one.""" + cfg = _config( + display_currency="USD", display_currency_overrides={"binance": "USDT"} + ) + assert resolve_currency(None, cfg) == "USD" + + +def test_display_currency_defaults_to_usd_with_empty_overrides_and_rates() -> None: + """A fresh config never fails to resolve/convert — everything defaults empty.""" + cfg = AppConfig.model_validate({}) + assert cfg.display_currency == "USD" + assert cfg.display_currency_overrides == {} + assert cfg.conversion_rates == {} + + +def test_zero_conversion_rate_rejected_at_config_parse() -> None: + """A non-positive rate is rejected — the money-field pattern.""" + with pytest.raises(ValidationError): + AppConfig.model_validate({"conversion_rates": {"EUR": "0"}}) + + +def test_negative_conversion_rate_rejected_at_config_parse() -> None: + """A negative rate is rejected — the money-field pattern.""" + with pytest.raises(ValidationError): + AppConfig.model_validate({"conversion_rates": {"EUR": "-1.08"}}) + + +def test_blank_display_currency_rejected_at_config_parse() -> None: + """An empty/whitespace ``display_currency`` is rejected.""" + with pytest.raises(ValidationError): + AppConfig.model_validate({"display_currency": " "}) + + +# --------------------------------------------------------------------------- +# API shape: /api/positions, /api/strategies, /api/kpi +# --------------------------------------------------------------------------- + + +#: The real book layout (see ``configs/dashboard.yaml``): Kraken quotes in USD, +#: Binance quotes in USDT. Each strategy declares an ``allocation`` so +#: ``contributed``/``unrealised``/``total_value`` populate (non-``None``), +#: exercising the strategy-row conversion pair. +def _two_venue_config( + *, + display_currency: str = "USD", + display_currency_overrides: dict[str, str] | None = None, + conversion_rates: dict[str, str] | None = None, +) -> AppConfig: + return AppConfig.model_validate( + { + "mode": "paper", + "display_currency": display_currency, + "display_currency_overrides": display_currency_overrides or {}, + "conversion_rates": conversion_rates or {}, + "brokers": [ + {"name": "kraken", "exchange": "kraken"}, + {"name": "binance", "exchange": "binance"}, + ], + "strategies": [ + { + "name": "btc-kraken", + "symbol": "BTC/USD", + "data": {"exchange": "kraken", "span": 60}, + "signal": {"ref": "ma_crossover", "params": {"fast": 3, "slow": 6}}, + "reference_qty": "2", + "lookback": 6, + "allocation": "1000", + }, + { + "name": "eth-binance", + "symbol": "ETH/USDT", + "data": {"exchange": "binance", "span": 60}, + "signal": {"ref": "ma_crossover", "params": {"fast": 3, "slow": 6}}, + "reference_qty": "2", + "lookback": 6, + "allocation": "1000", + }, + ], + } + ) + + +def _two_venue_client() -> _FakeDccdClient: + """Offline dccd client — `start()` never imports the real dccd (absent in CI).""" + return _FakeDccdClient( + {"BTC/USD": _dccd_ohlc(_trend()), "ETH/USDT": _dccd_ohlc(_trend())} + ) + + +async def _seeded_supervisor(**config_kwargs: object) -> StrategySupervisor: + """Two running units (Kraken USD, Binance USDT), each with an open, marked position. + + A single opening buy per unit (position stays open, unlike a closing round + trip) plus a directly-seeded :class:`~trading_bot.application.mark_cache. + MarkCache` entry (the leaf-02 contract-test technique) so ``value`` / + ``unrealised`` / ``total_value`` are non-``None`` on every row — otherwise + the ``_display`` conversion would be trivially ``None`` regardless of + config, and prove nothing. + """ + sup = StrategySupervisor( + _two_venue_config(**config_kwargs), dccd_client=_two_venue_client() + ) + await sup.start("btc-kraken") + await sup.start("eth-binance") + + kraken_unit = sup._units["btc-kraken"] # noqa: SLF001 — direct seed, test-only + kraken_unit.engine.bus.emit( + FillEvent( + Fill( + "K-F1", + "K-c1", + Instrument(Symbol("BTC", "USD")), + OrderSide.BUY, + money("2"), + money("100"), + money("1"), + 1, + ) + ) + ) + kraken_unit.engine.mark_cache.update(Symbol("BTC", "USD"), money("110"), 9_000) + + binance_unit = sup._units["eth-binance"] # noqa: SLF001 — direct seed, test-only + binance_unit.engine.bus.emit( + FillEvent( + Fill( + "B-F1", + "B-c1", + Instrument(Symbol("ETH", "USDT")), + OrderSide.BUY, + money("2"), + money("200"), + money("1"), + 1, + ) + ) + ) + binance_unit.engine.mark_cache.update(Symbol("ETH", "USDT"), money("220"), 9_000) + return sup + + +def test_api_positions_display_fields_default_config() -> None: + """No override: Kraken's own USD quote matches the global default (identity, + non-``null``); Binance's USDT quote has no declared rate (``null``), native + ``value``/``unrealised`` untouched. + """ + sup = asyncio.run(_seeded_supervisor()) + rows = TestClient(create_dashboard_app(sup)).get("/api/positions").json() + by_exchange = {r["exchange"]: r for r in rows} + kraken_row = by_exchange["kraken"] + binance_row = by_exchange["binance"] + + assert kraken_row["fee_ccy"] == "USD" + assert kraken_row["display_currency"] == "USD" + assert kraken_row["value_display"] == kraken_row["value"] + assert kraken_row["unrealised_display"] == kraken_row["unrealised"] + + assert binance_row["fee_ccy"] == "USDT" + assert binance_row["display_currency"] == "USD" + assert binance_row["value"] is not None # native untouched + assert binance_row["value_display"] is None + assert binance_row["unrealised_display"] is None + + +def test_api_positions_display_fields_with_override_and_identity_rate() -> None: + """``overrides={"binance":"USDT"}`` + ``rates={"USDT":"1"}``: Binance rows + label + display in USDT (identity vs native); Kraken rows stay USD. + """ + sup = asyncio.run( + _seeded_supervisor( + display_currency_overrides={"binance": "USDT"}, + conversion_rates={"USDT": "1"}, + ) + ) + rows = TestClient(create_dashboard_app(sup)).get("/api/positions").json() + by_exchange = {r["exchange"]: r for r in rows} + kraken_row = by_exchange["kraken"] + binance_row = by_exchange["binance"] + + assert binance_row["display_currency"] == "USDT" + assert binance_row["value_display"] == binance_row["value"] + assert binance_row["unrealised_display"] == binance_row["unrealised"] + + assert kraken_row["display_currency"] == "USD" + assert kraken_row["value_display"] == kraken_row["value"] + assert kraken_row["unrealised_display"] == kraken_row["unrealised"] + + +def test_api_positions_display_fields_empty_rates_goes_null() -> None: + """Declaring ``conversion_rates={}`` (no rate for USDT): Binance's displays + go ``null`` while its native ``value``/``unrealised`` stay untouched; + Kraken's own quote (``USD``) still identity-matches the global default (a + same-currency conversion never needs a declared rate). + """ + sup = asyncio.run( + _seeded_supervisor( + display_currency_overrides={"binance": "USDT"}, conversion_rates={} + ) + ) + rows = TestClient(create_dashboard_app(sup)).get("/api/positions").json() + by_exchange = {r["exchange"]: r for r in rows} + kraken_row = by_exchange["kraken"] + binance_row = by_exchange["binance"] + + assert binance_row["value_display"] is None + assert binance_row["unrealised_display"] is None + assert binance_row["value"] is not None + assert binance_row["unrealised"] is not None + + assert kraken_row["value_display"] == kraken_row["value"] + assert kraken_row["unrealised_display"] == kraken_row["unrealised"] + + +def test_api_positions_contract_regression() -> None: + """`/api/positions` carries every pre-existing field, byte-identical, plus the + new ``display_currency`` / ``value_display`` / ``unrealised_display`` (leaf 03). + """ + sup = asyncio.run(_seeded_supervisor()) + [row] = [ + r + for r in TestClient(create_dashboard_app(sup)).get("/api/positions").json() + if r["exchange"] == "kraken" + ] + assert set(row) == { + "strategy", + "exchange", + "instrument", + "base", + "net_qty", + "avg_entry_price", + "realised_pnl", + "fees_paid", + "mark", + "mark_asof_ts", + "mark_source", + "value", + "unrealised", + "fee_ccy", + "display_currency", + "value_display", + "unrealised_display", + } + assert row["strategy"] == "btc-kraken" + assert row["exchange"] == "kraken" + assert row["instrument"] == "BTC/USD" + assert row["base"] == "BTC" + assert row["net_qty"] == "2" + assert row["avg_entry_price"] == "100" + assert row["realised_pnl"] == "-1" # the opening fill's fee reduces realised PnL + assert row["fees_paid"] == "1" + assert row["mark"] == "110" + assert row["mark_asof_ts"] == 9_000 + assert row["mark_source"] == "bar_close" + assert row["value"] == "220" + assert row["unrealised"] == "20" + assert row["fee_ccy"] == "USD" + + +def test_api_strategies_display_fields_and_contract_regression() -> None: + """`/api/strategies` rows carry ``display_currency`` / ``total_value_display`` / + ``unrealised_display`` (mirroring the position-row pair) with every + pre-existing field untouched. + """ + sup = asyncio.run( + _seeded_supervisor( + display_currency_overrides={"binance": "USDT"}, + conversion_rates={"USDT": "1"}, + ) + ) + rows = TestClient(create_dashboard_app(sup)).get("/api/strategies").json() + by_name = {r["name"]: r for r in rows} + kraken_row = by_name["btc-kraken"] + binance_row = by_name["eth-binance"] + + assert set(kraken_row) == { + "name", + "kind", + "exchange", + "span", + "quote", + "mode", + "running", + "realised_pnl", + "open_orders", + "last_eval_ts", + "last_asof_ts", + "allocation", + "contributed", + "unrealised", + "total_value", + "capital_policy", + "health", + "health_detail", + "display_currency", + "total_value_display", + "unrealised_display", + } + # Pre-existing fields, unchanged. + assert kraken_row["quote"] == "USD" + assert kraken_row["allocation"] == "1000" + assert kraken_row["contributed"] == "1000" + assert kraken_row["realised_pnl"] == "-1" + assert kraken_row["unrealised"] == "20" + assert kraken_row["total_value"] == "1019" # 1000 - 1 + 20 + # New fields: Kraken's own USD quote identity-matches the global default. + assert kraken_row["display_currency"] == "USD" + assert kraken_row["total_value_display"] == kraken_row["total_value"] + assert kraken_row["unrealised_display"] == kraken_row["unrealised"] + + assert binance_row["quote"] == "USDT" + assert binance_row["total_value"] == "1039" # 1000 - 1 + 40 + # New fields: Binance's override + identity rate -> displayed in USDT. + assert binance_row["display_currency"] == "USDT" + assert binance_row["total_value_display"] == binance_row["total_value"] + assert binance_row["unrealised_display"] == binance_row["unrealised"] + + +def test_api_kpi_display_fields_and_contract_regression() -> None: + """`/api/kpi` (``level="strategy"``) rows carry ``display_currency`` / + ``realised_pnl_display`` / ``fees_paid_display`` with every pre-existing + field untouched. + """ + sup = asyncio.run( + _seeded_supervisor( + display_currency_overrides={"binance": "USDT"}, + conversion_rates={"USDT": "1"}, + ) + ) + rows = TestClient(create_dashboard_app(sup)).get("/api/kpi?level=strategy").json() + by_strategy = {r["strategy"]: r for r in rows} + kraken_row = by_strategy["btc-kraken"] + binance_row = by_strategy["eth-binance"] + + assert set(kraken_row) == { + "level", + "key", + "strategy", + "exchange", + "quote", + "realised_pnl", + "fees_paid", + "sharpe", + "sortino", + "calmar", + "max_drawdown", + "display_currency", + "realised_pnl_display", + "fees_paid_display", + } + assert kraken_row["level"] == "strategy" + assert kraken_row["quote"] == "USD" + assert kraken_row["realised_pnl"] == "-1" + assert kraken_row["fees_paid"] == "1" + assert kraken_row["display_currency"] == "USD" + assert kraken_row["realised_pnl_display"] == kraken_row["realised_pnl"] + assert kraken_row["fees_paid_display"] == kraken_row["fees_paid"] + + assert binance_row["quote"] == "USDT" + assert binance_row["display_currency"] == "USDT" + assert binance_row["realised_pnl_display"] == binance_row["realised_pnl"] + assert binance_row["fees_paid_display"] == binance_row["fees_paid"] + + +def test_api_kpi_total_level_mixed_quote_and_no_single_exchange_go_null() -> None: + """``level="total"`` folds both venues: ``quote`` is ``None`` (mixed currencies) + and ``exchange`` is ``None`` (no single venue) — the ``_display`` fields + never guess a conversion, but ``display_currency`` still resolves to the + global default (an unlisted/absent exchange falls back, per + :func:`~trading_bot.application.display_ccy.resolve_currency`). + """ + sup = asyncio.run( + _seeded_supervisor( + display_currency_overrides={"binance": "USDT"}, + conversion_rates={"USDT": "1"}, + ) + ) + [row] = TestClient(create_dashboard_app(sup)).get("/api/kpi?level=total").json() + assert row["quote"] is None + assert row["exchange"] is None + assert row["display_currency"] == "USD" + assert row["realised_pnl_display"] is None + assert row["fees_paid_display"] is None + # Native fields stay untouched even though the display pair is null. + assert row["realised_pnl"] == "-2" + assert row["fees_paid"] == "2" diff --git a/trading_bot/tests/application/test_mark_cache.py b/trading_bot/tests/application/test_mark_cache.py new file mode 100644 index 00000000..e959d5a5 --- /dev/null +++ b/trading_bot/tests/application/test_mark_cache.py @@ -0,0 +1,77 @@ +"""Tests for :mod:`trading_bot.application.mark_cache`. + +Proves the cache's whole contract: + +* ``update``/``get`` round-trip the exact ``Decimal`` price, the ``asof_ms`` + and the ``"bar_close"`` source (:func:`test_update_get_round_trips_exact_decimal`); +* ``get`` on a never-published symbol is ``None`` + (:func:`test_get_unknown_symbol_is_none`); +* a second ``update`` for the same symbol overwrites the first — latest write + wins, never accumulates (:func:`test_second_update_overwrites_the_first`); +* ``all`` snapshots every published symbol and mutating the returned dict never + affects the cache (:func:`test_all_snapshots_every_symbol_and_is_a_copy`). +""" + +from __future__ import annotations + +# Local +from trading_bot.application.mark_cache import Mark, MarkCache +from trading_bot.domain.instrument import Symbol +from trading_bot.domain.money import money + +BTC = Symbol("BTC", "USDT") +ETH = Symbol("ETH", "USDT") + + +def test_update_get_round_trips_exact_decimal() -> None: + """``update`` then ``get`` returns the exact price, asof and bar_close source.""" + cache = MarkCache() + + cache.update(BTC, money("50000.12345678"), 1_700_000_000_000) + + mark = cache.get(BTC) + assert mark is not None + assert mark.price == money("50000.12345678") + assert mark.asof_ms == 1_700_000_000_000 + assert mark.source == "bar_close" + assert mark == Mark( + price=money("50000.12345678"), asof_ms=1_700_000_000_000, source="bar_close" + ) + + +def test_get_unknown_symbol_is_none() -> None: + """A symbol never published to the cache returns ``None``, not a KeyError.""" + cache = MarkCache() + + assert cache.get(BTC) is None + + +def test_second_update_overwrites_the_first() -> None: + """A later ``update`` for the same symbol replaces the mark — no accumulation.""" + cache = MarkCache() + cache.update(BTC, money("50000"), 1_700_000_000_000) + + cache.update(BTC, money("51000"), 1_700_000_060_000) + + mark = cache.get(BTC) + assert mark is not None + assert mark.price == money("51000") + assert mark.asof_ms == 1_700_000_060_000 + + +def test_all_snapshots_every_symbol_and_is_a_copy() -> None: + """``all`` returns every published symbol's mark, and mutating it is inert.""" + cache = MarkCache() + cache.update(BTC, money("50000"), 1_700_000_000_000) + cache.update(ETH, money("2500"), 1_700_000_000_000) + + snapshot = cache.all() + + assert snapshot == { + BTC: Mark(price=money("50000"), asof_ms=1_700_000_000_000, source="bar_close"), + ETH: Mark(price=money("2500"), asof_ms=1_700_000_000_000, source="bar_close"), + } + + # Mutating the returned dict must never leak back into the cache. + del snapshot[BTC] + assert cache.get(BTC) is not None diff --git a/trading_bot/tests/application/test_portfolio_runner.py b/trading_bot/tests/application/test_portfolio_runner.py index eed182c5..9135297f 100644 --- a/trading_bot/tests/application/test_portfolio_runner.py +++ b/trading_bot/tests/application/test_portfolio_runner.py @@ -43,6 +43,7 @@ RiskManager, ) from trading_bot.application.config import RiskConfig +from trading_bot.application.mark_cache import MarkCache from trading_bot.brokers import PaperBroker from trading_bot.domain import ( CapitalEvent, @@ -1174,3 +1175,92 @@ async def test_resolver_requires_exchange() -> None: assert "exchange" in str(exc) else: # pragma: no cover - the guard must fire raise AssertionError("expected ValueError for a resolver with no exchange") + + +# --- mark cache publish ------------------------------------------------------ # + + +async def test_rebalance_publishes_marks_with_the_ticks_asof() -> None: + """A rebalance publishes every traded symbol's close + this tick's asof. + + The engine's :class:`~trading_bot.application.mark_cache.MarkCache`, when + threaded in, ends up holding exactly the ``prices`` the tick sized against + (read via :meth:`PortfolioRunner._latest_closes`) keyed by symbol, each + stamped with the same ``asof_ms`` :meth:`PortfolioRunner._derive_asof_ms` + derives for the tick's signals — and ``source == "bar_close"`` (this + runner never publishes a fill-derived mark). + """ + weights = {BTC: money("0.5"), ETH: money("-0.25")} + router, tracker, bus, _broker = _engine() + mark_cache = MarkCache() + runner = PortfolioRunner( + _strategy(_weights_signal(weights)), + _ListFeed([_frames()], asof=1_700), + router, + tracker, + event_bus=bus, + mark_cache=mark_cache, + ) + + await runner.rebalance(_frames()) + + expected_asof = 1_700_000_000_000_000_000 // 1_000_000 + btc_mark = mark_cache.get(BTC) + eth_mark = mark_cache.get(ETH) + assert btc_mark is not None + assert btc_mark.price == BTC_PRICE + assert btc_mark.asof_ms == expected_asof + assert btc_mark.source == "bar_close" + assert eth_mark is not None + assert eth_mark.price == ETH_PRICE + assert eth_mark.asof_ms == expected_asof + + +async def test_second_rebalance_updates_marks_in_place() -> None: + """A later rebalance's closes and asof overwrite the prior tick's marks.""" + weights = {BTC: money("0.5"), ETH: money("-0.25")} + router, tracker, bus, _broker = _engine() + mark_cache = MarkCache() + runner = PortfolioRunner( + _strategy(_weights_signal(weights)), + _ListFeed([_frames()], asof=1_700), + router, + tracker, + event_bus=bus, + mark_cache=mark_cache, + ) + await runner.rebalance(_frames()) + + second_time_ns = 1_700_000_060_000_000_000 + second_frames = { + BTC: _frame(51000.0, time_ns=second_time_ns), + ETH: _frame(2600.0, time_ns=second_time_ns), + } + await runner.rebalance(second_frames) + + btc_mark = mark_cache.get(BTC) + eth_mark = mark_cache.get(ETH) + assert btc_mark is not None + assert btc_mark.price == money("51000") + assert btc_mark.asof_ms == second_time_ns // 1_000_000 + assert eth_mark is not None + assert eth_mark.price == money("2600") + assert eth_mark.asof_ms == second_time_ns // 1_000_000 + + +async def test_no_mark_cache_is_a_legacy_noop() -> None: + """Omitting ``mark_cache`` (the default) rebalances exactly as before.""" + weights = {BTC: money("0.5"), ETH: money("-0.25")} + router, tracker, bus, _broker = _engine() + runner = PortfolioRunner( + _strategy(_weights_signal(weights)), + _ListFeed([_frames()], asof=1_700), + router, + tracker, + event_bus=bus, + ) + + result = await runner.rebalance(_frames()) + + assert result.submitted == 2 + assert result.failed == 0 diff --git a/trading_bot/tests/application/test_service_factory.py b/trading_bot/tests/application/test_service_factory.py index b5d92e91..e6a8ae4e 100644 --- a/trading_bot/tests/application/test_service_factory.py +++ b/trading_bot/tests/application/test_service_factory.py @@ -19,8 +19,9 @@ import pytest -from trading_bot.application.config import AppConfig, BrokerConfig +from trading_bot.application.config import AppConfig, BrokerConfig, RiskConfig from trading_bot.application.events import EventBus, FillEvent +from trading_bot.application.mark_cache import MarkCache from trading_bot.application.order_router import OrderRouter from trading_bot.application.performance_service import PerformanceService from trading_bot.application.position_tracker import PositionTracker @@ -83,6 +84,23 @@ def test_default_config_builds_paper_engine() -> None: assert engine.store is None +def test_engine_carries_mark_cache() -> None: + """Every built engine gets a fresh, empty :class:`MarkCache` by default. + + Mirrors ``spec_resolver`` — a default-factory field, so direct + ``Engine(...)`` constructions (tests wire engines by hand) keep working + unchanged, and every ``build_engine`` call gets its own cache (never + shared across units). + """ + engine_a = build_engine(AppConfig()) + engine_b = build_engine(AppConfig()) + + assert isinstance(engine_a.mark_cache, MarkCache) + assert engine_a.mark_cache.all() == {} + # Two engines never share the same cache instance. + assert engine_a.mark_cache is not engine_b.mark_cache + + async def test_paper_engine_fills_carry_the_wall_clock() -> None: """An engine-built PaperBroker stamps fills with the REAL wall clock. @@ -602,6 +620,67 @@ def test_brokerconfig_testnet_defaults_false() -> None: assert BrokerConfig(name="bn", exchange="binance").testnet is False +# --- BrokerConfig.symbols → the adapter's per-symbol fill endpoints ---------- # + + +def test_testnet_threads_symbols_to_binance_adapter(monkeypatch) -> None: + """``BrokerConfig.symbols`` reaches the testnet adapter, parsed canonically. + + Binance has no account-wide trade history (``myTrades`` is per-symbol), so + ``broker.fills()`` — reconciliation's and the canary venue oracle's + re-fetch channel — needs the symbol set threaded from the config. + """ + monkeypatch.setenv("BINANCE_API_KEY", "tk") + monkeypatch.setenv("BINANCE_API_SECRET", "ts") + cfg = AppConfig( + mode="live", + brokers=[ + BrokerConfig( + name="bn", + exchange="binance", + testnet=True, + symbols=["BTC/USDT", "ETHUSDT"], + ) + ], + ) + engine = build_engine(cfg) + assert isinstance(engine.broker, BinanceBroker) + assert engine.broker.symbols == (Symbol("BTC", "USDT"), Symbol("ETH", "USDT")) + + +def test_live_threads_symbols_to_binance_adapter(monkeypatch) -> None: + """The live path threads ``symbols`` too (same seam as testnet).""" + monkeypatch.setenv("BINANCE_API_KEY", "tk") + monkeypatch.setenv("BINANCE_API_SECRET", "ts") + cfg = AppConfig( + mode="live", + live_enabled=True, + risk=RiskConfig( + max_order=money("0.01"), + max_position=money("0.01"), + max_daily_loss=money("10"), + ), + brokers=[BrokerConfig(name="bn", exchange="binance", symbols=["BTC/USDT"])], + ) + engine = build_engine(cfg) + assert isinstance(engine.broker, BinanceBroker) + assert engine.broker.symbols == (Symbol("BTC", "USDT"),) + + +def test_unparseable_symbols_entry_refuses_at_build(monkeypatch) -> None: + """An unparseable ``symbols`` entry refuses with a clear ``BrokerError``.""" + monkeypatch.setenv("BINANCE_API_KEY", "tk") + monkeypatch.setenv("BINANCE_API_SECRET", "ts") + cfg = AppConfig( + mode="live", + brokers=[ + BrokerConfig(name="bn", exchange="binance", testnet=True, symbols=["???"]) + ], + ) + with pytest.raises(BrokerError, match="cannot parse symbols entry"): + build_engine(cfg) + + # --- the daily-loss circuit breaker is wired to live PnL ------------------- # diff --git a/trading_bot/tests/application/test_supervisor.py b/trading_bot/tests/application/test_supervisor.py index 002581e3..d1f9dc3d 100644 --- a/trading_bot/tests/application/test_supervisor.py +++ b/trading_bot/tests/application/test_supervisor.py @@ -394,6 +394,190 @@ async def test_positions_carry_strategy_and_exchange_tags() -> None: assert row.net_qty == money("3") +# --- position rows: mark / value / unrealised / fee_ccy (api-completeness 02) -- # + + +async def test_position_row_mark_from_cache() -> None: + """A cached bar-close mark prices the row: ``value``/``unrealised`` exact, short-safe. + + A SHORT (a SELL when flat) with the mark cache above the entry loses money — + the arithmetic must mirror :meth:`StrategySupervisor._unrealised_of` exactly + (never the naive long-only sign). + """ + pytest.importorskip("fynance") + sup = _supervisor() + await sup.start("btc-ma") + unit = sup._units["btc-ma"] # noqa: SLF001 + inst = Instrument(Symbol("BTC", "USD")) + unit.engine.bus.emit( # noqa: SLF001 — seed a short directly on the engine bus + FillEvent( + Fill( + "F1", + "c1", + inst, + OrderSide.SELL, + money("3"), + money("100"), + money("0"), + 1, + ) + ) + ) + unit.engine.mark_cache.update(Symbol("BTC", "USD"), money("90"), 5_000) + + [row] = sup.positions() + assert row.mark == money("90") + assert row.mark_asof_ts == 5_000 + assert row.mark_source == "bar_close" + assert row.value == money("270") # 90 * |−3| + # Short entered at 100, marked at 90 (price dropped) → the short profits. + assert row.unrealised == money("30") # (90 − 100) * −3 + + +async def test_position_row_falls_back_to_last_fill(tmp_path) -> None: # noqa: ANN001 + """No mark-cache entry for the symbol → the row falls back to the last own fill.""" + pytest.importorskip("fynance") + db = str(tmp_path / "book.sqlite") + sup = await _started_alloc_unit(db, allocation="1000") + unit = sup._units["btc-ma"] # noqa: SLF001 + inst = Instrument(Symbol("BTC", "USD")) + unit.engine.bus.emit( # noqa: SLF001 + FillEvent( + Fill( + "F1", + "c1", + inst, + OrderSide.BUY, + money("2"), + money("100"), + money("1"), + 42, + ) + ) + ) + # No mark_cache.update() call — the cache holds nothing for BTC/USD. + + [row] = sup.positions() + assert row.mark == money("100") + assert row.mark_asof_ts == 42 + assert row.mark_source == "last_fill" + assert row.value == money("200") + assert row.unrealised == money("0") # mark == avg_entry_price + + +async def test_position_row_no_mark_is_null() -> None: + """No cache entry and no reachable fill history → mark/value/unrealised null, no crash.""" + pytest.importorskip("fynance") + sup = _supervisor() # no storage db_path configured + await sup.start("btc-ma") + unit = sup._units["btc-ma"] # noqa: SLF001 + inst = Instrument(Symbol("BTC", "USD")) + unit.engine.bus.emit( # noqa: SLF001 + FillEvent( + Fill( + "F1", "c1", inst, OrderSide.BUY, money("2"), money("100"), money("1"), 1 + ) + ) + ) + # The bus-emitted fill lands in the tracker but the unit has no store, so it is + # unreachable via `_stored_fills_of` — and the mark cache was never populated. + + [row] = sup.positions() + assert row.mark is None + assert row.mark_asof_ts is None + assert row.mark_source is None + assert row.value is None + assert row.unrealised is None + # The position itself is unaffected — only the mark-derived fields go null. + assert row.net_qty == money("2") + assert row.avg_entry_price == money("100") + + +async def test_fee_ccy_is_quote() -> None: + """``fee_ccy`` is the instrument's quote asset (fees are charged in quote terms).""" + pytest.importorskip("fynance") + cfg = AppConfig.model_validate( + { + "mode": "paper", + "brokers": [{"name": "kraken", "exchange": "kraken"}], + "strategies": [ + { + "name": "eth-btc", + "symbol": "ETH/BTC", + "data": {"exchange": "kraken", "span": 60}, + "signal": {"ref": "ma_crossover", "params": {"fast": 3, "slow": 6}}, + "reference_qty": "2", + "lookback": 6, + } + ], + } + ) + sup = StrategySupervisor( + cfg, dccd_client=_FakeDccdClient({"ETH/BTC": _dccd_ohlc(_trend())}) + ) + await sup.start("eth-btc") + inst = Instrument(Symbol("ETH", "BTC")) + sup._units["eth-btc"].engine.bus.emit( # noqa: SLF001 + FillEvent( + Fill( + "F1", + "c1", + inst, + OrderSide.BUY, + money("1"), + money("0.05"), + money("0"), + 1, + ) + ) + ) + [row] = sup.positions() + assert row.base == "ETH" + assert row.fee_ccy == "BTC" + + +async def test_aggregate_and_row_agree() -> None: + """The strategy-aggregate ``unrealised`` equals Σ row ``unrealised`` — one mark policy. + + Two instruments (a long and a short) on the same unit, both marked from the + cache: :meth:`StrategySupervisor._unrealised_of` (the roster aggregate) and + :meth:`StrategySupervisor.positions` (the per-row figures) must never diverge. + """ + pytest.importorskip("fynance") + sup = _supervisor() + await sup.start("btc-ma") + unit = sup._units["btc-ma"] # noqa: SLF001 + btc = Instrument(Symbol("BTC", "USD")) + eth = Instrument(Symbol("ETH", "USD")) + unit.engine.bus.emit( # noqa: SLF001 + FillEvent( + Fill( + "F1", "c1", btc, OrderSide.BUY, money("2"), money("100"), money("0"), 1 + ) + ) + ) + unit.engine.bus.emit( # noqa: SLF001 + FillEvent( + Fill( + "F2", "c2", eth, OrderSide.SELL, money("1"), money("50"), money("0"), 1 + ) + ) + ) + unit.engine.mark_cache.update(Symbol("BTC", "USD"), money("110"), 1_000) + unit.engine.mark_cache.update(Symbol("ETH", "USD"), money("55"), 1_000) + + rows = sup.positions() + by_base = {row.base: row for row in rows} + assert by_base["BTC"].unrealised == money("20") # (110 − 100) * 2 + assert by_base["ETH"].unrealised == money("-5") # (55 − 50) * −1 + + row_total = sum( + (row.unrealised for row in rows if row.unrealised is not None), money("0") + ) + status = sup.status("btc-ma")[0] + assert status.unrealised == row_total == money("15") + + async def test_open_orders_carry_strategy_and_exchange_tags() -> None: """`open_orders()` rows are tagged with strategy + exchange across the units.""" pytest.importorskip("fynance") diff --git a/trading_bot/tests/brokers/test_binance_rest.py b/trading_bot/tests/brokers/test_binance_rest.py index e827f147..0712b17b 100644 --- a/trading_bot/tests/brokers/test_binance_rest.py +++ b/trading_bot/tests/brokers/test_binance_rest.py @@ -762,6 +762,77 @@ async def test_binance_error_body_raises_broker_error( await broker.place_order(order) +# --- HTTP-4xx rejections map to the same domain errors (real-venue shape) - # + + +async def test_http_400_rejection_maps_to_specific_domain_error( + httpx_mock, monkeypatch: pytest.MonkeyPatch +) -> None: + """A venue rejection sent as HTTP 400 + JSON body maps like an in-band error. + + On the real venue Binance delivers its ``{"code","msg"}`` rejections with + a **4xx status** (observed live on the testnet: a filter failure arrives + as HTTP 400), which the transport surfaces as ``HTTPError`` — the adapter + must recover the body and raise the same specific domain error the 200 + shape would have produced, never leak a transport exception. + """ + httpx_mock.add_response( + status_code=400, + json={"code": -2010, "msg": "Account has insufficient balance."}, + ) + broker = _broker(monkeypatch) + order = Order( + client_order_id="strat-400", + instrument=BTC_USDT, + side=OrderSide.BUY, + qty=money("1"), + type=OrderType.MARKET, + ) + + with pytest.raises(InsufficientBalance, match="insufficient balance"): + await broker.place_order(order) + + +async def test_http_400_filter_failure_maps_to_broker_error( + httpx_mock, monkeypatch: pytest.MonkeyPatch +) -> None: + """The real testnet's -1013 filter rejection surfaces as a ``BrokerError``. + + The exact shape the live canary hit: ``HTTP 400`` + + ``{"code":-1013,"msg":"Filter failure: PERCENT_PRICE_BY_SIDE"}`` on the + probe's far-below limit. It must arrive as a mapped ``BrokerError`` + carrying the venue's diagnostic (so the canary records it as a failed + check instead of crashing out of the scenario). + """ + httpx_mock.add_response( + status_code=400, + json={"code": -1013, "msg": "Filter failure: PERCENT_PRICE_BY_SIDE"}, + ) + broker = _broker(monkeypatch) + order = Order( + client_order_id="strat-1013", + instrument=BTC_USDT, + side=OrderSide.BUY, + qty=money("1"), + type=OrderType.LIMIT, + limit_price=money("30000"), + ) + + with pytest.raises(BrokerError, match="PERCENT_PRICE_BY_SIDE"): + await broker.place_order(order) + + +async def test_http_400_non_binance_body_degrades_to_generic_broker_error( + httpx_mock, monkeypatch: pytest.MonkeyPatch +) -> None: + """A 4xx whose body is not the Binance error shape still maps (generic).""" + httpx_mock.add_response(status_code=404, text="not found") + broker = _broker(monkeypatch) + + with pytest.raises(BrokerError, match="account"): + await broker.balances() + + # --- B-7: array-wrapped Binance errors are detected & mapped -------------- # diff --git a/trading_bot/tests/brokers/test_paper.py b/trading_bot/tests/brokers/test_paper.py index 1b002001..4fdb8a86 100644 --- a/trading_bot/tests/brokers/test_paper.py +++ b/trading_bot/tests/brokers/test_paper.py @@ -764,3 +764,110 @@ async def test_non_strict_default_still_permissive() -> None: id2 = await broker.place_order(_strict_limit_buy(qty="0.0002", cid="p")) assert id1 != id2 # a second order was created (no dedup) assert len(await broker.fills()) == 2 + + +# --- strict marketability: a passive limit rests when a mark is known ------- # + + +async def test_strict_far_below_limit_buy_rests_with_a_mark() -> None: + """Strict + mark: a limit BUY strictly below the mark rests — no fill. + + The real-venue behaviour the canary's cancel probe relies on: a far-below + limit buy sits on the book (open, zero fills, balances untouched) until + cancelled — it never crosses the mark. + """ + broker = PaperBroker( + prices={BTC_USD: money("30000")}, + starting_balances={"USD": money("100000")}, + strict=True, + ) + + venue_id = await broker.place_order(_limit_buy(qty="1", price="15000")) + + assert await broker.fills() == [] + assert (await broker.balances())["USD"] == Decimal("100000") + open_orders = await broker.open_orders() + assert len(open_orders) == 1 + assert open_orders[0].venue_order_id == venue_id + assert open_orders[0].filled_qty == Decimal("0") + + # The resting order is cancellable — the kill path works on it. + await broker.cancel_order(venue_id) + assert await broker.open_orders() == [] + assert await broker.fills() == [] + + +async def test_strict_far_above_limit_sell_rests_with_a_mark() -> None: + """Strict + mark: a limit SELL strictly above the mark rests symmetrically.""" + broker = PaperBroker( + prices={BTC_USD: money("30000")}, + starting_balances={"BTC": money("2")}, + strict=True, + ) + + await broker.place_order(_limit_sell(qty="1", price="60000")) + + assert await broker.fills() == [] + assert (await broker.balances())["BTC"] == Decimal("2") + assert len(await broker.open_orders()) == 1 + + +async def test_strict_marketable_limit_still_fills_at_limit() -> None: + """Strict + mark: a limit BUY at/above the mark crosses and fills as before.""" + broker = PaperBroker( + prices={BTC_USD: money("30000")}, + starting_balances={"USD": money("100000")}, + strict=True, + ) + + await broker.place_order(_limit_buy(qty="1", price="30000")) + + fills = await broker.fills() + assert len(fills) == 1 + assert fills[0].price == Decimal("30000") + assert await broker.open_orders() == [] + + +async def test_strict_limit_without_mark_keeps_fill_at_limit_shortcut() -> None: + """Strict but NO mark injected: the historical fill-at-limit shortcut holds. + + Regression pin for the self-contained pattern the runners and most tests + rely on (a maker LIMIT priced at the close, no seeded mark): strict mode + must not change it — marketability only applies when a mark is known. + """ + broker = PaperBroker(starting_balances={"USD": money("100000")}, strict=True) + + await broker.place_order(_limit_buy(qty="1", price="15000")) + + fills = await broker.fills() + assert len(fills) == 1 + assert fills[0].price == Decimal("15000") + + +async def test_permissive_far_below_limit_still_fills_with_a_mark() -> None: + """Non-strict + mark: the permissive model never rests (unchanged).""" + broker = PaperBroker( + prices={BTC_USD: money("30000")}, + starting_balances={"USD": money("100000")}, + ) + + await broker.place_order(_limit_buy(qty="1", price="15000")) + + assert len(await broker.fills()) == 1 + assert await broker.open_orders() == [] + + +async def test_strict_resting_order_dedups_repeated_client_order_id() -> None: + """A resting strict order's client-order-id is deduped on a retry too.""" + broker = PaperBroker( + prices={BTC_USD: money("30000")}, + starting_balances={"USD": money("100000")}, + strict=True, + ) + + id1 = await broker.place_order(_limit_buy(qty="1", price="15000", cid="rest-dup")) + id2 = await broker.place_order(_limit_buy(qty="1", price="15000", cid="rest-dup")) + + assert id2 == id1 + assert len(await broker.open_orders()) == 1 + assert await broker.fills() == [] diff --git a/trading_bot/tests/interfaces/test_cli_commands.py b/trading_bot/tests/interfaces/test_cli_commands.py index 219684df..6d3b7d94 100644 --- a/trading_bot/tests/interfaces/test_cli_commands.py +++ b/trading_bot/tests/interfaces/test_cli_commands.py @@ -28,6 +28,7 @@ import pytest from typer.testing import CliRunner +from trading_bot.application.canary import CanaryCheck, CanaryReport, size_for_minimums from trading_bot.application.performance_service import PerformanceService from trading_bot.domain.fill import Fill from trading_bot.domain.instrument import Instrument, Symbol @@ -35,6 +36,7 @@ from trading_bot.domain.order import Order, OrderSide, OrderType from trading_bot.domain.position import Position from trading_bot.interfaces.cli import _render +from trading_bot.interfaces.cli import main as cli_main from trading_bot.interfaces.cli.main import _ensure_cwd_importable, app from trading_bot.storage.sqlite_store import SqliteStore @@ -445,6 +447,268 @@ def test_kpi_capital_flag_beats_config_starting_capital( assert "201000" not in result.output # not the config anchor +# --- canary ------------------------------------------------------------------ # + +#: A spec-carrying instrument shaped like the real Binance BTC/USDT venue spec +#: (mirrors trading_bot/tests/application/test_canary.py's own fixture), so a +#: monkeypatched, offline ``_resolve_canary_sizing`` still exercises the real +#: sizing/quantization path the CLI would hit against the live resolver. +_CANARY_INSTRUMENT = Instrument( + Symbol("BTC", "USDT"), + price_precision=2, + qty_precision=5, + min_qty=money("0.00001"), + min_notional=money("5"), +) +_CANARY_MARK = money("50000") +_CANARY_QTY = size_for_minimums(_CANARY_INSTRUMENT, _CANARY_MARK) + + +def _patch_offline_sizing(monkeypatch: pytest.MonkeyPatch) -> None: + """Monkeypatch ``_resolve_canary_sizing`` so a canary test never hits the + network: same fixed ``(instrument, mark, qty)`` every time, computed + through the real :func:`~trading_bot.application.canary.size_for_minimums`. + """ + + async def _fake_resolve(exchange: str, symbol: str): + return _CANARY_INSTRUMENT, _CANARY_MARK, _CANARY_QTY + + monkeypatch.setattr(cli_main, "_resolve_canary_sizing", _fake_resolve) + + +def test_canary_paper_run_all_green( + tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """`canary` (paper, offline sizing) exits 0 and prints the all-PASS evidence.""" + _patch_offline_sizing(monkeypatch) + db = tmp_path / "canary.db" + + result = runner.invoke(app, ["canary", "--db", str(db)]) + + assert result.exit_code == 0, result.output + assert "BTC/USDT" in result.output + assert "cancel_probe.resting" in result.output + assert "roundtrip.flat" in result.output + assert "oracle.fill_count" in result.output + assert "FAIL" not in result.output + assert "result: PASS" in result.output + + +def test_canary_seeded_failure_exits_nonzero_with_fail_line( + tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A run whose scenario reports a failed check exits 1 with the FAIL line. + + ``run_canary`` itself is exhaustively tested against the real engine in + ``tests/application/test_canary.py``; this only checks the CLI's own + responsibility — rendering a failed report and setting a non-zero exit — + so the scenario is replaced with a canned report carrying one seeded + failure, deterministically, offline. + """ + _patch_offline_sizing(monkeypatch) + + async def _fake_run_canary(engine, instrument, *, qty, probe_offset_pct): + report = CanaryReport( + exchange="paper", mode="paper", instrument=instrument, qty=qty + ) + report.checks.append( + CanaryCheck( + name="seeded.failure", + expected="status=open", + observed="status=filled", + passed=False, + ) + ) + report.cost = money("0") + return report + + monkeypatch.setattr(cli_main, "run_canary", _fake_run_canary) + db = tmp_path / "canary.db" + + result = runner.invoke(app, ["canary", "--db", str(db)]) + + assert result.exit_code == 1, result.output + assert "seeded.failure" in result.output + assert "FAIL" in result.output + assert "result: FAIL" in result.output + + +def test_canary_testnet_kraken_refused_no_sandbox( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """`--mode testnet --exchange kraken` refuses clearly: no public spot testnet. + + The factory's ``_build_testnet_venue`` raises before any I/O (constructing + an adapter sends nothing), so this runs fully offline and places no order. + """ + result = runner.invoke(app, ["canary", "--mode", "testnet", "--exchange", "kraken"]) + + assert result.exit_code != 0 + assert "refusing to run canary" in result.output + assert "no testnet" in result.output + + +def test_canary_testnet_without_credentials_refused( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """`--mode testnet` without testnet credentials refuses before any order. + + With every Binance key stripped from the environment, the factory's + credential gate raises (the adapter is built keyless, no request goes + out), and the CLI surfaces it as a clean non-zero exit — offline. + """ + for var in ( + "BINANCE_TESTNET_API_KEY", + "BINANCE_TESTNET_API_SECRET", + "BINANCE_API_KEY", + "BINANCE_API_SECRET", + ): + monkeypatch.delenv(var, raising=False) + + result = runner.invoke(app, ["canary", "--mode", "testnet"]) + + assert result.exit_code != 0 + assert "refusing to run canary" in result.output + assert "credentials" in result.output + + +def test_canary_live_without_config_refused(monkeypatch: pytest.MonkeyPatch) -> None: + """`--mode live` without --config refuses before any prompt or build.""" + + def _must_not_build(*args, **kwargs): + raise AssertionError("live canary must refuse before building an engine") + + monkeypatch.setattr(cli_main, "build_engine", _must_not_build) + + result = runner.invoke(app, ["canary", "--mode", "live"]) + + assert result.exit_code != 0 + assert "--config" in result.output + assert "no order was placed" in result.output + + +def test_canary_live_without_live_enabled_refused( + tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """`--mode live` with a config missing live_enabled refuses, pre-prompt. + + Mirrors the ``run --live`` gate: the off-by-default ``live_enabled`` opt-in + must come from the operator's config — the canary never sets it. Refused + before the typed confirmation is even asked and before any engine exists. + """ + + def _must_not_build(*args, **kwargs): + raise AssertionError("live canary must refuse before building an engine") + + monkeypatch.setattr(cli_main, "build_engine", _must_not_build) + cfg = tmp_path / "live.yaml" + cfg.write_text("mode: live\nbrokers:\n - {name: kraken, exchange: kraken}\n") + + result = runner.invoke(app, ["canary", "--mode", "live", "--config", str(cfg)]) + + assert result.exit_code != 0 + assert "live is off by default" in result.output + assert "live_enabled" in result.output + assert "no order was placed" in result.output + + +def test_canary_live_confirmation_mismatch_refused( + tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """`--mode live` with the gates open still refuses without the typed phrase. + + The typed confirmation is the dashboard's go-live pattern: the exact + acknowledgement phrase, verbatim — anything else refuses with no engine + built and no order placed. + """ + + def _must_not_build(*args, **kwargs): + raise AssertionError("live canary must refuse before building an engine") + + monkeypatch.setattr(cli_main, "build_engine", _must_not_build) + cfg = tmp_path / "live.yaml" + cfg.write_text( + "mode: live\nlive_enabled: true\n" + "brokers:\n - {name: kraken, exchange: kraken}\n" + ) + + result = runner.invoke( + app, ["canary", "--mode", "live", "--config", str(cfg)], input="nope\n" + ) + + assert result.exit_code != 0 + assert "typed acknowledgement" in result.output + assert "no order was placed" in result.output + + +def test_canary_live_confirmed_reaches_existing_factory_gates( + tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A correctly-typed live confirmation proceeds — into the factory's gates. + + With the opt-in set and the exact phrase typed, the CLI goes on to build + the engine, where the EXISTING factory gates take over: with every Kraken + key stripped, the credential gate raises and the run refuses cleanly — + still offline, still no order. (Proves the confirmation path is wired to + the same guarded factory, not around it.) + """ + monkeypatch.delenv("KRAKEN_API_KEY", raising=False) + monkeypatch.delenv("KRAKEN_API_SECRET", raising=False) + cfg = tmp_path / "live.yaml" + cfg.write_text( + "mode: live\nlive_enabled: true\n" + "brokers:\n - {name: kraken, exchange: kraken}\n" + "risk: {max_order: '0.01', max_position: '0.01', max_daily_loss: '10'}\n" + ) + + result = runner.invoke( + app, + ["canary", "--mode", "live", "--exchange", "kraken", "--config", str(cfg)], + input="I UNDERSTAND\n", + ) + + assert result.exit_code != 0 + assert "refusing to run canary" in result.output + assert "credentials" in result.output + + +def test_canary_live_ack_phrase_mirrors_dashboard() -> None: + """The CLI's typed phrase is the dashboard's go-live phrase, verbatim. + + The CLI deliberately does not import the API module (FastAPI stays out of + the CLI's import graph), so this test is what keeps the two constants from + drifting apart. + """ + from trading_bot.interfaces.api import app as api_app + + assert cli_main._LIVE_ACK_PHRASE == api_app._LIVE_ACK_PHRASE + + +def test_canary_max_cost_refuses_before_any_order( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A ``--max-cost`` below the implied cost refuses before building anything. + + The implied paper cost for the fixed fixture is exact: ``2 * qty * mark * + fee_bps / 10000 = 2 * 0.0001 * 50000 * 10 / 10000 = 0.01``. A bound of + ``0.001`` must refuse — and ``run_canary`` must never be called (no engine + built, no order placed). + """ + _patch_offline_sizing(monkeypatch) + + async def _must_not_be_called(engine, instrument, *, qty, probe_offset_pct): + raise AssertionError("--max-cost refusal must happen before any order") + + monkeypatch.setattr(cli_main, "run_canary", _must_not_be_called) + + result = runner.invoke(app, ["canary", "--max-cost", "0.001"]) + + assert result.exit_code == 1, result.output + assert "refusing to run canary" in result.output + assert "exceeds" in result.output + assert "--max-cost" in result.output + + # --- _render helpers (no CLI) ---------------------------------------------- # diff --git a/trading_bot/tests/interfaces/test_dashboard.py b/trading_bot/tests/interfaces/test_dashboard.py index 6c5493b3..4bf6c6cc 100644 --- a/trading_bot/tests/interfaces/test_dashboard.py +++ b/trading_bot/tests/interfaces/test_dashboard.py @@ -167,6 +167,51 @@ def test_every_page_carries_the_health_pill_helper(path: str) -> None: assert "healthPillHtml" in html, path +@pytest.mark.parametrize("path", _PAGES) +def test_every_page_carries_the_expandable_row_helper(path: str) -> None: + """Every page's shell (base.html) carries the shared expandable-row helper. + + Leaf 01 (dashboard-tables-ux): clicking a table row opens a detail panel + beneath it (the pinned UX pattern — not a hover popin, not a per-order + page); `tbExpander()` is defined once in base.html so every page carries + it, including the ones this leaf doesn't wire yet (leaf 02 reuses it for + orders/fills tables). + """ + html = _client().get(path).text + assert "function tbExpander" in html, path + + +@pytest.mark.parametrize("path", _PAGES) +def test_every_page_carries_the_order_execution_cell_builders(path: str) -> None: + """Every page's shell (base.html) carries the shared order-execution cells. + + Leaf 02 (dashboard-tables-ux): the Filled % / Value cells and the expanded + per-order detail panel (ids, limit/stop, Σ fees, the fills join) are built + by `orderFilledPctCell()` / `orderValueCell()` / `orderDetailHtml()`, + defined once in base.html so the three order surfaces (Overview + open-orders, Orders page, strategy detail) render execution identically. + """ + html = _client().get(path).text + assert "function orderFilledPctCell" in html, path + assert "function orderValueCell" in html, path + assert "function orderDetailHtml" in html, path + + +@pytest.mark.parametrize("path", _PAGES) +def test_every_page_carries_the_bar_timing_chip_helper(path: str) -> None: + """Every page's shell (base.html) carries the shared bar-timing chip helper. + + Leaf 03 (dashboard-tables-ux): the "last bar X ago -> next in Y" chip is + fed by `barTimingChipHtml()`, defined once in base.html (mirroring + `healthPillHtml`'s shape) so both the roster and the detail header render + it identically, and so the epoch-aligned `nextBarCloseMs` guess it + replaces stays retired everywhere. + """ + html = _client().get(path).text + assert "function barTimingChipHtml" in html, path + assert "nextBarCloseMs" not in html, path + + @pytest.mark.parametrize("path", _PAGES) def test_nav_lists_four_tabs_and_no_pnl(path: str) -> None: """Every page's nav lists the four surviving tabs; the retired PnL tab is gone. @@ -672,10 +717,12 @@ def test_strategy_detail_serves_the_shell_with_name_and_sections() -> None: # The stats table's Return column + the muted starting-capital note. assert 'Return' in html assert 'id="pnl-v0-note"' in html - # This strategy's positions / orders / fills tables. + # This strategy's positions / orders tables. No standalone fills table — + # leaf 02 (dashboard-tables-ux) demoted fills into the orders' expanded + # detail (see test_strategy_detail_standalone_fills_table_is_gone). assert 'id="positions-table"' in html assert 'id="orders-table"' in html - assert 'id="fills-table"' in html + assert 'id="fills-table"' not in html # The control surface + go-live modal moved here from the roster row. assert "mode-select" in html assert 'id="live-modal"' in html @@ -740,6 +787,85 @@ def test_strategy_detail_carries_the_health_pill_hook() -> None: assert "healthPillHtml(s)" in html +def test_strategy_detail_carries_the_bar_timing_chip_hook() -> None: + """The detail header wires the shared bar-timing chip next to the pills (leaf 03). + + `#detail-bar-timing` sits alongside `#detail-run-pill`; `renderHeader()` + fills it from `barTimingChipHtml(s)` — the same helper the roster uses. + """ + html = _client().get("/strategies/btc-ma").text + assert 'id="detail-bar-timing"' in html + assert "barTimingChipHtml(s)" in html + + +def test_strategy_detail_positions_table_is_asset_first_with_expandable_rows() -> None: + """The detail page's positions table reads asset-first with expandable rows. + + Leaf 01 (dashboard-tables-ux): Asset | Qty | Avg entry | Price (as-of) | + Value | Unrealised | Realised (net) — Instrument and Fees are no longer + their own columns (the native pair and cumulative fees move into the + expanded detail); this page is already scoped to one strategy/exchange so + (unlike the overview variant) it carries no Strategy/Exchange column. + """ + html = _client().get("/strategies/btc-ma").text + assert "Asset" in html + assert 'Qty' in html + assert 'Avg entry' in html + assert "Price Value' in html + assert 'Unrealised' in html + assert 'Realised (net)' in html + assert "Net qty" not in html # the old column header is gone + # The shared expandable-row helper, wired for this table (keyed by + # instrument — one row per instrument within this strategy). + assert "tbExpander(document.getElementById('positions-body')" in html + assert "posExpander.rowHtml(r.instrument" in html + # Expanded detail: native pair, cumulative fees, gross/fees breakdown, mark + # provenance (gross = realised_pnl + fees_paid, computed client-side). + assert "function positionDetailHtml" in html + assert "Native pair" in html + assert "Realised gross" in html + assert "Mark source" in html + + +def test_strategy_detail_orders_table_shows_execution_with_expandable_fills() -> None: + """The detail page's orders table reads the execution blueprint, rows expandable. + + Leaf 02 (dashboard-tables-ux): Time | Instrument | Side | Type | Qty | + Filled % | Avg fill | Value | Status; a row expands (the shared tbExpander) + into that order's fills + client/venue ids, limit/stop, Σ fees — joined + client-side by client_order_id from the fills payload the page already + fetches (no new endpoint, no per-expansion request). + """ + html = _client().get("/strategies/btc-ma").text + assert 'Filled %' in html + assert 'Avg fill' in html + assert 'Value' in html + assert 'Filled' not in html # the bare-qty column is gone + # The shared expandable-row helper, wired for the orders table (keyed by + # client_order_id — the idempotency key is the natural row identity). + assert "tbExpander(document.getElementById('orders-body')" in html + assert "orderExpander.rowHtml(o.client_order_id" in html + # The client-side fills join feeding the shared detail builder (base.html). + assert "_fillsByOrder" in html + assert "function orderDetailHtml" in html + assert "/api/fills" in html # the join source is still fetched + + +def test_strategy_detail_standalone_fills_table_is_gone() -> None: + """The detail page's standalone "Recent fills" table is retired (leaf 02). + + Fills now live under their order (the expanded detail); the underlying + /api/fills fetch survives as the client_order_id join source. + """ + html = _client().get("/strategies/btc-ma").text + assert 'id="fills-card"' not in html + assert 'id="fills-table"' not in html + assert 'id="fills-body"' not in html + assert "Recent fills" not in html + assert "/api/fills" in html # the fetch survives — it feeds the join + + def test_strategy_detail_read_only_hides_capital_controls() -> None: """A read-only detail page drops the capital mutation controls, keeps the figures.""" html = _client(read_only=True).get("/strategies/btc-ma").text @@ -961,6 +1087,329 @@ def test_positions_unknown_group_by_is_422() -> None: assert _client().get("/api/positions?group_by=bogus").status_code == 422 +def test_api_positions_contract_regression(tmp_path) -> None: # noqa: ANN001 + """`/api/positions` carries every pre-existing field, byte-identical, plus the new ones. + + A public-contract regression check (mirrors `test_api_health_worst_and_count`): + proves the ``mark``/``mark_asof_ts``/``mark_source``/``value``/``unrealised``/ + ``fee_ccy`` additions (api-completeness leaf 02) never displaced or renamed any + of the eight fields the endpoint already shipped. + """ + from trading_bot.storage.sqlite_store import SqliteStore + + db = str(tmp_path / "book.sqlite") + inst = Instrument(Symbol("BTC", "USD")) + store = SqliteStore(db) + store.record_fill( + Fill("SF1", "sc1", inst, OrderSide.BUY, money("2"), money("100"), money("1"), 1) + ) + config = AppConfig.model_validate( + { + "mode": "paper", + "storage": {"db_path": db}, + "brokers": [{"name": "kraken", "exchange": "kraken"}], + "strategies": [ + { + "name": "btc-ma", + "symbol": "BTC/USD", + "data": {"exchange": "kraken", "span": 60}, + "signal": {"ref": "ma_crossover", "params": {"fast": 3, "slow": 6}}, + "reference_qty": "2", + "lookback": 6, + } + ], + } + ) + sup = StrategySupervisor(config, dccd_client=_FakeStartClient()) + asyncio.run(sup.start("btc-ma")) + # Seed the mark cache directly (the bar_close path) so every new field populates. + sup._units["btc-ma"].engine.mark_cache.update( # noqa: SLF001 + Symbol("BTC", "USD"), money("110"), 9_000 + ) + + [row] = TestClient(create_dashboard_app(sup)).get("/api/positions").json() + # Every pre-existing field, unchanged. + assert row["strategy"] == "btc-ma" + assert row["exchange"] == "kraken" + assert row["instrument"] == "BTC/USD" + assert row["base"] == "BTC" + assert row["net_qty"] == "2" + assert row["avg_entry_price"] == "100" + assert row["realised_pnl"] == "-1" # the opening fill's fee reduces realised PnL + assert row["fees_paid"] == "1" + # The new fields. + assert row["mark"] == "110" + assert row["mark_asof_ts"] == 9_000 + assert row["mark_source"] == "bar_close" + assert row["value"] == "220" + assert row["unrealised"] == "20" + assert row["fee_ccy"] == "USD" + + +# --- Balances (broker-reported free balances, one row per running unit) --- # + + +async def test_api_balances_running_unit_reports_broker_balances() -> None: + """`/api/balances` relays the running unit's own broker-reported balances. + + Seeds the paper simulator's ledger directly (mirroring the leaf-02 + mark-cache seeding technique) so the row is proven to come from + `Broker.balances()` itself, not the locally-tracked position — the whole + point of this endpoint (the positions<->balances cross-check seam). + """ + sup = StrategySupervisor(_two_venue_config(), dccd_client=_two_venue_client()) + await sup.start("btc-kraken") + unit = sup._units["btc-kraken"] # noqa: SLF001 — direct broker seed, test-only + unit.engine.broker._balances.update( # noqa: SLF001 + {"USD": money("998"), "BTC": money("2")} + ) + + [row] = TestClient(create_dashboard_app(sup)).get("/api/balances").json() + assert row["strategy"] == "btc-kraken" + assert row["exchange"] == "kraken" + assert row["mode"] == "paper" + assert row["balances"] == {"USD": "998", "BTC": "2"} + assert row["error"] is None + + +async def test_api_balances_stopped_unit_absent() -> None: + """A stopped unit contributes no row; empty list before anything starts.""" + sup = StrategySupervisor(_two_venue_config(), dccd_client=_two_venue_client()) + client = TestClient(create_dashboard_app(sup)) + assert client.get("/api/balances").json() == [] + + await sup.start("btc-kraken") + sup._units["btc-kraken"].engine.broker._balances.update( # noqa: SLF001 + {"USD": money("100")} + ) + rows = client.get("/api/balances").json() + assert {r["strategy"] for r in rows} == {"btc-kraken"} + + +async def test_api_balances_strategy_filter() -> None: + """`?strategy=` narrows the balances rows to one unit, mirroring orders/fills.""" + sup = StrategySupervisor(_two_venue_config(), dccd_client=_two_venue_client()) + await sup.start("btc-kraken") + await sup.start("eth-binance") + sup._units["btc-kraken"].engine.broker._balances.update( # noqa: SLF001 + {"USD": money("100")} + ) + sup._units["eth-binance"].engine.broker._balances.update( # noqa: SLF001 + {"USDT": money("200")} + ) + client = TestClient(create_dashboard_app(sup)) + rows = client.get("/api/balances?strategy=btc-kraken").json() + assert [r["strategy"] for r in rows] == ["btc-kraken"] + + +async def test_api_balances_broker_error_degrades_to_error_row() -> None: + """A broker error is never a 500 — it degrades to an ``error`` row (poll-safe).""" + from trading_bot.domain.errors import BrokerError + + sup = StrategySupervisor(_two_venue_config(), dccd_client=_two_venue_client()) + await sup.start("btc-kraken") + unit = sup._units["btc-kraken"] # noqa: SLF001 — direct broker patch, test-only + + async def _boom() -> dict[str, object]: + raise BrokerError("kraken: rate limited") + + unit.engine.broker.balances = _boom # type: ignore[method-assign] + + resp = TestClient(create_dashboard_app(sup)).get("/api/balances") + assert resp.status_code == 200 + [row] = resp.json() + assert row["strategy"] == "btc-kraken" + assert row["balances"] == {} + assert row["error"] == "kraken: rate limited" + + +# --- Epic-wide contract sweep (api-completeness leaf 04) ------------------- # + + +async def test_api_completeness_contract_sweep(tmp_path) -> None: # noqa: ANN001 + """Every pre-existing field on the six read endpoints still holds, in one pass. + + The additive-only proof for the whole `api-completeness` epic, right before + the API contract freeze (road-to-1.0 #5): `/api/strategies`, `/api/positions` + and `/api/kpi` each already have a dedicated, per-field contract-regression + test proving the leaf-02/03 additions never displaced a pre-existing field + (`test_api_positions_contract_regression`, + `test_api_strategies_display_fields_and_contract_regression`, + `test_api_kpi_display_fields_and_contract_regression` — test_display_ccy.py) + and `/api/health`'s shape is pinned by `test_health_shape_and_values` (above). + This sweep extends that established pattern to `/api/fills` and + `/api/orders` (untouched by any leaf, but still part of the frozen contract) + and exercises all six together against one running unit, so the whole + epic's additive-only guarantee is proven in a single place. + """ + from trading_bot.domain.order import Order, OrderType + from trading_bot.storage.sqlite_store import SqliteStore + + db = str(tmp_path / "book.sqlite") + inst = Instrument(Symbol("BTC", "USD")) + store = SqliteStore(db) + store.record_fill( + Fill( + "SWEEP-F1", + "sweep-c1", + inst, + OrderSide.BUY, + money("2"), + money("100"), + money("1"), + 1, + ) + ) + store.close() + config = AppConfig.model_validate( + { + "mode": "paper", + "storage": {"db_path": db}, + "brokers": [{"name": "kraken", "exchange": "kraken"}], + "strategies": [ + { + "name": "btc-ma", + "symbol": "BTC/USD", + "data": {"exchange": "kraken", "span": 60}, + "signal": {"ref": "ma_crossover", "params": {"fast": 3, "slow": 6}}, + "reference_qty": "2", + "lookback": 6, + } + ], + } + ) + sup = StrategySupervisor(config, dccd_client=_FakeStartClient()) + await sup.start("btc-ma") + unit = sup._units["btc-ma"] # noqa: SLF001 — direct seed, test-only + unit.engine.mark_cache.update(Symbol("BTC", "USD"), money("110"), 9_000) + # An open (non-terminal) order so `/api/orders` (default: open only) has a row. + unit.engine.router.restore( + [ + Order( + "sweep-open-1", + inst, + OrderSide.BUY, + money("1"), + OrderType.LIMIT, + limit_price=money("90"), + ) + ] + ) + + client = TestClient(create_dashboard_app(sup)) + + [strategy_row] = client.get("/api/strategies").json() + assert set(strategy_row) == { + "name", + "kind", + "exchange", + "span", + "quote", + "mode", + "running", + "realised_pnl", + "open_orders", + "last_eval_ts", + "last_asof_ts", + "allocation", + "contributed", + "unrealised", + "total_value", + "capital_policy", + "health", + "health_detail", + "display_currency", + "total_value_display", + "unrealised_display", + } + + [position_row] = client.get("/api/positions").json() + assert set(position_row) == { + "strategy", + "exchange", + "instrument", + "base", + "net_qty", + "avg_entry_price", + "realised_pnl", + "fees_paid", + "mark", + "mark_asof_ts", + "mark_source", + "value", + "unrealised", + "fee_ccy", + "display_currency", + "value_display", + "unrealised_display", + } + + [fill_row] = client.get("/api/fills").json() + assert set(fill_row) == { + "strategy", + "exchange", + "base", + "fill_id", + "client_order_id", + "instrument", + "side", + "qty", + "price", + "fee", + "ts", + } + + [order_row] = client.get("/api/orders").json() + assert set(order_row) == { + "strategy", + "exchange", + "base", + "ts", + "client_order_id", + "venue_order_id", + "instrument", + "side", + "type", + "qty", + "limit_price", + "stop_price", + "status", + "filled_qty", + "avg_fill_price", + "reject_reason", + } + + [kpi_row] = client.get("/api/kpi?level=strategy").json() + assert set(kpi_row) == { + "level", + "key", + "strategy", + "exchange", + "quote", + "realised_pnl", + "fees_paid", + "sharpe", + "sortino", + "calmar", + "max_drawdown", + "display_currency", + "realised_pnl_display", + "fees_paid_display", + } + + health_body = client.get("/api/health").json() + assert set(health_body) == { + "status", + "mode", + "strategies", + "read_only", + "next_tick_ts", + "tick", + "worst", + "unhealthy", + } + + def test_orders_endpoint_present_and_empty() -> None: """`GET /api/orders` exists and is an empty list when nothing is open.""" assert _client().get("/api/orders").json() == [] @@ -1256,9 +1705,51 @@ def test_orders_page_has_tables_and_filters() -> None: # history read comes back at exactly the server's default ?limit= cap. assert 'id="orders-cap"' in html and "showing the most recent 200" in html assert 'id="fills-cap"' in html - # The Orders table's Time column (order date/time, this leaf) — mirrors the - # Fills table's own Time column. - assert html.count("Time") == 2 # one in Orders, one in Fills + # The Orders table's Time column (order date/time) — mirrors the Fills + # table's own Time column. Three occurrences since dashboard-tables-ux + # leaf 02: the Orders + Fills table headers, plus base.html's shared + # orderDetailHtml builder whose nested per-order fills table carries its + # own Time header inside the inline script. + assert html.count("Time") == 3 + + +def test_orders_page_orders_table_shows_execution_with_expandable_fills() -> None: + """The Orders page's orders table reads the execution blueprint, rows expandable. + + Leaf 02 (dashboard-tables-ux): Qty | Filled % | Avg fill | Value replace the + bare Filled column; a row expands (the shared tbExpander) into that order's + fills joined by client_order_id from the fills payload the page already + fetches. The Strategy/Exchange tag columns stay — this is the cross-strategy + audit view (like the flat Fills table below it). + """ + html = _client().get("/orders").text + assert 'Filled %' in html + assert 'Avg fill' in html + assert 'Value' in html + assert 'Filled' not in html # the bare-qty column is gone + assert "Strategy" in html and "Exchange" in html + # The shared expandable-row helper, wired for the orders table (keyed by + # client_order_id) + the client-side fills join. + assert "tbExpander(document.getElementById('orders-body')" in html + assert "orderExpander.rowHtml(o.client_order_id" in html + assert "_fillsByOrder" in html + + +def test_orders_page_fills_table_gains_order_and_value() -> None: + """The Orders page's flat fills view (audit) gains Order + Value columns. + + Leaf 02 (dashboard-tables-ux): Order = the owning client_order_id truncated + to ~12 chars (the full id in the tooltip — the tie back to the order the + expansions key on); Value = qty × price (display-only arithmetic, exact + factors in the tooltip). + """ + html = _client().get("/orders").text + assert "Order" in html + # Value appears twice: once in the orders table, once in the fills table. + assert html.count('Value') == 2 + assert "function orderIdCell" in html # truncation, full id in tooltip + assert "slice(0, 12)" in html + assert "function fillValueCell" in html # qty × price def test_logs_page_has_feed_and_subscribes_to_sse() -> None: @@ -1313,6 +1804,57 @@ def test_overview_page_has_kpi_strip_and_tables() -> None: assert "/api/kpi" in html +def test_overview_positions_table_is_asset_first_with_expandable_rows() -> None: + """The overview positions table reads asset-first with expandable rows. + + Leaf 01 (dashboard-tables-ux): Asset | Qty | Avg entry | Price (as-of) | + Value | Unrealised | Realised (net) — plus Strategy/Exchange on this + (overview) variant, matching the existing grouping behaviour (Strategy + dropped when grouped by strategy; Exchange always shown). + """ + html = _client().get("/").text + assert "Asset" in html + assert 'Qty' in html + assert 'Avg entry' in html + assert "Price Value' in html + assert 'Unrealised' in html + assert 'Realised (net)' in html + assert "Net qty" not in html # the old column header is gone + # The shared expandable-row helper, wired for this table (keyed by + # strategy+exchange+instrument — positions are never merged across units). + assert "tbExpander(document.getElementById('positions-body')" in html + assert "posExpander.rowHtml(positionKey(r)" in html + assert "function positionDetailHtml" in html + # Value/Unrealised prefer the display-currency conversion when present. + assert "value_display" in html + assert "unrealised_display" in html + assert "display_currency" in html + + +def test_overview_open_orders_shows_execution_columns_without_expansion() -> None: + """The overview's open-orders table gains the execution columns, rows plain. + + Leaf 02 (dashboard-tables-ux): Filled % | Avg fill | Value via base.html's + shared cell builders, keeping the Strategy/Exchange tags. No row expansion + on THIS surface: the expanded panel's content is the order's fills and the + overview does not fetch /api/fills — a per-refresh fills fetch solely for + an optional affordance is disproportionate, and a fills-less panel would + fork the shared detail builder. The Orders page carries the full expansion. + """ + html = _client().get("/").text + assert 'Filled %' in html + assert 'Avg fill' in html + assert 'Filled' not in html # the bare-qty column is gone + # The shared execution cells are used by the row renderer (the call sites, + # not just base.html's `function ...` definitions)... + assert "+ orderFilledPctCell(o) +" in html + assert "+ orderValueCell(o) +" in html + # ...but the orders tbody is NOT wired to the expander (positions still is). + assert "tbExpander(document.getElementById('orders-body')" not in html + assert "tbExpander(document.getElementById('positions-body')" in html + + async def _never_disconnect() -> dict[str, object]: """An ASGI ``receive`` that never reports a disconnect (the consumer stays up).""" import asyncio @@ -1617,9 +2159,14 @@ def test_strategies_page_is_a_linked_roster() -> None: # Rows link to the per-strategy detail page (client-rendered in the roster JS). assert 'href="/strategies/' in html assert "strat-link" in html - # The kept roster columns (cadence / next-bar / last-eval), plus the stamp. + # The kept roster columns (cadence / bar-timing / last-eval), plus the stamp. + # "Next bar" retired (leaf 03, dashboard-tables-ux) — the epoch-aligned + # guess had no relation to the real cadence; "Bar timing" replaces it with + # the honest last_asof_ts-derived chip (see test_strategies_roster_... + # _bar_timing_chip_hook below). assert "Cadence" in html - assert "Next bar" in html + assert "Bar timing" in html + assert "Next bar" not in html assert "Last eval" in html # The condensed Total-value column (leaf 08) — replaces nothing; Realised # PnL stays alongside it so the same numbers read at every altitude. @@ -1643,6 +2190,18 @@ def test_strategies_roster_carries_the_health_pill_hook() -> None: assert "healthPillHtml(s)" in html +def test_strategies_roster_carries_the_bar_timing_chip_hook() -> None: + """The roster's row renderer calls the shared bar-timing chip helper (leaf 03). + + Placed in the "Bar timing" column that replaced "Next bar" — the old + `nextBarCell()` / `tbTime.nextBarCloseMs()` hooks are gone from the page. + """ + html = _client().get("/strategies").text + assert "barTimingChipHtml(s)" in html + assert "nextBarCell" not in html + assert "nextBarCloseMs" not in html + + def test_strategies_page_read_only_note_and_no_deploy_link() -> None: """A read-only roster advertises disabled controls and hides the Deploy link.""" html = _client(read_only=True).get("/strategies").text diff --git a/trading_bot/tests/storage/test_sqlite_store.py b/trading_bot/tests/storage/test_sqlite_store.py index f68a25f5..a78a15fc 100644 --- a/trading_bot/tests/storage/test_sqlite_store.py +++ b/trading_bot/tests/storage/test_sqlite_store.py @@ -680,6 +680,69 @@ def test_migration_is_idempotent_and_preserves_new_schema(tmp_path) -> None: assert (rec.mode, rec.venue) == ("paper", "kraken") +def test_fee_asset_round_trips_and_backfills_none(tmp_path) -> None: + """``Fill.fee_asset`` persists exactly; absent/legacy rows read back ``None``. + + A venue may denominate a commission in the base asset (Binance charges a + market buy's fee in BTC on BTC/USDT) — the denomination must survive the + store round trip for venue-truth accounting (the canary's identity + oracle). A fill recorded without one (the historic quote assumption) and + a pre-migration row (the column backfills NULL) both read back ``None``. + """ + db = tmp_path / "fees.db" + store = SqliteStore(db) + base_fee_fill = Fill( + fill_id="BF1", + client_order_id="cid-bf", + instrument=BTC_USD, + side=OrderSide.BUY, + qty=money("0.001"), + price=money("50000"), + fee=money("0.000001"), + ts=1_700_000_000_000, + fee_asset="BTC", + ) + store.record_fill(base_fee_fill) + store.record_fill(_fill(fill_id="QF1")) # no fee_asset: quote-denominated + + by_id = {f.fill_id: f for f in store.fills()} + assert by_id["BF1"].fee_asset == "BTC" + assert by_id["BF1"].fee == money("0.000001") + assert by_id["QF1"].fee_asset is None + + # A legacy (pre-fee_asset) database: the migration adds the column and the + # old row reads back fee_asset=None — exactly what it meant when written. + legacy_db = tmp_path / "legacy_fee.db" + legacy = sqlite3.connect(str(legacy_db)) + try: + legacy.executescript( + """ + CREATE TABLE fills ( + fill_id TEXT PRIMARY KEY, + client_order_id TEXT NOT NULL, + instrument TEXT NOT NULL, + side TEXT NOT NULL, + qty TEXT NOT NULL, + price TEXT NOT NULL, + fee TEXT NOT NULL, + ts INTEGER NOT NULL + ); + """ + ) + legacy.execute( + "INSERT INTO fills VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + ("OLD-FA", "c1", "BTC/USD", "buy", "1", "30000", "3", 1_700), + ) + legacy.commit() + finally: + legacy.close() + migrated = SqliteStore(legacy_db) + [old_fill] = migrated.fills() + assert old_fill.fill_id == "OLD-FA" + assert old_fill.fee_asset is None + assert old_fill.fee == money("3") + + # --- orders: schema migration (add ts/reject_reason/fill_tolerance) --------- #