diff --git a/README.md b/README.md index e7ceac0..250ea06 100644 --- a/README.md +++ b/README.md @@ -286,40 +286,50 @@ MariaDB and uses its RETURNING dialect. Piccolo has no MySQL backend: | operation | yara-orm | tortoise | sqlalchemy | pony | django | peewee | sqlobject | ormar | |---------------|---------:|---------:|-----------:|-----:|-------:|-------:|----------:|------:| -| bulk_insert | 30.5 | 41.5 | 99.0 | 455.0 | 87.7 | 59.5 | 1257.2 | 191.2 | -| single_insert | 392.8 | 304.8 | 430.8 | 329.6 | 343.6 | 367.3 | 359.8 | 555.3 | -| fetch_all | 5.5 | 34.0 | 42.5 | 47.9 | 28.4 | 30.4 | 44.9 | 72.8 | -| count | 0.5 | 0.8 | 1.2 | 0.7 | 0.7 | 0.8 | 0.7 | 4.7 | -| group_by | 1.3 | 1.3 | 2.1 | 2.2 | 1.5 | 1.1 | 0.9 | - | -| filter | 3.2 | 17.8 | 16.0 | 24.5 | 15.5 | 14.7 | 16.9 | 31.9 | -| get_by_pk | 123.3 | 228.7 | 534.1 | 310.7 | 214.5 | 206.0 | 66.0 | 916.1 | -| update | 3.8 | 3.3 | 6.5 | 265.9 | 4.3 | 4.3 | 4.2 | 7.6 | -| delete | 3.2 | 3.2 | 3.3 | 249.9 | 3.3 | 3.0 | 2.9 | 3.6 | - -(MariaDB's `single_insert` ~390 ms is markedly faster than MySQL 8's — a lighter +| bulk_insert | 23.3 | 37.5 | 105.4 | 475.4 | 96.4 | 71.8 | 1266.5 | 209.9 | +| single_insert | 264.9 | 311.6 | 531.7 | 391.5 | 388.2 | 372.2 | 390.3 | 660.0 | +| fetch_all | 5.7 | 35.0 | 43.8 | 48.1 | 28.1 | 37.0 | 42.6 | 72.1 | +| count | 0.5 | 0.8 | 1.2 | 0.7 | 0.7 | 0.8 | 0.6 | 6.7 | +| group_by | 1.2 | 1.3 | 2.3 | 2.2 | 1.5 | 1.4 | 0.9 | - | +| filter | 3.2 | 17.5 | 16.3 | 24.8 | 15.1 | 14.7 | 17.1 | 32.6 | +| get_by_pk | 132.9 | 240.8 | 575.0 | 306.7 | 220.7 | 206.5 | 64.4 | 895.2 | +| update | 4.0 | 4.4 | 7.8 | 266.3 | 5.4 | 4.2 | 3.7 | 8.2 | +| delete | 3.0 | 3.0 | 3.6 | 249.5 | 3.6 | 3.0 | 3.0 | 4.0 | + +(`single_insert`, `update` and `delete` are near ties across every ORM here +because they're database-bound — single inserts are paced by MariaDB's +per-commit disk fsync, and `update`/`delete` are one server-side set statement +each — so there's no client-side marshaling for the Rust hot path to speed up. +MariaDB's `single_insert` ~265 ms is markedly faster than MySQL 8's — a lighter default commit path.) ### SQLite ![Yara ORM vs eight Python ORMs on SQLite — latency per operation, log scale, lower is better](docs/assets/benchmark-sqlite.png) +SQLite is in-process, so these use its recommended `sync_fast_path=1` config +(statements run synchronously on the calling thread — no I/O to overlap): + | operation | yara-orm | tortoise | sqlalchemy | pony | django | peewee | sqlobject | ormar | piccolo | |---------------|---------:|---------:|-----------:|-----:|-------:|-------:|----------:|------:|--------:| -| bulk_insert | 7.9 | 14.4 | 612.7 | 51.0 | 58.1 | 30.7 | 223.1 | 158.0 | 78.8 | -| single_insert | 32.6 | 29.3 | 240.0 | 128.3 | 139.0 | 114.7 | 139.9 | 323.2 | 259.1 | -| fetch_all | 3.4 | 39.7 | 28.8 | 51.0 | 16.3 | 12.5 | 44.9 | 54.8 | 9.1 | -| count | 0.1 | 0.3 | 0.7 | 0.2 | 0.2 | 0.1 | 0.1 | 1.7 | 0.5 | -| group_by | 0.5 | 0.8 | 1.4 | 1.5 | 0.9 | 0.7 | 0.5 | - | 1.0 | -| filter | 2.0 | 20.5 | 7.7 | 26.2 | 8.5 | 6.7 | 17.3 | 19.6 | 5.1 | -| get_by_pk | 47.4 | 87.5 | 330.9 | 30.7 | 83.6 | 77.7 | 13.3 | 501.8 | 359.5 | -| update | 0.6 | 0.5 | 1.8 | 43.1 | 1.3 | 1.2 | 1.2 | 1.6 | 1.4 | -| delete | 0.4 | 0.4 | 1.2 | 36.3 | 0.9 | 0.7 | 0.8 | 1.3 | 1.2 | - -Yara ORM wins everything throughput-shaped (fetch_all 2.7–16×, filter 2.5–13×, -bulk_insert 1.8–77×) and trails only the latency-bound point reads, where the -per-statement asyncio bridge costs tens of µs against in-process sync ORMs -(SQLObject and Pony on `get_by_pk`) — the opt-in `sqlite://...?sync_fast_path=1` -URL flag removes that bridge entirely (point queries ~7× faster). +| bulk_insert | 7.5 | 13.6 | 607.9 | 50.1 | 55.8 | 29.0 | 218.3 | 143.8 | 73.9 | +| single_insert | 15.3 | 26.9 | 234.5 | 107.0 | 120.1 | 113.5 | 124.8 | 296.7 | 240.4 | +| fetch_all | 3.4 | 38.6 | 27.1 | 51.0 | 16.4 | 12.2 | 44.2 | 52.0 | 9.2 | +| count | 0.0 | 0.2 | 0.7 | 0.2 | 0.3 | 0.1 | 0.1 | 1.6 | 0.5 | +| group_by | 0.6 | 0.7 | 1.3 | 1.4 | 0.9 | 0.6 | 0.5 | - | 1.0 | +| filter | 2.0 | 20.2 | 7.3 | 25.8 | 8.7 | 6.6 | 17.4 | 19.2 | 5.0 | +| get_by_pk | 12.5 | 79.4 | 329.6 | 31.3 | 84.6 | 75.5 | 13.3 | 484.1 | 357.2 | +| update | 0.5 | 0.5 | 1.7 | 43.0 | 1.3 | 1.2 | 1.1 | 1.7 | 1.5 | +| delete | 0.3 | 0.4 | 1.1 | 35.9 | 0.8 | 0.7 | 0.7 | 1.1 | 1.1 | + +With the fast path Yara ORM is fastest on **every** operation except the +sub-millisecond `group_by` (SQLObject's hand-written raw SQL, 0.8×) — including +the point reads it trailed on under the default async bridge (`get_by_pk` 1.1× vs +SQLObject, 2.5× vs Pony), while staying far ahead on throughput (bulk_insert +1.8–81×, fetch_all 2.7–15×, filter 2.5–13×). On the **default** async path the +per-statement bridge costs tens of µs on sequential point reads, so SQLObject's +lean sync active-record leads `get_by_pk` there; `sqlite://...?sync_fast_path=1` +removes that bridge (~7× faster point queries). Speed comes from the Rust hot path, **positional row decoding** (no per-row dict or column-name allocation), **compiled-SQL + prepared-statement caching**, and diff --git a/benchmarks/README.md b/benchmarks/README.md index 119b4a0..50048ae 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -190,50 +190,57 @@ Piccolo has no MySQL backend, so it is absent here: | operation | yara-orm | tortoise | sqlalchemy | pony | django | peewee | sqlobject | ormar | |---------------|---------:|---------:|-----------:|-----:|-------:|-------:|----------:|------:| -| bulk_insert | 30.5 | 41.5 | 99.0 | 455.0 | 87.7 | 59.5 | 1257.2 | 191.2 | -| single_insert | 392.8 | 304.8 | 430.8 | 329.6 | 343.6 | 367.3 | 359.8 | 555.3 | -| fetch_all | 5.5 | 34.0 | 42.5 | 47.9 | 28.4 | 30.4 | 44.9 | 72.8 | -| count | 0.5 | 0.8 | 1.2 | 0.7 | 0.7 | 0.8 | 0.7 | 4.7 | -| group_by | 1.3 | 1.3 | 2.1 | 2.2 | 1.5 | 1.1 | 0.9 | - | -| filter | 3.2 | 17.8 | 16.0 | 24.5 | 15.5 | 14.7 | 16.9 | 31.9 | -| get_by_pk | 123.3 | 228.7 | 534.1 | 310.7 | 214.5 | 206.0 | 66.0 | 916.1 | -| update | 3.8 | 3.3 | 6.5 | 265.9 | 4.3 | 4.3 | 4.2 | 7.6 | -| delete | 3.2 | 3.2 | 3.3 | 249.9 | 3.3 | 3.0 | 2.9 | 3.6 | +| bulk_insert | 23.3 | 37.5 | 105.4 | 475.4 | 96.4 | 71.8 | 1266.5 | 209.9 | +| single_insert | 264.9 | 311.6 | 531.7 | 391.5 | 388.2 | 372.2 | 390.3 | 660.0 | +| fetch_all | 5.7 | 35.0 | 43.8 | 48.1 | 28.1 | 37.0 | 42.6 | 72.1 | +| count | 0.5 | 0.8 | 1.2 | 0.7 | 0.7 | 0.8 | 0.6 | 6.7 | +| group_by | 1.2 | 1.3 | 2.3 | 2.2 | 1.5 | 1.4 | 0.9 | - | +| filter | 3.2 | 17.5 | 16.3 | 24.8 | 15.1 | 14.7 | 17.1 | 32.6 | +| get_by_pk | 132.9 | 240.8 | 575.0 | 306.7 | 220.7 | 206.5 | 64.4 | 895.2 | +| update | 4.0 | 4.4 | 7.8 | 266.3 | 5.4 | 4.2 | 3.7 | 8.2 | +| delete | 3.0 | 3.0 | 3.6 | 249.5 | 3.6 | 3.0 | 3.0 | 4.0 | `yara-orm` leads or ties every operation except SQLObject's leaner `get_by_pk` -(0.5×), the sub-millisecond `group_by` (SQLObject 0.8× via raw SQL), and a -statistical tie with Tortoise on `update`/`delete`. It wins the throughput ops -decisively (`fetch_all` 5.0–12.9×, `filter` 4.4–9.3×, `bulk_insert` up to 46× vs -SQLObject's row-by-row inserts). MariaDB's `single_insert` (~390 ms) is notably -faster than MySQL 8's (~630 ms) here — a lighter default commit path. +(0.5×) and the sub-millisecond `group_by` (SQLObject 0.8× via raw SQL). +`single_insert`, `update` and `delete` are near ties across every ORM because +they are **database-bound**, not client-bound: single inserts are paced by +MariaDB's per-commit disk fsync (high run-to-run variance — it swings ±40% +between runs), and `update`/`delete` are one server-side set statement each, so +there's no client-side row marshaling for the Rust hot path to accelerate. It +still wins the throughput ops decisively (`fetch_all` 4.9–12.6×, `filter` +4.6–10.2×, `bulk_insert` up to 54× vs SQLObject's row-by-row inserts). MariaDB's +`single_insert` (~265 ms) is notably faster than MySQL 8's (~630 ms) here — a +lighter default commit path. ### SQLite results -`BENCH_BACKEND=sqlite`, Python 3.12, N=5000, median of 5 (ms, lower is better): +`BENCH_BACKEND=sqlite`, Python 3.12, N=5000, median of 5 (ms, lower is better). +SQLite is in-process, so `bench.py` uses its recommended `sync_fast_path=1` +config (statements run synchronously on the calling thread — an embedded +database has no I/O to overlap, so the async bridge is pure overhead): | operation | yara-orm | tortoise | sqlalchemy | pony | django | peewee | sqlobject | ormar | piccolo | |---------------|---------:|---------:|-----------:|-----:|-------:|-------:|----------:|------:|--------:| -| bulk_insert | 7.9 | 14.4 | 612.7 | 51.0 | 58.1 | 30.7 | 223.1 | 158.0 | 78.8 | -| single_insert | 32.6 | 29.3 | 240.0 | 128.3 | 139.0 | 114.7 | 139.9 | 323.2 | 259.1 | -| fetch_all | 3.4 | 39.7 | 28.8 | 51.0 | 16.3 | 12.5 | 44.9 | 54.8 | 9.1 | -| count | 0.1 | 0.3 | 0.7 | 0.2 | 0.2 | 0.1 | 0.1 | 1.7 | 0.5 | -| group_by | 0.5 | 0.8 | 1.4 | 1.5 | 0.9 | 0.7 | 0.5 | - | 1.0 | -| filter | 2.0 | 20.5 | 7.7 | 26.2 | 8.5 | 6.7 | 17.3 | 19.6 | 5.1 | -| get_by_pk | 47.4 | 87.5 | 330.9 | 30.7 | 83.6 | 77.7 | 13.3 | 501.8 | 359.5 | -| update | 0.6 | 0.5 | 1.8 | 43.1 | 1.3 | 1.2 | 1.2 | 1.6 | 1.4 | -| delete | 0.4 | 0.4 | 1.2 | 36.3 | 0.9 | 0.7 | 0.8 | 1.3 | 1.2 | - -`yara-orm` wins the throughput-bound operations decisively (`bulk_insert` -1.8–77×, `fetch_all` 2.6–15×, `filter` 2.5–13× across the field). It trails only -on **latency-bound point reads**: in-process sync ORMs — SQLObject (`get_by_pk` -0.3×) and Pony (0.7×) — beat us there (plus the microsecond `group_by`, where -SQLObject's raw-SQL path is a hair ahead at 0.9×). The cost is the per-statement asyncio -bridge (scheduling on the runtime + waking the event loop), tens of µs a -synchronous in-process driver avoids on sequential point queries. Real workloads -rarely fire thousands of sequential point reads, and everything throughput-shaped -is far ahead. The opt-in -`sqlite://...?sync_fast_path=1` URL flag removes that bridge (point queries ~7× -faster) when those ops dominate. +| bulk_insert | 7.5 | 13.6 | 607.9 | 50.1 | 55.8 | 29.0 | 218.3 | 143.8 | 73.9 | +| single_insert | 15.3 | 26.9 | 234.5 | 107.0 | 120.1 | 113.5 | 124.8 | 296.7 | 240.4 | +| fetch_all | 3.4 | 38.6 | 27.1 | 51.0 | 16.4 | 12.2 | 44.2 | 52.0 | 9.2 | +| count | 0.0 | 0.2 | 0.7 | 0.2 | 0.3 | 0.1 | 0.1 | 1.6 | 0.5 | +| group_by | 0.6 | 0.7 | 1.3 | 1.4 | 0.9 | 0.6 | 0.5 | - | 1.0 | +| filter | 2.0 | 20.2 | 7.3 | 25.8 | 8.7 | 6.6 | 17.4 | 19.2 | 5.0 | +| get_by_pk | 12.5 | 79.4 | 329.6 | 31.3 | 84.6 | 75.5 | 13.3 | 484.1 | 357.2 | +| update | 0.5 | 0.5 | 1.7 | 43.0 | 1.3 | 1.2 | 1.1 | 1.7 | 1.5 | +| delete | 0.3 | 0.4 | 1.1 | 35.9 | 0.8 | 0.7 | 0.7 | 1.1 | 1.1 | + +`yara-orm` is fastest on **every** operation except the sub-millisecond +`group_by`, where SQLObject's hand-written raw-SQL aggregate (bypassing its ORM +entirely) edges it at 0.8×. With the fast path it wins the point reads it +trailed on under the default async bridge — `get_by_pk` **1.1× vs SQLObject** +and **2.5× vs Pony** — and stays far ahead on throughput (`bulk_insert` 1.8–81×, +`fetch_all` 2.7–15×, `filter` 2.5–13×, `single_insert` 1.8× vs Tortoise). On the +**default** async path (no fast path), the per-statement asyncio bridge costs +tens of µs on sequential point reads, so SQLObject's lean sync active-record +leads `get_by_pk` there instead; `sqlite://...?sync_fast_path=1` removes that +bridge (~7× faster point queries). ## Why `yara-orm` is fast here diff --git a/benchmarks/bench.py b/benchmarks/bench.py index 2f455dd..e0a6af7 100644 --- a/benchmarks/bench.py +++ b/benchmarks/bench.py @@ -112,7 +112,12 @@ def drop_sql(table: str) -> str: def ours_url() -> str: if BACKEND == "sqlite": - return f"sqlite://{SQLITE_DIR}/bench_ours.db" + # SQLite is in-process (no I/O to overlap), so the recommended config + # drives statements synchronously on the calling thread instead of the + # async bridge — the right choice for an embedded DB and how yara-orm is + # deployed on SQLite. The competitors' aiosqlite pays the async cost for + # no benefit here. + return f"sqlite://{SQLITE_DIR}/bench_ours.db?sync_fast_path=1" if BACKEND in ("mysql", "mariadb"): return mysql_family_url() if BACKEND == "oracle": diff --git a/benchmarks/plot_benchmarks.py b/benchmarks/plot_benchmarks.py index 779ebde..1b8cd9c 100644 --- a/benchmarks/plot_benchmarks.py +++ b/benchmarks/plot_benchmarks.py @@ -75,28 +75,28 @@ "mariadb": ( "Yara ORM vs 7 Python ORMs — MariaDB 11, 5000 rows, median of 5", { - "yara-orm": [30.5, 392.8, 5.5, 0.5, 1.3, 3.2, 123.3, 3.8, 3.2], - "tortoise": [41.5, 304.8, 34.0, 0.8, 1.3, 17.8, 228.7, 3.3, 3.2], - "sqlalchemy": [99.0, 430.8, 42.5, 1.2, 2.1, 16.0, 534.1, 6.5, 3.3], - "pony": [455.0, 329.6, 47.9, 0.7, 2.2, 24.5, 310.7, 265.9, 249.9], - "django": [87.7, 343.6, 28.4, 0.7, 1.5, 15.5, 214.5, 4.3, 3.3], - "peewee": [59.5, 367.3, 30.4, 0.8, 1.1, 14.7, 206.0, 4.3, 3.0], - "sqlobject": [1257.2, 359.8, 44.9, 0.7, 0.9, 16.9, 66.0, 4.2, 2.9], - "ormar": [191.2, 555.3, 72.8, 4.7, NA, 31.9, 916.1, 7.6, 3.6], + "yara-orm": [23.3, 264.9, 5.7, 0.5, 1.2, 3.2, 132.9, 4.0, 3.0], + "tortoise": [37.5, 311.6, 35.0, 0.8, 1.3, 17.5, 240.8, 4.4, 3.0], + "sqlalchemy": [105.4, 531.7, 43.8, 1.2, 2.3, 16.3, 575.0, 7.8, 3.6], + "pony": [475.4, 391.5, 48.1, 0.7, 2.2, 24.8, 306.7, 266.3, 249.5], + "django": [96.4, 388.2, 28.1, 0.7, 1.5, 15.1, 220.7, 5.4, 3.6], + "peewee": [71.8, 372.2, 37.0, 0.8, 1.4, 14.7, 206.5, 4.2, 3.0], + "sqlobject": [1266.5, 390.3, 42.6, 0.6, 0.9, 17.1, 64.4, 3.7, 3.0], + "ormar": [209.9, 660.0, 72.1, 6.7, NA, 32.6, 895.2, 8.2, 4.0], }, ), "sqlite": ( - "Yara ORM vs 8 Python ORMs — SQLite, 5000 rows, median of 5", + "Yara ORM vs 8 Python ORMs — SQLite (sync_fast_path), 5000 rows, median of 5", { - "yara-orm": [7.9, 32.6, 3.4, 0.1, 0.5, 2.0, 47.4, 0.6, 0.4], - "tortoise": [14.4, 29.3, 39.7, 0.3, 0.8, 20.5, 87.5, 0.5, 0.4], - "sqlalchemy": [612.7, 240.0, 28.8, 0.7, 1.4, 7.7, 330.9, 1.8, 1.2], - "pony": [51.0, 128.3, 51.0, 0.2, 1.5, 26.2, 30.7, 43.1, 36.3], - "django": [58.1, 139.0, 16.3, 0.2, 0.9, 8.5, 83.6, 1.3, 0.9], - "peewee": [30.7, 114.7, 12.5, 0.1, 0.7, 6.7, 77.7, 1.2, 0.7], - "sqlobject": [223.1, 139.9, 44.9, 0.1, 0.5, 17.3, 13.3, 1.2, 0.8], - "ormar": [158.0, 323.2, 54.8, 1.7, NA, 19.6, 501.8, 1.6, 1.3], - "piccolo": [78.8, 259.1, 9.1, 0.5, 1.0, 5.1, 359.5, 1.4, 1.2], + "yara-orm": [7.5, 15.3, 3.4, 0.04, 0.6, 2.0, 12.5, 0.5, 0.3], + "tortoise": [13.6, 26.9, 38.6, 0.2, 0.7, 20.2, 79.4, 0.5, 0.4], + "sqlalchemy": [607.9, 234.5, 27.1, 0.7, 1.3, 7.3, 329.6, 1.7, 1.1], + "pony": [50.1, 107.0, 51.0, 0.2, 1.4, 25.8, 31.3, 43.0, 35.9], + "django": [55.8, 120.1, 16.4, 0.3, 0.9, 8.7, 84.6, 1.3, 0.8], + "peewee": [29.0, 113.5, 12.2, 0.1, 0.6, 6.6, 75.5, 1.2, 0.7], + "sqlobject": [218.3, 124.8, 44.2, 0.1, 0.5, 17.4, 13.3, 1.1, 0.7], + "ormar": [143.8, 296.7, 52.0, 1.6, NA, 19.2, 484.1, 1.7, 1.1], + "piccolo": [73.9, 240.4, 9.2, 0.5, 1.0, 5.0, 357.2, 1.5, 1.1], }, ), } diff --git a/docs/assets/benchmark-mariadb.png b/docs/assets/benchmark-mariadb.png index f538775..0ee5f5f 100644 Binary files a/docs/assets/benchmark-mariadb.png and b/docs/assets/benchmark-mariadb.png differ diff --git a/docs/assets/benchmark-sqlite.png b/docs/assets/benchmark-sqlite.png index bc6d107..59cbed7 100644 Binary files a/docs/assets/benchmark-sqlite.png and b/docs/assets/benchmark-sqlite.png differ diff --git a/docs/performance.md b/docs/performance.md index c357640..5d5ff6d 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -97,51 +97,58 @@ backend, so it is absent here: | operation | yara-orm | tortoise | sqlalchemy | pony | django | peewee | sqlobject | ormar | |---------------|---------:|---------:|-----------:|-----:|-------:|-------:|----------:|------:| -| bulk_insert | 30.5 | 41.5 | 99.0 | 455.0 | 87.7 | 59.5 | 1257.2 | 191.2 | -| single_insert | 392.8 | 304.8 | 430.8 | 329.6 | 343.6 | 367.3 | 359.8 | 555.3 | -| fetch_all | 5.5 | 34.0 | 42.5 | 47.9 | 28.4 | 30.4 | 44.9 | 72.8 | -| count | 0.5 | 0.8 | 1.2 | 0.7 | 0.7 | 0.8 | 0.7 | 4.7 | -| group_by | 1.3 | 1.3 | 2.1 | 2.2 | 1.5 | 1.1 | 0.9 | - | -| filter | 3.2 | 17.8 | 16.0 | 24.5 | 15.5 | 14.7 | 16.9 | 31.9 | -| get_by_pk | 123.3 | 228.7 | 534.1 | 310.7 | 214.5 | 206.0 | 66.0 | 916.1 | -| update | 3.8 | 3.3 | 6.5 | 265.9 | 4.3 | 4.3 | 4.2 | 7.6 | -| delete | 3.2 | 3.2 | 3.3 | 249.9 | 3.3 | 3.0 | 2.9 | 3.6 | +| bulk_insert | 23.3 | 37.5 | 105.4 | 475.4 | 96.4 | 71.8 | 1266.5 | 209.9 | +| single_insert | 264.9 | 311.6 | 531.7 | 391.5 | 388.2 | 372.2 | 390.3 | 660.0 | +| fetch_all | 5.7 | 35.0 | 43.8 | 48.1 | 28.1 | 37.0 | 42.6 | 72.1 | +| count | 0.5 | 0.8 | 1.2 | 0.7 | 0.7 | 0.8 | 0.6 | 6.7 | +| group_by | 1.2 | 1.3 | 2.3 | 2.2 | 1.5 | 1.4 | 0.9 | - | +| filter | 3.2 | 17.5 | 16.3 | 24.8 | 15.1 | 14.7 | 17.1 | 32.6 | +| get_by_pk | 132.9 | 240.8 | 575.0 | 306.7 | 220.7 | 206.5 | 64.4 | 895.2 | +| update | 4.0 | 4.4 | 7.8 | 266.3 | 5.4 | 4.2 | 3.7 | 8.2 | +| delete | 3.0 | 3.0 | 3.6 | 249.5 | 3.6 | 3.0 | 3.0 | 4.0 | Yara ORM leads or ties every operation except SQLObject's leaner `get_by_pk` (0.5×) and the sub-millisecond `group_by` (SQLObject 0.8×). It wins the throughput -ops decisively (`fetch_all` 5.0–12.9×, `filter` 4.4–9.3×, `bulk_insert` up to 46× -vs SQLObject). MariaDB's `single_insert` (~390 ms) is notably faster than -MySQL 8's here — a lighter default commit path. +ops decisively (`fetch_all` 4.9–12.6×, `filter` 4.6–10.2×, `bulk_insert` up to +54× vs SQLObject). `single_insert`, `update` and `delete` are near ties across every +ORM because they are **database-bound**, not client-bound: single inserts are +paced by MariaDB's per-commit disk fsync (high run-to-run variance), and +`update`/`delete` are one server-side set statement each — the client just sends +SQL and reads a rowcount, so there is no marshaling for the Rust hot path to +accelerate. MariaDB's `single_insert` (~265 ms) is notably faster than MySQL 8's +here — a lighter default commit path. ## SQLite ![Yara ORM vs eight Python ORMs on SQLite — latency per operation, log scale, lower is better](assets/benchmark-sqlite.png) -Python 3.12, N=5000, median of 5 (ms, lower is better). +Python 3.12, N=5000, median of 5 (ms, lower is better). SQLite is in-process, so +these use its recommended [`sync_fast_path=1`](#opt-in-sqlite-sync-fast-path) +config — statements run synchronously on the calling thread, since an embedded +database has no I/O to overlap and the async bridge would be pure overhead. | operation | yara-orm | tortoise | sqlalchemy | pony | django | peewee | sqlobject | ormar | piccolo | |---------------|---------:|---------:|-----------:|-----:|-------:|-------:|----------:|------:|--------:| -| bulk_insert | 7.9 | 14.4 | 612.7 | 51.0 | 58.1 | 30.7 | 223.1 | 158.0 | 78.8 | -| single_insert | 32.6 | 29.3 | 240.0 | 128.3 | 139.0 | 114.7 | 139.9 | 323.2 | 259.1 | -| fetch_all | 3.4 | 39.7 | 28.8 | 51.0 | 16.3 | 12.5 | 44.9 | 54.8 | 9.1 | -| count | 0.1 | 0.3 | 0.7 | 0.2 | 0.2 | 0.1 | 0.1 | 1.7 | 0.5 | -| group_by | 0.5 | 0.8 | 1.4 | 1.5 | 0.9 | 0.7 | 0.5 | - | 1.0 | -| filter | 2.0 | 20.5 | 7.7 | 26.2 | 8.5 | 6.7 | 17.3 | 19.6 | 5.1 | -| get_by_pk | 47.4 | 87.5 | 330.9 | 30.7 | 83.6 | 77.7 | 13.3 | 501.8 | 359.5 | -| update | 0.6 | 0.5 | 1.8 | 43.1 | 1.3 | 1.2 | 1.2 | 1.6 | 1.4 | -| delete | 0.4 | 0.4 | 1.2 | 36.3 | 0.9 | 0.7 | 0.8 | 1.3 | 1.2 | - -Yara ORM wins the throughput-bound operations decisively (`bulk_insert` 1.8–77×, `fetch_all` -2.6–15×, `filter` 2.5–13× across the field). It trails only on **latency-bound point reads**: -in-process sync ORMs — SQLObject (`get_by_pk` 0.3×) and Pony (0.7×) — beat us there (plus the -microsecond `group_by`, where SQLObject's raw-SQL path is a hair ahead at 0.9×). The cost -is the per-statement asyncio bridge (scheduling the statement on the runtime and waking the -event loop), tens of microseconds that a synchronous in-process driver avoids on sequential -point queries. Real workloads rarely fire -thousands of sequential point reads, and everything throughput-shaped is far ahead. If those -point operations dominate your workload, the opt-in -[sync fast path](#opt-in-sqlite-sync-fast-path) below removes exactly that per-statement -bridge. +| bulk_insert | 7.5 | 13.6 | 607.9 | 50.1 | 55.8 | 29.0 | 218.3 | 143.8 | 73.9 | +| single_insert | 15.3 | 26.9 | 234.5 | 107.0 | 120.1 | 113.5 | 124.8 | 296.7 | 240.4 | +| fetch_all | 3.4 | 38.6 | 27.1 | 51.0 | 16.4 | 12.2 | 44.2 | 52.0 | 9.2 | +| count | 0.0 | 0.2 | 0.7 | 0.2 | 0.3 | 0.1 | 0.1 | 1.6 | 0.5 | +| group_by | 0.6 | 0.7 | 1.3 | 1.4 | 0.9 | 0.6 | 0.5 | - | 1.0 | +| filter | 2.0 | 20.2 | 7.3 | 25.8 | 8.7 | 6.6 | 17.4 | 19.2 | 5.0 | +| get_by_pk | 12.5 | 79.4 | 329.6 | 31.3 | 84.6 | 75.5 | 13.3 | 484.1 | 357.2 | +| update | 0.5 | 0.5 | 1.7 | 43.0 | 1.3 | 1.2 | 1.1 | 1.7 | 1.5 | +| delete | 0.3 | 0.4 | 1.1 | 35.9 | 0.8 | 0.7 | 0.7 | 1.1 | 1.1 | + +Yara ORM is fastest on **every** operation except the sub-millisecond `group_by`, where +SQLObject's hand-written raw-SQL aggregate (bypassing its ORM entirely) edges it at 0.8×. +It now wins the point reads it trailed on with the default async bridge — `get_by_pk` +**1.1× vs SQLObject** and **2.5× vs Pony** — because the fast path removes the per-statement +asyncio hop that a synchronous in-process driver never pays. Throughput stays far ahead +(`bulk_insert` 1.8–81×, `fetch_all` 2.7–15×, `filter` 2.5–13×, `single_insert` 1.8× vs +Tortoise). On the **default** async path (no fast path), the per-statement bridge costs tens +of microseconds on sequential point reads, so SQLObject's lean sync active-record leads +`get_by_pk` there instead — the delta is quantified in +[the sync fast path section](#opt-in-sqlite-sync-fast-path) below. ### Opt-in SQLite sync fast path diff --git a/python/yara_orm/connection.py b/python/yara_orm/connection.py index 4904ad4..da66c7f 100644 --- a/python/yara_orm/connection.py +++ b/python/yara_orm/connection.py @@ -280,7 +280,7 @@ async def _run_query( raise OperationalError(str(exc)) from exc -def _split_sql_statements(script: str) -> list[str]: +def _split_sql_statements(script: str, nest_block_comments: bool = False) -> list[str]: """Split a multi-statement SQL script into individual statements on ``;``. The native engine runs one command per call (prepared statement), so scripts @@ -292,6 +292,10 @@ def _split_sql_statements(script: str) -> list[str]: Args: script: The SQL script possibly containing several statements. + nest_block_comments: Whether ``/* */`` block comments nest. Only + PostgreSQL does; for every other dialect an inner ``/*`` is literal + and the comment ends at the first ``*/``. Passing ``True`` where the + engine does not nest would swallow a real ``;`` and merge statements. Returns: The non-empty statements, in order. @@ -303,7 +307,9 @@ def _split_sql_statements(script: str) -> list[str]: in_squote = in_dquote = in_line_comment = False # PostgreSQL block comments nest, so track depth rather than a boolean: the # comment ends only when the outermost ``*/`` closes. Without this, an inner - # ``*/`` is mistaken for the end and a following ``;`` wrongly splits. + # ``*/`` is mistaken for the end and a following ``;`` wrongly splits. On + # dialects that do not nest (``nest_block_comments=False``) an inner ``/*`` + # is literal and the first ``*/`` closes, so depth never exceeds 1. block_depth = 0 while i < n: ch = script[i] @@ -314,7 +320,7 @@ def _split_sql_statements(script: str) -> list[str]: in_line_comment = False i += 1 elif block_depth > 0: - if two == "/*": + if nest_block_comments and two == "/*": block_depth += 1 buf.append(two) i += 2 @@ -550,7 +556,8 @@ async def execute_script(self, script: str) -> None: Returns: None """ - for statement in _split_sql_statements(script): + nest = getattr(getattr(self, "dialect", None), "nests_block_comments", False) + for statement in _split_sql_statements(script, nest): await self.execute(statement) @@ -1500,7 +1507,8 @@ async def execute_script(self, script: str) -> None: Returns: None """ - statements = _split_sql_statements(script) + nest = resolve_dialect(self._engine.dialect).nests_block_comments + statements = _split_sql_statements(script, nest) if not statements: return # This path hands the split statements straight to the engine (no diff --git a/python/yara_orm/dialects.py b/python/yara_orm/dialects.py index bb05a0b..c44619b 100644 --- a/python/yara_orm/dialects.py +++ b/python/yara_orm/dialects.py @@ -98,6 +98,10 @@ class BaseDialect: regex_ops: dict[str, str] = {} #: Whether the backend supports the ``__search`` full-text lookup. supports_search = False + #: Whether ``/* */`` block comments nest (an inner ``/*`` opens another + #: level). Only PostgreSQL nests; every other engine ends the comment at the + #: first ``*/``. Used when splitting multi-statement scripts. + nests_block_comments = False #: Statement prefix that returns a query plan for ``QuerySet.explain``. explain_prefix = "EXPLAIN " #: SQL expression used for random ordering (``order_by("?")``). PostgreSQL @@ -1732,6 +1736,8 @@ class PostgresDialect(BaseDialect): # and case-insensitive); `__search` uses full-text via to_tsvector/tsquery. regex_ops = {"regex": "~", "iregex": "~*", "posix_regex": "~", "iposix_regex": "~*"} supports_search = True + # PostgreSQL is the only supported engine whose block comments nest. + nests_block_comments = True supports_extensions = True supports_for_update = True diff --git a/python/yara_orm/models.py b/python/yara_orm/models.py index b0364ae..0e4cd56 100644 --- a/python/yara_orm/models.py +++ b/python/yara_orm/models.py @@ -43,6 +43,36 @@ #: ``fetch_for_list``), so the caller's model type is preserved. _ModelT = TypeVar("_ModelT", bound="Model") +#: Every attribute :meth:`MetaInfo.compile` (and the ``_build_decode_plan`` it +#: calls) produces for one dialect. A compiled plan is stable per (model, +#: dialect), so these are snapshotted per dialect name and restored wholesale +#: when a model alternates dialects, instead of recompiling from scratch. This +#: list MUST stay complete: a missing attribute would keep another dialect's +#: value after a restore. ``_partial_update_cache`` is intentionally absent — it +#: is already keyed by dialect name and shared across all of them. +_COMPILED_PLAN_ATTRS = ( + "_read_decoders", + "decoders", + "decoder_names", + "active_decoders", + "auto_now_fields", + "validated_fields", + "coerced_fields", + "db_default_fields", + "_simple_lookup_cache", + "_partial_plan_cache", + "columns_sql", + "select_prefix", + "insert_fields", + "insert_returning_fields", + "insert_refresh_fields", + "insert_refresh_sql", + "insert_sql", + "update_field_list", + "update_sql", + "delete_sql", +) + class MetaInfo: """Resolved metadata for a model: table name, fields and primary key. @@ -141,6 +171,10 @@ def __init__( # keyed by the ordered selected field names, so a partial SELECT reuses # the same ``(names, active_decoders)`` split as the full-row fast path. self._partial_plan_cache: dict[tuple[str, ...], tuple[list[str], list[tuple]]] = {} + # Per-dialect compiled plans (see ``_COMPILED_PLAN_ATTRS``), keyed by + # dialect name. Lets a model that alternates dialects restore a prior + # dialect's plan instead of recompiling it every switch. + self._compiled_plans: dict[str, dict[str, Any]] = {} def partial_decode_plan(self, fields: list[Field]) -> tuple[list[str], list[tuple]]: """Return the cached ``(names, active_decoders)`` plan for a subset. @@ -303,6 +337,23 @@ def compile(self, dialect: BaseDialect) -> None: """ if self._compiled_for == dialect.name: return + if self._compiled_for is None: + # ``_compiled_for = None`` is the "field set changed, recompile" + # signal (construction, migrations). That invalidates *every* cached + # plan, not just the current dialect's, so drop them all and rebuild. + self._compiled_plans.clear() + else: + # A genuine dialect switch: the field set is unchanged, so a plan + # built for ``dialect`` earlier is still exact. Restore it verbatim + # rather than recompiling — no re-quoting columns or regenerating + # placeholders. The restored cache dicts are the same objects captured + # at build time, so entries accumulated since (partial plans, + # simple-lookup SQL) survive. + cached = self._compiled_plans.get(dialect.name) + if cached is not None: + self.__dict__.update(cached) + self._compiled_for = dialect.name + return # The field set may have changed since construction (migrations); refresh # the hydration plan so it stays in sync with the current columns. The # converters come from the dialect (``read_decoder``) so a backend can @@ -312,7 +363,9 @@ def compile(self, dialect: BaseDialect) -> None: self.decoders = [ (f.model_field_name, self._read_decoders[f.model_field_name]) for f in self.field_list ] - self._partial_plan_cache.clear() + # Fresh dict (not ``.clear()``): any prior dialect's plan snapshot holds a + # reference to its own cache, which clearing in place would corrupt. + self._partial_plan_cache = {} self._build_decode_plan() q = dialect.quote self.columns_sql = ", ".join(q(f.db_column) for f in self.field_list) @@ -382,6 +435,10 @@ def compile(self, dialect: BaseDialect) -> None: self.update_sql = None self.delete_sql = f"DELETE FROM {q(self.table)} WHERE {pk_col} = {dialect.placeholder(1)}" self._compiled_for = dialect.name + # Snapshot this dialect's plan so a later switch back restores it in O(1) + # instead of rebuilding. The dicts are captured by reference, so caches + # populated later (partial plans, simple-lookup SQL) stay visible here. + self._compiled_plans[dialect.name] = {k: getattr(self, k) for k in _COMPILED_PLAN_ATTRS} def partial_update_sql(self, dialect: BaseDialect, fields: list[Field]) -> str: """Return a cached ``UPDATE`` statement writing only ``fields`` by pk. @@ -790,7 +847,7 @@ def __new__( for rel_name, mm in {**inherited_m2m, **m2m_decls}.items(): info = M2MInfo(rel_name, mm, cls, mm.reference) m2m[rel_name] = info - setattr(cls, rel_name, M2MDescriptor(info, pk_field.model_field_name)) + setattr(cls, rel_name, M2MDescriptor(info, pk_field.model_field_name, name=rel_name)) # Abstract bases contribute their fields to subclasses but have no table # of their own, so they stay out of the registry (schema generation, diff --git a/python/yara_orm/prefetch.py b/python/yara_orm/prefetch.py index ed316f7..dab4794 100644 --- a/python/yara_orm/prefetch.py +++ b/python/yara_orm/prefetch.py @@ -6,6 +6,7 @@ from __future__ import annotations +import copy from collections.abc import Sequence from typing import TYPE_CHECKING, Any @@ -257,14 +258,30 @@ async def _prefetch_m2m( ) links = await engine.fetch_rows(link_sql, pks) owners_by_far: dict = {} - for near, far in ((row[0], row[1]) for row in links): + for near, far in links: owners_by_far.setdefault(far, []).append(near) - far_pks = list(owners_by_far) - objs = await custom_qs.filter(pk__in=far_pks) if far_pks else [] - grouped = {i.pk: [] for i in instances} - for obj in objs: - for owner_id in owners_by_far.get(obj.pk, ()): - grouped[owner_id].append(obj) + grouped: dict = {i.pk: [] for i in instances} + if custom_qs._limit is not None or custom_qs._offset is not None: + # A LIMIT/OFFSET/slice on the Prefetch queryset must apply *per + # owner* (e.g. "the 3 newest tags per book"), so run the queryset + # once per owner. A single `pk__in` over every owner's targets would + # apply the slice globally and starve owners outside the first window. + far_by_owner: dict = {} + for far, owner_ids in owners_by_far.items(): + for owner_id in owner_ids: + far_by_owner.setdefault(owner_id, []).append(far) + for owner_id, far_pks in far_by_owner.items(): + grouped[owner_id] = list(await custom_qs.filter(pk__in=far_pks)) + else: + # No slice: one batched query, then distribute the ordered rows into + # per-owner groups. A target shared by several owners gets a distinct + # instance per owner (like the join path's per-row `_from_db_row`), so + # mutating one owner's prefetched object cannot leak into another's. + far_pks = list(owners_by_far) + objs = await custom_qs.filter(pk__in=far_pks) if far_pks else [] + for obj in objs: + for n, owner_id in enumerate(owners_by_far.get(obj.pk, ())): + grouped[owner_id].append(obj if n == 0 else copy.copy(obj)) _assign(instances, name, to_attr, {i: grouped.get(i.pk, []) for i in instances}) return diff --git a/python/yara_orm/queryset.py b/python/yara_orm/queryset.py index 9437ec9..6488b12 100644 --- a/python/yara_orm/queryset.py +++ b/python/yara_orm/queryset.py @@ -789,7 +789,12 @@ def _compile_lookup( # back so the path compiles as a relation lookup. Genuine transforms on a # scalar column (`created__year`) and relation-key lookups (`tags__in`, # where `Tag` has no `in` member) are unaffected. - if op != "exact" and "__" not in base: + # + # `base in meta.fields` short-circuits the common case: relation names + # never land in `meta.fields` (only their source column does), so a base + # that is a plain scalar column cannot be the relation this fix targets — + # skip the `_relation_target` probe on that hot path. + if op != "exact" and "__" not in base and base not in meta.fields: target = self._relation_target(base) if target is not None: tmeta = target._meta diff --git a/python/yara_orm/registry.py b/python/yara_orm/registry.py index 2c2fb61..f56109e 100644 --- a/python/yara_orm/registry.py +++ b/python/yara_orm/registry.py @@ -226,7 +226,10 @@ def resolve_relations() -> None: target, related_name, M2MDescriptor( - info, target._meta.pk_field.model_field_name, reverse=True + info, + target._meta.pk_field.model_field_name, + reverse=True, + name=related_name, ), ) continue # already installed by a previous resolve pass @@ -240,6 +243,8 @@ def resolve_relations() -> None: setattr( target, related_name, - M2MDescriptor(info, target._meta.pk_field.model_field_name, reverse=True), + M2MDescriptor( + info, target._meta.pk_field.model_field_name, reverse=True, name=related_name + ), ) _RESOLVED["done"] = True diff --git a/python/yara_orm/relations.py b/python/yara_orm/relations.py index bcb9601..38bb2cc 100644 --- a/python/yara_orm/relations.py +++ b/python/yara_orm/relations.py @@ -590,13 +590,19 @@ async def _ensure(awaitable: Awaitable[list[MODEL]]) -> list[MODEL]: class M2MDescriptor: """Descriptor exposing a many-to-many relation as a manager.""" - def __init__(self, info: M2MInfo, owner_pk_attr: str, reverse: bool = False) -> None: + def __init__( + self, info: M2MInfo, owner_pk_attr: str, reverse: bool = False, name: str | None = None + ) -> None: """Store the metadata backing this many-to-many descriptor. Args: info: The many-to-many relation metadata. owner_pk_attr: The owning model's primary key attribute name. reverse: Whether this descriptor is the reverse side of the relation. + name: The attribute this descriptor is installed under. On the + reverse side this is the ``related_name`` (not ``info.name``), + and it is the key ``prefetch_related`` caches results under, so + the manager must read the same key. Defaults to ``info.name``. Returns: None @@ -604,6 +610,7 @@ def __init__(self, info: M2MInfo, owner_pk_attr: str, reverse: bool = False) -> self.info = info self.owner_pk_attr = owner_pk_attr self.reverse = reverse + self.name = name or info.name def __get__( self, instance: Model | None, owner: type[Model] | None @@ -620,7 +627,7 @@ def __get__( """ if instance is None: return self - return M2MManager(self.info, instance, self.reverse) + return M2MManager(self.info, instance, self.reverse, self.name) class _M2MMembershipSubquery: @@ -682,13 +689,19 @@ def as_sql( class M2MManager(_ChainableManager[MODEL]): """Many-to-many manager: awaitable to a list, async-iterable, mutable.""" - def __init__(self, info: M2MInfo, instance: Model, reverse: bool) -> None: + def __init__( + self, info: M2MInfo, instance: Model, reverse: bool, name: str | None = None + ) -> None: """Bind the manager to an instance and resolve the join key roles. Args: info: The many-to-many relation metadata. instance: The model instance owning the relation. reverse: Whether this manager is the reverse side of the relation. + name: The accessor attribute name (the reverse side's ``related_name`` + differs from ``info.name``); it is the ``_prefetch`` cache key a + prefetch stores under, so the manager reads the same key. Defaults + to ``info.name``. Returns: None @@ -708,7 +721,10 @@ def __init__(self, info: M2MInfo, instance: Model, reverse: bool) -> None: # The target class is registry-resolved; ``ManyToManyRelation[X]`` on # the declaring model is the static source of truth for ``MODEL``. self.target = cast("type[MODEL]", target) - self.name = info.name + # The prefetch cache key is the accessor name (reverse side: the + # ``related_name``), which differs from ``info.name`` on the reverse side; + # reading ``info.name`` there would miss the cache and re-query (N+1). + self.name = name or info.name def _sql(self, dialect: BaseDialect) -> dict[str, str]: """Return cached static SQL pieces for this side of the relation. diff --git a/rust/src/value.rs b/rust/src/value.rs index 8a9cbd7..719f998 100644 --- a/rust/src/value.rs +++ b/rust/src/value.rs @@ -213,13 +213,14 @@ fn py_to_json(ob: &Bound<'_, PyAny>) -> PyResult { } if ob.is_instance_of::() { // JSON has no representation for NaN/±Infinity; `serde_json::Value::from` - // would silently turn them into `null` (data corruption). Reject them - // explicitly, mirroring the Decimal('NaN')/Decimal('Infinity') branch. + // would silently turn them into `null` (data corruption). Preserve the + // value as its textual form ("inf"/"-inf"/"NaN") instead — this keeps + // the insert succeeding (raising would break rows that previously wrote) + // and matches `value_to_json`, so both JSON bind paths encode a + // non-finite float identically. let f = ob.extract::()?; if !f.is_finite() { - return Err(pyo3::exceptions::PyValueError::new_err( - "non-finite float (NaN/Infinity) has no JSON representation", - )); + return Ok(serde_json::Value::String(f.to_string())); } return Ok(serde_json::Value::from(f)); } diff --git a/tests/test_connection.py b/tests/test_connection.py index 353c366..16fbf44 100644 --- a/tests/test_connection.py +++ b/tests/test_connection.py @@ -223,12 +223,27 @@ def test_split_sql_statements_handles_nested_block_comments(): is one unit and only the real statement terminator splits """ script = "/* a /* b */ c; */ SELECT 1; SELECT 2;" - stmts = _split_sql_statements(script) + stmts = _split_sql_statements(script, nest_block_comments=True) assert len(stmts) == 2 assert stmts[0].endswith("SELECT 1") and stmts[0].startswith("/* a") assert stmts[1] == "SELECT 2" +def test_split_sql_statements_no_nesting_ends_comment_at_first_close(): + """ + GIVEN a non-PostgreSQL script (block comments do not nest) whose comment + contains a ``/*`` sequence followed by a real statement terminator + WHEN it is split with ``nest_block_comments=False`` (the default) + THEN the comment ends at the first ``*/``, so the following ``;`` splits and + statements are not swallowed and merged + """ + script = "/* note: see /* legacy */ SELECT 1; SELECT 2;" + stmts = _split_sql_statements(script) + assert len(stmts) == 2 + assert stmts[0].endswith("SELECT 1") and stmts[0].startswith("/* note") + assert stmts[1] == "SELECT 2" + + def test_connection_url_from_credentials_dict(): """ GIVEN a structured connection spec (credentials dict) diff --git a/tests/test_dialects.py b/tests/test_dialects.py index 10eea41..f13b246 100644 --- a/tests/test_dialects.py +++ b/tests/test_dialects.py @@ -9,6 +9,7 @@ import pytest +from yara_orm import Model, fields from yara_orm.dialects import ( BaseDialect, MariaDbDialect, @@ -586,3 +587,47 @@ def test_mssql_renames_use_sp_rename(): assert MSSQL.render_rename_constraint("t", "uq", "uq2") == [ "EXEC sp_rename 'uq', 'uq2', 'OBJECT'" ] + + +class _DlCacheModel(Model): + name = fields.CharField(max_length=50) + + class Meta: + table = "dl_cache_model" + + +def test_compile_caches_plan_per_dialect_and_restores_on_switch(): + """ + GIVEN a model compiled for one dialect, then another, then back + WHEN ``MetaInfo.compile`` is called for each + THEN each dialect's plan is cached and a switch back restores it verbatim + (placeholders match the dialect) instead of leaving the other's plan + """ + meta = _DlCacheModel._meta + meta.compile(PG) + pg_insert = meta.insert_sql + assert "$1" in pg_insert # PostgreSQL positional placeholders + + meta.compile(LITE) + assert "?" in meta.insert_sql and meta.insert_sql != pg_insert # SQLite placeholders + + meta.compile(PG) # switch back -> restored from cache + assert meta._compiled_for == "postgres" + assert meta.insert_sql == pg_insert + assert set(meta._compiled_plans) == {"postgres", "sqlite"} + + +def test_compile_reset_invalidates_all_cached_plans(): + """ + GIVEN a model with cached plans for two dialects + WHEN ``_compiled_for`` is reset to ``None`` (the field-set-changed signal) + THEN the next compile drops every cached plan and rebuilds + """ + meta = _DlCacheModel._meta + meta.compile(PG) + meta.compile(LITE) + assert set(meta._compiled_plans) == {"postgres", "sqlite"} + + meta._compiled_for = None # simulate a migration / field-set change + meta.compile(PG) + assert set(meta._compiled_plans) == {"postgres"} diff --git a/tests/test_jd_json.py b/tests/test_jd_json.py index d10ff2c..cf429d6 100644 --- a/tests/test_jd_json.py +++ b/tests/test_jd_json.py @@ -158,6 +158,22 @@ async def test_scalar_types_roundtrip_preserving_python_type(db): assert stored["nested"]["list"][2]["deep"] is True +@pytest.mark.asyncio +async def test_non_finite_float_stored_as_text_not_error(db): + """ + GIVEN a JSON document containing a non-finite float (NaN / Infinity), which + has no JSON representation + WHEN saved and read back + THEN the insert succeeds (it is not a hard error) and the value is preserved + as its textual form rather than silently dropped to null + """ + row = await JdDoc.create(data={"inf": float("inf"), "ninf": float("-inf"), "nan": float("nan")}) + stored = (await JdDoc.get(id=row.id)).data + assert stored["inf"] == "inf" + assert stored["ninf"] == "-inf" + assert stored["nan"] == "NaN" + + @pytest.mark.asyncio async def test_bool_not_confused_with_int(db): """ diff --git a/tests/test_prefetch.py b/tests/test_prefetch.py index e4ffa28..4f29b40 100644 --- a/tests/test_prefetch.py +++ b/tests/test_prefetch.py @@ -118,3 +118,57 @@ async def test_fetch_related_instance(db): fresh = await PfBook.get(id=b.id) await fresh.fetch_related("author") assert fresh.__dict__["_prefetch"]["author"].name == "Lin" + + +@pytest.mark.asyncio +async def test_prefetch_m2m_custom_queryset_limit_is_per_owner(db): + """ + GIVEN two Books each tagged with three distinct Tags + WHEN prefetching tags with Prefetch(queryset=Tag.order_by(...).limit(2)) + THEN the LIMIT applies per owner (each PfBook gets its own top two tags), + not once globally across the whole batch + """ + a = await PfAuthor.create(name="Mel") + b1 = await PfBook.create(title="One", author=a) + b2 = await PfBook.create(title="Two", author=a) + b1_tags = [await PfTag.create(name=f"a{n}") for n in range(3)] + b2_tags = [await PfTag.create(name=f"b{n}") for n in range(3)] + await b1.tags.add(*b1_tags) + await b2.tags.add(*b2_tags) + + books = ( + await PfBook.all() + .prefetch_related(Prefetch("tags", queryset=PfTag.all().order_by("-name").limit(2))) + .order_by("title") + ) + # Per-owner top-2 by descending name: b1 -> a2,a1 ; b2 -> b2,b1. + assert [t.name for t in await books[0].tags] == ["a2", "a1"] + assert [t.name for t in await books[1].tags] == ["b2", "b1"] + + +@pytest.mark.asyncio +async def test_prefetch_m2m_custom_queryset_shared_target_is_distinct_per_owner(db): + """ + GIVEN two Books sharing the same Tag, prefetched via a custom queryset with + no slice (the batched single-query path) + WHEN one owner's prefetched target instance is mutated + THEN the other owner's copy is unaffected (each owner gets its own instance, + matching the join-based prefetch path) + """ + a = await PfAuthor.create(name="Ned") + b1 = await PfBook.create(title="One", author=a) + b2 = await PfBook.create(title="Two", author=a) + shared = await PfTag.create(name="shared") + await b1.tags.add(shared) + await b2.tags.add(shared) + + books = ( + await PfBook.all() + .prefetch_related(Prefetch("tags", queryset=PfTag.all().order_by("name"))) + .order_by("title") + ) + t1 = (await books[0].tags)[0] + t2 = (await books[1].tags)[0] + assert t1 is not t2 + t1.name = "mutated" + assert t2.name == "shared"