Skip to content

Fix transaction wrapping for sqlite-utils 4.0 compatibility - #2871

Open
bunlongheng wants to merge 1 commit into
simonw:mainfrom
bunlongheng:fix/sqlite-utils-4-transaction-compat
Open

Fix transaction wrapping for sqlite-utils 4.0 compatibility#2871
bunlongheng wants to merge 1 commit into
simonw:mainfrom
bunlongheng:fix/sqlite-utils-4-transaction-compat

Conversation

@bunlongheng

Copy link
Copy Markdown

Problem

sqlite-utils 4.0 changed transaction behaviour: every write (insert_all, upsert_all, execute, etc.) now checks conn.in_transaction at call-time. If that flag is False, sqlite-utils opens and immediately commits its own transaction, bypassing Datasette's outer BEGIN IMMEDIATE. A subsequent failure leaves the committed rows behind with no rollback path.

The old Datasette pattern:

with conn:
    conn.execute("BEGIN IMMEDIATE")
    result = fn(conn)

relies on Python's sqlite3.Connection context manager calling conn.commit() on exit. This is fragile: it does not guarantee that conn.in_transaction is True during fn(), and if an inner with conn: inside the write function already committed, the outer with conn: exit becomes a no-op commit on a closed transaction.

Fix

Replace both the non-threaded path (database.py:352) and the write-thread path (database.py:491) with an explicit try/except:

conn.execute("BEGIN IMMEDIATE")
try:
    result = fn(conn)
    conn.commit()
except BaseException:
    conn.rollback()
    raise

Key properties:

  • conn.in_transaction is True from BEGIN IMMEDIATE until commit()/rollback(), so sqlite-utils 4.0 correctly detects the outer transaction and uses SAVEPOINT instead of auto-commit
  • Python's conn.commit() / conn.rollback() are idempotent no-ops when no transaction is active, so write functions that manage their own inner with conn: (e.g. alter_table) continue to work

Testing

  • All 147 test_api_write.py tests pass
  • All 67 test_internals_database.py tests pass

Closes #2831

…mpat

sqlite-utils 4.0 checks conn.in_transaction at call-time; if False it
opens and immediately commits its own transaction, bypassing Datasette's
outer BEGIN IMMEDIATE. The old 'with conn: conn.execute("BEGIN IMMEDIATE")'
pattern was fragile - Python's context manager only calls commit/rollback
on exit, it does not guarantee in_transaction is True during fn().

Replace both the non-threaded path and the write-thread path with an
explicit try/except: execute("BEGIN IMMEDIATE") before fn(), then
conn.commit() on success and conn.rollback() on failure. Python's
conn.commit()/conn.rollback() methods are idempotent no-ops when no
transaction is active, so write functions that manage their own inner
'with conn:' block (e.g. alter_table) continue to work correctly.

Closes simonw#2831
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update transactions code for sqlite-utils 4.0

1 participant