diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f786d9..34bbeab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ Pre-1.0 note: while `pg_durable` is in major version `0`, minor releases may inc ## [0.2.4] - Unreleased +### Added + +- **`df.start_autonomous()` — Oracle-style autonomous transactions:** a new function that starts a durable function with autonomous-transaction semantics. It persists and enqueues the work on a separate PostgreSQL session, so it commits independently and **survives a rollback of the caller's transaction** (the PostgreSQL equivalent of Oracle's `PRAGMA AUTONOMOUS_TRANSACTION`). Use it for audit trails, error logging, and other "must persist even if the caller rolls back" work. Ordinary `df.start()` is unchanged and still participates in the caller's transaction. Takes the same arguments as `df.start()` (`fut`, optional `label`, optional `database`) and runs under the calling role's identity. Added additively as a new C symbol, so the new `.so` stays backward compatible with all previous schemas in this provider line (see `docs/upgrade-testing.md`). + ### Changed - **`df.wait_for_schedule()` cron timing:** the next cron tick is now computed at execution time using duroxide's deterministic clock (`ctx.utc_now()`) inside the `execute_function_graph` orchestration, instead of being pre-computed at `df.start()` time. This makes recurring `@>` schedules and any start-to-execution delay target the correct upcoming tick (#130). diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 4ce1b9b..be4a071 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -152,8 +152,37 @@ SELECT 'SELECT 1' ~> 'SELECT 2'; SELECT df.start('SELECT 1' ~> 'SELECT 2'); ``` +### Transaction Semantics & Autonomous Transactions + +`df.start()` participates in the **caller's transaction**: the instance and its +graph are written via the caller's session, so if the caller's transaction +rolls back, the durable function is rolled back with it and never runs. + +`df.start_autonomous()` provides the opposite guarantee — the PostgreSQL +equivalent of Oracle's `PRAGMA AUTONOMOUS_TRANSACTION`. It persists and enqueues +the durable function on a **separate session**, so it commits independently and +**survives a rollback of the caller's transaction**. Use it for audit trails, +error logging, and other "must persist even if the caller rolls back" work. + +```sql +BEGIN; + INSERT INTO employees (id, name) VALUES (999, 'Test User'); + + -- Persists even though the surrounding transaction rolls back below. + SELECT df.start_autonomous( + 'INSERT INTO audit_log (message) VALUES (''Inserted employee 999'')', + 'audit' + ); + + ROLLBACK; -- employees insert is undone; the audit_log entry remains +``` + +`df.start_autonomous()` takes the same arguments as `df.start()` and runs under +the calling role's identity and privileges. + --- + ## DSL Reference ### Auto-Wrap SQL @@ -182,6 +211,7 @@ df.sql('SELECT 1') ~> df.sql('SELECT 2') | `df.break()` | Exit enclosing loop | `df.break()` | | `df.break(value)` | Exit loop with **literal** return value (not auto-wrapped as SQL) | `df.break('{"done": true}')` | | `df.start(func, label, database)` | Start function (optionally in another database) | `df.start('SELECT 1', 'job')` | +| `df.start_autonomous(func, label, database)` | Start function with autonomous-transaction semantics (survives caller rollback) | `df.start_autonomous('INSERT INTO audit ...', 'audit')` | | `df.cancel(id, reason)` | Cancel function | `df.cancel('a1b2c3d4', 'Done')` | | `df.status(id)` | Get status by instance_id (not label) | `df.status('a1b2c3d4')` | | `df.result(id)` | Get result by instance_id (not label) | `df.result('a1b2c3d4')` | diff --git a/docs/upgrade-testing.md b/docs/upgrade-testing.md index e690034..10c68d3 100644 --- a/docs/upgrade-testing.md +++ b/docs/upgrade-testing.md @@ -205,6 +205,13 @@ what the upgrade script handles, and any backward compatibility considerations. ### v0.2.3 → v0.2.4 +#### Add `df.start_autonomous()` — Oracle-style autonomous transactions +- **DDL change (df schema):** Adds a new SQL-callable function `df.start_autonomous(text, text, text)` (bound to the new C symbol `start_autonomous_wrapper`). It persists and enqueues a durable function on a *separate* PostgreSQL session so it commits independently and survives a rollback of the caller's transaction — the PostgreSQL equivalent of Oracle's `PRAGMA AUTONOMOUS_TRANSACTION`. Ordinary `df.start(text, text, text)` is unchanged and still participates in the caller's transaction. +- **Upgrade script:** `sql/pg_durable--0.2.3--0.2.4.sql` runs a single `CREATE FUNCTION df.start_autonomous(...)`, copied verbatim from the pgrx-generated fresh-install DDL (same argument list, defaults, `RETURNS TEXT`, `LANGUAGE c`, and wrapper symbol). New `df.*` functions retain PostgreSQL's default PUBLIC `EXECUTE`, gated by `USAGE ON SCHEMA df`, so no explicit `GRANT` is needed. +- **Scenario A considerations:** The upgrade path creates exactly the same `df.start_autonomous` overload as a fresh install (identical signature and wrapper binding), so the Scenario A snapshot matches. +- **Scenario B1 considerations:** Pure additive change — a brand-new function on a new symbol; every pre-existing function and its C symbol is untouched. The new `.so` therefore runs correctly against all previous schemas in this provider line (0.2.2, 0.2.3); an un-upgraded schema simply does not expose `df.start_autonomous` yet, and `df.start()` keeps binding to the unchanged `start_wrapper`. `df.start_autonomous()` reads/writes only columns (`df.instances`, `df.nodes`) that exist in every shipped schema on this line. +- **Scenario B2 considerations:** No data migration; instances created before the upgrade are unaffected. + #### Simplify `df.grant_usage()` — drop the explicit function allowlist - **DDL change (df schema):** `df.grant_usage()` no longer loops over a hard-coded `func_sigs` array issuing `GRANT EXECUTE` per function. Fresh installs (`src/lib.rs`) and the upgrade script (`sql/pg_durable--0.2.3--0.2.4.sql`) both `CREATE OR REPLACE` the function with a body that grants `USAGE ON SCHEMA df` plus the table privileges, and conditionally grants `df.http()` / the admin helpers. The signature `df.grant_usage(text, boolean, boolean)` is unchanged. - **DDL change (df schema):** `df.revoke_usage()` is made symmetric with the new `grant_usage()`. It no longer loops over every `df.*` function in `pg_proc` issuing `REVOKE EXECUTE` (which, post-simplification, only produced "no privileges could be revoked" warnings since ordinary functions are never granted per-function EXECUTE). The new body revokes only what `grant_usage()` grants: schema `USAGE`, EXECUTE on the sensitive functions (`df.http`, `df.grant_usage`, `df.revoke_usage`), and the table privileges. The signature `df.revoke_usage(text)` is unchanged. diff --git a/sql/pg_durable--0.2.3--0.2.4.sql b/sql/pg_durable--0.2.3--0.2.4.sql index 8ae8436..7eda54e 100644 --- a/sql/pg_durable--0.2.3--0.2.4.sql +++ b/sql/pg_durable--0.2.3--0.2.4.sql @@ -391,3 +391,32 @@ CREATE FUNCTION df."list_instances"( LANGUAGE c /* Rust */ AS 'MODULE_PATHNAME', 'list_instances_paged_wrapper'; + +-- ============================================================================ +-- Add df.start_autonomous(): Oracle-style autonomous transaction support. +-- +-- df.start_autonomous() persists and enqueues the durable function on a +-- separate PostgreSQL session, so it commits independently and survives a +-- rollback of the caller's transaction (the PostgreSQL equivalent of Oracle's +-- PRAGMA AUTONOMOUS_TRANSACTION). Ordinary df.start() is unchanged and still +-- participates in the caller's transaction. +-- +-- Pure additive change: a brand-new function bound to a new C symbol +-- (start_autonomous_wrapper). Existing functions and their symbols are +-- untouched, so the new .so stays backward compatible with all previous +-- schemas in this provider line (Scenario B1) — an un-upgraded schema simply +-- does not expose df.start_autonomous yet. The CREATE FUNCTION block is the +-- pgrx-generated fresh-install DDL for src/dsl.rs::start_autonomous copied +-- verbatim, so the Scenario A snapshot matches a fresh 0.2.4 install. New df.* +-- functions retain PostgreSQL's default PUBLIC EXECUTE (gated by USAGE ON +-- SCHEMA df), so no explicit GRANT is needed. +-- ============================================================================ +-- pg_durable::dsl::start_autonomous +CREATE FUNCTION df."start_autonomous"( + "fut" TEXT, /* &str */ + "label" TEXT DEFAULT NULL, /* core::option::Option<&str> */ + "database" TEXT DEFAULT NULL /* core::option::Option<&str> */ +) RETURNS TEXT /* alloc::string::String */ + +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'start_autonomous_wrapper'; diff --git a/src/client.rs b/src/client.rs index 3d10dfb..4cb38a1 100644 --- a/src/client.rs +++ b/src/client.rs @@ -16,7 +16,9 @@ use duroxide::Client; use pgrx::prelude::*; use tokio::runtime::Runtime; -use crate::types::{backend_duroxide_schema, new_backend_provider, postgres_connection_string}; +use crate::types::{ + backend_duroxide_schema, connect_as_user, new_backend_provider, postgres_connection_string, +}; /// Cached tokio runtime for client operations. static CLIENT_RUNTIME: OnceLock = OnceLock::new(); @@ -220,6 +222,72 @@ pub fn start_durable_function( }) } +/// Start a durable function with **autonomous transaction** semantics. +/// +/// This provides the Oracle `PRAGMA AUTONOMOUS_TRANSACTION` equivalent: the +/// durable function's graph is persisted and enqueued on a *separate* +/// PostgreSQL session, so it commits independently and **survives a rollback of +/// the caller's transaction**. +/// +/// Mechanism: open a fresh loopback connection authenticated as `user` and run +/// the ordinary `df.start(...)` there. Because sqlx runs each statement in +/// autocommit mode, that inner `df.start` commits in its own transaction the +/// moment it returns — regardless of what the outer caller later does. This is +/// the same "separate backend" technique `pg_background` uses to achieve +/// autonomy, but it reuses pg_durable's existing graph construction and +/// duroxide enqueue path unchanged. +/// +/// Returns the new instance id. +/// +/// FUTURE EXPLORATION (alternative implementation): instead of opening another +/// client connection just to run `df.start` in a new backend, move the graph +/// persistence into the parent orchestration itself. With autonomy, the full +/// function graph would be encoded directly into the `start_orchestration` +/// input payload (which duroxide already commits out-of-band), and the +/// orchestration's first activity would become a `store-function-graph` +/// activity that writes `df.instances` / `df.nodes` from the background worker +/// — replacing today's `load-function-graph` first activity. That would remove +/// the extra backend connection entirely and collapse the current dual write +/// (graph in the caller's txn, enqueue out-of-band) into a single durable +/// source of truth, eliminating the orphaned-orchestration race for good. See +/// the PR description for the trade-offs worth evaluating. +pub fn start_autonomous( + fut: &str, + label: Option<&str>, + database: Option<&str>, + user: &str, +) -> Result { + use sqlx::Row; + + let rt = get_client_runtime(); + + let fut = fut.to_string(); + let label = label.map(|s| s.to_string()); + let database = database.map(|s| s.to_string()); + let user = user.to_string(); + + rt.block_on(async { + // Connect to the *current* database (database=None resolves to the + // control/target database) as the submitting role, so identity and + // privilege checks inside the inner df.start run as the caller. + let mut conn = connect_as_user(&user, None).await?; + + let row = sqlx::query("SELECT df.start($1, $2, $3) AS id") + .bind(&fut) + .bind(&label) + .bind(&database) + .fetch_one(&mut conn) + .await + .map_err(|e| format!("autonomous df.start failed: {e}"))?; + + let id: String = row + .try_get("id") + .map_err(|e| format!("autonomous df.start returned no instance id: {e}"))?; + + Ok(id) + }) +} + /// Cancel a durable function. pub fn cancel_durable_function(instance_id: &str, reason: &str) -> Result<(), String> { let inst_id = instance_id.to_string(); diff --git a/src/dsl.rs b/src/dsl.rs index 9a3ef22..b28aa4f 100644 --- a/src/dsl.rs +++ b/src/dsl.rs @@ -1074,6 +1074,42 @@ pub fn start( instance_id } +/// Starts a durable SQL function with **autonomous transaction** semantics. +/// +/// This is the PostgreSQL equivalent of Oracle's `PRAGMA AUTONOMOUS_TRANSACTION`: +/// the durable function is persisted and enqueued on a *separate* PostgreSQL +/// session, so it commits independently and **survives a rollback of the +/// caller's transaction**. Use it for audit/error logging and other +/// "must-persist-even-if-the-caller-rolls-back" work. +/// +/// Ordinary [`start`] participates in the caller's transaction: if the caller +/// rolls back, the durable function is rolled back with it. `start_autonomous` +/// is the opposite — it always persists once this call returns. +/// +/// Accepts the same arguments as [`start`] (`fut`, optional `label`, optional +/// `database`) and returns the new instance id. +#[pg_extern(schema = "df")] +pub fn start_autonomous( + fut: &str, + label: default!(Option<&str>, "NULL"), + database: default!(Option<&str>, "NULL"), +) -> String { + // Capture the calling role so the separate session runs df.start() with the + // same identity (and therefore the same privileges / RLS scope). + let user = unsafe { + let oid = pgrx::pg_sys::GetUserId(); + let name_ptr = pgrx::pg_sys::GetUserNameFromId(oid, false); + std::ffi::CStr::from_ptr(name_ptr) + .to_string_lossy() + .into_owned() + }; + + match crate::client::start_autonomous(fut, label, database, &user) { + Ok(id) => id, + Err(e) => pgrx::error!("{}", e), + } +} + /// Cancels a running durable function. #[pg_extern(schema = "df")] pub fn cancel(instance_id: &str, reason: default!(&str, "'Cancelled by user'")) -> String { diff --git a/tests/e2e/sql/54_autonomous_transaction.sql b/tests/e2e/sql/54_autonomous_transaction.sql new file mode 100644 index 0000000..5ca1371 --- /dev/null +++ b/tests/e2e/sql/54_autonomous_transaction.sql @@ -0,0 +1,116 @@ +-- Copyright (c) Microsoft Corporation. +-- Licensed under the PostgreSQL License. + +-- Oracle-style autonomous transaction support via df.start_autonomous(). +-- +-- Demonstrates that df.start_autonomous(...) commits the durable function +-- independently of the caller's transaction: it SURVIVES a caller ROLLBACK +-- (like Oracle PRAGMA AUTONOMOUS_TRANSACTION), whereas the default df.start() +-- is rolled back with the caller. +SET SESSION AUTHORIZATION df_e2e_user; + +DROP TABLE IF EXISTS test_autonomous_audit; +CREATE TABLE test_autonomous_audit (id SERIAL, message TEXT); + +DROP TABLE IF EXISTS test_autonomous_main; +CREATE TABLE test_autonomous_main (id INT); + +-- === Part 1: autonomous => true SURVIVES a caller rollback === + +BEGIN; + -- Main-transaction work that will be rolled back. + INSERT INTO test_autonomous_main (id) VALUES (999); + + -- Autonomous durable function: commits on a separate session. + SELECT df.start_autonomous( + 'INSERT INTO test_autonomous_audit (message) VALUES (''logged from autonomous txn'')', + 'test-autonomous-survives' + ); + + -- Simulate a failure in the surrounding transaction. + ROLLBACK; + +DO $$ +DECLARE + inst_id TEXT; + status TEXT; + main_count INT; + audit_count INT; +BEGIN + -- The main-transaction insert must have been rolled back. + SELECT count(*) INTO main_count FROM test_autonomous_main; + IF main_count <> 0 THEN + RAISE EXCEPTION 'TEST FAILED: main insert should have rolled back, got % rows', main_count; + END IF; + + -- The autonomous instance must have survived the rollback. + SELECT id INTO inst_id + FROM df.instances + WHERE label = 'test-autonomous-survives' + ORDER BY created_at DESC + LIMIT 1; + + IF inst_id IS NULL THEN + RAISE EXCEPTION 'TEST FAILED: autonomous instance did not survive caller rollback'; + END IF; + + SELECT df.await_instance(inst_id) INTO status; + IF status <> 'completed' THEN + RAISE EXCEPTION 'TEST FAILED: autonomous instance status = %', status; + END IF; + + -- The audit row must have persisted independently of the rollback. + SELECT count(*) INTO audit_count + FROM test_autonomous_audit + WHERE message = 'logged from autonomous txn'; + + IF audit_count <> 1 THEN + RAISE EXCEPTION 'TEST FAILED: audit row missing, count = % (autonomous txn did not persist)', audit_count; + END IF; + + RAISE NOTICE 'PASSED: autonomous => true survived caller rollback'; +END $$; + +-- === Part 2: default df.start() does NOT survive rollback === + +BEGIN; + SELECT df.start( + 'INSERT INTO test_autonomous_audit (message) VALUES (''should never persist'')', + 'test-autonomous-transactional' + ); + ROLLBACK; + +DO $$ +DECLARE + inst_count INT; + audit_count INT; +BEGIN + -- The instance row was written via SPI in the caller's transaction, so the + -- rollback removes it. (A dangling duroxide orchestration may briefly exist + -- and then fail to load the graph — it never runs the SQL.) + SELECT count(*) INTO inst_count + FROM df.instances + WHERE label = 'test-autonomous-transactional'; + + IF inst_count <> 0 THEN + RAISE EXCEPTION 'TEST FAILED: transactional instance should not survive rollback, found %', inst_count; + END IF; + + -- Give any dangling orchestration a moment; the SQL must never run. + PERFORM pg_sleep(1); + + SELECT count(*) INTO audit_count + FROM test_autonomous_audit + WHERE message = 'should never persist'; + + IF audit_count <> 0 THEN + RAISE EXCEPTION 'TEST FAILED: transactional df.start persisted work across rollback, count = %', audit_count; + END IF; + + RAISE NOTICE 'PASSED: default df.start() rolled back with the caller'; +END $$; + +DROP TABLE test_autonomous_audit; +DROP TABLE test_autonomous_main; + +SELECT 'TEST PASSED' AS result;