Use async wait timeout for SEA when direct results disabled (fix hybrid-path truncation)#1476
Merged
Conversation
When direct results are disabled, the SEA execute request used waitTimeout=10s (the old hybrid path) with onWaitTimeout=CONTINUE. For results that span multiple chunks but compress small (highly-compressible payloads), the server returns only the first chunk inline with no external links, and the driver returns just that chunk -- silently truncating the result. Switch to waitTimeout=0s (async) when direct results are disabled so the server delivers results via external links, which the driver fetches in full. This matches the intended contract: DirectResults=0 -> WaitTimeout=0 (async); DirectResults=1 -> WaitTimeout unset. Thrift is unaffected (no waitTimeout; it paginates). Related: ES-1714092 Co-authored-by: Isaac Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
gopalldb
reviewed
Jun 2, 2026
| if (executeAsync) { | ||
| request.setWaitTimeout(ASYNC_TIMEOUT_VALUE); | ||
| } else { | ||
| // Only set timeout if direct results mode is not enabled |
Collaborator
Author
There was a problem hiding this comment.
Added the unit tests
gopalldb
reviewed
Jun 2, 2026
| request.setWaitTimeout(SYNC_TIMEOUT_VALUE); | ||
| request.setWaitTimeout(ASYNC_TIMEOUT_VALUE); | ||
| } | ||
| request.setOnWaitTimeout(ExecuteStatementRequestOnWaitTimeout.CONTINUE); |
Collaborator
There was a problem hiding this comment.
why is this needed for async case?
Collaborator
Author
There was a problem hiding this comment.
This is pre existing line - unchanged by this PR. It might not be required for async mode, but its better to be explicit than missing it out for some edge case (in any future changes)
Asserts getRequest() sets waitTimeout=0s when direct results are disabled and leaves it unset when enabled. Addresses review feedback on #1476. Co-authored-by: Isaac Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
gopalldb
approved these changes
Jun 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When direct results are disabled on the SEA (SQL Execution API) path, the driver
built the
ExecuteStatementrequest withwaitTimeout=10s(SYNC_TIMEOUT_VALUE)and
onWaitTimeout=CONTINUE— the old SEA hybrid direct-results path.For results that span multiple chunks but compress small (highly-compressible
payloads), the server returns only the first chunk inline with no external
links (
hasInlineAttachment=true, numExternalLinks=0). The driver's inline pathreturns just that first chunk and never fetches the rest, so the result is
silently truncated.
This change sets
waitTimeout=0s(ASYNC_TIMEOUT_VALUE) when direct results aredisabled, avoiding the hybrid inline path. The server then delivers results via
external links, which the driver downloads in full.
Resulting contract (SEA):
DirectResults = 0→WaitTimeout = 0(async)DirectResults = 1→WaitTimeoutunset (true direct results)Thrift is unaffected — it has no
waitTimeoutand paginates correctly.Related: ES-1714092 (server-side compressed-vs-uncompressed byte limit on the
hybrid path). This driver change stops using that path; the server-side fix is
tracked separately.
Requirement / Motivation
Reported via Slack: SEA with direct results disabled silently truncated a large,
highly-compressible result (200MB logical → ~0.8MB compressed) to a handful of
rows, while Thrift returned all rows.
Testing done
Repro (manual, against a serverless SQL warehouse) — query
SELECT repeat('A', 1024 * 1024) AS payload FROM range(200)with direct resultsdisabled:
isCloudFetchUsed=false(inline) ❌isCloudFetchUsed=true(external links) ✅Additional disambiguation runs (repeating the real table via
UNION ALLup to 64×,9.6M rows, 40s execution) confirmed the trigger is the inline-vs-external-links
delivery (compressed size) — not query time or polling: a fast no-poll query
truncated while a slow 6-poll query did not.
Regression — SEA unit + fakeservice suites (all pass, 0 failures):
DatabricksSdkClientTest(42)SqlExecApiHybridResultsIntegrationTests(2)DatabricksMetadataQueryClientTest(57)CommandBuilderTest(21)DatabricksEmptyMetadataClientTest(11)SeaCircuitBreakerManagerTest(13)Notes / caveats
execute → poll → fetch (one extra round-trip for small results) instead of
returning inline on the first response.
affected path rather than fixing the server.