Skip to content

feat(spec): add source_task_size to RefreshMaterializedViewRequest#355

Merged
brendanclement merged 1 commit into
lance-format:mainfrom
justinrmiller:gen-476-refresh-source-task-size
Jun 12, 2026
Merged

feat(spec): add source_task_size to RefreshMaterializedViewRequest#355
brendanclement merged 1 commit into
lance-format:mainfrom
justinrmiller:gen-476-refresh-source-task-size

Conversation

@justinrmiller

@justinrmiller justinrmiller commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Context

Chunker-backed materialized-view refresh bounds per-actor memory by controlling how many source row IDs go into one expansion work item. That knob (source_task_size) needs to travel over the enterprise db:// path, but RefreshMaterializedViewRequest had no field for it — so it was silently dropped on remote refresh.

Change

Adds an optional source_task_size integer to RefreshMaterializedViewRequest, mirroring the existing optional overrides (max_rows_per_fragment, intra_applier_concurrency, …). Regenerated the Python (urllib3) and Rust (reqwest) client models and their docs; updated the spec model doc and the urllib3 test stub.

Geneva feature-detects the field, so it forwards source_task_size once a release carries it and degrades gracefully on older versions.

Test plan

  • openapi-spec-validator docs/src/spec.yaml → OK
  • urllib3 model round-trips source_task_size (set / omitted / from_dict)

Follow-ups (separate repos)

  • lance: bump lance-namespace-reqwest-client, rebuild RestNamespace bindings, publish a pylance beta.
  • geneva + sophon: pin the new versions so the field transits end-to-end at runtime.

🤖 Generated with Claude Code

Optional field bounding per-actor memory for chunker materialized-view
refresh by controlling the number of source row ids per expansion work item.
Mirrors the existing optional integer overrides. Regenerated the Python
urllib3 and Rust reqwest client models and docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added enhancement New feature or request python Python features rust Rust features labels Jun 11, 2026
@brendanclement brendanclement merged commit 89e0cab into lance-format:main Jun 12, 2026
4 of 5 checks passed
justinrmiller added a commit to justinrmiller/lance-namespace that referenced this pull request Jun 12, 2026
The source_task_size description in spec.yaml is a multi-line block
scalar, so openapi-generator folds it with a trailing space (same as
the sibling output_limit/manifest fields). PR lance-format#355 committed Rust files
with that trailing space stripped, so the `Check no difference in
codegen` CI step failed once full `make gen` regenerated them.

Regenerated via `make gen-rust` so the committed files match canonical
generator output. No semantic change — whitespace only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
brendanclement pushed a commit that referenced this pull request Jun 12, 2026
## Context

[#355](#355) added
the optional `source_task_size` field to
`RefreshMaterializedViewRequest` in the spec and regenerated only the
Python (urllib3) and Rust (reqwest) clients.

The three generated **Java** modules were left out:

- `lance-namespace-apache-client`
- `lance-namespace-async-client`
- `lance-namespace-springboot-server`

As a result the field was silently dropped on the Java REST path, even
though `docs/src/spec.yaml`, the Java-derived model doc
(`docs/src/namespace/operations/models/RefreshMaterializedViewRequest.md`),
and the Python/Rust clients all carry it. (Per CLAUDE.md the model doc
is *generated from the Java Apache client*, so the doc and the Java
source had drifted apart.)

## Change

Regenerated the Java modules from the spec via `make gen-java`, adding
`sourceTaskSize` and its accessors to all three
`RefreshMaterializedViewRequest` models (plus the embedded
`api/openapi.yaml` copies). No hand edits — pure generator output.

## Test plan

- `make gen-java` → BUILD SUCCESS
- `./mvnw compile -pl
lance-namespace-apache-client,lance-namespace-async-client,lance-namespace-springboot-server
-am` → OK
- `./mvnw checkstyle:check spotless:check` on the same modules → exit 0

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request python Python features rust Rust features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants