fix(gemini): type mp4/mov/webm uploads so Gemini actually receives them

[mkubenka]

The bug

GEMINI_UPLOAD_MIME_TYPES covers only images and PDF. Everything else falls through to application/octet-stream, and Gemini silently discards uploads it cannot type.

The failure is invisible from the outside, which is what makes it expensive:

$ oracle --engine browser -m gemini-3.1-pro --browser-attachments always \
    --file clip.mp4 -p "Describe this video. If you did not receive a video, say NO VIDEO RECEIVED."

Answer:
NO VIDEO RECEIVED.

7.4s · gemini-3.1-pro[browser] · ↑51 ↓5
files=1        <-- reports the file as sent

The upload succeeds, the payload is built, files=1 is printed — and the model gets nothing. Reading the source reinforces the wrong conclusion, because the upload path genuinely is complete and correct. Only the MIME label is wrong.

Declaring video/mp4 makes the same file work on the first attempt:

Answer:
The video features an interview setup in a studio environment.
* **Visuals:** A man wearing glasses and a black shirt sits in front of a microphone...
  Chinese and English subtitles appear at the bottom of the screen...
### Verbatim English Subtitles (First 15 Seconds):
* "so it snows."

The change

Adds exactly three entries — .mp4, .mov, .webm — and extracts the lookup into an exported resolveGeminiUploadMimeType() so it is unit-testable. No behaviour change for currently-mapped types; the application/octet-stream fallback is preserved.

Why only three

I first wrote a much larger table from the Gemini API's documented formats, then probed each candidate against the live endpoint with a sentinel prompt separating "received" from "actually viewable". Most of it was wrong:

Format	Result
.mp4 .mov .webm	viewable — describes people, clothing, on-screen text
.avi	received but not viewable; only echoed container metadata (Lavf62.3.100)
.m4v	dropped, despite byte-identical content to a working .mp4 and the same declared video/mp4
.mkv .3gp .heic	Gemini returns an error
.wav .mp3 .m4a	never delivered — tried audio/mpeg, audio/mp3, audio/wav, audio/x-wav
.txt	never delivered — tried text/plain and text/md

Two conclusions worth recording:

1. Audio and plain text are not supported by this endpoint at all, whatever type is declared. Mapping them would advertise support that does not exist.
2. The endpoint gates on the file extension, not just the declared MIME type. The .m4v case is the proof: same bytes, same video/mp4, still dropped. So this table cannot be extrapolated from the Gemini API docs — each entry needs probing.

I have kept the entries to what I verified. If you have a broader whitelist from the web app, more can be added safely.

Verification

Against gemini-3.1-pro:

• 1.7 MB / 60 s MP4 — before: NO VIDEO RECEIVED; after: accurate scene description plus verbatim on-screen subtitles, cross-checked against frames read locally.
• A full 99-minute video in ten ~8.6 MB chunks (scale=640:-2,fps=8, crf 32). Answers to timestamp-specific questions matched an independently produced OCR transcript, including subtitles at 5:00 and 5:48 and both speakers' names.

Checks: vitest (5 new tests plus existing tests/gemini.test.ts — 15 passing), tsc --noEmit, oxfmt --check, oxlint.

Notes for the maintainer

• The 1 MB DEFAULT_MAX_FILE_SIZE_BYTES still applies, so video realistically needs --max-file-size-bytes / ORACLE_MAX_FILE_SIZE_BYTES. Left alone as a separate policy decision.
• uploadGeminiFile() reads the whole file into a Buffer→Blob, so large media will exhaust the heap. Chunking is the practical answer; out of scope here.
• The deeper issue is the silent failure, not the missing entries. An extension falling back to application/octet-stream arguably deserves a warning, since the current behaviour reports success while the model receives nothing — that is what made video look impossible rather than merely broken. There is no logger plumbed into client.ts's upload path, so I kept this PR to the data fix. Happy to follow up if you want the warning.

[mkubenka]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29943205924
• Updated: 2026-07-22T17:39:11.512Z

[clawsweeper]

Codex review: needs changes before merge. Reviewed July 31, 2026, 5:40 PM ET / 21:40 UTC.

ClawSweeper review

What this changes

This PR maps .mp4, .mov, and .webm Gemini browser attachments to video MIME types, extracts the lookup into a testable resolver, adds regression tests, and records the user-visible fix in the changelog.

Merge readiness

⚠️ Needs maintainer review before merge - 3 items remain

The Gemini video-upload bug remains on current main, and this PR has a narrow, source-aligned fix with credible after-fix live output. One merge blocker remains: its changelog entry targets the now-released 0.16.1 section instead of the current 0.16.2 unreleased section.

Priority: P1
Reviewed head: 4ddc53c70a81e8e06709a0ead4f527acddc0a290

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The live evidence and narrow code change are strong, but the stale release-note placement is a small blocking correction.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (live_output): The PR body provides concrete before/after Gemini browser live output for the same video workflow, including a positive scene-and-subtitle result after MIME typing; redact account or media details in any future proof updates.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR body provides concrete before/after Gemini browser live output for the same video workflow, including a positive scene-and-subtitle result after MIME typing; redact account or media details in any future proof updates.
Evidence reviewed	4 items	Current-main bug remains: Current main maps image and PDF extensions only; an .mp4, .mov, or .webm file still reaches the application/octet-stream fallback before upload metadata is sent to Gemini. PR repair is narrowly scoped: The proposed branch adds only the three validated video mappings, routes the uploader through the exported resolver, and adds five focused resolver cases while retaining the existing octet-stream fallback. Feature provenance: The existing extension-based MIME path originated with Gemini upload support and later image MIME repair work; current blame attributes the preserved upload implementation to the release-tree commit.
Findings	1 actionable finding	[P2] Move the release note to the active unreleased section
Security	None	None.

How this fits together

Oracle’s Gemini web client reads CLI file attachments, uploads each file to Gemini, then includes the declared MIME type in the generated chat request. A missing extension mapping currently falls back to application/octet-stream, so Gemini may accept the upload request while omitting the attachment from the model input.

flowchart LR
  A[CLI file attachment] --> B[Gemini web executor]
  B --> C[Gemini upload client]
  C --> D[Extension to MIME lookup]
  D --> E[Gemini upload request]
  E --> F[Chat request with attachment metadata]
  F --> G[Gemini model response]

Loading

Before merge

• Move the release note to the active unreleased section (P2) - main released 0.16.1 on July 23, 2026 and now records new fixes under 0.16.2 — Unreleased. Leaving this hunk at the old location makes the merged changelog state that the video MIME repair shipped in v0.16.1; move the bullet under the current unreleased ### Fixed heading.
• Resolve merge risk (P1) - The changelog bullet currently belongs to the released 0.16.1 section; it must be moved to 0.16.2 — Unreleased so release history does not claim the video fix shipped in v0.16.1.
• Complete next step (P2) - A single mechanical changelog relocation resolves the remaining merge blocker without changing the validated Gemini upload repair.

Findings

• [P2] Move the release note to the active unreleased section — CHANGELOG.md:8-11

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Video MIME coverage	3 new mappings, 5 focused test cases	The change limits behavior expansion to the three formats the contributor reports validating against the live Gemini web endpoint.
Patch scope	3 files; 71 additions, 2 deletions	The implementation is small, and the only remaining correction is the release-note placement.

Merge-risk options

Maintainer options:

1. Decide the mitigation before merge
   Keep the three validated video mappings and focused resolver tests, then move the release note into the current 0.16.2 — Unreleased Fixed section before merging.
2. Pause or close
   Do not merge this PR until maintainers decide whether the risk is worth taking.

Technical review

Best possible solution:

Keep the three validated video mappings and focused resolver tests, then move the release note into the current 0.16.2 — Unreleased Fixed section before merging.

Do we have a high-confidence way to reproduce the issue?

Yes—current source deterministically sends these video extensions through the octet-stream fallback, and the PR supplies a concrete authenticated Gemini browser command plus before/after live output. A reviewer can reproduce with a supported video fixture under the 1 MB limit or with an explicit higher file-size setting.

Is this the best way to solve the issue?

No—not until the changelog hunk is moved. The MIME mappings, resolver extraction, and fallback-preserving tests are the narrowest maintainable repair, but the release note must target 0.16.2 — Unreleased.

Full review comments:

• [P2] Move the release note to the active unreleased section — CHANGELOG.md:8-11
main released 0.16.1 on July 23, 2026 and now records new fixes under 0.16.2 — Unreleased. Leaving this hunk at the old location makes the merged changelog state that the video MIME repair shipped in v0.16.1; move the bullet under the current unreleased ### Fixed heading.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.98

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 68b8c51b0ee0.

Labels

Label justifications:

• P1: Current Gemini browser runs can report a video attachment as sent while the model receives no video, breaking an established attachment workflow for affected users.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body provides concrete before/after Gemini browser live output for the same video workflow, including a positive scene-and-subtitle result after MIME typing; redact account or media details in any future proof updates.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides concrete before/after Gemini browser live output for the same video workflow, including a positive scene-and-subtitle result after MIME typing; redact account or media details in any future proof updates.

Evidence

Acceptance criteria:

• [P1] pnpm oxfmt --check CHANGELOG.md.

What I checked:

• Current-main bug remains: Current main maps image and PDF extensions only; an .mp4, .mov, or .webm file still reaches the application/octet-stream fallback before upload metadata is sent to Gemini. (src/gemini-web/client.ts:47, 68b8c51b0ee0)
• PR repair is narrowly scoped: The proposed branch adds only the three validated video mappings, routes the uploader through the exported resolver, and adds five focused resolver cases while retaining the existing octet-stream fallback. (src/gemini-web/client.ts:49, 4ddc53c70a81)
• Feature provenance: The existing extension-based MIME path originated with Gemini upload support and later image MIME repair work; current blame attributes the preserved upload implementation to the release-tree commit. (src/gemini-web/client.ts:190, a791643ee944)
• Release-boundary mismatch: The PR puts its release note under 0.16.1 — Unreleased, but current main shows 0.16.1 — 2026-07-23 and has opened 0.16.2 — Unreleased; retaining the branch hunk would document a post-release fix as shipped in the prior release. (CHANGELOG.md:8, 4ddc53c70a81)

Likely related people:

• Peter Steinberger: Introduced the original Gemini web upload path, merged the prior MIME-metadata repair, and owns the current release-tree implementation by blame. (role: recent area contributor; confidence: high; commits: 46d979c6b08b, a791643ee944, 5daa6ce8c352; files: src/gemini-web/client.ts, CHANGELOG.md)
• Trịnh Đức Hoàng: Authored the earlier extension-based MIME detection change that established the upload contract this PR extends. (role: introduced prior MIME behavior; confidence: high; commits: 120cb89ad8a5; files: src/gemini-web/client.ts)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Move the Gemini changelog bullet from 0.16.1 to 0.16.2 — Unreleased, then refresh the PR against current main.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (11 earlier review cycles; latest 8 shown)

• reviewed 2026-07-30T15:21:05.147Z sha 4ddc53c :: needs changes before merge. :: [P2] Remove the release-owned changelog entry
• reviewed 2026-07-31T00:01:48.752Z sha 4ddc53c :: needs changes before merge. :: [P2] Remove the release-owned changelog entry
• reviewed 2026-07-31T01:49:49.999Z sha 4ddc53c :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T05:23:36.452Z sha 4ddc53c :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T11:27:37.424Z sha 4ddc53c :: needs changes before merge. :: [P2] Remove the release-owned changelog entry
• reviewed 2026-07-31T14:27:11.744Z sha 4ddc53c :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T18:43:38.789Z sha 4ddc53c :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T20:15:42.163Z sha 4ddc53c :: needs maintainer review before merge. :: none