server: add read_media tool by parabelboi · Pull Request #25877 · ggml-org/llama.cpp

parabelboi · 2026-07-18T22:32:56Z

This adds a server-tool that allows vision models to analyze server-side images. It fixes #25875

Overview

This tool is reading only a single file for now:
The image data is base64 encoded and passed to the UI, which decodes it, fills the <img> tag and removes the data URI before passing the tool result back to the model.

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, brainstorming, analyzing and coding

This adds a server-tool that allows vision models to analyze server-side images. This tool is reading only a single file for now: The image data is base64 encoded and passed to the UI, which decodes it, fills the <img> tag and removes the data URI before passing the tool result back to the model.

ggml-gh-bot · 2026-07-18T22:37:40Z

Hi @parabelboi, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

parabelboi · 2026-07-18T23:05:08Z

collapsed:

expanded:

allozaur

few architectural remarks. but i really like it :) great stuff. @ngxson @ServeurpersoCom u guys can do a more thorough review/testing on the server side + security aspects

allozaur · 2026-07-20T14:18:33Z

+		const extras = section.toolResultExtras;
+		if (!extras || extras.length === 0) return null;
+		// Extract the attachment name from the cleaned result text
+		const match = section.toolResult?.match(/\[Attachment saved: ([^\]]+)\]/);


let's move the regex to constants

allozaur · 2026-07-20T14:18:58Z

+		if (trimmed.startsWith('Image: ')) {
+			path = trimmed.slice('Image: '.length).trim();
+			fileName = path.split('/').pop() ?? path;
+		} else if (trimmed.startsWith('Size: ')) {
+			const match = trimmed.match(/Size:\s*(\d+)\s*bytes/);
+			if (match) sizeBytes = parseInt(match[1], 10);
+		} else if (trimmed.startsWith('MIME: ')) {
+			mimeType = trimmed.slice('MIME: '.length).trim();


magic strings/regexes — let's move to $lib/constants

allozaur · 2026-07-20T14:19:11Z

+export function parseReadImageMeta(section: AgenticSection): ReadImageMeta | null {
+	if (!section.toolResult) return null;
+
+	const lines = section.toolResult.split('\n');


should use existing NEWLINE constant

- Add dedicated constants file: tools/ui/src/lib/constants/read-image.ts with PREFIX_IMAGE, PREFIX_SIZE, PREFIX_MIME constants - Use ATTACHMENT_SAVED_REGEX from agentic.ts in ChatMessageToolCallBlockReadImage.svelte - Use NEWLINE constant from code.ts instead of hardcoded '\n' - Use PREFIX_SIZE in regex pattern for size parsing - Add SERVER_TOOL_READ_IMAGE_PREFIX_* constants in C++ server-tools.cpp to match the TypeScript PREFIX_* constants for consistency

ngxson

I don't think it should be a dedicated tool. Instead, it should be read_file tool with a parameter type:

type: can be "text" or "media"; if "media", returns file content as base64 with mimetype; media file can be image, audio or video file

parabelboi · 2026-07-21T22:33:38Z

Unfortunately I see the image twice, but only directly after the tool-call. I did not notice until now, because switching to another conversation or reloading the tab makes the duplicate go away.

That must have been caused by an unclean source or build directory. After a clean checkout and rebuild the problem is gone.

parabelboi · 2026-07-22T10:01:03Z

I don't think it should be a dedicated tool. Instead, it should be read_file tool with a parameter type:
* `type`: can be `"text"` or `"media"`; if "media", returns file content as base64 with mimetype; media file can be image, audio or video file

I also thought about this, but decided to have a separate tool, because for two reasons:

First (technical) point is, that users can decide to disable read_image (or read_file) separately in the UI.
For instance, users of text-only models can simply disable read_image.
That saves some tokens when processing the context, because unneeded parameters (i.e. a potentially implemented scale_down_to_max_resolution parameter) do not have to be described to the model in that case.
The same argument would be true, if one wanted to further extend the read_file tool for videos: there you might want to have a cut_to_max_length or a skip_seconds parameter. That description would also have to be consumed by the model, if you only want it to analyze images. Also the UI should look different for videos. Does it make sense to load multiple videos at once? Maybe not, but for images it does.

Second (non-technical) point was, that it might be quicker to integrate a new tool, than to extend an existing one. It would create less friction because of separation of concerns.

ngxson · 2026-07-22T14:39:01Z

+    return (it != mime_map.end()) ? it->second : "application/octet-stream";
+}
+
+struct server_tool_read_image : server_tool {


first, I'd expect to have a generic read_media tool that allow reading image/audio/video

second, if we expect to keep read_media / read_file as separate tools, I'd expect server_tool_read_media to be a derived class of server_tool_read_file

the read_media::invoke() should do only one job: convert the output to base64 and provide the appropriate mime type

Would it be sufficient if the server_tool_read_media class is a derived class of server_tool_read_file, but would only support images (for now)?

And do you still want to have a media-type parameter and if so, would it be ok, if it's optional? Because making it mandatory and defining media-specific options based on media-type would make the json-schema more complicated.

hmm I think it's ok to allow image-only server_tool_read_media for now, maybe we should add a param type that can be image/audio/video in the future, tbd

Very good, I'll rename the tool and leave the type parameter out for now.

Also I'd concentrate to only a single file for now:
I already have a branch that reads in multiple images successfully, but that required a small fix in agentic.svelte.ts (directly after the tool call, attachments have been displayed duplicated, after a page reload everything was fine again). That could lead to a regression in mcp tools and needsmore testing especially with multiple attachments. Also it's way more code to review and test. Especially the code that collects the list of files based on a glob expression still contains bugs. So it's probably a good idea to leave that code out for now.

I managed to also get support for audio working. I currently don't have an audio capable model to test (it's still downloading), but at least the plain-text message was passed to the LLM and the audio data could be played successfully.

ngxson · 2026-07-22T14:46:54Z

The same argument would be true, if one wanted to further extend the read_file tool for videos: there you might want to have a cut_to_max_length or a skip_seconds parameter.

hmm ok maybe better to skip video for now, as the file can be large. but audio might still be a valid use case

for video, we might be better to allow /v1/chat/completion to read the file directly somehow, but I need to think more about it because that can introduce a huge security risk. the tools system doesn't apply because it's already allowed to read arbitrary files by default.

- Rename server_tool_read_image to server_tool_read_media in C++ - Rename enum BuiltInTool.READ_IMAGE to READ_MEDIA - Rename UI constants, parser, and Svelte component files - Update display label from 'Read image' to 'Read media'

- rename parseToolResultWithImages to parseToolResultWithMedia - update all consumers (Default, ExecShellCommand blocks) to use line.media - rename image to media in ToolResultLine and parseToolResultWithMedia, it now holds images and audio - server-tools.cpp: add audio MIME types to get_mime_from_extension() - extend extractBase64Attachments() to detect audio/ MIME types and create AudioFile extras with base64Url - add AUDIO_MIME_TO_EXTENSION to mcp-resource.ts constants alongside IMAGE_MIME_TO_EXTENSION - widen ToolResultLine.image type to include DatabaseMessageExtraAudioFile - ChatMessageToolCallBlockReadMedia renders now <audio> for audio files - ChatMessageToolCallBlockDefault fallback renderer now also handles audio attachments

ngxson · 2026-07-24T03:02:30Z

 }

+//
+// read_media: read a media file (image or audio) and return base64-encoded data with metadata


this is not the good placement in the file, move it to after the last tool definition above

yes, indeed. section has been moved

ngxson · 2026-07-25T08:28:38Z

@ggml-gh-bot review

ggml-gh-bot · 2026-07-25T08:28:41Z

Automated code review

Review of PR #25877 - read_media server tool + UI audio-attachment support.

I ran the always-on scope/security/general checks plus the Server checklist (touched paths are tools/server/ and tools/ui/). No new model/ggml/public-API changes.

Blocking

(point 1) Audio attachments are stored with the wrong field, breaking the type contract, model input, and UI rendering. tools/ui/src/lib/stores/agentic.svelte.ts:1036-1041 pushes

{ type: AttachmentType.AUDIO, name, mimeType, base64Url: trimmedLine }

But DatabaseMessageExtraAudioFile (tools/ui/src/lib/types/database.d.ts:21-27) requires base64Data: string and has no base64Url field. Image extras use a full data URI in base64Url; audio extras use raw base64 in base64Data (see tools/ui/src/lib/utils/convert-files-to-extra.ts:80-87). Consequences:

This object literal does not satisfy the DatabaseMessageExtra union (missing required base64Data), so the change does not typecheck as written.
chat.service.ts:1252 sends audio.base64Data to the model, which is undefined here.
ChatMessageToolCallBlockReadMedia.svelte (~line 53-54) and ChatMessageToolCallBlockDefault.svelte:118 render mediaAttachment.base64Url / line.media.base64Url, which is undefined for audio, so the <audio><source src=...> is broken.

Decide on one shape and apply it consistently: either push base64Data (stripping the data:...;base64, prefix) and have the two UI components build the data URL as `data:${mimeType};base64,${base64Data}`, or extend the audio extra type to also carry a base64Url and use it everywhere. As committed the feature cannot work for audio.

(point 2) Server advertises audio formats the model-input path cannot handle. get_mime_from_extension in server-tools.cpp returns audio/opus, audio/ogg, audio/flac, audio/mp4 (m4a). But AUDIO_MIME_TO_EXTENSION (tools/ui/src/lib/constants/mcp-resource.ts) lacks audio/opus, audio/ogg, audio/flac (so those files get a wrong .mp3 attachment name), and getAudioInputFormat (chat.service.ts:46-62) only ever returns WAV or MP3 - any other audio mime is sent to the model as format: MP3 with non-MP3 data, which the input_audio API rejects. Either restrict the server tool's mime map to .wav/.mp3 (and document that), or extend both the extension map and the format mapping. Also note: files whose extension yields application/octet-stream (e.g. .svg, or any uppercase extension) are not stripped by extractBase64Attachments (agentic.svelte.ts:1000-1044), so the full multi-MB base64 data URI stays in the tool result text that flows into the model context - bound the set of accepted extensions and have the unknown case return an error instead of a data URI.

Will slow the review

(point 3) --tools help text and README list of available tools were not updated. common/arg.cpp:3168 and tools/server/README.md:200 enumerate read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_datetime - add read_media to both so --help and the docs stay in sync.

(point 4) No test for the new builtin tool. tools/server/tests/unit/test_tools_builtin.py covers read_file/write_file/edit_file/grep_search. Add a small test invoking read_media against a known fixture (assert the File:/Size:/MIME: lines and the data: URI shape, plus the too-large and bad-path error paths), matching the existing style.

Nits

(point 5) Stray duplicated // comment line above // public API in server-tools.cpp:1216-1217, and two consecutive blank lines after the server_tool_read_media struct (server-tools.cpp:1164-1166). Clean both up.

(point 6) Svelte indentation in ChatMessageToolCallBlockDefault.svelte:119-122 (and the same block in the new ChatMessageToolCallBlockReadMedia.svelte): the inner {/if} closing the audio vs img branch is indented one level deeper than its {:else}, making the if/else hard to read. Align it with the rest of the block.

(point 7) get_mime_from_extension (server-tools.cpp:1069) compares the raw extension against lowercase map keys, so .PNG/.WAV fall through to application/octet-stream. auto ext = fs::path(path).extension().string(); then lowercase ext before the lookup (consistent with IMAGE_FILE_EXTENSION_REGEX/AUDIO_FILE_EXTENSION_REGEX being case-insensitive on the UI side).

(point 8) {"size_bytes", (int)file_size} (server-tools.cpp:1163) narrows a uintmax_t to int. The 16 MB cap keeps it safe today, but prefer (size_t) (or (int64_t)) to match the (size_t) cast used in the error string just above and avoid a latent truncation if the cap is ever raised beyond INT_MAX.

Overall the metadata/parsing wiring (prefix constants on both sides, ATTACHMENT_SAVED_REGEX ↔ extractBase64Attachments ↔ parseReadMediaMeta) is a reasonable design and reuses the existing image-attachment machinery; the integration is split correctly across BuiltInTool/BUILTIN_TOOL_UI/the dispatch block. The blocking issues are the audio data-shape mismatch and the unsupported audio formats - both need resolving before this is mergeable.

This review was generated automatically by pi coding agent using zai-org/GLM-5.2. It may contain mistakes. Maintainers make the final call.

ngxson · 2026-07-25T08:38:18Z

please address point 1, 2, 3; testing is optional, but recommended to add

parabelboi requested review from a team as code owners July 18, 2026 22:32

github-actions Bot added server server/ui labels Jul 18, 2026

parabelboi changed the title ~~server: add read_image tool (#25875)~~ server: add read_image tool Jul 18, 2026

allozaur requested changes Jul 20, 2026

View reviewed changes

parabelboi requested a review from allozaur July 20, 2026 19:13

allozaur requested review from ServeurpersoCom and ngxson July 20, 2026 21:58

ngxson requested changes Jul 21, 2026

View reviewed changes

ngxson reviewed Jul 22, 2026

View reviewed changes

parabelboi requested a review from ngxson July 23, 2026 18:47

parabelboi changed the title ~~server: add read_image tool~~ server: add read_media tool Jul 23, 2026

ngxson reviewed Jul 24, 2026

View reviewed changes

server: move read_media tool to the other tools

402c47b

Uh oh!

Conversation

parabelboi commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

ggml-gh-bot Bot commented Jul 18, 2026

Uh oh!

parabelboi commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

allozaur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

parabelboi commented Jul 21, 2026

Uh oh!

parabelboi commented Jul 22, 2026

Uh oh!

ngxson Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parabelboi Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parabelboi Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson commented Jul 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson commented Jul 25, 2026

Uh oh!

ggml-gh-bot Bot commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Blocking

Will slow the review

Nits

Uh oh!

ngxson commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

parabelboi commented Jul 18, 2026 •

edited

Loading

parabelboi commented Jul 18, 2026 •

edited

Loading

ngxson Jul 22, 2026 •

edited

Loading

parabelboi Jul 22, 2026 •

edited

Loading

parabelboi Jul 23, 2026 •

edited

Loading

ggml-gh-bot Bot commented Jul 25, 2026 •

edited

Loading