-
Notifications
You must be signed in to change notification settings - Fork 21.4k
server: add read_media tool #25877
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
parabelboi
wants to merge
5
commits into
ggml-org:master
Choose a base branch
from
parabelboi:add-read_image-tool
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
server: add read_media tool #25877
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
942f364
server: add read_image tool (#25875)
2256071
cleanup read_image tool: move magic strings to constants
28d0170
server: rename read_image tool to read_media for images and audio
95fa887
server: add audio file support to read_media tool
402c47b
server: move read_media tool to the other tools
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
93 changes: 93 additions & 0 deletions
93
...hat/ChatMessages/ChatMessage/ChatMessageToolCall/ChatMessageToolCallBlockReadMedia.svelte
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| <script lang="ts"> | ||
| import { Eye } from '@lucide/svelte'; | ||
| import { AttachmentType } from '$lib/enums'; | ||
| import { ATTACHMENT_SAVED_REGEX } from '$lib/constants/agentic'; | ||
| import type { DatabaseMessageExtra, DatabaseMessageExtraImageFile, DatabaseMessageExtraAudioFile } from '$lib/types'; | ||
| import { type AgenticSection } from '$lib/utils'; | ||
| import { parseReadMediaMeta } from './parsers/read-media'; | ||
| import ToolCallBlock from './ToolCallBlock.svelte'; | ||
|
|
||
| interface Props { | ||
| section: AgenticSection; | ||
| open: boolean; | ||
| isStreaming: boolean; | ||
| onToggle?: () => void; | ||
| } | ||
|
|
||
| let { section, open, isStreaming, onToggle }: Props = $props(); | ||
|
|
||
| const readMediaMeta = $derived(parseReadMediaMeta(section)); | ||
|
|
||
| // Find the attachment from toolResultExtras (attached to the tool result message. | ||
| // The extractBase64Attachments function in agentic.svelte.ts replaces the data URI line | ||
| // with [Attachment saved: name] and stores the base64 as an extra. | ||
| const mediaAttachment = $derived.by(() => { | ||
| const extras = section.toolResultExtras; | ||
| if (!extras || extras.length === 0) return null; | ||
| // Extract the attachment name from the cleaned result text | ||
| const match = section.toolResult?.match(ATTACHMENT_SAVED_REGEX); | ||
| if (!match) return null; | ||
| const attachmentName = match[1]; | ||
| return extras.find( | ||
| (e): e is DatabaseMessageExtraImageFile | DatabaseMessageExtraAudioFile => | ||
| (e.type === AttachmentType.IMAGE || e.type === AttachmentType.AUDIO) && | ||
| e.name === attachmentName | ||
| ) ?? null; | ||
| }); | ||
|
|
||
| const isAudio = $derived(mediaAttachment?.type === AttachmentType.AUDIO); | ||
| </script> | ||
|
|
||
| <ToolCallBlock {section} {open} {isStreaming} meta={readMediaMeta} {onToggle}> | ||
| {#snippet titleSnippet()} | ||
| <span class="text-muted-foreground">Read media </span> | ||
| <span class="font-mono">{readMediaMeta?.fileName}</span> | ||
| {/snippet} | ||
|
|
||
| {#snippet children(_meta, _ctx)} | ||
| {#if section.toolResult} | ||
| {#if mediaAttachment} | ||
| {#if isAudio} | ||
| <div class="mt-2"> | ||
| <audio controls class="w-full rounded-lg"> | ||
| <source src={mediaAttachment.base64Url} type={readMediaMeta?.mimeType ?? 'audio/mpeg'} /> | ||
| Your browser does not support the audio element. | ||
| </audio> | ||
| </div> | ||
| {:else} | ||
| <div class="mt-2"> | ||
| <img | ||
| src={mediaAttachment.base64Url} | ||
| alt={readMediaMeta?.fileName ?? 'media'} | ||
| class="max-h-[60vh] max-w-full rounded-lg object-contain shadow-lg" | ||
| loading="lazy" | ||
| /> | ||
| </div> | ||
| {/if} | ||
| {:else} | ||
| <div class="rounded bg-muted/20 p-2 text-xs text-muted-foreground/70 italic"> | ||
| Media attachment not found in message extras | ||
| </div> | ||
| {/if} | ||
|
|
||
| {#if readMediaMeta?.sizeBytes || readMediaMeta?.mimeType} | ||
| <div class="mt-2 flex gap-4 text-xs text-muted-foreground"> | ||
| {#if readMediaMeta?.sizeBytes} | ||
| <span>Size: {readMediaMeta.sizeBytes} bytes</span> | ||
| {/if} | ||
| {#if readMediaMeta?.mimeType} | ||
| <span>MIME: {readMediaMeta.mimeType}</span> | ||
| {/if} | ||
| </div> | ||
| {/if} | ||
|
|
||
| {#if readMediaMeta?.path} | ||
| <div class="mt-1 text-xs text-muted-foreground/60 font-mono">{readMediaMeta.path}</div> | ||
| {/if} | ||
| {:else} | ||
| <div class="rounded bg-muted/20 p-2 text-xs text-muted-foreground/70 italic"> | ||
| Waiting for media data... | ||
| </div> | ||
| {/if} | ||
| {/snippet} | ||
| </ToolCallBlock> |
48 changes: 48 additions & 0 deletions
48
...ib/components/app/chat/ChatMessages/ChatMessage/ChatMessageToolCall/parsers/read-media.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| import type { AgenticSection } from '$lib/utils'; | ||
| import { NEWLINE } from '$lib/constants/code'; | ||
| import { PREFIX_FILE, PREFIX_SIZE, PREFIX_MIME } from '$lib/constants/read-media'; | ||
|
|
||
| export interface ReadMediaMeta { | ||
| fileName: string; | ||
| path: string; | ||
| sizeBytes?: number; | ||
| mimeType?: string; | ||
| } | ||
|
|
||
| /** | ||
| * Parse read_media tool result to extract metadata. | ||
| * Expected format (after extractBase64Attachments processing): | ||
| * File: /path/to/file.png | ||
| * Size: 12345 bytes | ||
| * MIME: image/png | ||
| * [Attachment saved: mcp-attachment-xxx.png] | ||
| * | ||
| * The data URI line is replaced by the attachment marker by | ||
| * agenticStore.extractBase64Attachments before storage. | ||
| */ | ||
| export function parseReadMediaMeta(section: AgenticSection): ReadMediaMeta | null { | ||
| if (!section.toolResult) return null; | ||
|
|
||
| const lines = section.toolResult.split(NEWLINE); | ||
| let fileName = ''; | ||
| let path = ''; | ||
| let sizeBytes: number | undefined; | ||
| let mimeType: string | undefined; | ||
|
|
||
| for (const line of lines) { | ||
| const trimmed = line.trim(); | ||
| if (trimmed.startsWith(PREFIX_FILE)) { | ||
| path = trimmed.slice(PREFIX_FILE.length).trim(); | ||
| fileName = path.split('/').pop() ?? path; | ||
| } else if (trimmed.startsWith(PREFIX_SIZE)) { | ||
| const match = trimmed.match(new RegExp(`${PREFIX_SIZE}\\s*(\\d+)\\s*bytes`)); | ||
| if (match) sizeBytes = parseInt(match[1], 10); | ||
| } else if (trimmed.startsWith(PREFIX_MIME)) { | ||
| mimeType = trimmed.slice(PREFIX_MIME.length).trim(); | ||
| } | ||
| } | ||
|
|
||
| if (!path) return null; | ||
|
|
||
| return { fileName, path, sizeBytes, mimeType }; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not the good placement in the file, move it to after the last tool definition above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, indeed. section has been moved