Skip to content

Make detection of binary files more robust #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xenodium merged 2 commits into from
Jun 3, 2026
+26 −23
Merged

Make detection of binary files more robust #515

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
91c28fe
Make detection of binary files more robust
martenlienen Apr 9, 2026
f03c280
Encode binary files as binary resources
martenlienen Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 26 additions & 23 deletions agent-shell.el
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
(require 'dired)
(require 'diff)
(require 'json)
(require 'mailcap)
(require 'map)
(unless (require 'markdown-overlays nil 'noerror)
(error "Please update 'shell-maker' to v0.91.2 or newer"))
Expand Down Expand Up @@ -5268,13 +5269,13 @@ Returns list of alists with :start, :end, and :path for each mention."
(mimeType . ,(map-elt file :mime-type))
(uri . ,(concat "file://" resolved-path)))
content-blocks))
;; Text file, small enough, text file capabilities granted and embeddedContext supported
;; Small enough, text file capabilities granted and embeddedContext supported
;; Use ContentBlock::Resource
((and agent-shell-text-file-capabilities supports-embedded-context (map-elt file :size)
(< (map-elt file :size) agent-shell-embed-file-size-limit))
(push `((type . "resource")
(resource . ((uri . ,(concat "file://" resolved-path))
(text . ,(map-elt file :content))
(,(if (map-elt file :base64-p) 'blob 'text) . ,(map-elt file :content))
Copy link
Copy Markdown
Contributor Author

@martenlienen martenlienen Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This embeds small non-text files as binary resources now as documented here. And I think at least claude would also support embedded PDFs, if these API docs also apply to ACP.

Regarding and agent-shell-text-file-capabilities above, why is this necessary to work with embedded resources? From the naming and the pattern in the image case, I would think that supports-embedded-context would be the right test to check if the agent can deal with embedded context.

Copy link
Copy Markdown
Owner

@xenodium xenodium May 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding and agent-shell-text-file-capabilities above, why is this necessary to work with embedded resources?

IIRC, this was somewhat relevant to running agents in a container and preventing access to files outside the container, but this is a moot point since we're crafting a prompt so the user intent is indeed to send the file. We can likely remove it.

(mimeType . ,(map-elt file :mime-type)))))
content-blocks))
;; File too large, no text file capabilities granted or embeddedContext not supported
Expand Down Expand Up @@ -5311,30 +5312,32 @@ Returns an alist with:
:size - file size in bytes
:extension - file extension (lowercase)
:mime-type - MIME type based on extension
:base64-p - t if content is base64-encoded (binary image), nil otherwise
:base64-p - t if content is base64-encoded (binary file), nil otherwise
:content - file content (omitted when SHALLOW is non-nil)"
(let* ((ext (downcase (or (file-name-extension file-path) "")))
(mime-type (or (agent-shell--image-type-to-mime file-path)
"text/plain"))
;; Only treat supported binary image formats as binary
;; SVG is XML/text and should not be base64-encoded
;; API only supports: image/png, image/jpeg, image/gif, image/webp
(is-binary (member mime-type '("image/png" "image/jpeg" "image/gif" "image/webp")))
(file-size (file-attribute-size (file-attributes file-path)))
(content (unless shallow
(with-temp-buffer
(if is-binary
(progn
(insert-file-contents-literally file-path)
(base64-encode-string (buffer-string) t))
(insert-file-contents file-path)
(buffer-string))))))
(append (list (cons :size file-size)
(cons :extension ext)
(cons :mime-type mime-type)
(cons :base64-p is-binary))
(unless shallow
(list (cons :content content))))))
(mime-type (mailcap-extension-to-mime ext))
(content-fields
(unless shallow
(let ((raw-content (with-temp-buffer
(set-buffer-multibyte nil)
(insert-file-contents-literally file-path)
(buffer-string)))
;; Same heuristic that git uses
(is-binary (string-search "\0" raw-content))
(content (if is-binary
(base64-encode-string raw-content)
(decode-coding-string raw-content 'undecided t))))
;; Set a better default MIME type for unknown extensions
;; based on the content
(unless mime-type
(setq mime-type (if is-binary "application/octet-stream" "text/plain")))
`((:base64-p . ,is-binary)
(:content . ,content))))))
`((:size . ,file-size)
(:extension . ,ext)
(:mime-type . ,(or mime-type "text/plain"))
,@content-fields)))

(cl-defun agent-shell--load-image (&key file-path (max-width 200))
"Load image from FILE-PATH and return the image object.
Expand Down