Skip to content

Attach images to outgoing prompts #271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dnouri wants to merge 1 commit into
base: fix/prompt-ownership-settlement
Choose a base branch
from
Open

Attach images to outgoing prompts #271

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 41 additions & 14 deletions README.org
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,8 +130,9 @@ two windows:

The input header line shows the current model, thinking level,
activity phase, cost and context usage, session name, and extension
status when available. The model and thinking fields can be clicked
to change their values.
status when available. When a prompt image is attached, it also shows
the image's basename and source size. The model and thinking fields
can be clicked to change their values.

Type in the input buffer and press =C-c C-c= to send. If Pi is
already working, =C-c C-c= queues the text as a follow-up and sends it
Expand All @@ -154,6 +155,27 @@ registers, spell checks, Evil, or whatever else your setup already
gives you. Your prompt stays in the bottom window while the
conversation streams above it.

Press =C-c C-a= to attach one image: select a file, or paste its path
into the file prompt. Attaching another image replaces the first;
=C-u C-c C-a= clears it. The header line keeps the attached basename
and size visible. The file is read and materialized when attached,
and PNG, JPEG, GIF, and WebP are recognized from their content rather
than their extension. The default source limit is 3 MiB. Send it
with a nonempty ordinary prompt while using a vision-capable model.

Image-bearing drafts can be sent only as direct prompts while Pi is
idle. They are not queued while Pi is busy, used as steering
messages, or combined with slash commands. A refusal preserves both
prompt text and image. Image bytes are sent as attached, without
resizing or format conversion.
Clipboard image extraction, automatic detection of paths typed into
the prompt, and multiple attachments are deferred.

Image files are read by Emacs, so an Emacs-readable path may use a
file-name handler such as TRAMP. Pi receives encoded bytes, not the
path. Remote reads still depend on the configured handler and are not
broadly tested.

Slash commands work with completion: type =/= then =TAB= to complete
built-in commands and pi commands such as prompt templates, skills,
and extension commands. Prompt templates are discovered from places
Expand All @@ -178,19 +200,19 @@ tool block to expand or collapse it. Long-running commands stream
output live, file operations (=read=, =write=, =edit=) get syntax
highlighting, and edit diffs highlight what changed.

Image content returned by completed tool results is shown inline in graphical
Emacs and as a useful type-and-size placeholder in terminals. Pi's built-in
=read= already returns raster images as image content. For SVG, the preview is
made only from complete, standalone SVG text returned by =read=; pi-coding-agent
never reopens the argument path, and text with obvious scripts or external
resources is left as text. Previews are *display-only*: outgoing prompts have
no image field, so prompt image attachments remain separate
[[https://github.com/dnouri/pi-coding-agent/issues/261][issue #261]] work.
Image content in sent user turns and completed tool results is shown inline in
graphical Emacs and as a useful type-and-size placeholder in terminals. Sent
and returned image content share the same bounded renderer. Pi's built-in
=read= already returns raster images as image content. For SVG, a returned
preview is made only from complete, standalone SVG text supplied by =read=;
pi-coding-agent never reopens the argument path, and text with obvious scripts
or external resources is left as text.

Current limits: images in partial tool updates appear when the final result
arrives; custom tools can return animated or highly compressed data whose
decoder cost is not bounded by the source-byte cap (Pi's built-in =read=
resizes raster images to at most 2000x2000). Moving a terminal-rendered chat to
arrives; unresized prompt images and custom tools can supply animated or highly
compressed data whose decoder cost is not bounded by the source-byte cap (Pi's
built-in =read= resizes raster images to at most 2000x2000). Moving a
terminal-rendered chat to
a GUI, resizing previews, or applying a later extension replacement may require
a toggle or history reload.

Expand Down Expand Up @@ -266,6 +288,8 @@ configured warning and error thresholds.
| Key | Context | Description |
|------------------+---------+------------------------------------------------|
| =C-c C-c= | input | 📮 Send prompt, or queue follow-up if busy |
| =C-c C-a= | input | 🖼️ Attach or replace one prompt image |
| =C-u C-c C-a= | input | 🧹 Clear the attached prompt image |
| =C-c C-s= | input | 🐎 Send steering message while Pi is busy |
| =C-c C-k= | input | 🪓 Abort current response or compaction |
| =C-c C-p= | input | 🎛️ Open transient menu |
Expand Down Expand Up @@ -504,8 +528,11 @@ Less common tuning knobs:
;; (setopt pi-coding-agent-tool-preview-lines 20)
;; (setopt pi-coding-agent-bash-preview-lines 10)

;; Lower the 3 MiB source limit for an outgoing prompt image:
;; (setopt pi-coding-agent-prompt-image-max-bytes (* 2 1024 1024))

;; Cap inline image previews to 640 pixels as well as the chat window width;
;; lower the 10 MiB per-image returned-source limit if desired:
;; lower the 10 MiB per-image preview-source limit if desired:
;; (setopt pi-coding-agent-image-preview-max-width 640)
;; (setopt pi-coding-agent-image-preview-max-bytes (* 5 1024 1024))

Expand Down
4 changes: 1 addition & 3 deletions pi-coding-agent-browse.el
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2202,9 +2202,7 @@ blocks sending until the switch settles, so the text cannot leak into
the outgoing session. Failures are non-fatal."
(when (buffer-live-p input-buf)
(condition-case err
(with-current-buffer input-buf
(erase-buffer)
(when text (insert text)))
(pi-coding-agent--replace-input-draft input-buf text)
(error
(message "Pi: Failed to prefill prompt - %s"
(error-message-string err))))))
Expand Down
19 changes: 14 additions & 5 deletions pi-coding-agent-core.el
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,15 +466,24 @@ Maps request IDs to command type strings."
(defun pi-coding-agent--rpc-async (process command callback)
"Send COMMAND to pi PROCESS asynchronously.
COMMAND is a plist that will be augmented with a unique ID.
CALLBACK is called with the response plist when received."
CALLBACK is called with the response plist when received.
Encoding or scheduling failures leave no pending request behind."
(let* ((id (pi-coding-agent--next-request-id))
(full-command (plist-put (copy-sequence command) :id id))
;; Encode before registration so serialization failures cannot create
;; pending state that no response could ever resolve.
(encoded-command (pi-coding-agent--encode-command full-command))
(pending (pi-coding-agent--get-pending-requests process))
(pending-types (pi-coding-agent--get-pending-command-types process)))
(puthash id callback pending)
(puthash id (plist-get command :type) pending-types)
(pi-coding-agent--send-string
process (pi-coding-agent--encode-command full-command))))
(condition-case err
(progn
(puthash id callback pending)
(puthash id (plist-get command :type) pending-types)
(pi-coding-agent--send-string process encoded-command))
((error quit)
(remhash id pending)
(remhash id pending-types)
(signal (car err) (cdr err))))))

(defun pi-coding-agent--send-extension-ui-response (process response)
"Send extension UI RESPONSE to pi PROCESS.
Expand Down
181 changes: 155 additions & 26 deletions pi-coding-agent-input.el
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
;;
;; Key entry points:
;; `pi-coding-agent-send' Send prompt (C-c C-c)
;; `pi-coding-agent-attach-image' Attach one prompt image (C-c C-a)
;; `pi-coding-agent-abort' Abort current operation (C-c C-k)
;; `pi-coding-agent-quit' Close session
;; `pi-coding-agent-previous-input' History backward (M-p)
Expand Down Expand Up @@ -286,14 +287,120 @@ markup visibility, mode identity, and keybindings. Set
(pi-coding-agent--call-in-visible-chat-window
#'pi-coding-agent-previous-message))

;;;; Prompt Images

(defun pi-coding-agent--prompt-image-byte-limit ()
"Return the configured nonnegative byte limit for a prompt image."
(if (natnump pi-coding-agent-prompt-image-max-bytes)
pi-coding-agent-prompt-image-max-bytes
(* 3 1024 1024)))

(defun pi-coding-agent--sniff-prompt-image-mime-type (data)
"Return the supported MIME type sniffed from unibyte DATA, or nil."
(let ((length (length data)))
(cond
((and (>= length 8)
(= (aref data 0) #x89)
(equal (substring data 1 8) "PNG\r\n\x1a\n"))
"image/png")
((and (>= length 3)
(= (aref data 0) #xff)
(= (aref data 1) #xd8)
(= (aref data 2) #xff))
"image/jpeg")
((and (>= length 6)
(member (substring data 0 6) '("GIF87a" "GIF89a")))
"image/gif")
((and (>= length 12)
(equal (substring data 0 4) "RIFF")
(equal (substring data 8 12) "WEBP"))
"image/webp"))))

(defun pi-coding-agent--read-prompt-image (path)
"Read and materialize supported prompt image PATH.
The file is read literally through Emacs, including through file-name
handlers, and is never handed to the Pi process as a path."
(let* ((path (pi-coding-agent--route-preserving-expand-file-name path))
(limit (pi-coding-agent--prompt-image-byte-limit))
(attributes (file-attributes path 'string))
(reported-size (and attributes (file-attribute-size attributes))))
(unless (and attributes (file-regular-p path) (file-readable-p path))
(user-error "Prompt image is not a readable regular file: %s" path))
(when (> reported-size limit)
(user-error "Prompt image is too large (%s; limit %s)"
(file-size-human-readable reported-size 'iec " " "B")
(file-size-human-readable limit 'iec " " "B")))
(let ((data (with-temp-buffer
(set-buffer-multibyte nil)
(let ((coding-system-for-read 'no-conversion))
(insert-file-contents-literally
path nil 0
(and (< limit most-positive-fixnum) (1+ limit))))
(buffer-string))))
(when (> (length data) limit)
(user-error "Prompt image exceeds the %s byte limit"
(file-size-human-readable limit 'iec " " "B")))
(let ((mime-type (pi-coding-agent--sniff-prompt-image-mime-type data)))
(unless mime-type
(user-error "Unsupported prompt image format: %s" path))
(pi-coding-agent--make-prompt-image
:name (file-name-nondirectory path)
:mime-type mime-type
:byte-size (length data)
:data (base64-encode-string data t))))))

;;;###autoload
(defun pi-coding-agent-attach-image (&optional clear)
"Attach one materialized image to the current prompt draft.
With prefix argument CLEAR, remove the attached image instead. A new image
replaces the previous draft image."
(interactive "P")
(let ((input-buffer (pi-coding-agent--get-input-buffer)))
(unless (buffer-live-p input-buffer)
(user-error "No pi input buffer for this command"))
(with-current-buffer input-buffer
(let ((chat-buf (pi-coding-agent--get-chat-buffer)))
(when (and (buffer-live-p chat-buf)
(with-current-buffer chat-buf
(pi-coding-agent--prompt-start-wait-active-p)))
(user-error
"Cannot change prompt image while prompt acceptance is pending"))
(if clear
(progn
(pi-coding-agent--clear-prompt-image)
(message "Pi: Prompt image cleared"))
(let* ((path (read-file-name "Attach prompt image: " nil nil t))
(image (pi-coding-agent--read-prompt-image path)))
(pi-coding-agent--set-prompt-image image)
(message "Pi: Attached image %s"
(pi-coding-agent--prompt-image-name image))))))))

(defun pi-coding-agent--model-supports-image-input-p (chat-buffer)
"Return non-nil only when CHAT-BUFFER's model advertises image input."
(let* ((state (and (buffer-live-p chat-buffer)
(buffer-local-value 'pi-coding-agent--state chat-buffer)))
(model (and (listp state) (plist-get state :model))))
(condition-case nil
(and (listp model)
(plist-member model :input)
(let ((input (plist-get model :input)))
(and (or (vectorp input) (listp input))
(member "image" (if (vectorp input)
(append input nil)
input))
t)))
(error nil))))

;;;; Sending Prompts

(defun pi-coding-agent--accept-input-text (text)
"Accept TEXT from input buffer state.
Adds TEXT to history, resets history navigation, and clears input."
(defun pi-coding-agent--accept-input-text (text &optional prompt-image)
"Accept TEXT from input buffer state, consuming optional PROMPT-IMAGE.
Adds only TEXT to history, resets history navigation, and clears input."
(pi-coding-agent--history-add text)
(setq pi-coding-agent--input-ring-index nil
pi-coding-agent--input-saved nil)
(when prompt-image
(pi-coding-agent--clear-prompt-image))
(erase-buffer))

(defun pi-coding-agent--queue-followup-text (chat-buf text)
Expand All @@ -306,26 +413,45 @@ Adds TEXT to history, resets history navigation, and clears input."
"Send the current input buffer contents to pi.
Clears the input buffer after sending. Does nothing if buffer is empty.
If pi is busy (sending, streaming, or compacting), queues a local follow-up.
An attached image is accepted only with a direct, ordinary, idle prompt.
All built-in slash commands are handled locally; other slash commands are
sent to pi."
(interactive)
(let* ((text (string-trim (buffer-string)))
(chat-buf (pi-coding-agent--get-chat-buffer))
(prompt-image (pi-coding-agent--get-prompt-image))
(transitioning (and chat-buf
(pi-coding-agent--session-transition-active-p
chat-buf)))
(busy (and chat-buf (pi-coding-agent--session-busy-p chat-buf))))
(cond
((string-empty-p text) nil)
((string-empty-p text)
(when prompt-image
(message "Pi: Add prompt text before sending the attached image")))
(transitioning
(message "Pi: Cannot send while session is switching"))
((and prompt-image
(pi-coding-agent--model-change-pending-p chat-buf))
(message "Pi: Wait for the pending model change before sending an image"))
((and prompt-image busy)
(message "Pi: Cannot send an attached image while Pi is busy"))
((and prompt-image (string-prefix-p "/" text))
(message "Pi: Attached images cannot be sent with slash commands"))
((and prompt-image
(not (pi-coding-agent--model-supports-image-input-p chat-buf)))
(message "Pi: Current model does not support known image input"))
((and busy (pi-coding-agent--builtin-command-text-p text))
(message "Pi: Cannot queue /%s while Pi is busy"
(pi-coding-agent--builtin-command-name text)))
(busy
(pi-coding-agent--queue-followup-text chat-buf text)
(pi-coding-agent--maybe-hide-input-window)
(message "Pi: Message queued (will send when Pi is ready)"))
(prompt-image
(pi-coding-agent--accept-input-text text prompt-image)
(pi-coding-agent--maybe-hide-input-window)
(with-current-buffer chat-buf
(pi-coding-agent--prepare-and-send text nil prompt-image)))
(t
(pi-coding-agent--accept-input-text text)
(pi-coding-agent--maybe-hide-input-window)
Expand Down Expand Up @@ -566,30 +692,33 @@ assistant output completes).

When compaction is in progress, steering text is queued as a local
follow-up. It is sent after non-retry compaction, or after Pi's
automatic overflow retry turn finishes."
automatic overflow retry turn finishes. Steering refuses a draft image."
(interactive)
(let ((text (string-trim (buffer-string))))
(unless (string-empty-p text)
(let ((chat-buf (pi-coding-agent--get-chat-buffer)))
(when chat-buf
(let ((status (buffer-local-value 'pi-coding-agent--status chat-buf)))
(cond
((pi-coding-agent--session-transition-active-p chat-buf)
(message "Pi: Cannot send steering while session is switching"))
((and (eq status 'idle)
(not (pi-coding-agent--session-busy-p chat-buf)))
(message "Pi: Nothing to interrupt - use C-c C-c to send"))
((or (eq status 'compacting)
(and (eq status 'idle)
(pi-coding-agent--session-busy-p chat-buf)))
(pi-coding-agent--queue-followup-text chat-buf text)
(message "Pi: Steering queued (will send when Pi is ready)"))
((memq status '(sending streaming))
(when (pi-coding-agent--send-steer-message text)
(pi-coding-agent--accept-input-text text)
(message "Pi: Steering message sent")))
(t
(message "Pi: Cannot steer while session status is %s" status)))))))))
(if (pi-coding-agent--get-prompt-image)
(message "Pi: Cannot steer with an attached image")
(unless (string-empty-p text)
(let ((chat-buf (pi-coding-agent--get-chat-buffer)))
(when chat-buf
(let ((status (buffer-local-value 'pi-coding-agent--status chat-buf)))
(cond
((pi-coding-agent--session-transition-active-p chat-buf)
(message "Pi: Cannot send steering while session is switching"))
((and (eq status 'idle)
(not (pi-coding-agent--session-busy-p chat-buf)))
(message "Pi: Nothing to interrupt - use C-c C-c to send"))
((or (eq status 'compacting)
(and (eq status 'idle)
(pi-coding-agent--session-busy-p chat-buf)))
(pi-coding-agent--queue-followup-text chat-buf text)
(message "Pi: Steering queued (will send when Pi is ready)"))
((memq status '(sending streaming))
(when (pi-coding-agent--send-steer-message text)
(pi-coding-agent--accept-input-text text)
(message "Pi: Steering message sent")))
(t
(message "Pi: Cannot steer while session status is %s"
status))))))))))

(defun pi-coding-agent-queue-followup ()
"Queue current input as a follow-up message.
Expand Down
Loading
Loading