diff --git a/README.org b/README.org index 865104f..61eea54 100644 --- a/README.org +++ b/README.org @@ -130,8 +130,9 @@ two windows: The input header line shows the current model, thinking level, activity phase, cost and context usage, session name, and extension -status when available. The model and thinking fields can be clicked -to change their values. +status when available. When a prompt image is attached, it also shows +the image's basename and source size. The model and thinking fields +can be clicked to change their values. Type in the input buffer and press =C-c C-c= to send. If Pi is already working, =C-c C-c= queues the text as a follow-up and sends it @@ -154,6 +155,27 @@ registers, spell checks, Evil, or whatever else your setup already gives you. Your prompt stays in the bottom window while the conversation streams above it. +Press =C-c C-a= to attach one image: select a file, or paste its path +into the file prompt. Attaching another image replaces the first; +=C-u C-c C-a= clears it. The header line keeps the attached basename +and size visible. The file is read and materialized when attached, +and PNG, JPEG, GIF, and WebP are recognized from their content rather +than their extension. The default source limit is 3 MiB. Send it +with a nonempty ordinary prompt while using a vision-capable model. + +Image-bearing drafts can be sent only as direct prompts while Pi is +idle. They are not queued while Pi is busy, used as steering +messages, or combined with slash commands. A refusal preserves both +prompt text and image. Image bytes are sent as attached, without +resizing or format conversion. +Clipboard image extraction, automatic detection of paths typed into +the prompt, and multiple attachments are deferred. + +Image files are read by Emacs, so an Emacs-readable path may use a +file-name handler such as TRAMP. Pi receives encoded bytes, not the +path. Remote reads still depend on the configured handler and are not +broadly tested. + Slash commands work with completion: type =/= then =TAB= to complete built-in commands and pi commands such as prompt templates, skills, and extension commands. Prompt templates are discovered from places @@ -178,19 +200,19 @@ tool block to expand or collapse it. Long-running commands stream output live, file operations (=read=, =write=, =edit=) get syntax highlighting, and edit diffs highlight what changed. -Image content returned by completed tool results is shown inline in graphical -Emacs and as a useful type-and-size placeholder in terminals. Pi's built-in -=read= already returns raster images as image content. For SVG, the preview is -made only from complete, standalone SVG text returned by =read=; pi-coding-agent -never reopens the argument path, and text with obvious scripts or external -resources is left as text. Previews are *display-only*: outgoing prompts have -no image field, so prompt image attachments remain separate -[[https://github.com/dnouri/pi-coding-agent/issues/261][issue #261]] work. +Image content in sent user turns and completed tool results is shown inline in +graphical Emacs and as a useful type-and-size placeholder in terminals. Sent +and returned image content share the same bounded renderer. Pi's built-in +=read= already returns raster images as image content. For SVG, a returned +preview is made only from complete, standalone SVG text supplied by =read=; +pi-coding-agent never reopens the argument path, and text with obvious scripts +or external resources is left as text. Current limits: images in partial tool updates appear when the final result -arrives; custom tools can return animated or highly compressed data whose -decoder cost is not bounded by the source-byte cap (Pi's built-in =read= -resizes raster images to at most 2000x2000). Moving a terminal-rendered chat to +arrives; unresized prompt images and custom tools can supply animated or highly +compressed data whose decoder cost is not bounded by the source-byte cap (Pi's +built-in =read= resizes raster images to at most 2000x2000). Moving a +terminal-rendered chat to a GUI, resizing previews, or applying a later extension replacement may require a toggle or history reload. @@ -266,6 +288,8 @@ configured warning and error thresholds. | Key | Context | Description | |------------------+---------+------------------------------------------------| | =C-c C-c= | input | ๐Ÿ“ฎ Send prompt, or queue follow-up if busy | +| =C-c C-a= | input | ๐Ÿ–ผ๏ธ Attach or replace one prompt image | +| =C-u C-c C-a= | input | ๐Ÿงน Clear the attached prompt image | | =C-c C-s= | input | ๐ŸŽ Send steering message while Pi is busy | | =C-c C-k= | input | ๐Ÿช“ Abort current response or compaction | | =C-c C-p= | input | ๐ŸŽ›๏ธ Open transient menu | @@ -504,8 +528,11 @@ Less common tuning knobs: ;; (setopt pi-coding-agent-tool-preview-lines 20) ;; (setopt pi-coding-agent-bash-preview-lines 10) +;; Lower the 3 MiB source limit for an outgoing prompt image: +;; (setopt pi-coding-agent-prompt-image-max-bytes (* 2 1024 1024)) + ;; Cap inline image previews to 640 pixels as well as the chat window width; -;; lower the 10 MiB per-image returned-source limit if desired: +;; lower the 10 MiB per-image preview-source limit if desired: ;; (setopt pi-coding-agent-image-preview-max-width 640) ;; (setopt pi-coding-agent-image-preview-max-bytes (* 5 1024 1024)) diff --git a/pi-coding-agent-browse.el b/pi-coding-agent-browse.el index b474278..0b98ea4 100644 --- a/pi-coding-agent-browse.el +++ b/pi-coding-agent-browse.el @@ -2202,9 +2202,7 @@ blocks sending until the switch settles, so the text cannot leak into the outgoing session. Failures are non-fatal." (when (buffer-live-p input-buf) (condition-case err - (with-current-buffer input-buf - (erase-buffer) - (when text (insert text))) + (pi-coding-agent--replace-input-draft input-buf text) (error (message "Pi: Failed to prefill prompt - %s" (error-message-string err)))))) diff --git a/pi-coding-agent-core.el b/pi-coding-agent-core.el index e1183dd..d328a6a 100644 --- a/pi-coding-agent-core.el +++ b/pi-coding-agent-core.el @@ -466,15 +466,24 @@ Maps request IDs to command type strings." (defun pi-coding-agent--rpc-async (process command callback) "Send COMMAND to pi PROCESS asynchronously. COMMAND is a plist that will be augmented with a unique ID. -CALLBACK is called with the response plist when received." +CALLBACK is called with the response plist when received. +Encoding or scheduling failures leave no pending request behind." (let* ((id (pi-coding-agent--next-request-id)) (full-command (plist-put (copy-sequence command) :id id)) + ;; Encode before registration so serialization failures cannot create + ;; pending state that no response could ever resolve. + (encoded-command (pi-coding-agent--encode-command full-command)) (pending (pi-coding-agent--get-pending-requests process)) (pending-types (pi-coding-agent--get-pending-command-types process))) - (puthash id callback pending) - (puthash id (plist-get command :type) pending-types) - (pi-coding-agent--send-string - process (pi-coding-agent--encode-command full-command)))) + (condition-case err + (progn + (puthash id callback pending) + (puthash id (plist-get command :type) pending-types) + (pi-coding-agent--send-string process encoded-command)) + ((error quit) + (remhash id pending) + (remhash id pending-types) + (signal (car err) (cdr err)))))) (defun pi-coding-agent--send-extension-ui-response (process response) "Send extension UI RESPONSE to pi PROCESS. diff --git a/pi-coding-agent-input.el b/pi-coding-agent-input.el index e5bdb78..a4cf628 100644 --- a/pi-coding-agent-input.el +++ b/pi-coding-agent-input.el @@ -33,6 +33,7 @@ ;; ;; Key entry points: ;; `pi-coding-agent-send' Send prompt (C-c C-c) +;; `pi-coding-agent-attach-image' Attach one prompt image (C-c C-a) ;; `pi-coding-agent-abort' Abort current operation (C-c C-k) ;; `pi-coding-agent-quit' Close session ;; `pi-coding-agent-previous-input' History backward (M-p) @@ -286,14 +287,120 @@ markup visibility, mode identity, and keybindings. Set (pi-coding-agent--call-in-visible-chat-window #'pi-coding-agent-previous-message)) +;;;; Prompt Images + +(defun pi-coding-agent--prompt-image-byte-limit () + "Return the configured nonnegative byte limit for a prompt image." + (if (natnump pi-coding-agent-prompt-image-max-bytes) + pi-coding-agent-prompt-image-max-bytes + (* 3 1024 1024))) + +(defun pi-coding-agent--sniff-prompt-image-mime-type (data) + "Return the supported MIME type sniffed from unibyte DATA, or nil." + (let ((length (length data))) + (cond + ((and (>= length 8) + (= (aref data 0) #x89) + (equal (substring data 1 8) "PNG\r\n\x1a\n")) + "image/png") + ((and (>= length 3) + (= (aref data 0) #xff) + (= (aref data 1) #xd8) + (= (aref data 2) #xff)) + "image/jpeg") + ((and (>= length 6) + (member (substring data 0 6) '("GIF87a" "GIF89a"))) + "image/gif") + ((and (>= length 12) + (equal (substring data 0 4) "RIFF") + (equal (substring data 8 12) "WEBP")) + "image/webp")))) + +(defun pi-coding-agent--read-prompt-image (path) + "Read and materialize supported prompt image PATH. +The file is read literally through Emacs, including through file-name +handlers, and is never handed to the Pi process as a path." + (let* ((path (pi-coding-agent--route-preserving-expand-file-name path)) + (limit (pi-coding-agent--prompt-image-byte-limit)) + (attributes (file-attributes path 'string)) + (reported-size (and attributes (file-attribute-size attributes)))) + (unless (and attributes (file-regular-p path) (file-readable-p path)) + (user-error "Prompt image is not a readable regular file: %s" path)) + (when (> reported-size limit) + (user-error "Prompt image is too large (%s; limit %s)" + (file-size-human-readable reported-size 'iec " " "B") + (file-size-human-readable limit 'iec " " "B"))) + (let ((data (with-temp-buffer + (set-buffer-multibyte nil) + (let ((coding-system-for-read 'no-conversion)) + (insert-file-contents-literally + path nil 0 + (and (< limit most-positive-fixnum) (1+ limit)))) + (buffer-string)))) + (when (> (length data) limit) + (user-error "Prompt image exceeds the %s byte limit" + (file-size-human-readable limit 'iec " " "B"))) + (let ((mime-type (pi-coding-agent--sniff-prompt-image-mime-type data))) + (unless mime-type + (user-error "Unsupported prompt image format: %s" path)) + (pi-coding-agent--make-prompt-image + :name (file-name-nondirectory path) + :mime-type mime-type + :byte-size (length data) + :data (base64-encode-string data t)))))) + +;;;###autoload +(defun pi-coding-agent-attach-image (&optional clear) + "Attach one materialized image to the current prompt draft. +With prefix argument CLEAR, remove the attached image instead. A new image +replaces the previous draft image." + (interactive "P") + (let ((input-buffer (pi-coding-agent--get-input-buffer))) + (unless (buffer-live-p input-buffer) + (user-error "No pi input buffer for this command")) + (with-current-buffer input-buffer + (let ((chat-buf (pi-coding-agent--get-chat-buffer))) + (when (and (buffer-live-p chat-buf) + (with-current-buffer chat-buf + (pi-coding-agent--prompt-start-wait-active-p))) + (user-error + "Cannot change prompt image while prompt acceptance is pending")) + (if clear + (progn + (pi-coding-agent--clear-prompt-image) + (message "Pi: Prompt image cleared")) + (let* ((path (read-file-name "Attach prompt image: " nil nil t)) + (image (pi-coding-agent--read-prompt-image path))) + (pi-coding-agent--set-prompt-image image) + (message "Pi: Attached image %s" + (pi-coding-agent--prompt-image-name image)))))))) + +(defun pi-coding-agent--model-supports-image-input-p (chat-buffer) + "Return non-nil only when CHAT-BUFFER's model advertises image input." + (let* ((state (and (buffer-live-p chat-buffer) + (buffer-local-value 'pi-coding-agent--state chat-buffer))) + (model (and (listp state) (plist-get state :model)))) + (condition-case nil + (and (listp model) + (plist-member model :input) + (let ((input (plist-get model :input))) + (and (or (vectorp input) (listp input)) + (member "image" (if (vectorp input) + (append input nil) + input)) + t))) + (error nil)))) + ;;;; Sending Prompts -(defun pi-coding-agent--accept-input-text (text) - "Accept TEXT from input buffer state. -Adds TEXT to history, resets history navigation, and clears input." +(defun pi-coding-agent--accept-input-text (text &optional prompt-image) + "Accept TEXT from input buffer state, consuming optional PROMPT-IMAGE. +Adds only TEXT to history, resets history navigation, and clears input." (pi-coding-agent--history-add text) (setq pi-coding-agent--input-ring-index nil pi-coding-agent--input-saved nil) + (when prompt-image + (pi-coding-agent--clear-prompt-image)) (erase-buffer)) (defun pi-coding-agent--queue-followup-text (chat-buf text) @@ -306,19 +413,33 @@ Adds TEXT to history, resets history navigation, and clears input." "Send the current input buffer contents to pi. Clears the input buffer after sending. Does nothing if buffer is empty. If pi is busy (sending, streaming, or compacting), queues a local follow-up. +An attached image is accepted only with a direct, ordinary, idle prompt. All built-in slash commands are handled locally; other slash commands are sent to pi." (interactive) (let* ((text (string-trim (buffer-string))) (chat-buf (pi-coding-agent--get-chat-buffer)) + (prompt-image (pi-coding-agent--get-prompt-image)) (transitioning (and chat-buf (pi-coding-agent--session-transition-active-p chat-buf))) (busy (and chat-buf (pi-coding-agent--session-busy-p chat-buf)))) (cond - ((string-empty-p text) nil) + ((string-empty-p text) + (when prompt-image + (message "Pi: Add prompt text before sending the attached image"))) (transitioning (message "Pi: Cannot send while session is switching")) + ((and prompt-image + (pi-coding-agent--model-change-pending-p chat-buf)) + (message "Pi: Wait for the pending model change before sending an image")) + ((and prompt-image busy) + (message "Pi: Cannot send an attached image while Pi is busy")) + ((and prompt-image (string-prefix-p "/" text)) + (message "Pi: Attached images cannot be sent with slash commands")) + ((and prompt-image + (not (pi-coding-agent--model-supports-image-input-p chat-buf))) + (message "Pi: Current model does not support known image input")) ((and busy (pi-coding-agent--builtin-command-text-p text)) (message "Pi: Cannot queue /%s while Pi is busy" (pi-coding-agent--builtin-command-name text))) @@ -326,6 +447,11 @@ sent to pi." (pi-coding-agent--queue-followup-text chat-buf text) (pi-coding-agent--maybe-hide-input-window) (message "Pi: Message queued (will send when Pi is ready)")) + (prompt-image + (pi-coding-agent--accept-input-text text prompt-image) + (pi-coding-agent--maybe-hide-input-window) + (with-current-buffer chat-buf + (pi-coding-agent--prepare-and-send text nil prompt-image))) (t (pi-coding-agent--accept-input-text text) (pi-coding-agent--maybe-hide-input-window) @@ -566,30 +692,33 @@ assistant output completes). When compaction is in progress, steering text is queued as a local follow-up. It is sent after non-retry compaction, or after Pi's -automatic overflow retry turn finishes." +automatic overflow retry turn finishes. Steering refuses a draft image." (interactive) (let ((text (string-trim (buffer-string)))) - (unless (string-empty-p text) - (let ((chat-buf (pi-coding-agent--get-chat-buffer))) - (when chat-buf - (let ((status (buffer-local-value 'pi-coding-agent--status chat-buf))) - (cond - ((pi-coding-agent--session-transition-active-p chat-buf) - (message "Pi: Cannot send steering while session is switching")) - ((and (eq status 'idle) - (not (pi-coding-agent--session-busy-p chat-buf))) - (message "Pi: Nothing to interrupt - use C-c C-c to send")) - ((or (eq status 'compacting) - (and (eq status 'idle) - (pi-coding-agent--session-busy-p chat-buf))) - (pi-coding-agent--queue-followup-text chat-buf text) - (message "Pi: Steering queued (will send when Pi is ready)")) - ((memq status '(sending streaming)) - (when (pi-coding-agent--send-steer-message text) - (pi-coding-agent--accept-input-text text) - (message "Pi: Steering message sent"))) - (t - (message "Pi: Cannot steer while session status is %s" status))))))))) + (if (pi-coding-agent--get-prompt-image) + (message "Pi: Cannot steer with an attached image") + (unless (string-empty-p text) + (let ((chat-buf (pi-coding-agent--get-chat-buffer))) + (when chat-buf + (let ((status (buffer-local-value 'pi-coding-agent--status chat-buf))) + (cond + ((pi-coding-agent--session-transition-active-p chat-buf) + (message "Pi: Cannot send steering while session is switching")) + ((and (eq status 'idle) + (not (pi-coding-agent--session-busy-p chat-buf))) + (message "Pi: Nothing to interrupt - use C-c C-c to send")) + ((or (eq status 'compacting) + (and (eq status 'idle) + (pi-coding-agent--session-busy-p chat-buf))) + (pi-coding-agent--queue-followup-text chat-buf text) + (message "Pi: Steering queued (will send when Pi is ready)")) + ((memq status '(sending streaming)) + (when (pi-coding-agent--send-steer-message text) + (pi-coding-agent--accept-input-text text) + (message "Pi: Steering message sent"))) + (t + (message "Pi: Cannot steer while session status is %s" + status)))))))))) (defun pi-coding-agent-queue-followup () "Queue current input as a follow-up message. diff --git a/pi-coding-agent-menu.el b/pi-coding-agent-menu.el index e9165e6..13528f9 100644 --- a/pi-coding-agent-menu.el +++ b/pi-coding-agent-menu.el @@ -216,6 +216,9 @@ and another transition may not be discarded." ((pi-coding-agent--prompt-start-wait-active-p) (message "Pi: Cannot start a new session while prompt acceptance is pending") nil) + ((pi-coding-agent--model-change-pending-p) + (message "Pi: Cannot start a new session while a model change is pending") + nil) ((or pi-coding-agent--followup-queue (pi-coding-agent--followup-drain-pending-p)) (message "Pi: Cannot start a new session with queued follow-ups") @@ -365,6 +368,7 @@ Call this when starting a new session to ensure no stale state persists." pi-coding-agent--thinking-block-order-counter 0) (pi-coding-agent--set-activity-phase "idle" 'reset t) (pi-coding-agent--clear-local-user-message-region) + (pi-coding-agent--invalidate-model-change) (pi-coding-agent--clear-unsupported-extension-ui-warnings) (pi-coding-agent--invalidate-history-loads) (pi-coding-agent--finish-session-transition @@ -551,6 +555,8 @@ buffer from session history." ((not session-file) (message "Pi: No session file available - cannot reload")) (t + (with-current-buffer chat-buf + (pi-coding-agent--cancel-model-change-and-restore-followups)) (message "Pi: Reloading...") (with-current-buffer chat-buf (let ((dir (pi-coding-agent--session-directory))) @@ -722,7 +728,11 @@ Optional INITIAL-INPUT pre-fills the completion prompt for filtering." (chat-buf (pi-coding-agent--get-chat-buffer))) (unless proc (user-error "No pi process running")) - (let* ((state (pi-coding-agent--menu-state)) + (when (pi-coding-agent--model-change-pending-p chat-buf) + (user-error "A model change is already pending")) + (when (pi-coding-agent--session-transition-ready-p + chat-buf "change models") + (let* ((state (pi-coding-agent--menu-state)) (response (pi-coding-agent--rpc-sync proc '(:type "get_available_models") 5)) (data (plist-get response :data)) (models (plist-get data :models)) @@ -764,20 +774,48 @@ Optional INITIAL-INPUT pre-fills the completion prompt for filtering." (format "Model (current: %s): " (or current-display "unknown")) names nil t))))) - (when (and choice (not (equal choice current-display))) + (when (and choice + (not (equal choice current-display)) + (pi-coding-agent--session-transition-ready-p + chat-buf "change models")) (let* ((selected-model (cdr (assoc choice model-alist))) (model-id (plist-get selected-model :id)) - (provider (plist-get selected-model :provider))) - (pi-coding-agent--rpc-async proc (list :type "set_model" - :provider provider - :modelId model-id) - (lambda (resp) - (when (and (eq (plist-get resp :success) t) - (buffer-live-p chat-buf)) + (provider (plist-get selected-model :provider)) + (token (pi-coding-agent--begin-model-change proc chat-buf))) + (if (not token) + (message "Pi: Process changed while selecting a model; try again") + (condition-case err + (pi-coding-agent--rpc-async + proc (list :type "set_model" + :provider provider + :modelId model-id) + (lambda (resp) + (when (pi-coding-agent--model-change-current-p token chat-buf) + (let ((success (eq (plist-get resp :success) t)) + (applied nil)) + (unwind-protect + (when success (with-current-buffer chat-buf (pi-coding-agent--update-state-from-response resp) (force-mode-line-update)) - (message "Pi: Model set to %s" choice))))))))) + (setq applied t)) + (when (pi-coding-agent--finish-model-change + token chat-buf) + (when (buffer-live-p chat-buf) + (with-current-buffer chat-buf + (if applied + (pi-coding-agent--schedule-followup-queue-processing) + (pi-coding-agent--restore-followup-queue-to-input)))) + (cond + (applied + (message "Pi: Model set to %s" choice)) + ((not success) + (message "Pi: Failed to set model: %s" + (or (plist-get resp :error) + "unknown error")))))))))) + ((error quit) + (pi-coding-agent--finish-model-change token chat-buf) + (signal (car err) (cdr err))))))))))) (defun pi-coding-agent--thinking-level-effective-value (level model) "Return LEVEL's provider value for MODEL. @@ -1217,9 +1255,7 @@ Captures chat and input buffers at call time (before the async RPC)." (when refresh-scheduled (condition-case err (when (buffer-live-p input-buf) - (with-current-buffer input-buf - (erase-buffer) - (when text (insert text)))) + (pi-coding-agent--replace-input-draft input-buf text)) (error (message "Pi: Failed to prefill fork prompt - %s" (error-message-string err)))))) diff --git a/pi-coding-agent-render.el b/pi-coding-agent-render.el index 17075c6..c77163b 100644 --- a/pi-coding-agent-render.el +++ b/pi-coding-agent-render.el @@ -103,15 +103,18 @@ call ID in `pi-coding-agent--live-tool-blocks'.") (unless (pi-coding-agent--history-postprocessing-deferred-p) (pi-coding-agent--decorate-tables-in-region start end))) -(defun pi-coding-agent--display-user-message (text &optional timestamp track-region) - "Display user message TEXT in the chat buffer. +(defun pi-coding-agent--display-user-message + (text &optional timestamp content track-region) + "Display user message TEXT and optional image CONTENT in the chat buffer. If TIMESTAMP (Emacs time value) is provided, display it in the header. When TRACK-REGION is non-nil, return a marker pair bounding the inserted turn." (let* ((chat-buffer (pi-coding-agent--get-chat-buffer)) - (start (with-current-buffer chat-buffer (point-max)))) + (start (with-current-buffer chat-buffer (point-max))) + (previews (pi-coding-agent--content-image-previews content))) (pi-coding-agent--append-to-chat (concat "\n" (pi-coding-agent--make-separator "You" timestamp) "\n" - text "\n")) + (or text "") "\n" + (or (pi-coding-agent--image-previews-text previews) ""))) (with-current-buffer chat-buffer (pi-coding-agent--decorate-tables-unless-deferred start (point-max)) (when track-region @@ -137,6 +140,25 @@ TRACK-REGION is non-nil, return a marker pair bounding the inserted turn." (pi-coding-agent--discard-local-user-message) (pi-coding-agent--schedule-followup-queue-processing))) +(defun pi-coding-agent--content-has-image-p (content) + "Return non-nil if CONTENT has an image block." + (seq-some (lambda (block) + (equal (plist-get block :type) "image")) + (pi-coding-agent--content-block-list content))) + +(defun pi-coding-agent--local-user-message-matches-p + (local-message text content) + "Return non-nil when LOCAL-MESSAGE exactly represents TEXT and CONTENT. +Strings retain the existing text-only echo contract. Image turns use their +full content vector, so an authoritative image transformation cannot be lost." + (cond + ((vectorp local-message) + (and (vectorp content) (equal local-message content))) + ((stringp local-message) + (and text + (not (pi-coding-agent--content-has-image-p content)) + (string= text local-message))))) + (defun pi-coding-agent--display-agent-start () "Display separator for new agent turn. Only shows the Assistant header once per prompt, even during retries. @@ -654,6 +676,7 @@ follow-up as a fresh prompt.") "Return non-nil when a queued follow-up may become the next prompt." (and pi-coding-agent--followup-queue (eq pi-coding-agent--status 'idle) + (not (pi-coding-agent--model-change-pending-p)) (not (pi-coding-agent--session-transition-active-p)) (not (pi-coding-agent--prompt-start-wait-active-p)) (null pi-coding-agent--local-user-message))) @@ -735,14 +758,15 @@ Returns non-nil if TEXT matched a built-in command and was handled." (_ (funcall handler))) t))))) -(defun pi-coding-agent--prepare-and-send (text &optional queued) - "Prepare chat buffer state and send TEXT to pi. +(defun pi-coding-agent--prepare-and-send (text &optional queued prompt-image) + "Prepare chat buffer state and send TEXT with optional PROMPT-IMAGE to pi. Built-in slash commands are dispatched locally via the dispatch table. Other slash commands (extensions, skills, prompts) are sent to pi without local transcript display. Regular text is displayed after prompt preflight accepts it. When QUEUED is non-nil, TEXT is the oldest local follow-up and is removed -from the queue only after prompt preflight succeeds. +from the queue only after prompt preflight succeeds. PROMPT-IMAGE is only +valid for a direct regular prompt. Must be called with chat buffer current. Pi events own streaming/idle turn transitions; prompt submission marks the local pre-event window as busy." (pi-coding-agent--invalidate-history-loads) @@ -769,13 +793,30 @@ transitions; prompt submission marks the local pre-event window as busy." ;; Regular text is displayed only after prompt preflight accepts it. That ;; keeps rejected prompts out of the transcript and lets us restore them to ;; the input buffer for user recovery. + (prompt-image + (let* ((image-block + (pi-coding-agent--prompt-image-content-block prompt-image)) + (user-content + (vector (list :type "text" :text text) image-block))) + (pi-coding-agent--send-prompt + text + (lambda () + (setq pi-coding-agent--local-user-message-region + (pi-coding-agent--display-user-message + text (current-time) user-content t)) + (setq pi-coding-agent--local-user-message user-content) + (setq pi-coding-agent--assistant-header-shown nil)) + (lambda () (pi-coding-agent--restore-input-text text prompt-image)) + #'pi-coding-agent--handle-no-turn-local-prompt + prompt-image))) (queued (pi-coding-agent--send-prompt text (lambda () (when (pi-coding-agent--drop-followup text) (setq pi-coding-agent--local-user-message-region - (pi-coding-agent--display-user-message text (current-time) t)) + (pi-coding-agent--display-user-message + text (current-time) nil t)) (setq pi-coding-agent--local-user-message text) (setq pi-coding-agent--assistant-header-shown nil))) #'pi-coding-agent--restore-followup-queue-to-input @@ -785,7 +826,8 @@ transitions; prompt submission marks the local pre-event window as busy." text (lambda () (setq pi-coding-agent--local-user-message-region - (pi-coding-agent--display-user-message text (current-time) t)) + (pi-coding-agent--display-user-message + text (current-time) nil t)) (setq pi-coding-agent--local-user-message text) (setq pi-coding-agent--assistant-header-shown nil)) (lambda () (pi-coding-agent--restore-input-text text)) @@ -1020,10 +1062,7 @@ Include optional STDERR in a text fence and optional DETAIL before it." "Handle set_editor_text method from EVENT." (let ((text (plist-get event :text))) (when-let* ((input-buf pi-coding-agent--input-buffer)) - (when (buffer-live-p input-buf) - (with-current-buffer input-buf - (erase-buffer) - (insert text)))))) + (pi-coding-agent--replace-input-draft input-buf text)))) (defun pi-coding-agent--extension-ui-set-status (event) "Handle setStatus method from EVENT." @@ -1208,6 +1247,7 @@ which asks upfront before any buffers are touched." (when pi-coding-agent--tool-args-cache (clrhash pi-coding-agent--tool-args-cache)) (pi-coding-agent--set-process nil) + (pi-coding-agent--invalidate-model-change) (pi-coding-agent--set-activity-phase "idle") (setq pi-coding-agent--local-user-message nil) (pi-coding-agent--clear-local-user-message-region) @@ -1253,17 +1293,18 @@ Updates buffer-local state and renders display updates." (timestamp (plist-get message :timestamp)) (text (when content (pi-coding-agent--extract-user-message-text content))) + (has-images (pi-coding-agent--content-has-image-p content)) (local-msg pi-coding-agent--local-user-message)) - ;; Clear local tracking + ;; Clear local tracking before rendering the authoritative turn. (setq pi-coding-agent--local-user-message nil) (pi-coding-agent--clear-local-user-message-region) - ;; Display if: no local message, OR pi's message differs (expanded template) - (when (and text - (or (null local-msg) - (not (string= text local-msg)))) + (when (and (or text has-images) + (not (pi-coding-agent--local-user-message-matches-p + local-msg text content))) (pi-coding-agent--display-user-message text - (pi-coding-agent--ms-to-time timestamp)) + (pi-coding-agent--ms-to-time timestamp) + content) ;; Reset so next assistant message shows its header (setq pi-coding-agent--assistant-header-shown nil)))) ("custom" @@ -1876,8 +1917,8 @@ path/error metadata for `pi-coding-agent-visit-file'." :max-height (max 1 (truncate (* 0.5 (window-pixel-height window))))))) -(defconst pi-coding-agent--image-previews-per-tool-limit 8 - "Maximum tool-result image blocks rendered for one tool invocation.") +(defconst pi-coding-agent--image-previews-per-content-limit 8 + "Maximum image blocks rendered from one message or tool result.") (defun pi-coding-agent--image-preview-byte-limit () "Return the nonnegative source-byte limit for one image preview." @@ -1919,8 +1960,8 @@ path/error metadata for `pi-coding-agent-visit-file'." (image-type-from-data data) (error nil)))) -(defun pi-coding-agent--render-tool-image-preview (block) - "Return a rendered preview string for tool-result image BLOCK." +(defun pi-coding-agent--render-content-image-preview (block) + "Return a rendered preview string for image content BLOCK." (let* ((mime-type (or (plist-get block :mimeType) (plist-get block :mime-type))) (data (plist-get block :data)) @@ -1984,17 +2025,17 @@ path/error metadata for `pi-coding-agent-visit-file'." (pi-coding-agent--image-preview-string (pi-coding-agent--image-preview-label mime-type "decode error")))))))) -(defun pi-coding-agent--tool-result-image-previews (content-blocks) - "Render a bounded number of images from tool result CONTENT-BLOCKS." +(defun pi-coding-agent--content-image-previews (content) + "Render a bounded number of image blocks from vector or list CONTENT." (let* ((blocks (seq-filter (lambda (block) (equal (plist-get block :type) "image")) - content-blocks)) - (limit (max 0 pi-coding-agent--image-previews-per-tool-limit)) + (pi-coding-agent--content-block-list content))) + (limit (max 0 pi-coding-agent--image-previews-per-content-limit)) (shown (seq-take blocks limit)) (omitted (- (length blocks) (length shown)))) (append - (mapcar #'pi-coding-agent--render-tool-image-preview shown) + (mapcar #'pi-coding-agent--render-content-image-preview shown) (when (> omitted 0) (list (pi-coding-agent--image-preview-string @@ -3214,7 +3255,7 @@ if none exists, render the result at point without a live overlay." (plist-get c :text))) text-blocks "\n")) (content-image-previews - (pi-coding-agent--tool-result-image-previews content-blocks)) + (pi-coding-agent--content-image-previews content-blocks)) (svg-preview (and (null content-image-previews) (pi-coding-agent--read-svg-preview @@ -6820,10 +6861,14 @@ Tool calls are rendered with headers, output, overlays, and toggles." (role (plist-get message :role))) (pcase role ("user" - (let* ((text (pi-coding-agent--extract-history-user-message-text message)) - (timestamp (pi-coding-agent--ms-to-time (plist-get message :timestamp)))) - (when text - (pi-coding-agent--display-user-message text timestamp))) + (let* ((content (plist-get message :content)) + (text (pi-coding-agent--extract-history-user-message-text message)) + (has-images + (pi-coding-agent--content-has-image-p content)) + (timestamp + (pi-coding-agent--ms-to-time (plist-get message :timestamp)))) + (when (or text has-images) + (pi-coding-agent--display-user-message text timestamp content))) (setq prev-role "user")) ("assistant" (when (not (equal prev-role "assistant")) diff --git a/pi-coding-agent-ui.el b/pi-coding-agent-ui.el index ccba54d..95a5244 100644 --- a/pi-coding-agent-ui.el +++ b/pi-coding-agent-ui.el @@ -64,6 +64,7 @@ ;; pi-coding-agent-input.el (input buffer commands) (declare-function pi-coding-agent-quit "pi-coding-agent-input") (declare-function pi-coding-agent-send "pi-coding-agent-input") +(declare-function pi-coding-agent-attach-image "pi-coding-agent-input") (declare-function pi-coding-agent-abort "pi-coding-agent-input") (declare-function pi-coding-agent-previous-input "pi-coding-agent-input") (declare-function pi-coding-agent-next-input "pi-coding-agent-input") @@ -207,8 +208,13 @@ Previews are also constrained to the visible chat window." :group 'pi-coding-agent) (defcustom pi-coding-agent-image-preview-max-bytes (* 10 1024 1024) - "Maximum source bytes retained for one inline image preview. -Larger tool-result images use a textual placeholder." + "Maximum source bytes decoded for one inline image preview. +Larger user-message or tool-result images use a textual placeholder." + :type 'natnum + :group 'pi-coding-agent) + +(defcustom pi-coding-agent-prompt-image-max-bytes (* 3 1024 1024) + "Maximum source bytes for the image attached to a prompt draft." :type 'natnum :group 'pi-coding-agent) @@ -879,6 +885,8 @@ This is a read-only buffer showing the conversation history." (setq-local pi-coding-agent--history-load-generation 0) (setq-local pi-coding-agent--session-transition-generation 0) (setq-local pi-coding-agent--session-transition-active nil) + (setq-local pi-coding-agent--model-change-generation 0) + (setq-local pi-coding-agent--model-change-active-token nil) (setq-local pi-coding-agent--local-user-message-region nil) ;; Disable hl-line-mode: its post-command-hook overlay update causes ;; scroll oscillation in buffers with invisible text + variable heights. @@ -919,6 +927,7 @@ removing the instructional header that would otherwise appear." (defvar pi-coding-agent-input-mode-map (let ((map (make-sparse-keymap))) (define-key map (kbd "C-c C-c") #'pi-coding-agent-send) + (define-key map (kbd "C-c C-a") #'pi-coding-agent-attach-image) (define-key map (kbd "TAB") #'pi-coding-agent-complete) (define-key map (kbd "C-c C-k") #'pi-coding-agent-abort) (define-key map (kbd "C-c C-p") #'pi-coding-agent-menu) @@ -1069,10 +1078,89 @@ of the current session in the selected frame." (defvar-local pi-coding-agent--process-version nil "Detected pi CLI version for the current process.") +(defvar-local pi-coding-agent--model-change-generation 0 + "Monotonic generation for asynchronous model-change callbacks.") + +(defvar-local pi-coding-agent--model-change-active-token nil + "Process-bound token owned by the active model change, or nil.") + +(defun pi-coding-agent--begin-model-change (process &optional chat-buffer) + "Begin a model change through PROCESS in CHAT-BUFFER and return its token. +Return nil if PROCESS is no longer current. CHAT-BUFFER defaults to the +current buffer." + (let ((buffer (or chat-buffer (current-buffer)))) + (when (buffer-live-p buffer) + (with-current-buffer buffer + (when (eq process pi-coding-agent--process) + (setq pi-coding-agent--model-change-generation + (1+ (or pi-coding-agent--model-change-generation 0))) + (setq pi-coding-agent--model-change-active-token + (cons pi-coding-agent--model-change-generation process))))))) + +(defun pi-coding-agent--model-change-owned-p (token &optional chat-buffer) + "Return non-nil when TOKEN owns CHAT-BUFFER's model-change gate. +CHAT-BUFFER defaults to the current buffer." + (let ((buffer (or chat-buffer (current-buffer)))) + (and token + (buffer-live-p buffer) + (with-current-buffer buffer + (and (eq token pi-coding-agent--model-change-active-token) + (eql (car token) pi-coding-agent--model-change-generation)))))) + +(defun pi-coding-agent--model-change-current-p (token &optional chat-buffer) + "Return non-nil when TOKEN owns CHAT-BUFFER's current-process model change. +CHAT-BUFFER defaults to the current buffer." + (let ((buffer (or chat-buffer (current-buffer)))) + (and (pi-coding-agent--model-change-owned-p token buffer) + (with-current-buffer buffer + (eq (cdr token) pi-coding-agent--process))))) + +(defun pi-coding-agent--finish-model-change (token &optional chat-buffer) + "Finish CHAT-BUFFER's model change only when TOKEN still owns it. +Unlike applying its response, cleanup does not require TOKEN's process to +remain current. CHAT-BUFFER defaults to the current buffer." + (let ((buffer (or chat-buffer (current-buffer)))) + (when (pi-coding-agent--model-change-owned-p token buffer) + (with-current-buffer buffer + (setq pi-coding-agent--model-change-active-token nil)) + t))) + +(defun pi-coding-agent--invalidate-model-change (&optional chat-buffer) + "Invalidate any model change in CHAT-BUFFER and return the new generation. +CHAT-BUFFER defaults to the current buffer." + (let ((buffer (or chat-buffer (current-buffer)))) + (when (buffer-live-p buffer) + (with-current-buffer buffer + (setq pi-coding-agent--model-change-generation + (1+ (or pi-coding-agent--model-change-generation 0)) + pi-coding-agent--model-change-active-token nil) + pi-coding-agent--model-change-generation)))) + +(defun pi-coding-agent--model-change-pending-p (&optional chat-buffer) + "Return whether CHAT-BUFFER has an active model change. +CHAT-BUFFER defaults to the current buffer." + (let ((buffer (or chat-buffer (current-buffer)))) + (and (buffer-live-p buffer) + (buffer-local-value 'pi-coding-agent--model-change-active-token buffer) + t))) + +(defun pi-coding-agent--cancel-model-change-and-restore-followups + (&optional chat-buffer) + "Cancel CHAT-BUFFER's model change and restore text queued behind it." + (let ((buffer (or chat-buffer (current-buffer)))) + (when (and (buffer-live-p buffer) + (pi-coding-agent--model-change-pending-p buffer)) + (with-current-buffer buffer + (pi-coding-agent--invalidate-model-change) + (pi-coding-agent--restore-followup-queue-to-input)) + t))) + (defun pi-coding-agent--set-process (process) "Set the pi RPC subprocess PROCESS for this session. Resets cached process version and starts a delayed version probe for new live processes in interactive sessions." + (unless (eq process pi-coding-agent--process) + (pi-coding-agent--invalidate-model-change)) (setq pi-coding-agent--process process pi-coding-agent--process-version nil) (when (and (processp process) @@ -1335,6 +1423,52 @@ execution; this slot remains only for older single-tool flows.") "Non-nil if Assistant header has been shown for current prompt. Used to avoid duplicate headers during retry sequences.") +(cl-defstruct (pi-coding-agent--prompt-image + (:constructor pi-coding-agent--make-prompt-image)) + "Materialized image attached to one input-buffer prompt draft." + name + mime-type + byte-size + data) + +(defvar-local pi-coding-agent--draft-prompt-image nil + "Materialized prompt image attached to the current input draft.") + +(defun pi-coding-agent--get-prompt-image (&optional input-buffer) + "Return the draft prompt image in INPUT-BUFFER or the current buffer." + (let ((buffer (or input-buffer (current-buffer)))) + (when (buffer-live-p buffer) + (buffer-local-value 'pi-coding-agent--draft-prompt-image buffer)))) + +(defun pi-coding-agent--set-prompt-image (image &optional input-buffer) + "Set IMAGE as the draft prompt image in INPUT-BUFFER or current buffer." + (let ((buffer (or input-buffer (current-buffer)))) + (when (buffer-live-p buffer) + (with-current-buffer buffer + (setq pi-coding-agent--draft-prompt-image image) + (force-mode-line-update t))) + image)) + +(defun pi-coding-agent--clear-prompt-image (&optional input-buffer) + "Clear the draft prompt image in INPUT-BUFFER or the current buffer." + (pi-coding-agent--set-prompt-image nil input-buffer)) + +(defun pi-coding-agent--prompt-image-content-block (image) + "Return the RPC image content block for prompt IMAGE." + (list :type "image" + :data (pi-coding-agent--prompt-image-data image) + :mimeType (pi-coding-agent--prompt-image-mime-type image))) + +(defun pi-coding-agent--replace-input-draft (input-buffer text) + "Replace INPUT-BUFFER's draft with TEXT and clear its prompt image." + (when (buffer-live-p input-buffer) + (with-current-buffer input-buffer + (erase-buffer) + (when text + (insert text)) + (pi-coding-agent--clear-prompt-image) + (goto-char (point-max))))) + (defvar-local pi-coding-agent--followup-queue nil "List of follow-up messages queued while agent is busy. Messages are added when the user sends while streaming, compacting, or @@ -1364,8 +1498,8 @@ Follow-ups are processed in FIFO order: first pushed, first sent." "Return queued follow-up messages in the order they would be sent." (reverse pi-coding-agent--followup-queue)) -(defun pi-coding-agent--restore-input-text (text) - "Restore TEXT to the linked input buffer for user recovery. +(defun pi-coding-agent--restore-input-text (text &optional prompt-image) + "Restore TEXT and optional PROMPT-IMAGE to the linked input buffer. Recovered text is older than any draft currently in the input buffer, so it is placed first and separated from the draft by a blank line." (when-let* ((input-buf pi-coding-agent--input-buffer) @@ -1376,6 +1510,8 @@ placed first and separated from the draft by a blank line." (insert text) (unless (string-empty-p draft) (insert "\n\n" draft)) + (when prompt-image + (pi-coding-agent--set-prompt-image prompt-image)) (goto-char (point-max)))))) (defun pi-coding-agent--restore-followup-queue-to-input () @@ -1411,12 +1547,12 @@ prompt preflight succeeds, so rejected queued prompts remain available." (and pi-coding-agent--followup-drain-timer t)) (defvar-local pi-coding-agent--local-user-message nil - "Text of user message we displayed locally, awaiting pi's echo. -Set when displaying a user message (normal send, follow-up). -Cleared when we receive message_start role=user from pi. -When nil and we receive message_start role=user, we display it. -When set but different from pi's message, we display pi's version -\(e.g., expanded template).") + "Locally displayed user turn awaiting pi's authoritative echo. +A string records an existing text-only turn. An image turn stores its full +normalized content vector so text and image blocks must both match. Nil means +there is no local echo to suppress. The value is cleared on message_start; +when the authoritative turn differs, pi's version is also displayed (for +example, after prompt or image transformation).") (defvar-local pi-coding-agent--local-user-message-region nil "Marker pair bounding the locally displayed user turn awaiting pi's echo.") @@ -1438,10 +1574,11 @@ When set but different from pi's message, we display pi's version (defun pi-coding-agent--session-busy-p (&optional chat-buf) "Return non-nil when CHAT-BUF has active or locally pending work. When CHAT-BUF is nil, inspect the current buffer. This includes Pi-owned -activity from `pi-coding-agent--status' plus session transitions, prompt -preflight, and follow-up drain waits." +activity from `pi-coding-agent--status' plus model changes, session +transitions, prompt preflight, and follow-up drain waits." (with-current-buffer (or chat-buf (current-buffer)) (or (memq pi-coding-agent--status '(sending streaming compacting)) + (pi-coding-agent--model-change-pending-p) (pi-coding-agent--session-transition-active-p) (pi-coding-agent--prompt-start-wait-active-p) (pi-coding-agent--followup-drain-pending-p)))) @@ -2187,6 +2324,7 @@ Stores the result in CHAT-BUF and emits a minibuffer notice when available." (concat separator "\n" "C-c C-c send prompt\n" + "C-c C-a attach image (C-u clears)\n" "C-c C-k abort\n" "C-c C-r sessions\n" "C-c C-p menu\n"))) @@ -2330,6 +2468,17 @@ when no extension info exists." (concat " โ”‚ " (mapconcat #'identity (nreverse parts) " ยท ")) ""))) +(defun pi-coding-agent--header-format-prompt-image (image) + "Format a leading-pipe header group for prompt IMAGE." + (if (not image) + "" + (let ((name (pi-coding-agent--header-escape-text + (pi-coding-agent--prompt-image-name image))) + (size (file-size-human-readable + (pi-coding-agent--prompt-image-byte-size image) + 'iec " " "B"))) + (format " โ”‚ image: %s (%s)" name size)))) + (defun pi-coding-agent--header-line-string () "Return formatted header-line string for input buffer. Accesses state from the linked chat buffer." @@ -2365,7 +2514,9 @@ Accesses state from the linked chat buffer." (pi-coding-agent--header-format-identity model-short thinking activity-phase-str) (pi-coding-agent--header-format-stats stats) (pi-coding-agent--header-format-context-group session-name) - (pi-coding-agent--header-format-extension-group ext-status working-message)))) + (pi-coding-agent--header-format-extension-group ext-status working-message) + (pi-coding-agent--header-format-prompt-image + (pi-coding-agent--get-prompt-image))))) ;;; State Management @@ -2533,29 +2684,39 @@ ON-NO-AGENT-START is called if the fallback actually fires." #'pi-coding-agent--clear-sending-if-no-agent-start chat-buf generation on-no-agent-start)))))) +(defun pi-coding-agent--handle-prompt-send-failure + (chat-buf generation on-failure &optional error-text) + "Finish the current failed prompt send owned by GENERATION. +Restore user input through ON-FAILURE, reset CHAT-BUF, and report ERROR-TEXT. +Return non-nil only when GENERATION still owned the prompt-start wait." + (let ((current-failure + (and (buffer-live-p chat-buf) + (with-current-buffer chat-buf + (pi-coding-agent--prompt-start-current-p generation))))) + (when current-failure + (pi-coding-agent--abort-send chat-buf on-failure) + (message "Pi: Send failed%s" + (if error-text (format ": %s" error-text) ""))) + current-failure)) + (defun pi-coding-agent--send-prompt - (text &optional on-success on-failure on-no-agent-start) - "Send TEXT as a prompt to the pi process. + (text &optional on-success on-failure on-no-agent-start prompt-image) + "Send TEXT and optional PROMPT-IMAGE to the pi process. Slash commands are sent literally - pi handles expansion. Shows an error message if process is unavailable. ON-SUCCESS is called in the chat buffer after prompt preflight accepts TEXT. -ON-FAILURE is called in the chat buffer if preflight rejects TEXT. -ON-NO-AGENT-START is called if success is not followed by agent_start." +ON-FAILURE is called in the chat buffer if preflight rejects TEXT or scheduling +fails synchronously. ON-NO-AGENT-START is called if success is not followed +by agent_start." (let ((proc (pi-coding-agent--get-process)) (chat-buf (pi-coding-agent--get-chat-buffer)) (prompt-generation nil)) (cond ((null proc) - (when (and on-failure (buffer-live-p chat-buf)) - (with-current-buffer chat-buf - (funcall on-failure))) - (pi-coding-agent--abort-send chat-buf) + (pi-coding-agent--abort-send chat-buf on-failure) (message "Pi: No process available - try M-x pi-coding-agent-reload or C-c C-p R")) ((not (process-live-p proc)) - (when (and on-failure (buffer-live-p chat-buf)) - (with-current-buffer chat-buf - (funcall on-failure))) - (pi-coding-agent--abort-send chat-buf) + (pi-coding-agent--abort-send chat-buf on-failure) (message "Pi: Process died - try M-x pi-coding-agent-reload or C-c C-p R")) (t (when (buffer-live-p chat-buf) @@ -2563,44 +2724,54 @@ ON-NO-AGENT-START is called if success is not followed by agent_start." (setq prompt-generation (pi-coding-agent--begin-prompt-start-wait)) (setq pi-coding-agent--status 'sending) (pi-coding-agent--set-activity-phase "thinking"))) - (pi-coding-agent--rpc-async - proc - (list :type "prompt" :message text) - (lambda (response) - (if (eq (plist-get response :success) t) - (when (buffer-live-p chat-buf) - (with-current-buffer chat-buf - (when (pi-coding-agent--prompt-start-current-p prompt-generation) - (when on-success - (funcall on-success)) - (pi-coding-agent--schedule-prompt-start-fallback - chat-buf prompt-generation on-no-agent-start)))) - (let ((current-failure nil)) - (when (buffer-live-p chat-buf) - (with-current-buffer chat-buf - (when (pi-coding-agent--prompt-start-current-p prompt-generation) - (setq current-failure t) - (pi-coding-agent--invalidate-prompt-start-wait) - (when on-failure - (funcall on-failure))))) - (when current-failure - (pi-coding-agent--abort-send chat-buf) - (message "Pi: Send failed%s" - (if-let* ((error-text (plist-get response :error))) - (format ": %s" error-text) - ""))))))))))) - -(defun pi-coding-agent--abort-send (chat-buf) + (condition-case err + (pi-coding-agent--rpc-async + proc + (append (list :type "prompt" :message text) + (when prompt-image + (list :images + (vector + (pi-coding-agent--prompt-image-content-block + prompt-image))))) + (lambda (response) + (if (eq (plist-get response :success) t) + (when (buffer-live-p chat-buf) + (with-current-buffer chat-buf + (when (pi-coding-agent--prompt-start-current-p + prompt-generation) + (when on-success + (funcall on-success)) + (pi-coding-agent--schedule-prompt-start-fallback + chat-buf prompt-generation on-no-agent-start)))) + (pi-coding-agent--handle-prompt-send-failure + chat-buf prompt-generation on-failure + (plist-get response :error))))) + ((error quit) + (if (eq (car err) 'quit) + (unwind-protect + (pi-coding-agent--handle-prompt-send-failure + chat-buf prompt-generation on-failure + (error-message-string err)) + (signal (car err) (cdr err))) + (pi-coding-agent--handle-prompt-send-failure + chat-buf prompt-generation on-failure + (error-message-string err))))))))) + +(defun pi-coding-agent--abort-send (chat-buf &optional on-failure) "Clean up after a failed send attempt in CHAT-BUF. -Resets activity phase and status to idle." +Call ON-FAILURE once after invalidating the prompt wait, then reset activity, +local echo state, and status to idle even if restoration signals." (when (buffer-live-p chat-buf) (with-current-buffer chat-buf (pi-coding-agent--invalidate-prompt-start-wait) - (setq pi-coding-agent--local-user-message nil) - (pi-coding-agent--clear-local-user-message-region) - (setq pi-coding-agent--pre-compaction-status nil) - (setq pi-coding-agent--status 'idle) - (pi-coding-agent--set-activity-phase "idle")))) + (unwind-protect + (when on-failure + (funcall on-failure)) + (setq pi-coding-agent--local-user-message nil) + (pi-coding-agent--clear-local-user-message-region) + (setq pi-coding-agent--pre-compaction-status nil) + (setq pi-coding-agent--status 'idle) + (pi-coding-agent--set-activity-phase "idle"))))) (provide 'pi-coding-agent-ui) diff --git a/pi-coding-agent.el b/pi-coding-agent.el index cc8782d..a1c3328 100644 --- a/pi-coding-agent.el +++ b/pi-coding-agent.el @@ -53,7 +53,8 @@ ;; ;; Key Bindings: ;; Input buffer: -;; C-c C-c Send prompt (queues as follow-up if busy) +;; C-c C-c Send prompt (queues text as follow-up if busy) +;; C-c C-a Attach/replace one prompt image (C-u clears) ;; C-c C-s Queue steering (interrupts after current tool; busy only) ;; C-c C-k Abort current operation ;; C-c C-p Open menu @@ -83,9 +84,12 @@ ;; Editor Features: ;; - File reference (@): Type @ to search project files (respects .gitignore) ;; - Path completion (Tab): Complete relative paths, ../, ~/, etc. -;; - Message queuing: Submit messages while agent is working: +;; - Prompt image: Attach one content-sniffed raster image to a direct, +;; idle, non-slash prompt; the input header shows its name and size. +;; - Message queuing: Submit text messages while agent is working: ;; C-c C-c queues follow-up (delivered after agent completes) ;; C-c C-s queues steering (interrupts after current tool) +;; Image-bearing drafts refuse these busy paths and remain intact. ;; ;; Press C-c C-p for the full transient menu with model selection, ;; thinking level, completed-thinking controls, session management, diff --git a/test/pi-coding-agent-core-test.el b/test/pi-coding-agent-core-test.el index 04baf2a..8df7fb6 100644 --- a/test/pi-coding-agent-core-test.el +++ b/test/pi-coding-agent-core-test.el @@ -632,6 +632,45 @@ (delete-process fake-proc))) (kill-buffer output-buffer)))) +(ert-deftest pi-coding-agent-test-rpc-async-scheduling-error-cleans-pending-state () + "Encoding and sending failures leave no orphaned pending request state." + (let ((pi-coding-agent--request-id-counter 0) + (real-encode (symbol-function 'pi-coding-agent--encode-command)) + (real-send (symbol-function 'pi-coding-agent--send-string)) + results) + (dolist (failure '(encode send)) + (let ((fake-proc (start-process "cat" nil "cat")) + attempted) + (unwind-protect + (cl-letf (((symbol-function 'pi-coding-agent--encode-command) + (lambda (command) + (if (eq failure 'encode) + (progn + (setq attempted t) + (error "synchronous encode failure")) + (funcall real-encode command)))) + ((symbol-function 'pi-coding-agent--send-string) + (lambda (process string) + (if (eq failure 'send) + (progn + (setq attempted t) + (error "synchronous send failure")) + (funcall real-send process string))))) + (condition-case nil + (pi-coding-agent--rpc-async + fake-proc '(:type "get_state") #'ignore) + (error nil)) + (push + (list failure attempted + (hash-table-count + (pi-coding-agent--get-pending-requests fake-proc)) + (hash-table-count + (pi-coding-agent--get-pending-command-types fake-proc))) + results)) + (ignore-errors (delete-process fake-proc))))) + (should (equal (nreverse results) + '((encode t 0 0) (send t 0 0)))))) + (ert-deftest pi-coding-agent-test-remote-rpc-queue-flushes-after-ready-marker () "Remote RPC writes queue until the ready marker, then flush FIFO." (let ((pi-coding-agent--request-id-counter 0) diff --git a/test/pi-coding-agent-fake-pi-test.el b/test/pi-coding-agent-fake-pi-test.el index bdc8a30..5fad6f6 100644 --- a/test/pi-coding-agent-fake-pi-test.el +++ b/test/pi-coding-agent-fake-pi-test.el @@ -1174,6 +1174,59 @@ SPEC is (SESSION SCENARIO &rest EXTRA-ARGS)." (with-current-buffer chat-buf (should (file-exists-p (plist-get pi-coding-agent--state :session-file))))))) +(ert-deftest pi-coding-agent-fake-pi-test-prompt-image-persists-canonical-content () + "A UI-attached PNG survives the fake prompt and canonical history contract." + (let* ((dir (make-temp-file "pi-coding-agent-fake-pi-image-" t)) + (path (pi-coding-agent-test--write-prompt-image + (expand-file-name "pixel.png" dir) 'png)) + (data (pi-coding-agent-test--prompt-image-base64 'png)) + (text "Describe this fake-contract pixel")) + (unwind-protect + (pi-coding-agent-fake-pi-test-with-session + (session "prompt-lifecycle") + (let ((chat-buf (plist-get session :chat-buffer)) + (input-buf (plist-get session :input-buffer)) + (proc (plist-get session :process))) + (pi-coding-agent-fake-pi-test--wait-or-fail + proc + (lambda () + (pi-coding-agent--model-supports-image-input-p chat-buf)) + "vision model state") + (with-current-buffer input-buf + (erase-buffer) + (insert text) + (cl-letf (((symbol-function 'read-file-name) + (lambda (&rest _) path))) + (call-interactively #'pi-coding-agent-attach-image)) + (delete-file path) + (pi-coding-agent-send)) + (pi-coding-agent-fake-pi-test--wait-or-fail + proc + (lambda () + (with-current-buffer chat-buf + (and (eq pi-coding-agent--status 'idle) + (not (pi-coding-agent--prompt-start-wait-active-p)) + (string-match-p "Fake reply for:" (buffer-string))))) + "image prompt settlement") + (with-current-buffer chat-buf + (should (string-match-p "Image: image/png" (buffer-string)))) + (let* ((response + (pi-coding-agent--rpc-sync + proc '(:type "get_messages") + pi-coding-agent-fake-pi-test--timeout)) + (messages (plist-get (plist-get response :data) :messages)) + (user (seq-find + (lambda (message) + (equal (plist-get message :role) "user")) + (append messages nil)))) + (should (eq (plist-get response :success) t)) + (should + (equal (plist-get user :content) + (vector (list :type "text" :text text) + (list :type "image" :data data + :mimeType "image/png"))))))) + (delete-directory dir t)))) + (ert-deftest pi-coding-agent-fake-pi-test-extension-confirm-displays-through-emacs-seam () "An extension confirm round-trip renders the follow-up message in chat." (cl-letf (((symbol-function 'yes-or-no-p) (lambda (_prompt) t))) diff --git a/test/pi-coding-agent-input-test.el b/test/pi-coding-agent-input-test.el index 36bc7bf..6964426 100644 --- a/test/pi-coding-agent-input-test.el +++ b/test/pi-coding-agent-input-test.el @@ -2717,6 +2717,550 @@ Pi handles command expansion on the server side." (should-not (plist-member rpc-message :images))) (delete-process fake-proc)))) +(ert-deftest pi-coding-agent-test-prompt-image-png-end-to-end () + "C-c C-a content-sniffs a misleadingly named PNG for exact RPC content." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let* ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "pixel.txt" dir) 'png)) + (data (pi-coding-agent-test--prompt-image-base64 'png)) + rpc-message) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image-via-key path) + (should (string-match-p "pixel.txt" (pi-coding-agent-test--input-header))) + (pi-coding-agent-test--attach-image-via-key path 'clear) + (should-not (string-match-p "pixel.txt" (pi-coding-agent-test--input-header))) + (pi-coding-agent-test--attach-image-via-key path) + (delete-file path) + (insert "Describe the pixel") + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (_process command _callback) + (setq rpc-message command)))) + (pi-coding-agent-send)) + (should (equal rpc-message + (list :type "prompt" :message "Describe the pixel" :images + (vector (list :type "image" :data data :mimeType "image/png"))))) + (should (equal (ring-ref pi-coding-agent--input-ring 0) "Describe the pixel")))))) + +(ert-deftest pi-coding-agent-test-prompt-image-sync-rpc-error-restores-draft () + "A synchronous RPC error restores the exact pending image draft." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let* ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "sync-error.png" dir) 'png)) + (text "Keep this image prompt") + attached-image + chat-before) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image path) + (setq attached-image (pi-coding-agent--get-prompt-image)) + (insert text)) + (setq chat-before (with-current-buffer chat-buf (buffer-string))) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (&rest _) (error "synchronous RPC failure"))) + ((symbol-function 'message) #'ignore)) + (with-current-buffer input-buf + (condition-case nil + (pi-coding-agent-send) + (error nil)))) + (with-current-buffer input-buf + (should (equal (buffer-string) text)) + (should (eq (pi-coding-agent--get-prompt-image) attached-image))) + (with-current-buffer chat-buf + (should-not (pi-coding-agent--prompt-start-wait-active-p)) + (should (eq pi-coding-agent--status 'idle)) + (should-not pi-coding-agent--local-user-message) + (should (equal (buffer-string) chat-before)))))) + +(ert-deftest pi-coding-agent-test-prompt-image-signatures-and-rejections () + "Other raster signatures attach; non-images and over-cap sources do not." + (pi-coding-agent-test-with-prompt-image-session (dir _chat-buf input-buf) + (with-current-buffer input-buf + (let (previous) + (dolist (spec '((jpeg "photo.jpg") (gif "pixel.gif") + (webp "pixel.webp"))) + (when previous + (pi-coding-agent-test--attach-image-via-key previous 'clear)) + (setq previous + (pi-coding-agent-test--write-prompt-image + (expand-file-name (cadr spec) dir) (car spec))) + (pi-coding-agent-test--attach-image previous) + (should (string-match-p + (regexp-quote (file-name-nondirectory previous)) + (pi-coding-agent-test--input-header)))) + (pi-coding-agent-test--attach-image-via-key previous 'clear)) + (let* ((not-image (expand-file-name "not-image.txt" dir)) + (too-large (pi-coding-agent-test--write-prompt-image + (expand-file-name "too-large.png" dir) 'png))) + (with-temp-file not-image (insert "not an image")) + (dolist (case `((,not-image nil "image\\|format") + (,too-large 1 "large\\|limit\\|byte\\|size"))) + (let (feedback) + (cl-letf (((symbol-function 'message) + (lambda (format-string &rest args) + (when format-string + (setq feedback (apply #'format format-string args)))))) + (condition-case error-data + (let ((pi-coding-agent-prompt-image-max-bytes + (or (cadr case) most-positive-fixnum))) + (pi-coding-agent-test--attach-image (car case))) + (user-error (setq feedback (error-message-string error-data))))) + (should (string-match-p (caddr case) (downcase (or feedback "")))) + (should-not (string-match-p + (regexp-quote (file-name-nondirectory (car case))) + (pi-coding-agent-test--input-header))))))))) + +(ert-deftest pi-coding-agent-test-prompt-image-refusal-matrix-preserves-draft () + "Capability, busy, slash, steering, and empty refusals retain the draft." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "guard.png" dir) 'png))) + (dolist (case '((text-model "Describe" idle send "model\\|support") + (missing-input "Missing metadata" idle send "model\\|support\\|load") + (unknown-model "Unknown model" idle send "model\\|support\\|load") + (busy "Wait" streaming send "busy\\|stream") + (slash "/new" idle send "slash\\|command") + (steering "Change" streaming steer "steer") + (empty "" idle send "empty\\|text\\|prompt"))) + (pcase-let ((`(,kind ,text ,status ,action ,reason) case)) + (with-current-buffer input-buf + (erase-buffer) + (pi-coding-agent-test--attach-image path) + (insert text)) + (with-current-buffer chat-buf + (setq pi-coding-agent--status status + pi-coding-agent--state + (pcase kind + ('text-model '(:model (:name "Text" :input ["text"]))) + ('missing-input '(:model (:name "Loading"))) + ('unknown-model nil) + (_ '(:model (:name "Vision" :input ["text" "image"])))))) + (let (feedback rpc-called builtin-called) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (&rest _) (setq rpc-called t))) + ((symbol-function 'pi-coding-agent-new-session) + (lambda () (setq builtin-called t))) + ((symbol-function 'message) + (lambda (format-string &rest args) + (when format-string + (setq feedback (apply #'format format-string args)))))) + (with-current-buffer input-buf + (pcase action + ('send (pi-coding-agent-send)) + ('steer (pi-coding-agent-queue-steering))) + (should (equal (buffer-string) text)) + (should (string-match-p "guard.png" + (pi-coding-agent-test--input-header)))) + (should-not rpc-called) + (should-not builtin-called) + (should (string-match-p reason (downcase (or feedback ""))))) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image-via-key path 'clear)))))))) + +(ert-deftest pi-coding-agent-test-prompt-image-waits-for-model-change () + "Image send waits for model selection, then uses the accepted model state." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let* ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "model-change.png" dir) 'png)) + (old-model '(:id "vision-old" :name "Vision Old" + :provider "fake" :input ["text" "image"])) + (new-model '(:id "vision-next" :name "Vision Next" + :provider "fake" :input ["text" "image"])) + (text "Wait for the selected model") + attached-image + model-callback + prompt-command) + (with-current-buffer chat-buf + (setq pi-coding-agent--process 'image-process + pi-coding-agent--state (list :model old-model))) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image path) + (setq attached-image (pi-coding-agent--get-prompt-image)) + (insert text)) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-sync) + (lambda (&rest _) + (list :success t :data + (list :models (vector old-model new-model))))) + ((symbol-function 'completing-read) + (lambda (_prompt collection &rest _) + (or (seq-find + (lambda (candidate) + (string-match-p "Vision Next" candidate)) + collection) + (ert-fail "Missing Vision Next model choice")))) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (_process command callback) + (pcase (plist-get command :type) + ("set_model" (setq model-callback callback)) + ("prompt" (setq prompt-command command))))) + ((symbol-function 'message) #'ignore)) + (with-current-buffer input-buf + (pi-coding-agent-select-model) + (should (functionp model-callback)) + (pi-coding-agent-send) + (should (equal (buffer-string) text)) + (should (eq (pi-coding-agent--get-prompt-image) attached-image))) + (should-not prompt-command) + (funcall model-callback + (list :success t :command "set_model" :data new-model)) + (with-current-buffer input-buf + (pi-coding-agent-send) + (should (string-empty-p (buffer-string))) + (should-not (pi-coding-agent--get-prompt-image))) + (should (equal (plist-get prompt-command :message) text)) + (should (plist-member prompt-command :images)))))) + +(ert-deftest pi-coding-agent-test-model-change-refuses-image-preflight () + "Model selection cannot overlap an image prompt awaiting acceptance." + (pi-coding-agent-test-with-prompt-image-session (_dir chat-buf _input-buf) + (let (rpc-called feedback) + (with-current-buffer chat-buf + (setq pi-coding-agent--process 'image-process + pi-coding-agent--status 'sending + pi-coding-agent--prompt-start-wait-active t)) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'pi-coding-agent--get-chat-buffer) + (lambda () chat-buf)) + ((symbol-function 'pi-coding-agent--rpc-sync) + (lambda (&rest _) + (setq rpc-called t))) + ((symbol-function 'message) + (lambda (format-string &rest args) + (when format-string + (setq feedback (apply #'format format-string args)))))) + (with-current-buffer chat-buf + (pi-coding-agent-select-model))) + (should-not rpc-called) + (should (string-match-p "Cannot change models" + (or feedback "")))))) + +(ert-deftest pi-coding-agent-test-model-change-aborts-if-process-changes-during-selection () + "A selector cannot acquire a model gate for a process that was replaced." + (pi-coding-agent-test-with-prompt-image-session (_dir chat-buf _input-buf) + (let* ((old-model '(:id "old" :name "Old" :provider "fake")) + (new-model '(:id "new" :name "New" :provider "fake")) + rpc-called + feedback) + (with-current-buffer chat-buf + (setq pi-coding-agent--process 'old-process + pi-coding-agent--state (list :model old-model))) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'old-process)) + ((symbol-function 'pi-coding-agent--get-chat-buffer) + (lambda () chat-buf)) + ((symbol-function 'pi-coding-agent--rpc-sync) + (lambda (&rest _) + (list :success t :data + (list :models (vector old-model new-model))))) + ((symbol-function 'completing-read) + (lambda (&rest _) + (with-current-buffer chat-buf + (setq pi-coding-agent--process 'new-process)) + "New [fake]")) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (&rest _) + (setq rpc-called t))) + ((symbol-function 'message) + (lambda (format-string &rest args) + (when format-string + (setq feedback (apply #'format format-string args)))))) + (with-current-buffer chat-buf + (pi-coding-agent-select-model))) + (should-not rpc-called) + (with-current-buffer chat-buf + (should-not (pi-coding-agent--model-change-pending-p))) + (should (equal feedback + "Pi: Process changed while selecting a model; try again"))))) + +(ert-deftest pi-coding-agent-test-model-cancellation-restores-gated-queue () + "Cancelling a model change makes text queued behind it visible." + (pi-coding-agent-test-with-prompt-image-session (_dir chat-buf input-buf) + (with-current-buffer chat-buf + (setq pi-coding-agent--process 'old-process) + (should (pi-coding-agent--begin-model-change + 'old-process chat-buf)) + (pi-coding-agent--push-followup "do not strand me") + (pi-coding-agent--cancel-model-change-and-restore-followups chat-buf) + (pi-coding-agent--set-process 'new-process) + (should-not (pi-coding-agent--model-change-pending-p)) + (should-not pi-coding-agent--followup-queue)) + (with-current-buffer input-buf + (should (equal (buffer-string) "do not strand me"))))) + +(ert-deftest pi-coding-agent-test-failed-model-change-restores-queued-text () + "A failed model change must not send queued text under the old model." + (pi-coding-agent-test-with-prompt-image-session (_dir chat-buf input-buf) + (let* ((old-model '(:id "old" :name "Old" :provider "fake")) + (new-model '(:id "new" :name "New" :provider "fake")) + model-callback + prompt-called + feedback) + (with-current-buffer chat-buf + (setq pi-coding-agent--process 'image-process + pi-coding-agent--state (list :model old-model))) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'pi-coding-agent--get-chat-buffer) + (lambda () chat-buf)) + ((symbol-function 'pi-coding-agent--rpc-sync) + (lambda (&rest _) + (list :success t :data + (list :models (vector old-model new-model))))) + ((symbol-function 'completing-read) + (lambda (&rest _) "New [fake]")) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (_process command callback) + (pcase (plist-get command :type) + ("set_model" (setq model-callback callback)) + ("prompt" (setq prompt-called t))))) + ((symbol-function 'message) + (lambda (format-string &rest args) + (when format-string + (setq feedback (apply #'format format-string args)))))) + (with-current-buffer chat-buf + (pi-coding-agent-select-model)) + (with-current-buffer input-buf + (insert "keep this queued") + (pi-coding-agent-send) + (should (string-empty-p (buffer-string)))) + (should (functionp model-callback)) + (funcall model-callback '(:success :false :error "model unavailable"))) + (should-not prompt-called) + (with-current-buffer chat-buf + (should-not (pi-coding-agent--model-change-pending-p)) + (should-not pi-coding-agent--followup-queue) + (should (equal (plist-get pi-coding-agent--state :model) old-model))) + (with-current-buffer input-buf + (should (equal (buffer-string) "keep this queued"))) + (should (equal feedback + "Pi: Failed to set model: model unavailable"))))) + +(ert-deftest pi-coding-agent-test-prompt-image-stale-model-callback-stays-gated () + "A replaced process's model callback cannot release the current gate." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let* ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "stale-model.png" dir) 'png)) + (old-model '(:id "vision-old" :name "Vision Old" + :provider "fake" :input ["text" "image"])) + (model-a '(:id "vision-a" :name "Vision A" + :provider "fake" :input ["text" "image"])) + (model-b '(:id "vision-b" :name "Vision B" + :provider "fake" :input ["text" "image"])) + (text "Keep gating this image") + (current-process 'image-process) + choice-name attached-image model-callbacks prompt-command) + (with-current-buffer chat-buf + (setq pi-coding-agent--process current-process + pi-coding-agent--state (list :model old-model))) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image path) + (setq attached-image (pi-coding-agent--get-prompt-image)) + (insert text)) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () current-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-sync) + (lambda (&rest _) + (list :success t :data + (list :models (vector old-model model-a model-b))))) + ((symbol-function 'completing-read) + (lambda (_prompt collection &rest _) + (or (seq-find + (lambda (candidate) + (string-match-p choice-name candidate)) + collection) + (ert-fail "Missing requested model choice")))) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (_process command callback) + (pcase (plist-get command :type) + ("set_model" + (push (cons (plist-get command :modelId) callback) + model-callbacks)) + ("prompt" (setq prompt-command command))))) + ((symbol-function 'message) #'ignore)) + (setq choice-name "Vision A") + (with-current-buffer input-buf + (pi-coding-agent-select-model)) + (setq current-process 'replacement-process) + (with-current-buffer chat-buf + (pi-coding-agent--set-process current-process)) + (setq choice-name "Vision B") + (with-current-buffer input-buf + (pi-coding-agent-select-model)) + (let ((callback-a (alist-get "vision-a" model-callbacks + nil nil #'equal)) + (callback-b (alist-get "vision-b" model-callbacks + nil nil #'equal))) + (should (functionp callback-a)) + (should (functionp callback-b)) + (funcall callback-a + (list :success t :command "set_model" :data model-a)) + (with-current-buffer input-buf + (pi-coding-agent-send) + (should (equal (buffer-string) text)) + (should (eq (pi-coding-agent--get-prompt-image) attached-image))) + (should-not prompt-command) + (funcall callback-b + (list :success t :command "set_model" :data model-b)) + (with-current-buffer chat-buf + (should (equal (plist-get (plist-get pi-coding-agent--state :model) + :id) + "vision-b"))) + (with-current-buffer input-buf + (pi-coding-agent-send) + (should (string-empty-p (buffer-string))) + (should-not (pi-coding-agent--get-prompt-image))) + (should (equal (plist-get prompt-command :message) text)) + (should (plist-member prompt-command :images))))))) + +(ert-deftest pi-coding-agent-test-prompt-image-preflight-ownership () + "No-process and rejected sends restore image bytes; acceptance consumes them." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let* ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "restored.png" dir) 'png)) + (data (pi-coding-agent-test--prompt-image-base64 'png)) + rpc-message rpc-callback attached-image) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image path) + (setq attached-image (pi-coding-agent--get-prompt-image)) + (delete-file path) + (insert "Recover this turn") + (cl-letf (((symbol-function 'pi-coding-agent--get-process) (lambda () nil)) + ((symbol-function 'message) #'ignore)) + (pi-coding-agent-send)) + (should (equal (buffer-string) "Recover this turn")) + (should (string-match-p "restored.png" + (pi-coding-agent-test--input-header))) + (cl-labels ((send () + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (_process command callback) + (setq rpc-message command + rpc-callback callback))) + ((symbol-function 'message) #'ignore)) + (pi-coding-agent-send)))) + (send) + (let ((pending-block (aref (plist-get rpc-message :images) 0))) + (should (equal (plist-get pending-block :data) data)) + (should-error (pi-coding-agent-attach-image 'clear) + :type 'user-error) + (funcall rpc-callback '(:success nil :error "rejected")) + (should (eq (pi-coding-agent--get-prompt-image) attached-image)) + (should (equal + (pi-coding-agent--prompt-image-content-block + (pi-coding-agent--get-prompt-image)) + pending-block))) + (should (equal (buffer-string) "Recover this turn")) + (should (string-match-p "restored.png" + (pi-coding-agent-test--input-header))) + (setq rpc-message nil rpc-callback nil) + (send) + (cl-letf (((symbol-function 'display-images-p) (lambda (&rest _) nil))) + (funcall rpc-callback '(:success t)))) + (should (string-empty-p (buffer-string))) + (should-not (string-match-p "restored.png" + (pi-coding-agent-test--input-header)))) + (with-current-buffer chat-buf + (should (string-match-p "Recover this turn" (buffer-string))) + (should (string-match-p "Image: image/png" (buffer-string))))))) + +(ert-deftest pi-coding-agent-test-prompt-image-authoritative-echo-compares-content () + "A same-text echo with a different image renders authoritative content." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let* ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "original.png" dir) 'png)) + (text "Inspect this image") + (jpeg-data (pi-coding-agent-test--prompt-image-base64 'jpeg)) + rpc-callback) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image path) + (insert text) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (_process _command callback) + (setq rpc-callback callback)))) + (pi-coding-agent-send))) + (should (functionp rpc-callback)) + (cl-letf (((symbol-function 'display-images-p) (lambda (&rest _) nil))) + (funcall rpc-callback '(:success t)) + (with-current-buffer chat-buf + (pi-coding-agent--handle-display-event '(:type "agent_start")) + (should (string-match-p "Image: image/png" (buffer-string))) + (pi-coding-agent--handle-display-event + (list :type "message_start" + :message + (list :role "user" :timestamp 1704067200000 + :content + (vector (list :type "text" :text text) + (list :type "image" :data jpeg-data + :mimeType "image/jpeg"))))) + (should (string-match-p "Image: image/jpeg" (buffer-string)))))))) + +(ert-deftest pi-coding-agent-test-prompt-image-no-turn-success-retracts-local-echo () + "An extension-handled image prompt leaves no phantom user turn." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let* ((path (pi-coding-agent-test--write-prompt-image + (expand-file-name "handled.png" dir) 'png)) + rpc-callback state-callback fallback-callback fallback-args) + (with-current-buffer chat-buf + (setq pi-coding-agent--process 'image-process)) + (with-current-buffer input-buf + (pi-coding-agent-test--attach-image path) + (insert "Handle this without a turn")) + (cl-letf (((symbol-function 'pi-coding-agent--get-process) + (lambda () 'image-process)) + ((symbol-function 'process-live-p) (lambda (_) t)) + ((symbol-function 'pi-coding-agent--rpc-async) + (lambda (_process command callback) + (pcase (plist-get command :type) + ("prompt" (setq rpc-callback callback)) + ("get_state" (setq state-callback callback))))) + ((symbol-function 'run-at-time) + (lambda (_secs _repeat function &rest args) + (if (eq function + 'pi-coding-agent--clear-sending-if-no-agent-start) + (setq fallback-callback function + fallback-args args) + 'fake-drain-timer) + 'fake-prompt-start-timer)) + ((symbol-function 'display-images-p) (lambda (&rest _) nil)) + ((symbol-function 'message) #'ignore)) + (with-current-buffer input-buf + (pi-coding-agent-send)) + (funcall rpc-callback '(:success t)) + (with-current-buffer chat-buf + (should pi-coding-agent--local-user-message) + (should (string-match-p "Handle this without a turn" + (buffer-string))) + (narrow-to-region (1+ (point-min)) (point-max))) + (apply fallback-callback fallback-args) + (should (functionp state-callback)) + (funcall state-callback + '(:success t + :data (:isStreaming :false :isCompacting :false)))) + (with-current-buffer chat-buf + (widen) + (should (eq pi-coding-agent--status 'idle)) + (should-not pi-coding-agent--local-user-message) + (should-not pi-coding-agent--local-user-message-region) + (should-not (string-match-p "Handle this without a turn" + (buffer-string))))))) + (ert-deftest pi-coding-agent-test-no-turn-fallback-keeps-server-active-prompt () "A delayed agent_start must not be mistaken for an extension-handled prompt." (let ((fake-proc (start-process "test-active-prompt" nil "cat"))) @@ -2729,7 +3273,7 @@ Pi handles command expansion on the server side." pi-coding-agent--followup-queue '("wait behind it")) (setq pi-coding-agent--local-user-message-region (pi-coding-agent--display-user-message - "slow prompt" (current-time) t)) + "slow prompt" (current-time) nil t)) (let ((generation (pi-coding-agent--begin-prompt-start-wait))) (cl-letf (((symbol-function 'pi-coding-agent--rpc-async) (lambda (_process command callback) diff --git a/test/pi-coding-agent-menu-test.el b/test/pi-coding-agent-menu-test.el index 523c698..debfaa5 100644 --- a/test/pi-coding-agent-menu-test.el +++ b/test/pi-coding-agent-menu-test.el @@ -2438,7 +2438,17 @@ replaced by the resumed or forked history." :timestamp 1704067201000)])) (pi-coding-agent-test--seed-stale-session-rebuild-state chat-buf "STALE FORK CONTENT") + (with-current-buffer chat-buf + (setq pi-coding-agent--state + (plist-put pi-coding-agent--state :model + '(:name "Vision" :input ["text" "image"])))) (with-current-buffer input-buf + (let ((path (make-temp-file "pi-prompt-attachment-" nil ".png"))) + (unwind-protect + (progn + (pi-coding-agent-test--write-prompt-image path 'png) + (pi-coding-agent-test--attach-image path)) + (delete-file path))) (insert "old input text")) (cl-letf (((symbol-function 'completing-read) (lambda (&rest _) selected-choice)) @@ -2483,7 +2493,10 @@ replaced by the resumed or forked history." (should (string-match-p "Second question" (buffer-string))) (should (string-match-p "Forked answer" (buffer-string)))) (with-current-buffer input-buf - (should (equal (buffer-string) "Second question"))) + (should (equal (buffer-string) "Second question")) + (should-not (string-match-p + "pi-prompt-attachment-" + (pi-coding-agent-test--input-header)))) (pi-coding-agent-test--assert-clean-session-rebuild chat-buf messages "STALE FORK CONTENT") (should (equal (nreverse rpc-calls) diff --git a/test/pi-coding-agent-render-test.el b/test/pi-coding-agent-render-test.el index 8f73098..97395b1 100644 --- a/test/pi-coding-agent-render-test.el +++ b/test/pi-coding-agent-render-test.el @@ -2132,6 +2132,26 @@ since we don't display them locally. Let pi's message_start handle it." "Prefilled text"))) (kill-buffer input-buf)))) +(ert-deftest pi-coding-agent-test-prompt-image-draft-replacements-clear-attachment () + "Extension and browser prefills cannot retain a stale prompt image." + (pi-coding-agent-test-with-prompt-image-session (dir chat-buf input-buf) + (let ((path (pi-coding-agent-test--write-prompt-image (expand-file-name "stale.png" dir) 'png))) + (with-current-buffer input-buf + (insert "old draft") + (pi-coding-agent-test--attach-image path)) + (with-current-buffer chat-buf + (pi-coding-agent--handle-extension-ui-request + '(:type "extension_ui_request" :id "replace-image-draft" + :method "set_editor_text" :text "Extension replacement"))) + (with-current-buffer input-buf + (should (equal (buffer-string) "Extension replacement")) + (should-not (string-match-p "stale.png" (pi-coding-agent-test--input-header))) + (pi-coding-agent-test--attach-image path)) + (pi-coding-agent--browse-prefill-input input-buf "Browser replacement") + (with-current-buffer input-buf + (should (equal (buffer-string) "Browser replacement")) + (should-not (string-match-p "stale.png" (pi-coding-agent-test--input-header))))))) + (ert-deftest pi-coding-agent-test-extension-ui-set-status () "extension_ui_request setStatus updates extension status storage." (with-temp-buffer @@ -2619,6 +2639,26 @@ See https://github.com/dnouri/pi-coding-agent/issues/176." "Image: image/png, 71 B" (buffer-substring-no-properties position (overlay-end overlay)))))))) +(ert-deftest pi-coding-agent-test-prompt-image-live-and-history-use-image-preview () + "Live and replayed user image blocks use the bounded #221 renderer." + (let ((image (list :type "image" :mimeType "image/png" + :data (pi-coding-agent-test--prompt-image-base64 'png)))) + (dolist (route '(live history)) + (with-temp-buffer + (pi-coding-agent-chat-mode) + (cl-letf (((symbol-function 'display-images-p) (lambda (&rest _) nil))) + (let ((message + (list :role "user" :timestamp 1704067200000 + :content (vector '(:type "text" :text "Visual question") + image)))) + (if (eq route 'live) + (pi-coding-agent--handle-display-event + (list :type "message_start" :message message)) + (pi-coding-agent--display-history-messages (vector message))))) + (should (string-match-p "Visual question" (buffer-string))) + (should (string-match-p "Image: image/png, 69 B" (buffer-string))) + (should (= 1 (length (pi-coding-agent-test--image-preview-positions)))))))) + (ert-deftest pi-coding-agent-test-tool-result-image-inserts-scaled-display-property () "A graphical result carries one scaled image display property." (with-temp-buffer @@ -2679,7 +2719,7 @@ See https://github.com/dnouri/pi-coding-agent/issues/176." (with-temp-buffer (pi-coding-agent-chat-mode) (let ((pi-coding-agent-image-preview-max-bytes 2) - (pi-coding-agent--image-previews-per-tool-limit 2)) + (pi-coding-agent--image-previews-per-content-limit 2)) (cl-letf (((symbol-function 'display-images-p) (lambda (&rest _) t)) ((symbol-function 'create-image) (lambda (&rest _) diff --git a/test/pi-coding-agent-test-common.el b/test/pi-coding-agent-test-common.el index 5d9cea6..268c923 100644 --- a/test/pi-coding-agent-test-common.el +++ b/test/pi-coding-agent-test-common.el @@ -260,6 +260,56 @@ Uses tool call ID \"call_1\" and contentIndex 0." (list (pi-coding-agent-test--toolcall "call_1" tool-name args)) "x")) +(defconst pi-coding-agent-test--prompt-image-fixtures + '((png . "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4z8AAAAMBAQDJ/pLvAAAAAElFTkSuQmCC") + (jpeg . "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAABAAEDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDi6KKK+ZP3E//Z") + (gif . "R0lGODdhAQABAIEAAP8AAAAAAAAAAAAAACwAAAAAAQABAAAIBAABBAQAOw==") + (webp . "UklGRjwAAABXRUJQVlA4IDAAAADQAQCdASoBAAEAAUAmJaACdLoB+AADsAD+8ut//NgVzXPv9//S4P0uD9Lg/9KQAAA=")) + "Valid one-pixel raster images used by prompt attachment tests.") + +(defun pi-coding-agent-test--prompt-image-base64 (type) + "Return the base64 fixture for image TYPE." + (or (alist-get type pi-coding-agent-test--prompt-image-fixtures) + (error "No prompt image fixture for %S" type))) + +(defun pi-coding-agent-test--write-prompt-image (path type) + "Write the binary prompt image fixture TYPE to PATH and return PATH." + (let ((coding-system-for-write 'no-conversion)) + (with-temp-file path + (set-buffer-multibyte nil) + (insert (base64-decode-string (pi-coding-agent-test--prompt-image-base64 type))))) + path) + +(defun pi-coding-agent-test--input-header () + "Return the current input header without properties." + (substring-no-properties (pi-coding-agent--header-line-string))) + +(defun pi-coding-agent-test--attach-image (path) + "Attach prompt image PATH through the public interactive command." + (cl-letf (((symbol-function 'read-file-name) (lambda (&rest _) path))) + (call-interactively #'pi-coding-agent-attach-image))) + +(defun pi-coding-agent-test--attach-image-via-key (path &optional clear) + "Invoke the input binding for PATH, with prefix argument when CLEAR." + (cl-letf (((symbol-function 'read-file-name) (lambda (&rest _) path))) + (let ((current-prefix-arg (and clear '(4)))) + (call-interactively (key-binding (kbd "C-c C-a")))))) + +(cl-defmacro pi-coding-agent-test-with-prompt-image-session + ((dir chat-buf input-buf) &rest body) + "Run BODY in a fresh vision-capable mock session." + (declare (indent 1) (debug ((symbolp symbolp symbolp) body))) + `(let ((,dir (pi-coding-agent-test--make-temp-directory "pi-prompt-image-"))) + (unwind-protect + (pi-coding-agent-test-with-mock-session ,dir + (let ((,chat-buf (get-buffer (pi-coding-agent-test--chat-buffer-name ,dir))) + (,input-buf (get-buffer (pi-coding-agent-test--input-buffer-name ,dir)))) + (with-current-buffer ,chat-buf + (setq pi-coding-agent--status 'idle + pi-coding-agent--state '(:model (:name "Vision" :input ["text" "image"])))) + ,@body)) + (delete-directory ,dir t)))) + ;;;; Mock Session (defmacro pi-coding-agent-test-with-mock-session (dir &rest body) diff --git a/test/pi-coding-agent-ui-test.el b/test/pi-coding-agent-ui-test.el index 0ce930c..c65985d 100644 --- a/test/pi-coding-agent-ui-test.el +++ b/test/pi-coding-agent-ui-test.el @@ -862,6 +862,7 @@ without an input window." (let ((header (pi-coding-agent--format-startup-header))) (should (string-match-p "C-c C-c" header)) (should (string-match-p "send" header)) + (should (string-match-p "C-c C-a attach image (C-u clears)" header)) (should (string-match-p "C-c C-r sessions" header)))) (ert-deftest pi-coding-agent-test-startup-header-shows-pi-label () @@ -985,6 +986,18 @@ without an input window." (should (equal captured-default-directory "/ssh:pi-host:/home/pi/project/"))))) +(ert-deftest pi-coding-agent-test-process-replacement-invalidates-model-change () + "A model callback cannot mutate state after its target process is replaced." + (with-temp-buffer + (pi-coding-agent-chat-mode) + (setq pi-coding-agent--process 'old-process) + (let ((token (pi-coding-agent--begin-model-change + 'old-process (current-buffer)))) + (should (pi-coding-agent--model-change-current-p token)) + (pi-coding-agent--set-process 'new-process) + (should-not (pi-coding-agent--model-change-current-p token)) + (should-not (pi-coding-agent--model-change-pending-p))))) + (ert-deftest pi-coding-agent-test-set-process-probes-version-for-current-process () "Setting process starts version probe and stores result for current process." (let ((callback nil) diff --git a/test/support/fake-pi-contract.md b/test/support/fake-pi-contract.md index 19974d8..3c413fd 100644 --- a/test/support/fake-pi-contract.md +++ b/test/support/fake-pi-contract.md @@ -133,7 +133,7 @@ higher-level events until a test genuinely needs them. Fields the current Emacs code or assertions actively read: -- `model` +- `model` (the fake model advertises `input: ["text", "image"]`) - `thinkingLevel` - `isStreaming` - `isCompacting` @@ -176,6 +176,18 @@ Required behavior: 7. update `get_state.isStreaming` and `messageCount` 8. persist enough session data to back session-file assertions +A `prompt` may include `images`, which must be a JSON array. Every item must +be an object with `type: "image"`, nonempty string `data`, and nonempty string +`mimeType`. The fake validates only this upstream RPC shape: it neither +decodes base64 nor restricts MIME values. Valid blocks are detached from the +request and persisted/emitted after the prompt's text block in request order. + +For `text_stream`, images belong only to the initial user turn; steering is +text-only and image-bearing `steer` commands fail. `tool_stream` preserves +prompt images on its ordinary user message. The extension-owned +`extension_dialog` and `custom_message` prompt behaviors reject nonempty image +arrays before reporting prompt success. No new scenario type is implied. + ### Tool execution path For deterministic GUI and benchmark tests, the fake must emit the current diff --git a/test/support/fake_pi.py b/test/support/fake_pi.py index 97a4968..a491237 100755 --- a/test/support/fake_pi.py +++ b/test/support/fake_pi.py @@ -84,6 +84,18 @@ def to_rpc(self) -> JsonDict: return data +@dataclass(frozen=True) +class PromptImageContent: + """Validated immutable image content from one prompt command.""" + + data: str + mime_type: str + + def to_rpc(self) -> JsonDict: + """Return a fresh RPC content block.""" + return {"type": "image", "data": self.data, "mimeType": self.mime_type} + + @dataclass(frozen=True) class TextStreamPrompt: """Scenario data for a simple streamed text reply.""" @@ -332,6 +344,7 @@ def __init__( "api": "fake-api", "contextWindow": 8192, "maxTokens": 1024, + "input": ["text", "image"], } self.state = SessionState(model=model) self.user_messages: list[dict[str, str]] = [] @@ -400,20 +413,62 @@ def handle(self, command: JsonDict) -> None: case _: self._fail(command, f"Unsupported fake-pi command: {command_type}") + @staticmethod + def _parse_prompt_images(command: JsonDict) -> tuple[PromptImageContent, ...]: + """Validate and detach optional image content from a prompt COMMAND.""" + if "images" not in command: + return () + raw_images = command["images"] + if not isinstance(raw_images, list): + raise ValueError("prompt images must be an array") + images: list[PromptImageContent] = [] + for index, block in enumerate(raw_images): + if not isinstance(block, dict): + raise ValueError(f"prompt image {index} must be an object") + block_type = block.get("type") + data = block.get("data") + mime_type = block.get("mimeType") + if not isinstance(block_type, str) or block_type != "image": + raise ValueError(f"prompt image {index} type must be 'image'") + if not isinstance(data, str) or not data: + raise ValueError(f"prompt image {index} data must be a nonempty string") + if not isinstance(mime_type, str) or not mime_type: + raise ValueError( + f"prompt image {index} mimeType must be a nonempty string" + ) + images.append(PromptImageContent(data=data, mime_type=mime_type)) + return tuple(images) + def _handle_prompt(self, command: JsonDict) -> None: - """Start the scenario-specific prompt behavior.""" + """Validate and start the scenario-specific prompt behavior.""" if self.state.is_streaming: self._fail(command, "Fake pi is already streaming") return + try: + prompt_images = self._parse_prompt_images(command) + except ValueError as exc: + self._fail(command, str(exc)) + return + behavior = self.scenario.prompt + if prompt_images and isinstance( + behavior, (ExtensionDialogPrompt, CustomMessagePrompt) + ): + self._fail( + command, + "Prompt images are not supported by extension-owned fake scenarios", + ) + return self._abort_requested.clear() message = str(command["message"]) - match self.scenario.prompt: + match behavior: case TextStreamPrompt() as behavior: self._respond(command) self._start_run( name=f"fake-pi-text-stream-{self.scenario.name}", target=lambda: self._run_text_prompt( - message, cast(TextStreamPrompt, behavior) + message, + cast(TextStreamPrompt, behavior), + prompt_images=prompt_images, ), ) case ExtensionDialogPrompt() as behavior: @@ -443,13 +498,18 @@ def _handle_prompt(self, command: JsonDict) -> None: self._respond(command) self._start_run( name=f"fake-pi-tool-stream-{self.scenario.name}", - target=lambda: self._run_tool_prompt(message, behavior), + target=lambda: self._run_tool_prompt( + message, behavior, prompt_images=prompt_images + ), ) case _: raise AssertionError("Unknown prompt behavior") def _handle_steer(self, command: JsonDict) -> None: - """Queue a steering message for the active text stream.""" + """Queue a text-only steering message for the active text stream.""" + if "images" in command: + self._fail(command, "Steering images are out of scope for this fake") + return if not self.state.is_streaming: self._fail(command, "Cannot steer when no prompt is streaming") return @@ -562,15 +622,23 @@ def _handle_extension_ui_response(self, command: JsonDict) -> None: self._extension_waiter.set() self._log("extension-response", command) - def _run_text_prompt(self, message: str, behavior: TextStreamPrompt) -> None: - """Run a streamed-text prompt scenario.""" + def _run_text_prompt( + self, + message: str, + behavior: TextStreamPrompt, + *, + prompt_images: tuple[PromptImageContent, ...], + ) -> None: + """Run a streamed-text prompt, imaging only its initial user turn.""" emitted_messages: list[JsonDict] = [] current_message = message + current_images = prompt_images self._write_json({"type": "agent_start"}) while True: completed, assistant_message = self._emit_text_turn( current_message, behavior=behavior, + prompt_images=current_images, assistant_text_template=( behavior.assistant_text if current_message == message @@ -586,6 +654,7 @@ def _run_text_prompt(self, message: str, behavior: TextStreamPrompt) -> None: if pending_steer is None: break current_message = pending_steer + current_images = () self._finish_run(emitted_messages) def _emit_text_turn( @@ -593,6 +662,7 @@ def _emit_text_turn( user_text: str, *, behavior: TextStreamPrompt, + prompt_images: tuple[PromptImageContent, ...], assistant_text_template: str, ) -> tuple[bool, JsonDict]: """Emit one user->assistant text exchange. @@ -601,7 +671,7 @@ def _emit_text_turn( ``message_start``, an aborted result contains the authoritative partial message that must be emitted before ``agent_end``. """ - user_message = self._build_user_message(user_text) + user_message = self._build_user_message(user_text, prompt_images) self._persist_user_message(user_message) if behavior.echo_user: self._write_json({"type": "message_start", "message": user_message}) @@ -675,10 +745,16 @@ def _run_custom_message_prompt( self._write_json({"type": "message_start", "message": followup}) self._write_json({"type": "message_end", "message": followup}) - def _run_tool_prompt(self, message: str, behavior: ToolStreamPrompt) -> None: + def _run_tool_prompt( + self, + message: str, + behavior: ToolStreamPrompt, + *, + prompt_images: tuple[PromptImageContent, ...], + ) -> None: """Run a prompt that emits tool-call and tool-execution events.""" self._write_json({"type": "agent_start"}) - user_message = self._build_user_message(message) + user_message = self._build_user_message(message, prompt_images) self._persist_user_message(user_message) if behavior.echo_user: self._write_json({"type": "message_start", "message": user_message}) @@ -996,11 +1072,15 @@ def _tool_result_payload(text: str) -> JsonDict: "details": {"truncation": None, "fullOutputPath": None}, } - def _build_user_message(self, text: str) -> JsonDict: - """Return a user message payload.""" + def _build_user_message( + self, text: str, images: tuple[PromptImageContent, ...] = () + ) -> JsonDict: + """Return a user message with detached ordered image content.""" + content: list[JsonDict] = [{"type": "text", "text": text}] + content.extend(image.to_rpc() for image in images) return { "role": "user", - "content": [{"type": "text", "text": text}], + "content": content, "timestamp": now_ms(), }