feat(assistant): read files attached in the panel (untrusted, capped, vision-aware) #66

2026-10-16T20:56:29Z

KofTwentyTwo commented

2026-10-16 20:56:29 +00:00

(Migrated from github.com)

Files attached to an assistant question now reach the model as content, so a user can ask for a draft blog post from text/PDF or ask a vision model to interpret an image. Unreadable files produce notices while readable companion files remain available.

Part of KofTwentyTwo/Website-Backend#305. This is the assistant-library implementation; host dependency adoption and frontend changes are separate.

Read only the signed-in user's own, unexpired upload references through core upload/storage APIs, with per-file and per-turn byte/character limits.
Decode text/Markdown/JSON, PDF page text using existing PDFBox, and CSV/TSV/XLSX previews using established parsers. Refuse executables, unsupported archives, macro workbooks and unsupported binary formats.
Frame extracted content as untrusted data; preserve existing tool authorization/approval boundaries. Attachments count as PHI in HIPAA mode.
Add provider image parts for Anthropic/Bedrock and OpenAI-compatible adapters; require model vision capability and remove images on non-vision fallback.
Validate PNG/JPEG/GIF/WebP with ImageIO before submission. Check dimensions before decoding, cap total decoded pixels at 16 million and frames at 100, and reject decoder warnings/truncated data. Preserve accepted original bytes. WebP uses an explicit TwelveMonkeys 3.12.0 decoder dependency.
Preserve the host's PDFBox font mapper; attachment extraction no longer replaces JVM-global font configuration.
Advertise attachment availability/types/limits through assistant config and document the API fields.

Limits: defaults are 5 files, 10 MiB/file, 25 MiB/turn, 50k characters/file, 150k characters/turn, 50 PDF pages, 20 preview rows, 5 MiB/image and 8000 pixels per side. PDFs provide extractable text, not OCR; encrypted PDFs are refused. XLSX reads the first sheet. Unsupported decoder variants receive a notice. PDF scratch storage is capped and memory-only; this does not impose a whole-process heap limit, and normal host font-cache behavior is preserved.

Validation:

Two pre-fix regressions failed as expected: header-only image accepted and host PDF font mapper replaced.
22 focused tests pass, 0 failures/errors/skips: real image formats, truncated data, dimensions/pixel/frame bounds, valid animated GIF, mapper preservation, ownership/caps, and a completed turn with a malformed image plus readable text.
Full reactor ./mvnw -B verify: passed all eight modules and coverage gates in 4m46s; 500 tests, 0 failures, 0 errors, 5 skipped.
Separate actual-library probe accepted a real two-frame animated WebP (ImageReader confirmed two frames).
Original PR also adds provider contract, PHI-routing and prompt-injection tests; final full verification includes them.

No deployment or dependency adoption is included in this PR.

Files attached to an assistant question now reach the model as content, so a user can ask for a draft blog post from text/PDF or ask a vision model to interpret an image. Unreadable files produce notices while readable companion files remain available. Part of KofTwentyTwo/Website-Backend#305. This is the assistant-library implementation; host dependency adoption and frontend changes are separate. - Read only the signed-in user's own, unexpired upload references through core upload/storage APIs, with per-file and per-turn byte/character limits. - Decode text/Markdown/JSON, PDF page text using existing PDFBox, and CSV/TSV/XLSX previews using established parsers. Refuse executables, unsupported archives, macro workbooks and unsupported binary formats. - Frame extracted content as untrusted data; preserve existing tool authorization/approval boundaries. Attachments count as PHI in HIPAA mode. - Add provider image parts for Anthropic/Bedrock and OpenAI-compatible adapters; require model vision capability and remove images on non-vision fallback. - Validate PNG/JPEG/GIF/WebP with ImageIO before submission. Check dimensions before decoding, cap total decoded pixels at 16 million and frames at 100, and reject decoder warnings/truncated data. Preserve accepted original bytes. WebP uses an explicit TwelveMonkeys 3.12.0 decoder dependency. - Preserve the host's PDFBox font mapper; attachment extraction no longer replaces JVM-global font configuration. - Advertise attachment availability/types/limits through assistant config and document the API fields. Limits: defaults are 5 files, 10 MiB/file, 25 MiB/turn, 50k characters/file, 150k characters/turn, 50 PDF pages, 20 preview rows, 5 MiB/image and 8000 pixels per side. PDFs provide extractable text, not OCR; encrypted PDFs are refused. XLSX reads the first sheet. Unsupported decoder variants receive a notice. PDF scratch storage is capped and memory-only; this does not impose a whole-process heap limit, and normal host font-cache behavior is preserved. Validation: - Two pre-fix regressions failed as expected: header-only image accepted and host PDF font mapper replaced. - 22 focused tests pass, 0 failures/errors/skips: real image formats, truncated data, dimensions/pixel/frame bounds, valid animated GIF, mapper preservation, ownership/caps, and a completed turn with a malformed image plus readable text. - Full reactor `./mvnw -B verify`: passed all eight modules and coverage gates in 4m46s; 500 tests, 0 failures, 0 errors, 5 skipped. - Separate actual-library probe accepted a real two-frame animated WebP (ImageReader confirmed two frames). - Original PR also adds provider contract, PHI-routing and prompt-injection tests; final full verification includes them. No deployment or dependency adoption is included in this PR.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-assistant#66