feat(proto): keyset pagination for all list/search RPCs (AIP-158) #350

2026-07-31T21:36:06Z

KofTwentyTwo commented

2026-07-31 21:36:06 +00:00

(Migrated from github.com)

Summary

Applies one uniform AIP-158 keyset pagination convention to every nuncio.v1 list/search RPC. Each request now carries page_size + page_token; each response carries next_page_token. The page_token is an opaque keyset cursor (not an offset), so paging is stable under concurrent inserts. Reconciles Audit.ListRecords off its previous unstable limit/offset.

Closes the inconsistency where ListMessages was limit-only, several lists were fully unbounded, and Audit alone used limit/offset.

RPCs paginated

Mail.ListMessages, Mail.ListFolders, Mail.SearchMessages, Filters.ListRules, Calendar.ListEvents, Contacts.ListContacts, Audit.ListRecords.

Cursor + limits

Encoding: opaque page_token = hex(JSON) of a typed cursor component list, private to the daemon (nunciod::pagination). Clients treat it as opaque and echo it back.
page_size: default 50, max 500 (0 = default; clamped).
Per-list sort key / cursor: messages (received_at DESC, id DESC); folders (folder_id ASC); rules (priority ASC, id ASC); events (start_time ASC, id ASC) over the recurrence-expanded set; contacts (interaction_count DESC, display_name ASC, id ASC); search (rank ASC, id ASC) via a rank-materializing subquery; audit (sequence ASC).
Malformed page_token -> typed VALIDATION_FAILED ErrorInfo (via WS-A1 nuncio_proto::errors) mapped to InvalidArgument -- never a panic or silent empty page.

Store

Added keyset *_page methods fetching page_size + 1 to detect a following page. Rule paging computes the boundary from raw (priority, id) so unparseable rows still advance the cursor without dupes/gaps. Search materializes bm25 rank in an inner query so the outer query can keyset-filter and order by (rank, id).

CLI

List/search commands transparently follow next_page_token to the end (output unchanged). Audit list replaces --limit/--offset with --page-size (per-page size; all pages followed).

Contract

nuncio.proto is the only proto edited this cycle; the committed golden descriptor.bin was regenerated and contract_stability::descriptor_matches_committed_golden passes.

Tests

list_messages_keyset_pagination_returns_every_message_once_no_gaps -- 7 messages (some sharing a timestamp) at page_size 2; asserts full set once, newest-first incl. id tiebreaker.
list_records_keyset_pagination_returns_every_record_once_no_gaps -- 7 records at page_size 2, ascending, once; plus malformed-token check.
list_messages_rejects_malformed_page_token -- asserts InvalidArgument + VALIDATION_FAILED.
search_messages_page_returns_every_hit_once_no_gaps (store) -- exercises the FTS (rank, id) keyset.
nunciod::pagination unit tests -- clamp, round-trip, malformed, wrong-shape rejection.

Gate (all green, 0 ignored)

cargo fmt --all -- --check   # clean
cargo check-all              # clippy -D warnings, 0 warnings
cargo test-all               # 524 passed; 0 failed; 0 ignored

Golden regenerated; contract_stability passes.

Notes / follow-ups

Filters.GetExecutionLogs still uses a limit-capped tail and was intentionally left out of this pass (not in the WS-A3 RPC list; it is a bounded newest-first ledger tail, not a full paginated listing). Worth a follow-up if a full paginated view is wanted.

Refs #275

## Summary Applies one uniform **AIP-158 keyset pagination** convention to every `nuncio.v1` list/search RPC. Each request now carries `page_size` + `page_token`; each response carries `next_page_token`. The `page_token` is an **opaque keyset cursor** (not an offset), so paging is stable under concurrent inserts. Reconciles `Audit.ListRecords` off its previous unstable `limit`/`offset`. Closes the inconsistency where `ListMessages` was limit-only, several lists were fully unbounded, and Audit alone used limit/offset. ### RPCs paginated `Mail.ListMessages`, `Mail.ListFolders`, `Mail.SearchMessages`, `Filters.ListRules`, `Calendar.ListEvents`, `Contacts.ListContacts`, `Audit.ListRecords`. ### Cursor + limits - **Encoding:** opaque `page_token` = hex(JSON) of a typed cursor component list, private to the daemon (`nunciod::pagination`). Clients treat it as opaque and echo it back. - **page_size:** default **50**, max **500** (0 = default; clamped). - **Per-list sort key / cursor:** messages `(received_at DESC, id DESC)`; folders `(folder_id ASC)`; rules `(priority ASC, id ASC)`; events `(start_time ASC, id ASC)` over the recurrence-expanded set; contacts `(interaction_count DESC, display_name ASC, id ASC)`; search `(rank ASC, id ASC)` via a rank-materializing subquery; audit `(sequence ASC)`. - **Malformed `page_token`** -> typed `VALIDATION_FAILED` `ErrorInfo` (via WS-A1 `nuncio_proto::errors`) mapped to `InvalidArgument` -- never a panic or silent empty page. ### Store Added keyset `*_page` methods fetching `page_size + 1` to detect a following page. Rule paging computes the boundary from raw `(priority, id)` so unparseable rows still advance the cursor without dupes/gaps. Search materializes bm25 `rank` in an inner query so the outer query can keyset-filter and order by `(rank, id)`. ### CLI List/search commands transparently follow `next_page_token` to the end (output unchanged). Audit `list` replaces `--limit/--offset` with `--page-size` (per-page size; all pages followed). ### Contract `nuncio.proto` is the only proto edited this cycle; the committed golden `descriptor.bin` was regenerated and `contract_stability::descriptor_matches_committed_golden` passes. ## Tests - `list_messages_keyset_pagination_returns_every_message_once_no_gaps` -- 7 messages (some sharing a timestamp) at page_size 2; asserts full set once, newest-first incl. id tiebreaker. - `list_records_keyset_pagination_returns_every_record_once_no_gaps` -- 7 records at page_size 2, ascending, once; plus malformed-token check. - `list_messages_rejects_malformed_page_token` -- asserts `InvalidArgument` + `VALIDATION_FAILED`. - `search_messages_page_returns_every_hit_once_no_gaps` (store) -- exercises the FTS `(rank, id)` keyset. - `nunciod::pagination` unit tests -- clamp, round-trip, malformed, wrong-shape rejection. ## Gate (all green, 0 ignored) ``` cargo fmt --all -- --check # clean cargo check-all # clippy -D warnings, 0 warnings cargo test-all # 524 passed; 0 failed; 0 ignored ``` Golden regenerated; `contract_stability` passes. ## Notes / follow-ups - `Filters.GetExecutionLogs` still uses a `limit`-capped tail and was intentionally left out of this pass (not in the WS-A3 RPC list; it is a bounded newest-first ledger tail, not a full paginated listing). Worth a follow-up if a full paginated view is wanted. Refs #275

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#350