Fetch, Coverage, and Sync: agreed design
Status: approved direction from the session Q&A; runtime implementation and local verification are recorded in the implementation plan. Live relay and authenticated timeline review remain pending. The architecture document describes the current implementation.
1. Decisions
- Compare first; publish eligible confirmed-missing copies only on explicit Sync.
- Prefer NIP-77 reconciliation; use ordinary Nostr timestamp paging as fallback.
- After initial backfill, Fetch gets newer events by default with a ten-second overlap. Recheck history is a separate action. Resume continues unfinished work.
- Preserve per-fetch/per-relay range history rather than overwriting one global relay status. Coverage primarily displays ranges, counts, and observation times.
- Range Refetch/Resume actions belong in a compact
⋮menu. - Fetch automatically updates Coverage and the per-relay Sync tree. This never automatically publishes events.
- Sync defaults to all eligible confirmed-missing archived events on configured write relays; users may narrow by relay, kind, or date.
- Sync records publication acknowledgements only; the user runs Recheck history afterward. Accepted is not storage evidence. Inconclusive acknowledgement is Unverified, distinct from Rejected. This supersedes immediate per-ID verification.
- Ordinary fetch defaults to 100 events per page, adjustable in fetch settings. The effective limit is saved with the run and reused on Resume.
- Keep query bounds, request/completion timestamps, and EOSE/NOTICE/rejection/ timeout/disconnect outcomes. Age alone does not erase presence observations.
- Specialized-relay kind-filter fallbacks and hostname-specific behavior are
postponed. There is no planned special case for
purplepag.es. - Coverage relay headers offer a fresh relay-only Recheck history, also exposed after a finished single-destination Sync pass. This scans beginning through now for one current fetch source; saved-range Refetch/Resume retain their old bounds. The global history action still checks all sources, and Fetch remains the deduplicated archive grouped by kind/year/month. Sync shortcut scope comes from the complete saved delivery snapshot, not a single outcome page.
2. Shared flow and relay roles
Account
|
+-- configured NIP-65 read/write relays
+-- extra fetch sources
|
write relays + extras
|
Fetch / Back up
|
deduplicated local archive
|
per-fetch range evidence/history
|
Coverage + Sync candidates
|
explicit Sync action
|
configured write relays only
|
Accepted -> user Recheck history -> Coverage
The local read/write list may differ from its published NIP-65 snapshot during a migration. Existing Reset to published relays and Publish NIP-65 remain separate local-replacement and signing/publication actions. Read-only relays are excluded from authored-history fetching/sync unless separately added as an extra source. An extra source never becomes a sync destination merely because it was queried.
3. Vocabulary and evidence
- Found: distinct event IDs established on a relay in this observation's scope.
- Fetched: found events whose original signed bodies are validated and archived.
- Not fetched: identified remote event IDs still awaiting a validated body. It does not mean events missing from the relay.
- Newly archived: bodies first inserted into the shared archive during the run.
- Received deliveries: raw repeat deliveries across pages/retries, shown only in details; these are not unique Found or newly archived counts.
- Missing copy: an eligible local event absent from a completed, applicable relay comparison. Used by the Sync tree, not as the headline Coverage vocabulary.
- Undetermined: insufficient applicable evidence to establish relay absence.
- Invalid observation: malformed/future observation metadata, reported separately.
This is distinct from an event's
created_atand from an incomplete network query.
Signature validation remains mandatory for archived/downloaded bodies. Reconciliation can identify remote IDs before their bodies are downloaded; display Found separately from Fetched while that transfer is pending.
Historical facts retain their timestamp and provenance. A failed query does not erase prior successful observations. Present is not a promise of permanent relay retention. A completed observation can update earlier evidence for the same scope; an incremental query does not alter conclusions about older, unqueried ranges.
4. Fetch modes and checkpoints
State is independent for each account/source relay, not one global account cursor. Snapshot source URLs, filters, chosen page limit, and run boundary when starting. Adding a source gives it its own initial-backfill state; edits do not retroactively change an existing run's query scope.
Initial history and fallback paging
Use a fixed run-start timestamp T as the first upper bound. Initial history has
no lower bound: Beginning represents an open historical bound, not an invented
timestamp for the author's first event.
REQ { authors: [account], until: T, limit: 100 }
|
archive verified returned events
|
literal EOSE arrived?
/ \
no yes
| |
partial/error page shorter than effective limit?
keep checkpoint / \
yes no
| |
history queried to end sort by created_at
under relay semantics earliest timestamp E
|
next REQ until=E (inclusive)
deduplicate repeated boundary IDs
Use the inclusive earliest timestamp to avoid silently skipping events sharing
that second. Persist boundary IDs/progress sufficient to detect repeated pages.
If paging stalls at a saturated timestamp, leave the scope partial and report the
reason; do not advance to E - 1 and pretend the skipped bucket is complete.
The normal strategy is backward until paging, replacing the current default
recursive window splitting.
A short page with EOSE is a completion signal under the relay's effective query limits, not proof of an immutable/global earliest event. Record the requested limit and completion basis. Relay caps, filters, and retention can limit recovery; reject inconclusive evidence rather than using it to establish Missing.
Incremental fetching
After initial history completes:
since = last successful forward boundary - 10 seconds
until = current run's start timestamp
limit = run's saved page size
Page backward within this fixed interval if needed. Advance the successful forward
boundary only after the entire interval finishes. Preserve since, fixed until,
current paging cursor, and page progress on failure/interruption. Resume uses those
saved values, not a new run boundary or changed settings. Newer activity can be
fetched in a subsequent run after the resumed snapshot completes.
Store history_complete separately from the last successful forward boundary.
Completed historical work does not mean future Fetch work is unnecessary.
Refetch and Recheck history
Old signed events can be added to a relay later with their old created_at intact.
An incremental timestamp query cannot discover every such import. Recheck history
revisits older scopes; a range's Refetch revisits that exact scope. Both create new
observation attempts, retaining the earlier results. Resume appends query attempts
to the same unfinished logical range/run.
5. NIP-77 preferred strategy
NIP-77 compares timestamp/ID sets for a standard filter:
local matching set <-> NEG-OPEN / NEG-MSG <-> relay matching set
|
remote-only IDs -> REQ -> validate/archive
local-only IDs -> candidate missing copies
common IDs -> relay presence evidence
NIP-77 discovers differences; it does not transfer event bodies or publish them. Fetching from an extra source consumes remote-only events without uploading local- only events. Sync candidates are computed only for configured write destinations.
Use the same logical ranges, history records, and UI for both strategies. Preserve method and outcome per attempt. Advertised NIP-11 support may guide strategy choice; runtime negotiation is authoritative. Unsupported negotiation uses the paging fallback; timeout/NEG-ERR remains recorded and cannot establish absence. Large reconciliation scopes may require bounded subscopes; their completion must be explicit, not inferred from one successful fragment.
Full-range rechecking can discover newly imported old events without redownloading common bodies. Recent default fetching remains incremental as agreed. Reconciliation of a recent interval must not update older history outside that filter.
6. Coverage UI
Coverage includes fetch-source relays (write relays plus extras), grouped by relay and logical range. Sync separately restricts destinations to write relays.
COVERAGE
v Relay A
> Beginning -> DATEX Found (n) Checked T1 Complete [⋮]
> DATEX -> DATEY Found (m) Checked T2 Complete [⋮]
> DATEY -> DATEZ Found (k) Checked T3 Partial [⋮]
Expanded range
Fetched n / Found n
> kind 1 (count)
> kind 7 (count)
Strategy: NIP-77 / ordinary REQ
Query bounds / cursor / chosen limit
Observation attempts and query outcome details
Menu: Refetch for an existing range; Resume for unfinished ranges. Disable actions
while that range is active. Failed/partial scopes remain visible, not represented
as complete zero-result ranges. Complete means the comparison/query scope
finished; show any remaining Not fetched bodies separately.
Display overlapping timestamp endpoints accurately. Inclusive boundary deliveries are deduplicated by ID within an observation; summing overlapping range counts must not be presented as a unique account-event total. The archive count is unique IDs across all runs and relays.
7. Comparison and Sync
Automatically recompute candidates after relevant fetch/comparison evidence is committed. Do not require a second Compare button. A scan on one relay says nothing about an unqueried relay. Newly archived events may be compared against applicable completed inventories from other relay/range observations; keep the evidence's observation time visible. Coverage is historical evidence, not a simultaneous snapshot of the whole network.
Only absence established by a completed applicable comparison becomes a candidate. Not returned by a limited/partial page is insufficient. Absence is neither created by age nor by connection failures. A limited IDs query likewise cannot establish absence if the response may have been truncated.
SYNC total missing copies
v Relay A (count)
> kind X (count)
> kind Y (count)
v Relay B (count)
> kind X (count)
[Sync missing events]
One event missing on two relays is two deliveries. Default scope is all eligible confirmed-missing copies; optional relay/kind/date narrowing includes collapsed branches. Preserve the exact selected candidate/scope snapshot with the sync run.
Send original signed events without re-signing or changing IDs/timestamps. Current Sync outcomes are Pending, Accepted, Rejected, and Unverified (uncertain acknowledgement). Historical Verified records remain readable, but Sync no longer performs immediate read-back, on initial publication or Resume.
Process continuously in bounded 200-copy chunks, with Stop/Resume. Pending copies precede retries; each copy is attempted at most once in one invocation, avoiding an automatic failure retry loop. Accepted and historical Verified deliveries are skipped. Publication completion does not assert storage: the user runs Recheck history after Sync, using existing NIP-77 or ordinary paging to update Coverage. Recent incremental Fetch may not cover historical event timestamps.
A newer Accepted acknowledgement can place an older missing/undetermined pair in Awaiting recheck. This is a publication overlay, never a positive storage observation. Keep it out of ordinary missing-copy selection until a later applicable comparison establishes present/absent evidence. Failed or unrelated queries do not clear it. Order equal-time observations by query sequence so a later recheck wins over the acknowledgement. Existing positive storage evidence is not erased by an acknowledgement.
Archive and replay eligibility differ: preserve history locally but avoid replaying expired/deleted events, ephemeral/private events, or superseded replaceable events. Honor deletion requests and latest replaceable/addressable versions before forming candidates. This does not implement new deletion-request creation workflows.
8. Persistence direction
Archive export is implemented separately from Sync: Fetch offers hierarchical and
individual selection, Select all/Clear selection, and full/selected JSONL downloads
named relays_backup_${timestamp}.jsonl. Scope rules include unloaded descendants;
the local archive is resolved at export time without replay-eligibility filtering.
Original signed values are preserved and overlapping scopes never duplicate events.
Selection lives in the account view rather than an extra database table.
Reuse the existing events archive and operations run identities. Add normalized
range/query/evidence records where provenance and scoped comparison need indexes;
retain JSON for checkpoint/settings payloads rather than a large opaque history blob.
Exact migrations/types are implementation-phase work, but these responsibilities
are required:
| Record | Responsibility |
|---|---|
| Per-account/relay fetch state | History completion and last successful incremental boundary |
| Fetch range/attempt | Run, relay, filter/bounds, strategy, settings snapshot, status, counts, timestamps, checkpoint; prior attempt link for Refetch |
| Query log | Range, exact REQ/NEG scope, sent/completed timestamps, limit, EOSE/NEG completion, NOTICE/error/timeout/disconnect, received/unique counts |
| Range-event evidence | Range/attempt/query provenance and found/common/local-only IDs; body availability through the shared archive |
| Current comparison projection | Latest applicable positive/negative evidence by event/relay, with provenance; rebuildable cache, no age TTL |
| Sync deliveries | Run/event/relay, attempts, publication-pass marker, acknowledgement/rejection/error; historical verification fields retained |
Reconciliation may identify an ID before an archive body exists; an evidence ID must therefore not require a body row prematurely. Failed attempts append outcomes without overwriting earlier evidence. Use query completion times/scopes rather than job-finalization time to order concurrent evidence accurately.
9. Migration from the original behavior
At planning time, the original implementation used 500-event split windows, operation-level pendingWindows JSON, one-hour presence expiry, and a global Coverage table. It did not implement NIP-77, incremental cursors, per-range Coverage, or Sync. The new runtime provides those flows. The user subsequently requested a complete reset of the Relay Manager database, including accounts and relay settings, and removal of legacy code. This supersedes the earlier migration/preservation requirement.
The plugin now initializes only the current schema. The old split-window runner, legacy Resume option/UI, scaffold CRUD, and legacy presence tables/migration are removed. Normal Resume still preserves current NIP-77/paging checkpoints; old 500-event checkpoints are not imported or reinterpreted.
10. Deferred work
- Specialized-relay kind-filter fallback and related capability policy.
- JSONL import. Archive selection/JSONL export were approved and implemented in a subsequent iteration; see the implementation plan.
- New deletion-request workflows, media-file backup, and unattended scheduling.
- Relay authentication and persisted preferred-signer references remain existing follow-ups; do not confuse authentication failure with a completed empty scan.
Implementation milestones and exit criteria live in the implementation plan. This document records the agreed direction; checked implementation milestones distinguish delivered work from pending review.