feat(peering): sync configured peers over NIP-77 and a live feed

.changeset/peering-sync.md

@@ -0,0 +1,25 @@
+---
+"nostream": minor
+---
+
+feat(peering): sync configured peers over NIP-77 and a live feed
+
+Adds an outbound peering-sync worker. For every enabled `peering.peers`
+entry it reads the peer's NIP-11 `peering.scopes`, authenticates with NIP-42
+using the relay's own keypair, reconciles each scope over NIP-77 negentropy,
+fetches only the missing ids, and keeps a live subscription for the same
+scopes. Periodic reconciliation and reconnect-and-resync close gaps the live
+feed can miss, and a new or restarted peer converges without a full dump.
+
+Fetched and live events are accepted only when they match the agreed scope,
+were requested by id, and pass id/signature verification. They are applied with
+the same kind rules as local writes (addressable `d`-tag keys, kind-5 deletes,
+kind-62 vanishes, ephemeral broadcast-only), and authors under an active vanish
+request are skipped. Live events are broadcast to local subscribers; backfill is
+not, so a full sync does not fan out a dump. Wide scopes are split by kind and
+time, overflowed fetch batches are retried in smaller pieces, live re-checks use
+the scope bounds rather than the subscription overlap, and fetch/live buffers are
+bounded by count and bytes. Per-peer sync state (status, last reconciliation,
+lag, missing count) is heartbeat-refreshed during long reconciliations, recorded
+in Redis, exposed at `GET /admin/peering`, shown in the admin console's
+Operations view, and included in diagnostics exports.

CONFIGURATION.md

@@ -348,3 +348,10 @@ terms, or contact details are exposed.
 only when a NIP-42 AUTH event proves one of those pubkeys; matching a URL or
 self-declaring a key is not enough. The relay's own identity, key storage, and
 the AUTH exchange are documented in [docs/PEERING.md](docs/PEERING.md).
+
+Each enabled `peering.peers` entry is also the outbound sync list: a single
+peering-sync worker reads that peer's NIP-11 `peering.scopes`, authenticates
+with NIP-42, reconciles history over NIP-77, and keeps a live subscription to
+those scopes. No extra feed configuration is required. Per-peer sync state is
+available at `GET /admin/peering` and in the admin console's Operations view.
+The peer must advertise NIP-77 and have it enabled.

docs/PEERING.md

@@ -5,8 +5,8 @@ side must be able to prove which relay it is: a URL is an address, not an
 identity. Nostream uses NIP-42 AUTH signed with the relay's own keypair for that
 proof, and a configured pubkey list for trust.
 
-This phase covers identity only. Feed treatment, sync, on-demand requests,
-payments, and the admin console build on it.
+This document covers relay identity, peer-aware feed treatment, and outbound
+sync. On-demand requests, payments, and the full admin console build on it.
 
 ## Relay keypair
 
@@ -81,6 +81,55 @@ authenticated for the caller to use for a feed or negentropy session. This
 proves our identity to the peer; it does not verify the peer's identity in
 return.
 
+## Outbound sync
+
+Sync is driven by the peer's advertisement, not by a second config-file feed
+list. For every enabled `peering.peers` entry the relay starts one peering-sync
+worker session:
+
+1. Read the peer's NIP-11 document and its `peering.scopes`. The document's
+   `self` must match the configured peer pubkey when it is present, and the
+   peer must advertise NIP-77. Redirects to loopback, private, or link-local
+   destinations are refused.
+2. Authenticate with NIP-42 using this relay's keypair.
+3. Open the live `REQ` first, starting from a short overlap so events published
+   while history is syncing are not lost. A rolling `windowSeconds` is checked
+   against the current time, and never widened past its own cutoff.
+4. Reconcile each advertised scope with a NIP-77 negentropy session. A scope is
+   split into filters by kind range and, if the peer still rejects it as too
+   large, by time range. Only the ids the peer has and we lack are fetched, in
+   `REQ` batches, so an empty peer converges without a full dump.
+5. Reconcile again every 15 minutes. Any disconnect reconnects and reruns the
+   reconciliation, which closes any gap the live feed missed.
+
+Fetched and live events are accepted only when they match the scope, were
+requested by id, and pass id and signature verification. They are then applied
+with the same kind rules as locally published events: addressable events keep
+their `d` tag as the deduplication key, kind-5 deletes hide their targets, and
+kind-62 requests delete the author's events and set the vanish flag only when
+their relay tags target this relay. Ephemeral events are broadcast live but
+never stored. Authors under an active vanish request are skipped, with the
+vanish check cached once per author per batch. Live events are also broadcast to
+local subscribers, so a peer feed is indistinguishable from a locally published
+event to this relay's clients. Backfilled history is not broadcast, because it
+predates any live subscription.
+
+If the peer rejects a filter as too large, the worker splits it by kind and then
+by time range and retries. A fetch batch that exceeds the byte budget is split
+into smaller batches without dropping the connection. The live queue and fetch
+buffers are bounded by both count and bytes, and a failing live event is logged
+without stalling the rest of the queue. The per-peer state heartbeat keeps a long
+reconciliation from being reported as stale.
+
+Each reconciliation writes the peer's state to Redis and exposes it at
+`GET /admin/peering`: status, last reconciliation, latest event age ("lag"),
+and the number of events the last reconciliation found missing. The admin
+console shows the same rows under Operations → Peering.
+
+NIP-77 must be enabled on the source relay (`nip77.enabled` and the PostgreSQL
+event store). `peering.peers` on the receiving relay is the only trust material
+an operator supplies.
+
 ## Feed treatment
 
 Once a connection has proven a configured peer pubkey, its subscription and
@@ -106,8 +155,10 @@ message limits come from `limits.peer`, not `limits.client` or `limits.message`:
   limits. An unauthenticated or unconfigured connection stays on the client
   limits exactly as before.
 
-Scope enforcement (only the agreed kinds and time ranges) is part of the sync
-work; this phase grants feed-sized limits to trusted peers.
+Scope enforcement (only the agreed kinds and time ranges) is applied on the
+outbound sync path: the sync worker reconciles only the peer's advertised
+scopes and drops fetched events that fall outside them. Inbound feed scope
+enforcement remains part of the sync work that is still in progress.
 
 Only the public half of the keypair is ever visible on the wire. Inbound AUTH is
 verified by `AuthMessageHandler`: event id, signature, timestamp, challenge, and

resources/admin/assets/admin-ops.js

@@ -10,6 +10,8 @@
   const elements = {
     workersSync: document.getElementById('ops-workers-sync'),
     workersBody,
+    peeringSync: document.getElementById('peering-sync'),
+    peeringBody: document.getElementById('peering-body'),
     nip05Error: document.getElementById('ops-nip05-error'),
     nip05Domain: document.getElementById('ops-nip05-domain'),
     nip05Verified: document.getElementById('ops-nip05-verified'),
@@ -100,8 +102,71 @@
       .join('')
   }
 
-  const renderNip05 = (rows, pagination) => {
+  const formatLag = (seconds) => {
+    if (seconds === undefined || seconds === null) {
+      return '—'
+    }
+    if (seconds < 60) {
+      return `${Math.max(0, Math.round(seconds))}s`
+    }
+    if (seconds < 3600) {
+      return `${Math.round(seconds / 60)}m`
+    }
+    if (seconds < 86400) {
+      return `${Math.round(seconds / 3600)}h`
+    }
+    return `${Math.round(seconds / 86400)}d`
+  }
+
+  const peeringSignalBadge = (row) => {
+    if (!row.configured || !row.enabled) {
+      return '<span class="status-badge"><span class="status-led" aria-hidden="true"></span>Disabled</span>'
+    }
+    if (row.state?.status === 'error') {
+      return '<span class="status-badge status-degraded"><span class="status-led" aria-hidden="true"></span>Error</span>'
+    }
+    if (row.stale) {
+      return '<span class="status-badge status-degraded"><span class="status-led" aria-hidden="true"></span>Stale</span>'
+    }
+
+    switch (row.state?.status) {
+      case 'live':
+        return signalBadge(false)
+      case 'connecting':
+      case 'reconciling':
+        return '<span class="status-badge"><span class="status-led" aria-hidden="true"></span>Syncing</span>'
+      case 'stopped':
+        return '<span class="status-badge status-degraded"><span class="status-led" aria-hidden="true"></span>Stopped</span>'
+      default:
+        return '<span class="status-badge"><span class="status-led" aria-hidden="true"></span>Pending</span>'
+    }
+  }
+
+  const renderPeering = (rows) => {
     if (!Array.isArray(rows) || !rows.length) {
+      elements.peeringBody.innerHTML =
+        '<tr><td colspan="6" class="admin-muted small">No peering peers configured.</td></tr>'
+      return
+    }
+
+    elements.peeringBody.innerHTML = rows
+      .map((row) => {
+        const state = row.state
+        const status = state?.status ? String(state.status) : 'no sync yet'
+        const title = [row.relayUrl, row.pubkey, status].filter(Boolean).join(' · ')
+        return `<tr>
+          <td><code class="small" title="${escapeAttr(title)}">${escapeHtml(shortPubkey(row.pubkey))}</code></td>
+          <td class="small">${escapeHtml(row.relayUrl)}</td>
+          <td class="small">${escapeHtml(formatTime(state?.lastReconciledAt))}</td>
+          <td>${escapeHtml(formatLag(state?.lagSeconds))}</td>
+          <td>${escapeHtml(state?.missingEvents ?? '—')}</td>
+          <td>${peeringSignalBadge(row)}</td>
+        </tr>`
+      })
+      .join('')
+  }
+
+  const renderNip05 = (rows, pagination) => {    if (!Array.isArray(rows) || !rows.length) {
       elements.nip05Body.innerHTML = '<tr><td colspan="7" class="admin-muted small">No matching verifications.</td></tr>'
     } else {
       elements.nip05Body.innerHTML = rows
@@ -204,6 +269,18 @@
     }
   }
 
+  const loadPeering = async () => {
+    const response = await adminFetch('/peering')
+    if (!response.ok) {
+      throw new Error('Failed to load peering state')
+    }
+    const payload = await response.json()
+    renderPeering(payload.peers)
+    if (elements.peeringSync) {
+      elements.peeringSync.innerHTML = `<span class="prompt-char">&gt;</span> peers: updated ${formatTime(new Date().toISOString())}`
+    }
+  }
+
   const buildNip05Query = () => {
     const params = new URLSearchParams()
     params.set('limit', String(nip05Limit))
@@ -284,9 +361,16 @@
   }
 
   const refreshAll = async () => {
-    try {
-      await Promise.all([loadWorkers(), loadNip05()])
-    } catch {
+    const results = await Promise.allSettled([loadWorkers(), loadPeering(), loadNip05()])
+
+    if (results[1]?.status === 'rejected') {
+      if (elements.peeringSync) {
+        elements.peeringSync.innerHTML =
+          '<span class="prompt-char">&gt;</span><span class="text-danger"> peers: unable to refresh sync state</span>'
+      }
+    }
+
+    if (results[0]?.status === 'rejected' || results[2]?.status === 'rejected') {
       showNip05Error('Unable to refresh operations data.')
     }
   }

resources/admin/dashboard.html

@@ -226,6 +226,29 @@ <h2 class="section-title">Worker ops</h2>
             </div>
           </section>
 
+          <section class="dashboard-section">
+            <h2 class="section-title">Peering</h2>
+            <p id="peering-sync" class="sync-line mb-2"><span class="prompt-char">&gt;</span> peers: not loaded</p>
+            <div class="table-responsive panel-card">
+              <table class="table table-sm mb-0" aria-describedby="peering-caption">
+                <caption id="peering-caption" class="visually-hidden">Configured peering peers and their sync state</caption>
+                <thead>
+                  <tr>
+                    <th scope="col">Peer</th>
+                    <th scope="col">Relay</th>
+                    <th scope="col">Last reconciliation</th>
+                    <th scope="col">Lag</th>
+                    <th scope="col">Missing</th>
+                    <th scope="col">Signal</th>
+                  </tr>
+                </thead>
+                <tbody id="peering-body">
+                  <tr><td colspan="6" class="admin-muted small">No peering peers configured.</td></tr>
+                </tbody>
+              </table>
+            </div>
+          </section>
+
           <section id="network-health-section" class="dashboard-section">
             <div class="d-flex flex-wrap justify-content-between align-items-center gap-2 mb-2">
               <h2 class="section-title mb-0">Tor / I2P &amp; network probes</h2>

resources/default-settings.yaml

@@ -198,7 +198,9 @@ peering:
   terms: []
   # Peering peers trusted by their relay pubkey (hex or npub), never by URL
   # alone. A connection that proves one of these keys with NIP-42 AUTH is
-  # recognized as a peer; any other key gets no peer recognition.
+  # recognized as a peer; any other key gets no peer recognition. Enabled
+  # peers are also synced: the peering-sync worker reads each peer's NIP-11
+  # peering scopes, reconciles them over NIP-77, and keeps a live feed.
   # Example:
   #   peers:
   #     - pubkey: replace-with-peer-relay-pubkey-in-hex

src/@types/peering.ts

@@ -0,0 +1,40 @@
+import { PeeringScope, PeeringTerm } from './settings'
+
+/** One scope advertised by a peer in its NIP-11 `peering` section. */
+export type AdvertisedPeeringScope = PeeringScope
+
+export interface PeeringAdvertisement {
+  enabled: boolean
+  scopes: AdvertisedPeeringScope[]
+  terms: PeeringTerm[]
+  /** The relay key the peer advertises as `self`, when it publishes one. */
+  self?: string
+  supportedNips: number[]
+}
+
+export type PeeringSyncStatus = 'idle' | 'connecting' | 'reconciling' | 'live' | 'error' | 'stopped'
+
+/**
+ * Per-peer sync state. Written by the peering-sync worker and read by the admin
+ * console; `lagSeconds` is how old the newest event we hold in a current scope
+ * is, so a healthy live feed stays near zero.
+ */
+export interface PeeringPeerSyncState {
+  pubkey: string
+  relayUrl: string
+  status: PeeringSyncStatus
+  scopeCount: number
+  updatedAt: string
+  lastSyncStartedAt?: string
+  lastSyncFinishedAt?: string
+  lastReconciledAt?: string
+  lastEventAt?: string
+  /** Newest `created_at` held in a scope that is still current (no past `until`). */
+  latestEventCreatedAt?: number
+  lagSeconds?: number
+  /** Events the last reconciliation found on the peer but not locally. */
+  missingEvents?: number
+  /** Events fetched from the peer during the last reconciliation. */
+  receivedEvents?: number
+  lastError?: string
+}