diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 150782185..300e3da12 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -163,8 +163,9 @@ This can release user database handles held by application components so an onli Also include these constraints in the restore procedure: -- **`target_database` requires the server stopped.** Restoring into a separate database rather than overwriting the source is CLI-only with Harper down. The target must not already exist or must be an empty directory. -- **An interrupted restore leaves the database unloadable.** On a crash or power loss mid-restore, Harper marks the database as incompletely restored and refuses to load it on the next start. Recover by rerunning `restore_backup` for the same database and `backup_id`. Do not try to load or hand-repair the directory. +- **`target_database` restores into a separate database.** The target must not already exist or must be an empty directory. As of v5.4.0 this works online too; before v5.4.0 it required the CLI with Harper down. +- **An interrupted restore leaves the database as it was, restored, or blocked.** As of v5.4.0, a restore that stopped before replacing anything is discarded at the next start, and one proven before the crash is finished. A crash or power loss while files are being replaced leaves the database marked as incompletely restored, and Harper refuses to load it. Recover by rerunning `restore_backup` for the same database and `backup_id`. Do not try to load or hand-repair the directory. +- **A restore refuses to run on a node with replication peers.** As of v5.4.0, peers in `system.hdb_nodes` or `replication.routes` block it, even when replication to them is blocked by the network. See the procedure below. ## Restoring in a replicated cluster @@ -187,7 +188,7 @@ flowchart TD For an in-place recovery: 1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. -2. **Isolate replication separately.** For example, use a network policy that blocks peer replication in both directions and closes existing peer connections. Keep it effective across restarts, and verify that transactions cannot cross the boundary. Removing a node from the load balancer or setting `"replicated": false` on one operation does not isolate replication. +2. **Isolate replication separately.** For example, use a network policy that blocks peer replication in both directions and closes existing peer connections. Keep it effective across restarts, and verify that transactions cannot cross the boundary. Removing a node from the load balancer or setting `"replicated": false` on one operation does not isolate replication. As of v5.4.0, `restore_backup` also refuses to run while this node lists any peer, so remove the node's peers (its `system.hdb_nodes` rows other than itself, and any `replication.routes`) before restoring; the network policy still matters, because a peer that still lists this node can reconnect to it. 3. Confirm the data authority and replay policy: which writes to preserve, repair, or reapply, and how peers will converge. 4. For an online managed restore allowed by the table above, use its `backup_id` (`1` here) and poll the returned job on the same node: @@ -198,7 +199,7 @@ For an in-place recovery: For an offline restore, use the [local CLI procedure](/reference/v5/backups/operations#running-backup-operations-from-the-cli). For a `get_backup` file, follow the [manual restore procedure](/reference/v5/backups/overview#example-download-a-snapshot-and-restore-it-manually) with Harper stopped and its process exited, including the blob roots. 5. Verify data expectations locally before any peer sees the node. -6. Reconnect replication under the policy you proved in an isolated multi-node drill. Verify convergence, then complete the traffic-admission checks before restoring traffic. If peers still hold the data this restore was meant to undo, do not reconnect this node to them. Rebuild the peers from it, as the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) describes for an authoritative node: once it passes the traffic-admission checks, it returns to rotation, still isolated, as the peers leave, and only the peers wait for convergence. +6. Add the peers back, and reconnect replication under the policy you proved in an isolated multi-node drill. Verify convergence, then complete the traffic-admission checks before restoring traffic. If peers still hold the data this restore was meant to undo, do not reconnect this node to them. Rebuild the peers from it, as the [long-absent node policy](./operating-replication.mdx#what-moves-automatically-and-what-does-not) describes for an authoritative node: once it passes the traffic-admission checks, it returns to rotation, still isolated, as the peers leave, and only the peers wait for convergence. :::danger **Draining traffic does not stop replication.** A drained node continues exchanging transactions with peers. An offline restore stops that exchange while Harper is down, but configured routes can reconnect as soon as it starts. Keep replication isolated until validation is complete and you are ready to execute the re-entry policy. diff --git a/reference/backups/operations.md b/reference/backups/operations.md index 9a56daf1a..bf62af782 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -30,10 +30,18 @@ A target set with `HARPER_CLI_TARGET` or `CLI_TARGET` is always forwarded too, i A target saved by `harper login` is also forwarded, with one exception. If the saved target is a loopback address (`localhost`, `127.x` or `::1`) and the local server is stopped: - `create_backup`, `list_backups` and `verify_backup` run offline against this installation's files. -- `restore_backup`, `delete_backup` and `purge_backups` fail without changing anything, because a loopback address can belong to a different instance, such as an SSH tunnel, a container or another local install. +- `restore_backup`, `delete_backup` and `purge_backups` run offline too when the saved target's port is this installation's operations API port (or secure port). +- On any other port they fail without changing anything, because that address can belong to a different instance, such as an SSH tunnel, a container or another local install. To run one of the failing commands on this installation's files, first run `harper logout `, which also removes that saved login. To run it on the saved target, pass `target=`. +
+Before v5.4.0 + +`restore_backup`, `delete_backup` and `purge_backups` failed for any saved loopback target, including one on this installation's own port, so a local `harper login` had to be logged out before restoring offline. + +
+ Offline invocation matters most for restore: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must be run with the server stopped. See ["when can a database be restored"](./overview.md#when-can-a-database-be-restored). ## `create_backup` @@ -80,10 +88,11 @@ harper list_backups database=data Verifies a managed backup. `backup_id` is required. - **RocksDB files:** checks file sizes and, when `verify_checksum` is `true`, checksums. Checksum verification takes longer. +- **Tables:** opens each table file and reads every table, so a backup this version of Harper cannot read, such as one with a newer table format, fails verification as it would fail a restore. - **Transaction log:** always checks the snapshot's framing. - **Blob snapshot:** checks that a snapshot is present if the backup recorded blobs. A missing snapshot fails verification. Individual blob sizes, checksums, and readability are not checked, so damaged blob contents can pass verification. -Through a running server, verification runs as a background [job](../operations-api/operations.md#jobs). Poll `get_job` for its result. +Through a running server, verification runs as a background [job](../operations-api/operations.md#jobs). Poll `get_job` for its result. As of v5.4.0, verification runs in a separate process, and only one restore or verification runs at a time on an instance; another one fails with status code 409 until it finishes. ```json { "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true } @@ -127,26 +136,45 @@ harper purge_backups database=data keep_count=3 Restores a database in place from a managed backup. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data, and — for a database with file-backed blobs — the blob roots are purged and rewritten from the backup's blob snapshot. - `allow_engine_only` (optional, default `false`) permits restoring a backup created with `exclude_blobs: true`. Without it, that restore is refused, including when using `target_database`. Enabling it accepts the risk that restored records reference missing or unrelated blob files; recover and validate those blobs separately. +Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, checks the backup, restores it, and reloads it. As of v5.4.0, the check and the restore run in a separate process, and only one restore or [verification](#verify_backup) runs at a time on an instance; another one fails with status code 409 until it finishes. This works only when no loaded component is holding the database open — if one does, the job ends in `ERROR` (surfaced by [`get_job`](../operations-api/operations.md#get_job)) telling you to restore offline. Restoring the `system` database is rejected up front, before a job is created. It requires running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) -Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, checks the backup, restores it, and reloads it. This works only when no loaded component is holding the database open — if one does, the job ends in `ERROR` (surfaced by [`get_job`](../operations-api/operations.md#get_job)) telling you to restore offline. Restoring the `system` database is rejected up front, before a job is created, as is `target_database` while the server is running. These cases require running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) +| Parameter | Description | +| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `backup_id` | The backup to restore. Defaults to the latest. | +| `target_database` | Restore into a separate, new database instead of overwriting the source. The target must not exist, or must be an empty directory. As of v5.4.0 this works with the server running too: the source stays online, and the new database loads on every thread when the restore publishes. | +| `allow_engine_only` | Default `false`. Permits restoring a backup created with `exclude_blobs: true`; without it that restore is refused, including with `target_database`. Enabling it accepts the risk that restored records reference missing or unrelated blob files; recover and validate those blobs separately. | +| `accept_incompatible_backup` | Restore a backup taken by a newer version of Harper whose archive manifest major is higher than this version understands. What it waived is logged and returned as `compatibility_waived`. A capability this version lacks is still refused. | +| `accept_unverified_blobs` | Publish the restore even if records reference blob files the backup does not contain. The result reports `blob_integrity: unverified`. | +| `immutable_source` | Restore from a repository Harper cannot write to, such as a read-only mount, without locking or pinning it. Use it only if nothing can modify the repository until the restore finishes. | As of v5.3.1, subscriptions opened before a restore do not carry over to the restored data. When Harper reloads the database, each one ends, and its last message is a `DatabaseGenerationChangedError` (status code 409, code `DATABASE_GENERATION_CHANGED`). Resubscribe to resynchronize against the restored state. A restore that fails before it changes anything reloads the database as it was; those subscriptions end instead with a retryable `DatabaseClosingError` (status code 503, code `DATABASE_CLOSING`); resubscribe to continue. -As of v5.4.0, a restore checks the backup before it deletes anything. It verifies the checksum of every database file in the backup (not blob files), checks the backup's transaction logs, and opens each of its table files to confirm this version of Harper can read their format and compression. For a backup that captured blobs, it also checks that the blob snapshot is present. A backup that fails any of these is refused: the restore fails with an error saying it did not modify the database, followed by the cause, and the database keeps its data and its blobs. A rerun over an earlier restore that did not finish is the exception: that database is already incomplete, so it stays blocked until a restore succeeds, and the error says so. [`verify_backup`](#verify_backup) does not open the table files, so a backup it reports as valid can still be refused here for a format this version cannot read. +### What a restore checks -The check reads the whole backup once before the restore copies it, so a restore takes longer than before but needs no extra disk space. Through a running server, the check runs after the database is closed, so the database is unavailable while the backup is checked as well as while it is copied. Writes during that time fail, as they do for the rest of the restore, rather than succeeding and then being lost. The contents of blob files are not checked: a blob that cannot be read fails the restore while the blobs are copied, after the database files have been replaced, and the database stays marked incompletely restored until a rerun succeeds. +As of v5.4.0, a restore checks the backup before it deletes anything. It verifies the checksum of every database file in the backup (not blob files), checks the backup's transaction logs, and opens each of its table files to confirm this version of Harper can read their format and compression. For a backup that captured blobs, it also checks that the blob snapshot is present. A backup that fails any of these is refused: the restore fails with an error saying it did not modify the database, followed by the cause, and the database keeps its data and its blobs. A rerun over an earlier restore that did not finish is the exception: that database is already incomplete, so it stays blocked until a restore succeeds, and the error says so. + +The pre-copy check reads the whole backup once before the restore copies it, so a restore takes longer than before but needs no extra disk space. Through a running server, the check runs after the database is closed, so the database is unavailable while the backup is checked as well as while it is copied. Writes during that time fail, as they do for the rest of the restore, rather than succeeding and then being lost. + +After the files are replaced, and before the restored database is published, the restore also proves it. The transaction log replays completely, with no skipped entry, torn transaction or corrupt frame. Every record decodes. Every file-backed blob a record references is present and the size the record declares, or is a capture marker the backup recorded for a blob that was still being written; the result reports `blob_integrity: unverified` when any marker was accepted. A reference to a blob the backup does not contain fails the proof unless `accept_unverified_blobs` is set. A restore that fails this proof is not published: the database stays blocked until a restore from a sound backup succeeds.
Before v5.4.0 -Previously the restore deleted the database first, so a backup this version could not read, such as one taken by a newer version of Harper, destroyed the database it was meant to restore. +Previously the restore deleted the database first, so a backup this version could not read, such as one taken by a newer version of Harper, destroyed the database it was meant to restore. Blob files were not checked at all, and `target_database` was refused while the server was running.
-From the CLI with the server stopped, `target_database=` restores into a separate database instead of overwriting the source. The target must not already exist, or must be an empty directory; Harper picks the new database up on the next start. +### If a restore is interrupted + +As of v5.4.0, every restore records its decision on disk before it changes anything, does the work in a separate process, and publishes the restored database only after that process has proven it. A crash at any point leaves one of three states: + +- **The database as it was**, if the restore stopped before it replaced anything. The next start discards the unfinished restore and loads the database normally. +- **The restored database**, if the restore was proven before the crash. The next start, or the next `restore_backup` of that database, finishes publishing it without repeating the work. +- **A blocked database**, if the restore stopped while replacing files. Harper refuses to load it and logs an incomplete-restore error. Recover by rerunning `restore_backup` for the same database and backup; do not try to load or hand-repair the directory. A backup the interrupted restore was using stays protected from `delete_backup` and `purge_backups` until a restore succeeds, unless the restore used `immutable_source`, which pins nothing. + +### Replication peers -If a restore is interrupted (crash, power loss) after it has started replacing the database's files, Harper marks the database as incompletely restored and refuses to load it on the next start, logging an incomplete-restore error. Recover by rerunning `restore_backup` for the same database and `backup_id` — do not try to load or hand-repair the directory. As of v5.4.0, an interruption while the backup is still being checked leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. +As of v5.4.0, a restore refuses to run on a node that has replication peers, whether from `system.hdb_nodes` or `replication.routes`, because replication would either undo the restore or spread it to the cluster. Through a running server the job ends in `ERROR`, with a message naming the peers; offline the command fails with the same message. Peers count even when replication to them is blocked by the network. Restore on a standalone node, then add fresh peers. While a restore runs, operations that add a peer, such as `add_node` or a `set_configuration` that sets replication routes, fail with status code 409. An offline restore of the `system` database checks only `replication.routes`, since it replaces the node list itself. ```json { "operation": "restore_backup", "database": "data", "backup_id": 1 } @@ -154,7 +182,7 @@ If a restore is interrupted (crash, power loss) after it has started replacing t ```bash harper restore_backup database=data backup_id=1 -# restore into a new database instead of overwriting the source (server stopped) +# restore into a new database instead of overwriting the source harper restore_backup database=data backup_id=1 target_database=data_restored ``` diff --git a/reference/backups/overview.md b/reference/backups/overview.md index d853f2fc4..64153040a 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -37,10 +37,10 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **One storage root per database.** Backup is rejected if the database's tables resolve to multiple storage roots. A custom database-level `storage.path`, or per-table paths that still resolve to a single root, are supported. - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. -- **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. +- **A restore is a point-in-time rollback.** A restore refuses to run on a node that has replication peers. Remove the node's peers before restoring, and add fresh ones afterwards; see [restoring in a replicated cluster](/learn/administration/backup-and-recovery#restoring-in-a-replicated-cluster). - **Full-text indexes rebuild locally after restore.** Ordinary table reads remain available. Writes can receive retryable `DERIVED_INDEX_LAGGING` responses while replay catches up after a long rebuild; see [write backpressure](../full-text-search/overview.md#write-backpressure). While the rebuild is active, full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`; branch on the code because other `503` responses are not necessarily rebuilds. -- **A restore checks the backup before it deletes anything.** A backup with a corrupt database file, or one this version of Harper cannot read, is refused and the database is left as it was (a database an earlier restore left incomplete stays blocked). The check reads the whole backup, which adds to how long an online restore keeps the database unavailable. Blob file contents are not checked. See [`restore_backup`](./operations.md#restore_backup). -- **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted after it has started replacing the database's files (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. +- **A restore checks the backup before it deletes anything.** A backup with a corrupt database file, or one this version of Harper cannot read, is refused and the database is left as it was (a database an earlier restore left incomplete stays blocked). The check reads the whole backup, which adds to how long an online restore keeps the database unavailable. Before it publishes, the restore also proves the restored database: its transaction log replays completely, every record decodes and every referenced blob file is present. Blob file contents are not checksummed. See [`restore_backup`](./operations.md#restore_backup). +- **An interrupted restore leaves the database as it was, restored, or blocked.** A restore that stopped before replacing anything is discarded at the next start, and one that was proven before the crash is finished. One interrupted while replacing the database's files leaves it marked as incompletely restored; Harper skips loading it and logs an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. See [if a restore is interrupted](./operations.md#if-a-restore-is-interrupted). ### When can a database be restored? @@ -55,7 +55,7 @@ An in-place restore purges and rewrites the database's files, which requires the - **Online** works only when no loaded component is holding the database open. Because `restore_backup` runs as a background job, the request returns a `job_id` first; if a component is holding the database open, the check fails inside the job, so the job ends in `ERROR` (surfaced by [`get_job`](../operations-api/operations.md#get_job)) with a message telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.) The `system` database is rejected up front, before a job is created. - **Offline** — the same [`restore_backup`](./operations.md#restore_backup) command run from the CLI with the server stopped — works for any database, because no components are loaded and nothing holds it open. This is the required path for the `system` database and for any database a component keeps open. -Online restore always restores in place. To restore into a new database instead of overwriting the source, run `restore_backup` with `target_database` from the CLI with the server stopped. +To restore into a new database instead of overwriting the source, pass `target_database` to `restore_backup`. As of v5.4.0 this works with the server running; before v5.4.0 it required the CLI with the server stopped. ## Example: incremental backups and restore diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index b64b22c0f..fdad3ec7f 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -180,7 +180,19 @@ The new [`storage.rocks.transactionLogWriteIntents`](/reference/v5/database/stor ### Restore Checks a Backup Before Deleting the Database -`restore_backup` no longer deletes a database before checking that the backup can be read. It first verifies the checksum of every database file in the backup, checks its transaction logs, opens each table file to confirm this version can read its format and compression, and, for a backup that captured blobs, checks that the blob snapshot is present. A backup that fails is refused and the database is left as it was (one an earlier restore left incomplete stays blocked); previously the restore deleted the database first and then failed, so a backup taken by a newer version, or one with a corrupt file, destroyed the database it was meant to restore. The check reads the whole backup once, so a restore takes longer, and an online restore keeps the database unavailable while it runs; it needs no extra disk space. A restore interrupted while it is still checking the backup leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. Blob file contents are not checked. See [`restore_backup`](/reference/v5/backups/operations#restore_backup). +`restore_backup` no longer deletes a database before checking that the backup can be read. It first verifies the checksum of every database file in the backup, checks its transaction logs, opens each table file to confirm this version can read its format and compression, and, for a backup that captured blobs, checks that the blob snapshot is present. A backup that fails is refused and the database is left as it was (one an earlier restore left incomplete stays blocked); previously the restore deleted the database first and then failed, so a backup taken by a newer version, or one with a corrupt file, destroyed the database it was meant to restore. The check reads the whole backup once, so a restore takes longer, and an online restore keeps the database unavailable while it runs; it needs no extra disk space. A restore interrupted while it is still checking the backup leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. Blob file contents are not checksummed; the restore later checks that every blob a record references is present (see below). See [`restore_backup`](/reference/v5/backups/operations#restore_backup). + +### Restores Survive a Crash, and Prove the Database Before Publishing It + +Every `restore_backup` now records its decision on disk before it changes anything, applies and checks the backup in a separate process, and publishes the restored database only after that process has proven it: the transaction log replays completely, every record decodes, and every file-backed blob a record references is present. A crash at any point leaves the database as it was, restored, or blocked until a rerun finishes it; previously a crash mid-copy could leave a mix of old and new files. Only one restore or verification runs at a time on an instance. See [if a restore is interrupted](/reference/v5/backups/operations#if-a-restore-is-interrupted) ([harper#3920](https://github.com/HarperFast/harper/pull/3920)). + +### Restores Refuse to Run on a Node With Replication Peers + +`restore_backup` now fails on a node that lists replication peers in `system.hdb_nodes` or `replication.routes`, since replication would undo the restore or spread it to the cluster. Remove the node's peers before restoring and add fresh ones afterwards. While a restore runs, operations that add a peer fail with status code 409. See [restoring in a replicated cluster](/learn/administration/backup-and-recovery#restoring-in-a-replicated-cluster). + +### New restore_backup Options, and target_database Online + +`accept_incompatible_backup` restores a backup taken by a newer Harper whose archive manifest major is higher than this version understands, logging what it waived. `accept_unverified_blobs` publishes a restore whose records reference blob files the backup lacks. `immutable_source` restores from a read-only repository without locking or pinning it. `target_database` now works with the server running, and `verify_backup` opens and reads every table, so it fails a backup whose tables this version cannot read, as a restore would. A saved `harper login` to localhost on this installation's own operations API port no longer blocks an offline `restore_backup`, `delete_backup` or `purge_backups`. See [`restore_backup`](/reference/v5/backups/operations#restore_backup). ## Analytics