[IN PROGRESS] Migration engine for the primary instance

[warwickschroeder]

What this adds

The migration engine moves an error instance's data from RavenDB to SQL Server or PostgreSQL. ServiceControl copies the data it needs while it is closed, opens on SQL, then copies the rest in the background. This is the base branch for the whole engine. It is not finished, and a real instance with Migration/Enabled turned on still refuses to copy anything until the last milestone below.

Architecture

The engine core knows nothing about RavenDB or SQL, so it can easily be extended in future to copy between any two persisters:

• The engine sits in the middle. MigrationEngine handles batching, resuming, failures and progress, and only talks to a source and a target.
• A source reads. It implements IMigrationSource, with one reader per kind of data. Today that is RavenDB.
• A target writes. It implements IMigrationTarget and IMigrationCheckpointStore, with one writer per kind of data. Today that is the shared EF Core target, which covers SQL Server and PostgreSQL.
• Adding a new pair takes a source or target for the new persister, a reader or writer for each kind of data, and a change to MigrationPairIsSupportedCheck, which is the one place that says RavenDB to SQL is the only pair allowed today.

Target State

• Is predictable. A dry run shows what will move and roughly how long ServiceControl will be closed.
• Is one setting. Turn on Migration/Enabled and start ServiceControl as usual.
• Never changes RavenDB. It only reads from it.
• Survives a stop at any point without losing or doubling a row, because each batch saves its rows and its progress together.
• Never decides on its own when something fails. A failed part waits to be retried or given up, so a restart cannot loop or lose data.
• Is visible while it runs, through ServicePulse, the log and read-only status and verify commands.
• Can be undone for free, right up until ServiceControl first opens on SQL.

The docs describe the target state in full:

• Overview: what moves, what is skipped, and what happens when something goes wrong.
• Instructions: how an operator runs the migration, step by step.
• System design: which class does what, and the order things run in.

Milestones

Each milestone lands on this branch as its own PR, stacked here and reviewed on its own. This branch goes to master once the last one is in.

• Foundation ([IN PROGRESS] Migration engine for the primary instance #5894): a safe, read-only way to open and count the old RavenDB data, and the copy engine tested against fakes.
• Saving progress ([Migration Engine Part 2] Add a migration checkpoint table to SQL Server and PostgreSQL #5897): a progress table on SQL Server and PostgreSQL, so a stopped copy resumes instead of starting over.
• Copying at startup ([Migration Engine Part 3] Implement the RavenDB to SQL migration machinery #5911): the engine wired into ServiceControl, the startup checks, the Done, Failed and Abandoned rules, and the first two parts of the data.
• Failed messages and their bodies (Copy unresolved failed messages and their bodies to SQL #6005): unresolved failed messages come across with their message bodies, which is usually the biggest part of the outage.
• The rest of the required data: throughput and licensing data, redirects, subscriptions, retry history, notification settings, custom checks, pending integration events and group comments.
• Copying in the background: once ServiceControl is open, the event log and archived and resolved messages copy quietly behind it.
• Finishing safely: --migration-retry and --migration-abandon, and a guard that stops the migration being switched off with anything unfinished.
• Seeing what is happening: progress in the ServicePulse activity feed, --migration-status, --migration-verify and the dry run.
• Ready to ship: embedded, container and cloud-hosted RavenDB all supported, the customer docs finished, and the lock that stops a real copy removed.

Tests

• ServiceControl.UnitTests/Migration: the engine, its halt and resume rules, each startup check and the refusal messages, against fakes and a fake clock.
• ServiceControl.Persistence.Tests/EFCore: the progress table, the target and its writers, on SQL Server and PostgreSQL.
• ServiceControl.Persistence.Tests.RavenDB/DataMigration: the read-only source and its readers against an embedded server.
• ServiceControl.Persistence.Tests.SqlServer: endpoint settings keys that differ only in letter case.
• ServiceControl.Migration.Tests (new): the source report, both persisters in one process, and every reader having a matching writer.
• ServiceControl.Migration.AcceptanceTests (new): a killed copy, a restart, each refusal and ServiceControl opening on SQL, on both providers.

[rbev]

Partial review - adding as comments for now.

[johnsimons]

A few question and comments but overall, lets keep going