[Migration Engine Part 3] Implement the RavenDB to SQL migration machinery

[warwickschroeder]

Stacked on #5897 (warwick/migration-engine-2). Review against that branch, not master.

What this adds

The machinery that copies an instance's data from RavenDB into SQL Server or PostgreSQL during startup, before the host opens. Only known endpoints and endpoint settings can be copied, so a real instance with Migration/Enabled set still refuses to start; message bodies are not read, and Migration/AllowIncompleteExit does nothing yet.

• The copy runs during startup, and the host does not open until every category has finished.
• Each batch commits its rows and checkpoint together, so a restart resumes from the last batch.
• Only categories both the source and the target support are attempted.
• Startup checks refuse before anything is copied, including one that blocks any build unable to copy every required category.
• A stalled or failing copy stops with an error that names the category, what to fix and how to resume.
• The moment the host first opens on the target is recorded, and every refusal says how to go back to RavenDB before then.
• An ingestion-only worker refuses to start while a copy into its database is unfinished.
• Docs gain a system design, and the overview and instructions are updated.

Tests

• ServiceControl.UnitTests/Migration: the engine, each startup check, the stall watchdog and the refusal messages, against fakes and a fake clock.
• ServiceControl.Persistence.Tests/EFCore/Migration: the target and its two writers on SQL Server and PostgreSQL, and a check that fails when an entity has no migration decision.
• ServiceControl.Persistence.Tests.RavenDB/DataMigration: the readers, the source and its data version against an embedded server.
• ServiceControl.Persistence.Tests.SqlServer: endpoint settings keys that differ only in case, and settings wiring.
• ServiceControl.Migration.AcceptanceTests (new): a killed copy, a restart, each refusal and the host opening on the target, on both providers.
• ServiceControl.Migration.Tests: every category with a reader has a writer, and the reverse.

[rbev]

This is a functional change that could impact licensing, is it actually ok?

[warwickschroeder]

Specifically on licensing - licensing never goes through the endpoint settings table. So merging of endpoint settings here doesnt affect that. Licensing is already case-blind on both stores. EF lowercases the licensing key (LicensingDataStore.cs:331, Normalize uses ToLowerInvariant), and RavenDB keys licensing documents on {Name}/{ThroughputSource} (EndpointExtensions.cs:9), and RavenDB ids ignore case. So the migration changes no licensing counts.

Known endpoints dont merge either. They are keyed on a GUID. The only thing that merges is the settings for the endpoints, which is only 1 TrackInstances flag.

SQL Server persistence already does this. EndpointSettingsStore.UpdateEndpointSettings upserts on that case-insensitive key, so a live SQL Server instance can't hold Sales and sales apart either. RavenDB and PostgreSQL keep both.

The question is, whats the impact of this. Its either fine, or its a product bug on SQL Server that the migration just inherited. @johnsimons - thoughts?

[warwickschroeder]

This feels like a defect we need to resolve..

1. The settings sync runs 20 seconds after start and then every 6 hours. It reads the stored names ({"Sales"}), compares them case-sensitively with the known endpoints ({"Sales", "sales"}), and decides sales has no setting.
2. It then upserts sales with the default value. The key ignores case, so that upsert overwrites the shared Sales row.
3. Result: both endpoints are reset to the instance-wide default at startup and every 6 hours after. Any choice an operator makes for either one in ServicePulse is lost by the next sync.
4. Old-instance cleanup only ever matches the stored spelling, so dead instances of sales are never cleaned up when tracking is off.