fix: SQLite join order for polling scans (0.16.1)

[cardmagic]

Why

Two polling queries chose bad plans on SQLite. A production app measured them on MySQL and on a SQLite copy of the same data:

Query	Executions per day	MySQL	SQLite copy
ActivationManager#claimed_instance_ids	470,245	0.67 ms	36.7 to 74 ms
EffectRecoveryCoordinator#recovery_candidates	251,593	0.68 ms	5.75 ms

On SQLite, the first query alone cost about 0.2 of a CPU core all the time and slowed actor delivery. The two queries have different causes, so they get different fixes.

Claimed-message scan: empty table at ANALYZE time

The query joins claimed messages to instances, and its WHERE clause has conditions on instance columns only. The claimed-messages table is a short queue, so it is often empty when ANALYZE or PRAGMA optimize runs. sqlite_stat1 then has no row for it, and SQLite assumes that the table is large. The planner used the analyzed instances table as the outer loop:

SCAN solid_objects_instances
SEARCH solid_objects_claimed_messages USING INDEX idx_so_claimed_instance (instance_id=?)
USE TEMP B-TREE FOR GROUP BY
USE TEMP B-TREE FOR ORDER BY

The query now uses CROSS JOIN with the equality in WHERE. SQLite keeps CROSS JOIN as a fixed join order:

SCAN solid_objects_claimed_messages USING INDEX idx_so_claimed_instance
SEARCH solid_objects_instances USING INTEGER PRIMARY KEY (rowid=?)
USE TEMP B-TREE FOR ORDER BY

On generated data with 71,863 instances and no claimed messages, the call went from 2.33 ms to 0.029 ms on a laptop. PostgreSQL and MySQL treat CROSS JOIN with an equality as an inner join. Their EXPLAIN ANALYZE plans and costs are the same before and after, with 0 and with 200 claimed rows (PostgreSQL 17, MySQL 8.4). The ids and their order do not change.

A sqlite_stat1 row that marks the table as small also fixes the plan, but the next ANALYZE of an empty queue deletes it.

Recovery candidates: skewed status statistics

This is not the empty-table problem, and it is not a missing index. sqlite_stat1 records only the average row count for each effect status. When most effects are complete, SQLite estimated that status = 'processing' matched most of the effects table. It then read every recovery row in key order to skip the sort:

SCAN solid_objects_effect_recoveries USING INDEX sqlite_autoindex_solid_objects_effect_recoveries_1
SEARCH solid_objects_effects USING INDEX idx_so_effects_effect_id (effect_id=?)
SEARCH solid_objects_processes USING INDEX sqlite_autoindex_solid_objects_processes_1 (id=?) LEFT-JOIN

An index on retired_at IS NULL AND recovery_operation IS NOT NULL does not help. Only a recovery retirement writes retired_at, so almost every recovery row matches that filter.

What I tried on generated SQLite data:

Data	Current	CROSS JOIN	unlikely()	likelihood(..., 0.000001)
100,000 completed effects, 10,000 recoveries	4.5 ms, scans recoveries	2.4 ms, scans effects	0.010 ms	0.010 ms
500,000 effects with 50 dead, 2,000 recoveries	1.0 ms, scans recoveries	0.011 ms	1.0 ms, scans recoveries	0.010 ms

On SQLite the status test now carries likelihood(..., 0.000001). SQLite uses that value in place of the sqlite_stat1 estimate for the term, so the plan starts with SEARCH solid_objects_effects USING INDEX idx_so_effects_poll (status=?). The PostgreSQL and MySQL SQL does not change.

Tests

The tests came first. Each SQLite plan test failed on the old code with the production plan:

ActivationCandidatesTest#test_SQLite_reads_claimed_candidates_from_the_claimed_messages_when_they_have_no_statistics
Expected /\A(SCAN|SEARCH) solid_objects_claimed_messages\b/ to match "SCAN solid_objects_instances".

EffectRecoveryCandidatesTest#test_SQLite_finds_recovery_candidates_through_processing_effects_when_most_effects_are_complete
Expected /\ASEARCH solid_objects_effects USING INDEX idx_so_effects_poll \(status=\?\)/ to match
"SCAN solid_objects_effect_recoveries USING INDEX sqlite_autoindex_solid_objects_effect_recoveries_1".

Each plan test also asserts its cause: sqlite_stat1 has no claimed-messages row, and the poll index statistics give 3,000 rows for each status.

The behaviour tests fix today's results on every adapter. They passed before and after the change:

• Claimed candidates skip paused instances and live leases, keep expired and unexpiring leases, order by claim time and then by claimed id, and stop at claim_scan_limit.
• Recovery candidates are processing effects with a recovery operation whose owner is absent or stale past the larger of the threshold and the recovery timeout. They come in effect id order and stop at claim_scan_limit.

Validation

Command	Result
bundle exec rake (SQLite)	803 runs, 0 failures, 0 errors, 28 skips; Standard, RuboCop, RBS, Steep, Brakeman clean
SOLID_OBJECTS_DATABASE_URL=postgresql://... bundle exec rake test (17)	803 runs, 0 failures, 0 errors, 20 skips
SOLID_OBJECTS_DATABASE_URL=mysql2://... bundle exec rake test (8.4)	803 runs, 0 failures, 0 errors, 38 skips
SOLID_OBJECTS_DATABASE_URL=trilogy://... bundle exec rake test (8.4)	803 runs, 0 failures, 0 errors, 38 skips

The two SQLite plan tests add one skip each on PostgreSQL and MySQL.

Effects

• API: none.
• Correctness: the same rows in the same order.
• Security: none. Brakeman flagged a first version that returned both SQL fragments from one case as an array, so each fragment has its own case.
• Migration: none.
• Compatibility: likelihood() exists in SQLite since 3.8.1.

Release

This PR prepares 0.16.1. An app that pins ~> 0.16.0 gets it with a bundle update.

The JS port has the same recovery query, and it gets the same fix in cardmagic/solid-objects-js#61. JS has no claimed-message scan.

Not in this PR

Recovery rows stay after their effects are deleted. solid_objects_effect_recoveries has a foreign key to instances only. The message pruner deletes old effects through the message cascade, but it leaves their recovery rows. So the table grows with every effect that sets on_recovery or on_status.

[greptile-apps]

Confidence Score: 5/5

[High risk] Database query optimization for polling scans.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR changes two SQLite polling-query plans and prepares the 0.16.1 release. The latest commit replaces destructive plan-test cleanup with a helper that saves and restores existing SQLite statistics. The previously reported cleanup issue is addressed; no new actionable issue was established.

_{Reviews (2) · Last reviewed commit: "test: restore SQLite statistics after pl..."}

[cardmagic]

@greptileai Please review the latest commit f02ec5d. It answers the finding about the plan-test cleanup: the tests now save and restore SQLite statistics.