Repository navigation
Conversation
added 12 commits
October 5, 2026 22:37
…port Reoptimization of the database layer: - Add block_states dictionary table and nullable block_state_ref / old_block_state_ref columns on actions; new writes reference the dictionary instead of repeating full state strings (CoreProtect-style int encoding). Legacy text columns stay null-compatible for reads. - Existing databases upgrade via instant ALTER TABLE ADD COLUMN (metadata-checked, idempotent). - Replace single-column indexes with query-oriented composite indexes (player_id,time), (action_id,time), (object_id,time), (old_object_id,time), (world_id,x,z,time); superseded indexes are dropped under the existing updateSchema config flag. - SQLite tuning: WAL journal + synchronous=NORMAL + 10s busy timeout. - Streaming rollback/restore support: currentMaxActionId, countActionsFor, keyset-paginated selectRollbackBatch / selectRestoreBatch (manual LessOp/GreaterOp predicates since SqlExpressionBuilder is deprecated at ERROR level in Exposed 1.0.0). - Search count cache (TTL 30s) so paging does not re-run full COUNTs. - compactBlockStates(): batched legacy text -> dictionary migration with progress callback; vacuumDatabase(): VACUUM / OPTIMIZE TABLE. - Shutdown drain: 4x larger batches and NonCancellable writes so a shutdown-time cancellation can never drop a half-flushed batch. (cherry picked from commit dc14afb)
Reoptimization of the command layer: - Rollback/restore no longer load the entire result set into memory and apply it in one monolithic main-thread loop. They now stream batches via keyset pagination (max 1000 rows in memory), commit the rolled_back flag per batch (crash-safe), yield one full tick whenever the main thread has worked for 25ms (adaptive tick budget, CoreProtect-style), and report progress every 5 batches. - Rollback takes MAX(id)+1 as an exclusive upper bound before starting, so actions logged during the rollback itself can never be picked up by later batches of the same rollback. - New /ledger compact command: migrates legacy text block states into the block_states dictionary (atomic per-row text->ref swap) and then reclaims disk space via VACUUM / OPTIMIZE TABLE. Safe on a live server; requires the purge permission level. - Lang keys for progress messages (en_us, en_gb, zh_cn, zh_tw). (cherry picked from commit eb7cf36)
(cherry picked from commit 005883e)
An earlier revision of this branch replaced upstream's single-column indexes with four (dimension, time) composites and widened actions_by_location to (world_id, x, z, time). Ledger stores timestamps as TEXT (~23 bytes), so every extra time column in an index costs ~23 bytes per row. Measured on a 13,500-row workload: index footprint 1.45 MB -> 3.29 MB, whole file 2.39 MB -> 4.26 MB (+78%), with no measurable lookup improvement. - Tables.kt: restore upstream index declarations - DatabaseManager: re-assert the upstream index set and drop the four composites; rebuild actions_by_location as (x, y, z, world_id) when the wide form is detected. Metadata-driven and idempotent. - Bytes per row: 315 -> 179 (upstream: 177) Verified by bench/test_index_migration.py: a database carrying the intermediate build's indexes is repaired on startup - composites dropped, single-column indexes restored, actions_by_location rebuilt, 13,500 rows intact. CHANGELOG: document the revert and the CoreProtect benchmark. (cherry picked from commit c0f6410)
…budget
The time column is the one place where a wide index key was worth removing rather
than avoiding. actions.time stored TEXT ('2026-10-05 13:22:21.702') and an index
over it cost ~35 bytes per row - measured at 19.6% of the whole database, the
largest non-table object.
- Tables.kt: add time_ms (INTEGER, epoch millis) carrying the index; drop
.index("actions_time") from the TEXT column.
- DatabaseManager: add the column, index it, backfill it in id-ordered batches,
then drop the superseded TEXT index. Resumable; queries fall back to the TEXT
column until the backfill completes so a partial migration stays correct. With
updateSchema=false nothing runs and inserts skip the new column.
- DatabaseManager: SQLite pragmas via SQLiteConfig (temp_store=MEMORY, 16 MiB
cache, 256 MiB mmap_size, 64 MiB journal_size_limit), reaching every pooled
connection; verified applied through JDBC.
- Rollback/Restore: tick budget sized from the server's smoothed tick time instead
of a fixed 25 ms. On an idle benchmark server this changes nothing measurable -
a rollback completes with zero yields, so the budget was never its bottleneck.
The benefit is backing off automatically under load. A previous comment claiming
a large share of wall time went into delay(1) was wrong and is removed.
Measured, 13,500-block workload, medians of 3:
time index 35.5 -> 14.6 B/row (-59%)
whole file 181.4 -> 159.3 B/row (-12.2%, after migration + VACUUM)
ingest 1,959 -> 2,125 rows/s
Independent 500,000-row synthetic run: -11.5% file, -53% index.
Rollback and lookup show no measurable change and are not claimed as wins.
Composite (time_ms, id) indexes were measured at 500k rows and rejected: still
USE TEMP B-TREE FOR ORDER BY, and +3.0-3.7% file size.
Verified: migration 8/8 checks, fresh-database schema path, and a rollback/restore
state round-trip on a live MC 26.3 server, all with zero errors.
(cherry picked from commit d56252d)
Two figures overstated the change: - the -12.2% database reduction compared a migrated+vacuumed database against an un-vacuumed one, crediting VACUUM (which any index drop would also produce) to the optimisation. Like-for-like the change is -7.9% (181.44 -> 167.2 B/row), with the index itself going 35.5 -> 16.4 B/row (-54%). - ingest throughput does not reproduce: reopt.2 measured 2,125 rows/s in one campaign and 1,917 in the next, with reopt.1 at 1,959 and 1,957. It is noise, so ingest/lookup/rollback are now reported as no measurable change rather than as wins. (cherry picked from commit cbdba32)
…heduling
Measured first, then fixed - two assumptions from the previous round turned out
to be wrong:
* rollback's 14 reads and 14 rolled_back commits: bypassed, not the subject here
* ingest: a 13,500-action burst logged 13,500 individual
"INSERT INTO actions" statements. Exposed's batchInsert was issuing one
statement per row. On the same schema and the same rows, a real JDBC batch
costs 23 us/row and an individual statement 44 us/row - yet the live path
measured 310-500 us/row, so statement machinery dominated.
* the scheduler unconditionally `delay(batchDelay)` (500 ms) on every pass even
with work already queued: 6.1 s of an 8.7 s ingest window was spent waiting,
against 2.6 s of actual writes.
Changes:
- DatabaseManager.insertActions now writes through a prepared statement on the
transaction's own JDBC connection with addBatch/executeBatch. Values still come
from Exposed column types via valueToDB, so the stored representation
("yyyy-MM-dd HH:mm:ss.SSS" text) is unchanged - verified byte-for-byte against
an archive written by the ORM path, 500/500 samples identical.
- ActionQueueService only skips the wait while the queue is being flooded (the
previous pass filled a complete batch). Once a pass comes up short the
configured batchDelay applies again, so trickle traffic still batches exactly as
upstream did; an idle server does not become one transaction per action.
Measured over 3 runs each (medians, 13,500 actions, fresh terrain):
drain window 4.534 s -> 3.622 s (-0.91 s; the 3.0 s settle window is a
constant, so actual write work went
1.53 s -> 0.62 s)
drain rate 2,977 -> 3,727 rows/s (+25%)
ingest window 6.713 s -> 5.733 s
rollback unchanged (3.85 s -> 4.10 s, within run-to-run noise)
Also evaluated and REJECTED: turning off SQLite foreign keys. A controlled
transaction-cost test showed FK checks cost 5.9 -> 5.7 ms per 1,000-row
transaction (3%), which is not worth weakening referential integrity for.
Verified: fresh-database schema, zero errors on a live MC 26.3 server, rollback ->
restore state round-trip passes, dictionary references written correctly, and
time/time_ms remain mutually consistent in both the old and new paths.
(cherry picked from commit 0cb9653)
… rows
Rollback was the one metric with a large gap to CoreProtect (about 2 s vs 0.6 s
for the same 13,500 actions), so the batch size was measured rather than left at
its long-standing default. Hiding 1,000 rows behind each read and each
rolled_back commit means the per-statement overhead is paid fourteen times for a
13,500-row undo; 5,000 pays it three times.
Measured on a 13,500-block rollback, two fresh-terrain runs per setting:
1,000 rows/batch : 3.857 s / 4.164 s
5,000 rows/batch : 1.991 s / 2.030 s (about 2x faster)
Safety was checked before adopting it, since a bigger batch means more work
between yields:
* zero "can't keep up" warnings in every run at both settings, so the larger
batch does not cost server responsiveness - the existing per-tick budget still
splits a batch and yields mid-way through it
* all four runs logged exactly 13,500 rows with zero errors
Worst-case memory is bounded by roughly 5,000 materialised actions.
Not changed, for the record: SQLite's page size cannot be altered on an existing
database (PRAGMA page_size is silently ignored once the file exists, confirmed by
measurement - every value collapsed back to 4,096), so it is not a lever this build
can pull. That same test did confirm that VACUUM alone reclaims about 4.7% of the
file, which /ledger compact already exposes to users.
(cherry picked from commit 4091b35)
Records this round's three changes (real JDBC batch insert, burst-aware queue scheduling, 1,000 -> 5,000 rollback batch), the combined before/after numbers, and the three options that were measured and rejected (foreign keys off, page_size, dropping the legacy time column) with the reason in each case. (cherry picked from commit 6fe0f56)
The publish helper reads a GitHub token from $GITHUB_TOKEN or from a .secrets/gh_token file. Ignore both patterns so a token placed inside the working tree can never be committed by accident. (cherry picked from commit af6806d)
Purely mechanical - trailing whitespace, trailing commas, and `page-1` spacing. Needed so the branch satisfies the project's own detekt configuration; safe to skip when reviewing. The same reformatting previously rode along inside the NBT commit. That commit was dropped when this branch was rebased onto 1.3.25 (upstream fixed the NBT issue in QuiltServerTools#402), so the formatting is restored here on its own.
Upstream moved while this branch was in flight: 22705ac fix: update block state NBT format for 26.3 (QuiltServerTools#402) dd947aa Update modVersion from 1.3.24 to 1.3.25 QuiltServerTools#402 fixes exactly what this branch's own NBT commit fixed, so that commit has been dropped in the rebase and the code is now upstream's - credit belongs there. The one piece that was ours alone, a read-side fallback to the legacy "Properties" key, was also dropped: on 26.3 writeBlockState emits "properties", so the fallback could never fire and was dead code. That also removed the unrelated MessageUtils reformatting from that commit, which is now its own change. The CHANGELOG heading and comparison baseline move to 1.3.25, and the NBT entry now records that the fix is upstream's rather than implying it is ours. Verified after rebasing: builds clean, detekt passes, and a live MC 26.3 server completes a rollback -> restore round-trip with the exact block state preserved (oak_stairs[facing=west,half=top]) and zero exceptions.
Pau1am
force-pushed
the
Ledger-Reoptimization
branch
from
October 5, 2026 14:42
af6806d to
c34174e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto
masterat 1.3.25. Every change below was measured on a live Fabric26.3 server against CoreProtect 24.1 on the same machine and the same workload; the
method and raw per-run data are in the accompanying report.
Note on the rebase
Upstream moved while this branch was in flight:
#402 fixes exactly what this branch's own NBT commit fixed, so that commit has been
dropped and the code is now upstream's - credit belongs there. The one part that was
ours alone, a read-side fallback to the legacy
Propertieskey, was dropped too: on26.3
writeBlockStateemitsproperties, so the fallback could never fire and wasdead code. This is also why the diff no longer touches
NbtUtils.kt.Storage
block_statestable and referenced by integer id. Measured on a stairs workload:31.3% smaller (55.8 bytes per row) than repeating the literal state text.
timeis TEXT ('2026-10-05 13:22:21.702', ~23 chars), soan index over it cost ~35 bytes per row - 19.6% of the whole database, the largest
object after the table itself. A new
time_mscolumn (epoch millis) carries theindex at ~16 bytes: the time index shrinks 54% and the file 7.9% like-for-like.
Migration is automatic, resumable, and queries fall back to the text column while a
backfill is still running, so a partially migrated database stays correct.
/ledger compactreclaims the freed pages.Performance
batchInsertwas emitting one statement per row - a13,500-action burst produced 13,500 individual
INSERT INTO actionsstatements.Writing through a prepared statement with
addBatch/executeBatch, with valuesstill produced by Exposed column types via
valueToDBso the stored representationis unchanged, cut the write window from 1.53 s to 0.62 s of real work
(drain rate 2,977 -> 3,727 rows/s).
batchDelayon everypass even with work already queued, which accounted for 6.1 s of an 8.7 s ingest
window. It now skips the wait only while the previous pass filled a complete batch,
and returns to the configured cadence as soon as one comes up short - trickle
traffic still batches exactly as before.
(4.01 s -> 2.01 s) with zero "can't keep up" warnings at either setting.
Net effect against the pre-optimisation build, medians of three runs:
Also evaluated and rejected (measured, not assumed)
(time_ms, id)/(time_ms, rolled_back, id)indexes - still force atemp B-tree sort for
ORDER BY id DESC LIMIT n, no query gain, +3% file sizethe integrity trade
page_size- silently ignored once the database exists (every valuecollapsed back to 4,096), so not a usable lever
timecolumn - ~13% more space, but needs a table rebuild plusa branched read path for a non-reversible migration; not worth it against the
correctness risk
Verification
Live MC 26.3 Fabric server: zero exceptions, rollback -> restore round-trip preserves
the exact block state (
execute if block ... oak_stairs[facing=west,half=top]passesafter restore), both migration paths pass their checks, and every benchmark run logs
exactly 13,500 rows with zero errors. The one
style:commit is detekt autocorrectand can be skipped when reviewing.