Repository navigation
feat(hardware): add daily hardware aggregates recomputed from raw - #2037
alanpeixinho wants to merge 3 commits into
Conversation
39d0d5b to
27eb2e6
Compare
alanpeixinho
changed the title
b5c7772 to
89f9401
Compare
There was a problem hiding this comment.
--origins does not apply to hardware_daily_* ?
Those tables are in the default --tables set and are deleted with only checkout_day < cutoff. running prune_db --older-than "30 days" --origins maestro (the docs example) can wipe every origin’s daily aggregates while leaving other tenants’ raw rows.
There was a problem hiding this comment.
Nice catch. Daily prune was ignoring --origins and deleting by checkout_day only. Fixed in c9392fb — hardware_daily_* now filter on checkout_origin the same way raw rows use origin.
ea02b2d to
c9392fb
Compare
alanpeixinho
changed the title
ac58f3a to
43c8bc7
Compare
|
@nuclearcat this PR creates new tables to deal with hardware aggregation for the changes required by #2034. |
ping me in discord when you want to deploy it |
Add hardware_daily_builds and hardware_daily_tests keyed by checkout day, origin, lab, and platform. Depends on the labs migration already on main, so no merge migration is required. Signed-off-by: Alan Peixinho <[email protected]>
Rebuild one UTC day at a time with DELETE + INSERT, an advisory lock, and a rollback when raw rows are gone. Materialize the day's tests once, ANALYZE them, and raise work_mem so the planner hashes instead of spilling. Cron refreshes today hourly and the last 15 days every 6 hours. Wire prune_db, seed_test_data, and update_db for the new tables. Closes #2034 Signed-off-by: Alan Peixinho <[email protected]>
Filter hardware_daily_* on checkout_origin the same way raw rows use origin, so an origin-scoped prune cannot wipe every tenant's aggregates. Signed-off-by: Alan Peixinho <[email protected]>
43c8bc7 to
62dc826
Compare
|
Depends on #2178. Merge that first. It adds the tables. |
|
Moved to #2181 so the PR is opened from profusion, not from kernelci. The recompute commits are the same. Closing this. |
3 participants
Depends on #2178. Merge that first. It adds the tables.
What it is
Adds daily hardware aggregate tables (hardware_daily_builds, hardware_daily_tests) built from raw checkouts / builds / tests.
exist, keeps the stale rows.
Closes #2034.
How to test
migrate
Run local database and apply migration
manual recompute
python manage.py recompute_hardware_daily --day=2025-07-18python manage.py recompute_hardware_daily --days=7verify rows
Check that the aggregate counts for builds, boots and tests match the sum of the raw tables (builds and tests).
prune dry-run (aggregates only)
python manage.py prune_db --older-than "30 days" \ --tables hardware_daily_builds,hardware_daily_tests --dry-run