Repository navigation
perf(labs): serve the listing from lab_daily - #2176
alanpeixinho wants to merge 5 commits into
Conversation
Add hardware_daily_builds and hardware_daily_tests keyed by checkout day, origin, lab, and platform. Depends on the labs migration already on main, so no merge migration is required. Signed-off-by: Alan Peixinho <[email protected]>
43620c0 to
4a8542d
Compare
Store one row per UTC day, origin, and lab so the listing can sum counts without scanning builds and tests. Signed-off-by: Alan Peixinho <[email protected]>
Read the labs listing and lab origins from lab_daily, with no JSON lab fallback. Cache the listing query, and refresh today hourly and the last 30 days every 6 hours. Assisted-by: Cursor Grok 4.7 Closes kernelci#2165 Signed-off-by: Alan Peixinho <[email protected]>
4a8542d to
4dd4b93
Compare
There was a problem hiding this comment.
How complex would be to add an integration test for the aggregation SQL? The current tests cover the day selection and cache calls, but don't test the counts. A small fixture with builds, boots, and tests would help catch incorrect totals or rows being dropped. If this would take too much time now we can open an issue to improve the integration tests coverage in the future
There was a problem hiding this comment.
Added it. It might also be a good idea to do something similar to other aggregations as well.
Seed those rows with recompute_lab_daily so the listing aggregate matches production. Assisted-by: Cursor Signed-off-by: Alan Peixinho <[email protected]>
Pytest refuses two collected files that share the basename recompute_lab_daily_test. Assisted-by: Cursor Signed-off-by: Alan Peixinho <[email protected]>
| SELECT DISTINCT origin | ||
| FROM lab_daily | ||
| WHERE day >= CURRENT_DATE - %(interval_in_days)s |
There was a problem hiding this comment.
The previous query used a rolling time window (NOW() - INTERVAL), but this one includes complete calendar days. Is this change intentional? For example, a five-day interval now includes data older than five days. Should we adjust the API semantics or frontend labels to reflect this?
There was a problem hiding this comment.
shouldnt we invalidate the list on cache after the aggregation run?
This branch has not been deployed
2 participants
Depends on #2179. Merge that first. It adds
lab_dailyas migration 0022, after #2180.What it is
The labs listing and the lab origins list read
lab_dailyinstead of scanningbuildsandtests. Each row is one UTC day, one origin, and one lab, with pass, fail, and inconclusive counts for builds, boots, and tests.recompute_lab_dailyrebuilds a day by deleting that day and inserting the counts again from raw rows. Rows with nolab_idare left out. Dummy builds are left out. The cron rebuilds today every hour, and the last 30 days every 6 hours.Both queries use the normal query cache. Build counts stay at 0 until those builds have a
lab_id.On production, scanning
testsfor the listing took 12.2 s, and scanningtestsfor the origins list took 66 to 75 s. The local database has 2,100,330 tests. The listing scan read 1,478,325 of those and returned 16 labs. The same listing read 80 rows fromlab_daily. The origins scan read 1,599,999 tests and returned 2 origins. The same list read 120 rows fromlab_daily.testslab_dailyHow to test
Run this on a local database.
0022_lab_daily.python manage.py recompute_lab_daily --days=30./labs, or callGET /api/labs/?origin=maestro&interval_in_days=5andGET /api/labs/origins/.lab_idset shows its counts. A lab withlab_idnull does not appear.python manage.py recompute_lab_daily --day=YYYY-MM-DDa second time. The same day is replaced, and the counts stay the same.Closes #2165
Made with Cursor