Repository navigation
test(backend): smoke-test kci-dev against the seeded API - #2145
alanpeixinho wants to merge 5 commits into
Conversation
* Pin kci-dev 0.1.11 as a backend dev dependency * Run results trees, summary, builds, build, tests, test, and issues against the seeded API * Fill listing aggregates in the seed so tree listing returns rows * Document the pin, how to bump it, and how to run the checks locally Closes kernelci#2143 Signed-off-by: Alan Peixinho <[email protected]>
|
|
||
| import pytest | ||
|
|
||
| API = os.environ.get("DASHBOARD_API", "http://localhost:8001/api/") |
There was a problem hiding this comment.
I guess the other tests are using TEST_BASE_URL
I'd use the same env, derive one from another or at least keep them together and well documented.
it is easy to forget that we have two envs and try to change only one of them in the future thinking will affect both test suites
There was a problem hiding this comment.
@felipebergamin The CLI URL is now TEST_BASE_URL plus /api/, the same variable the other integration clients use. DASHBOARD_API only overrides the full URL. Both are documented together in docs/IntegrationTests.md. See 281c37a.
| incidents = self.create_incidents(issues=issues, builds=builds, tests=tests) | ||
| rollup_rows = self.create_tests_rollup(tests=tests, incidents=incidents) | ||
| latest_checkouts = self.create_latest_checkouts(checkouts=checkouts) | ||
| self.stdout.write("Updating listing aggregates...") | ||
| aggregate_checkouts_and_pendings(checkouts, tests, builds) | ||
| hardware_rows = self.create_hardware_status(tests=tests) |
There was a problem hiding this comment.
aggregate_checkouts_and_pendings() creates tree_listing rows with zeroed counters and only queues builds and tests in the pending_* tables. The test stack starts Gunicorn but never runs process_pending_aggregations, so the API returns trees with zero status counts despite the seeded builds and tests.
I'm not sure if makes sense for a smoke test. But we could try a complete aggregation flow to test with complete data. But as I said, I'm not sure if makes sense here. Consider a nit.
There was a problem hiding this comment.
@felipebergamin The seed now fills the listing counts with a new recompute_tree_listing command, so the smoke stack returns real tree counts without running process_pending_aggregations. The seed also stopped queueing pending rows, so a worker started on seeded data does not add them twice. tree_listing_counts_test.py checks the counts. See 7881f9e.
| def _parse_stdout(stdout: str): | ||
| text = stdout.strip() | ||
| try: | ||
| return json.loads(text) | ||
| except json.JSONDecodeError: | ||
| return ast.literal_eval(text) |
There was a problem hiding this comment.
I think this might hide regressions on output payloads. Wouldn't be better removing the fallback and let it fail with the error?
There was a problem hiding this comment.
@felipebergamin Some kci-dev output is not JSON, even with --json: results build and results test print a Python repr. The fallback is gone. Every other command is parsed as JSON and fails on anything else. Only those two calls parse the repr, with a comment at each one. See 3f40ccc.
* Point the CLI at TEST_BASE_URL plus /api/, matching the other integration clients * Keep DASHBOARD_API as an override of the full URL * Document both variables together Assisted-by: Cursor Signed-off-by: Alan Peixinho <[email protected]>
The seed was summing those rows in Python. Call populate_tree_tests_rollup for each seeded checkout, and check the counts from a known set of tests. Assisted-by: Cursor Signed-off-by: Alan Peixinho <[email protected]>
The seed left listing counts at zero until process_pending_aggregations ran. Add recompute_tree_listing for one checkout and call it from the seed, which no longer queues pending rows. Assisted-by: Cursor Signed-off-by: Alan Peixinho <[email protected]>
Parse every command as JSON, except the single build and test commands, which print a Python repr. Assisted-by: Cursor Signed-off-by: Alan Peixinho <[email protected]>
This branch has not been deployed
2 participants
What it is
Integration tests run the installed kci-dev 0.1.11 CLI against the local Dashboard test API.
The commands are results trees, summary, builds, build, tests, test, and issues. Each one must exit 0, print a body, and include the fields that command builds. Where a seed id is stable, the test checks
that id too.
The seed now fills listing aggregates with aggregate_checkouts_and_pendings, so tree listing has rows for results trees. kci-dev is pinned as a Poetry dev dependency in backend/pyproject.toml.
Closes #2143
How to test
From the repo root, start the test stack and load this branch's seed:
docker compose -f docker-compose.test.yml up test_db redis -d
docker compose -f docker-compose.test.yml run --rm test_backend python manage.py migrate
docker compose -f docker-compose.test.yml run --rm test_backend python manage.py seed_test_data --clear --yes
docker compose -f docker-compose.test.yml up --build -d
From backend/:
poetry install
DB_HOST=127.0.0.1 DB_PORT=5435 TEST_BASE_URL=http://localhost:8001
DJANGO_SETTINGS_MODULE=kernelCI.test_settings
poetry run pytest -m integration --use-local-db
kernelCI_app/tests/integrationTests/kci_dev_results_test.py -v
All tests should pass. results issues uses --origin origin-fake-00000000, which is the origin IssueFactory writes. The CLI default maestro matches no seeded issue.