Repository navigation
Fixed env for image comparison testing, test via CI - #139
sadielbartholomew wants to merge 9 commits into
Conversation
|
OK so the first CI job runs show that the setup is correct and the tests start to run. I forgot the main step though to use a specific environment, so will get that implemented next. |
|
Opening and closing to see the CI job results after creating a separate workflow to run a separate job on Mac and Linux with latest Python version (3.14 at present) to run the image comparison, whereas the general matrix will run the tests without image comparison (for now, given we don't have reference images to cover multiple cases yet) across multiple versions excluding the latest to avoid the duplicate. |
|
Will pick this up on Monday, where the outcomes of the CI job should guide as to how to proceed. Note this PR is in 'draft' form effectively until I sort out the workflows so they pass which I am investigating on this PR by prodding them with a stick. |
|
OK saw the results before I clocked off. From the log the setup needs fixing for the Mac jobs. |
Towards a solution for #135 (but not closing it, as I have longer-term work to ensure the test results are reproducible). Also towards closing #18 (is useful to do this at the same time - see last point below) though I won't close that one either until I set up also workflows for linting and testing the source dist.
This PR in particular provides a short-term solution for image comparison test results varying by (mostly) small pixel-order differences:
--skip-image-comparisonor-Ioption topytestrun over the test modules to run only the functional aspect, skipping the image comparison. Allows basic testing to proceed regardless of environment as long as it is valid to run cf-plot.cfplot-test-env.ymlwhich users are encouraged to activate before running the tests inclusive of image comparison. See new README 'Testing' section for a guide to how to do this, though I will further document this in the dedicated developer docs soon enough. This env will need refreshing, ideally every release, to ensure it uses the latest and greatest libraries to test on those without getting stale. I will make that part of a release checklist one I write one up, shortly.fig_18), but they are known ones and at least we have a starting point towards fixing those and updating the reference images. I have put them in separate directories for now to help me track the minor differences per environment to help towards a more general solution based on comparing against directories of images by image hash against an isolated environment matrix (later work).