Skip to content

Fixed env for image comparison testing, test via CI - #139

Open
sadielbartholomew wants to merge 9 commits into
NCAS-CMS:mainfrom
sadielbartholomew:testing-ci-img-cmp-updates
Open

sadielbartholomew wants to merge 9 commits into
NCAS-CMS:mainfrom
sadielbartholomew:testing-ci-img-cmp-updates

Conversation

@sadielbartholomew

@sadielbartholomew sadielbartholomew commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Towards a solution for #135 (but not closing it, as I have longer-term work to ensure the test results are reproducible). Also towards closing #18 (is useful to do this at the same time - see last point below) though I won't close that one either until I set up also workflows for linting and testing the source dist.

This PR in particular provides a short-term solution for image comparison test results varying by (mostly) small pixel-order differences:

  • Add a --skip-image-comparison or -I option to pytest run over the test modules to run only the functional aspect, skipping the image comparison. Allows basic testing to proceed regardless of environment as long as it is valid to run cf-plot.
  • Provide a 'reference' conda environment cfplot-test-env.yml which users are encouraged to activate before running the tests inclusive of image comparison. See new README 'Testing' section for a guide to how to do this, though I will further document this in the dedicated developer docs soon enough. This env will need refreshing, ideally every release, to ensure it uses the latest and greatest libraries to test on those without getting stale. I will make that part of a release checklist one I write one up, shortly.
  • Add new reference images directories to show the results with this env. I have eyeballed these that they are equivalent to the ones in the test suite now bar those 'trivial' pixel order differences. Some bugs are hidden in these (such as the 'arc' in the NW of the polar plot for fig_18), but they are known ones and at least we have a starting point towards fixing those and updating the reference images. I have put them in separate directories for now to help me track the minor differences per environment to help towards a more general solution based on comparing against directories of images by image hash against an isolated environment matrix (later work).
  • It isn't clear if the platform makes a difference to the output plots for a given environment, so I'm putting up the test-running workflow with a platform matrix to allow us to see if there are platform-based differences.

@sadielbartholomew

Copy link
Copy Markdown
Member Author

OK so the first CI job runs show that the setup is correct and the tests start to run. I forgot the main step though to use a specific environment, so will get that implemented next.

@sadielbartholomew

sadielbartholomew commented Oct 16, 2026 •

Copy link
Copy Markdown
Member Author

Opening and closing to see the CI job results after creating a separate workflow to run a separate job on Mac and Linux with latest Python version (3.14 at present) to run the image comparison, whereas the general matrix will run the tests without image comparison (for now, given we don't have reference images to cover multiple cases yet) across multiple versions excluding the latest to avoid the duplicate.

@sadielbartholomew

Copy link
Copy Markdown
Member Author

Will pick this up on Monday, where the outcomes of the CI job should guide as to how to proceed. Note this PR is in 'draft' form effectively until I sort out the workflows so they pass which I am investigating on this PR by prodding them with a stick.

@sadielbartholomew

Copy link
Copy Markdown
Member Author

OK saw the results before I clocked off. From the log the setup needs fixing for the Mac jobs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant