Repository navigation
Reference files for the parallel ASCII output writer tests - #205
Merged
Merged
Conversation
Files written on 2 ranks by the previous ASCII writers for the output_writers tests of su2code/SU2 (flat plate of vandv/rans/flatplate). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
rois1995
added a commit
to su2code/SU2
that referenced
this pull request
Sep 23, 2026
Temporary, for the reference files of su2code/TestCases#205. To be reverted to develop before merging. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
6 tasks done
bigfooted
approved these changes
Sep 26, 2026
bigfooted
left a comment
bigfooted
left a comment
Contributor
There was a problem hiding this comment.
Looks good, thanks!
If you can already merge this in, then you can switch back to develop in the other PR.
joshkellyjak
added a commit
to su2code/SU2
that referenced
this pull request
Oct 8, 2026
…shes (#2924) ## Proposed Changes This PR fixes the output of large meshes and makes the output writing faster. The problems appeared on a mesh with 273M points and 496M cells (mostly prisms) run on 576 ranks: some output files were broken, the CGNS file could not be opened in ParaView, and some formats took very long to write. Each commit is self-contained, and its message gives the details. ### 1. Integer overflows Several writers stored sizes, offsets or MPI counts in a 32-bit `int`. All of them hit the same limit, 2^31 - 1 = 2 147 483 647: above it, the run crashed or, worse, wrote a broken file without any error. The limit is reached at different mesh sizes only because each writer counts something different: the data of one rank or of the whole file, in bytes or in values, with a different number of values per point or per element. | Writer | Counted in `int` | Over | Unit | Limit reached at | |---|---|---|---|---| | MPI-IO collective writes (binary restart, PARAVIEW, PARAVIEW_LEGACY) | data written by one rank in one call | one rank | bytes | 20 fields x 8 bytes: ~13.4M points per rank | | Sorting of the output data | buffer sizes, offsets, MPI counts | one rank | values | 100 fields: ~21.5M points per rank | | CGNS gather on the master rank | MPI counts of the connectivity | one rank | bytes | 8 nodes x 8 bytes: ~33.5M hexahedra per rank | | PARAVIEW (.vtu) | connectivity and offsets | whole file | entries | 6 nodes per prism: ~358M prisms | | PARAVIEW_LEGACY (.vtk, format 3.0) | cell array | whole file | entries | 7 entries per prism: ~307M prisms | The fixes (in all cases, the files of meshes below the limits are unchanged): - **MPI-IO collective writes**: the local data is now described with an MPI datatype of 1 MB blocks, so no count can overflow. Hybrid MPI+OpenMP runs, with few ranks and many points per rank, reach this limit first. - **Sorting of the output data**: the data of each point is sent as one element of an MPI datatype, and the sizes are `size_t`. - **PARAVIEW (.vtu)**: connectivity and offsets are written as Int64 when needed. - **PARAVIEW_LEGACY (.vtk)**: when the 3.0 cell array does not fit, the VTK 5.1 layout with Int64 offsets and connectivity is used (readable by ParaView >= 5.8). - **CGNS gather on the master rank**: this path is replaced by the parallel writer (section 4). ### 2. CGNS: sections above 2^31-1 connectivity entries The CGNS file of the large mesh is valid, but ParaView crashed when opening it. The VTK CGNS reader stores the connectivity size of each section in an `int` (reported in https://gitlab.kitware.com/vtk/vtk/-/work_items/20170). Until the reader is fixed, an element type above this limit is written as several sections with consecutive element ranges (`<name>_1`, `<name>_2`, ...). Smaller meshes still get one section per element type. ### 3. CGNS: named boundaries and double precision coordinates - **Boundaries**: the volume CGNS file had only the interior cells, and the surface file merged all markers in one unnamed zone. The volume file now also has one section per marker, named as the marker, with a `ZoneBC` and a `Family` that carries the BC type. ParaView shows each marker as a named patch, and SU2 can read the file back as a mesh with the same markers. SURFACE_CGNS writes one zone per marker. - **Coordinates**: they were written in single precision, which moves the points by up to ~1e-7 of the domain size, which can be larger than the smallest cells. When the output was used again as a mesh (a periodic case), the solution changed by up to 5e-5. Coordinates are now always written in double precision; with this, the same test reproduces the solution to ~1e-14. - **New option `VOLUME_OUTPUT_PRECISION`** (default `SINGLE`): `DOUBLE` writes the fields of CGNS, PARAVIEW and PARAVIEW_MULTIBLOCK in double precision. The default keeps single precision, so file sizes do not change. ### 4. Parallel writing - **CGNS**: the master rank gathered all data and wrote the whole file alone. This was slow, needed a lot of memory on the master node, and a large file could take longer than the job time left. All ranks now write their own part with the parallel CGNS API (`cgp_*`, parallel HDF5, already built by SU2 with MPI). Without MPI, the serial API is used as before. The write bandwidth of the CGNS files is now reported in the output summary (it always showed 0 MB/s). - **PARAVIEW_ASCII, RESTART_ASCII, TECPLOT_ASCII and the SU2 mesh writers** (ASCII and binary): the ranks wrote one after the other, with a barrier after each rank and each block of data, so the time grew with the number of ranks. Each rank now formats its part in memory, and all ranks write at the same time with one collective MPI-IO call. The files are byte-identical to the previous ones, also for multizone meshes. ### Testing The large mesh cannot be shared, and the overflow limits cannot be reached in CI (a single section above 2^31-1 entries needs more than 20 GB of memory). The tests therefore check that the parallel writers give the same files as before: - **New regression tests** (`TestCases/output_writers/ascii_output.cfg`, parallel regression on 2 ranks): the solution of `vandv/rans/flatplate` is written without iterating by SU2_CFD (RESTART_ASCII), SU2_SOL (PARAVIEW_ASCII, TECPLOT_ASCII and their surface versions) and SU2_DEF (SU2 mesh file). Each file is compared with a reference file written by the previous writers. The new files are byte-identical to the old ones, on 2 and 4 ranks. The reference files are in su2code/TestCases#205. - The parallel CGNS writer is run by the existing `oneram6` parallel regression test (CGNS and SURFACE_CGNS output). - On the large mesh above, the parallel CGNS file is written and read correctly by post-processing scripts. **Temporary:** `.github/workflows/regression.yml` uses the `fixCGNSOutput` branch of TestCases until su2code/TestCases#205 is merged. It will be set back to `develop` before merging. ### Question for the reviewers With MPI, the new writers always write in parallel; the old serial path (gather on the master rank, or ranks writing one after the other) is removed. Parallel HDF5 can be slow on some file systems (e.g. NFS). Should we add a config option to write serially (for example only for CGNS), or is the parallel writing enough? ## Related Work - VTK CGNS reader limit: https://gitlab.kitware.com/vtk/vtk/-/work_items/20170 - Reference files of the new tests: su2code/TestCases#205 ## PR Checklist - [x] I am submitting my contribution to the develop branch. - [x] My contribution generates no new compiler warnings (try with --warnlevel=3 when using meson). - [x] My contribution is commented and consistent with SU2 style (https://su2code.github.io/docs_v7/Style-Guide/). - [x] I used the pre-commit hook to prevent dirty commits and used `pre-commit run --all` to format old commits. - [x] I have added a test case that demonstrates my contribution, if necessary. - [x] I have updated appropriate documentation (Tutorials, Docs Page, config_template.cpp), if necessary. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: rois1995 <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Josh Kelly <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference files for the new regression tests of the parallel ASCII output writers, added in su2code/SU2#2924 (
TestCases/output_writers/ascii_output.cfg).The files were written on 2 ranks by the previous ASCII writers (current
develop) from the solution ofvandv/rans/flatplate, without iterating. The new parallel writers produce byte-identical files.restart_flow.csv.ref: RESTART_ASCII (SU2_CFD)flow.vtk.ref,surface_flow.vtk.ref: PARAVIEW_ASCII, SURFACE_PARAVIEW_ASCII (SU2_SOL)flow.dat.ref,surface_flow.dat.ref: TECPLOT_ASCII, SURFACE_TECPLOT_ASCII (SU2_SOL)mesh_out.su2.ref: SU2 mesh file (SU2_DEF, no deformation)🤖 Generated with Claude Code