Repository navigation
Fix integer overflows and parallelize the output writers for large meshes - #2924
Conversation
The CGNS writer gathered field and connectivity data on the master rank with a single MPI_CHAR message per rank, whose byte count was cast to int. With 64-bit cgsize_t, a rank holding more than ~33.5M hexahedra overflowed the count, which crashed the run or silently wrote corrupted connectivity. Send/receive the buffers in chunks of at most 1 GiB, so every MPI count fits in an int. MPI_CHAR is kept since typed transfers are not portable across AD builds and platforms where long is 32-bit. Also accumulate the element counts in CParallelDataSorter with an unsigned long initial value, to avoid int overflow of the sum. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
WriteMPIBinaryDataAll cast the local size in bytes to int for MPI_Type_contiguous and MPI_File_write_all. When one rank wrote more than INT_MAX bytes in one call (e.g. a binary restart with 20 fields and more than ~13M points per rank, typical of hybrid MPI+OpenMP runs) the write failed or silently wrote a truncated file. Describe the local data as 1 MiB blocks plus a remainder in a single datatype and write it with count 1, so no int count can overflow and the call stays collective. Output files are unchanged. Affects the SU2 binary restart and the PARAVIEW / PARAVIEW_LEGACY writers. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The PARAVIEW (.vtu) writer stored connectivity and offsets as Int32, and the cumulative connectivity counter of CParallelDataSorter was an int. Meshes with more than 2^31-1 connectivity entries (e.g. more than ~268M hexahedra or ~358M prisms) got overflowed offsets and unreadable files. - Write connectivity and offsets as Int64 when the global connectivity size does not fit in Int32; smaller meshes keep Int32 (files unchanged). - Store nElemConn_Send/nElemConn_Cum as unsigned long. - Write NumberOfPoints/NumberOfCells without casting to int. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The VTK/ParaView CGNS reader stores the connectivity size of each section in a 32-bit int, so it crashes on sections with more than 2^31-1 entries (e.g. more than ~358M prisms), although the file is valid. Write an element type that exceeds the limit as several sections with consecutive element ranges (<name>_1, <name>_2, ...). Blocks received from each rank are split at section boundaries. Smaller meshes still get one section per element type, so their files are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The PARAVIEW_LEGACY writer stored all cells in one Int32 array (the number of nodes followed by the node ids of each cell) and printed the header counts through int. Meshes with more than 2^31-1 entries in that array (e.g. more than ~238M hexahedra or ~307M prisms) got broken files. When the cell array does not fit in Int32, write the legacy 5.1 layout (readable by VTK >= 9.0 / ParaView >= 5.8) with separate Int64 OFFSETS and CONNECTIVITY arrays. Smaller meshes keep the 3.0 layout, so their files are unchanged. Header counts are no longer cast to int. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
SortOutputData and PrepareSendBuffers computed buffer sizes, offsets and MPI counts as (number of output fields) x (number of points) in int. When one rank sent or received more than INT_MAX values (e.g. 100 output fields and more than ~21M points per rank, as in hybrid MPI+OpenMP runs) the counts overflowed and the output was broken. Send the data of each point as one element of a contiguous MPI datatype, so the MPI counts are numbers of points, and use size_t for the buffer sizes and offsets. Output files are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The CGNS output had no boundary information: the volume file only held the interior cells, and the surface file merged all plotted markers into one unnamed zone. Volume file (CGNS): - After the interior sections, write one boundary section per marker (except send-receive), named as the marker. A marker with a single element type gets a TRI_3/QUAD_4/BAR_2 section, otherwise MIXED. Faces use the volume point numbering, so they share the coordinates and fields of the zone. - Each boundary element is kept on exactly one rank (no halo node and at least one owned node) and gathered on the master node. - Add a ZoneBC with one BC_t per marker (FamilySpecified, PointRange of the section, FaceCenter/EdgeCenter) and a Family_t per marker that carries the physical BC type (wall, farfield, inflow, ...). The ParaView/VTK CGNS reader shows each marker as a named patch, and SU2 can read the file back as a mesh with the same markers. Surface file (SURFACE_CGNS): - Write one zone per plotted marker, named as the marker; the surface data is sorted again for each marker. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…double The CGNS and Paraview XML writers stored coordinates and fields in single precision. Coordinates in single precision move the points by up to ~1e-7 of the size of the domain, which can exceed the size of the smallest cells, so a CGNS output file could not be reused as a mesh without losing accuracy (in a round trip of a periodic case the solution changed by up to 5e-5). - Write the CGNS coordinates always in double precision. - Add WRT_OUTPUT_DOUBLE_PRECISION (default NO) to write the fields of the volume and surface files in double precision, for CGNS, PARAVIEW and PARAVIEW_MULTIBLOCK. Single precision remains the default, so files and their sizes are unchanged for visualization use. With double coordinates the same round trip reproduces the coordinates exactly and the solution to ~1e-14. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The CGNS writer gathered all data on the master node, which wrote the whole file by itself: the master was an I/O and memory bottleneck, and a large file could take long enough for the job to end before the file was closed. Open the file with cgp_open on all ranks and write each rank's own range of points, elements and fields with cgp_coord_write_data, cgp_elements_write_data and cgp_field_write_data. The nodes of the file (base, zone, sections, solution, fields, boundary conditions, families) are metadata and are created by all ranks with the same arguments. Ranks without data take part in the collective calls and write nothing. Boundary sections are written the same way, so the gather of the boundary faces on the master node is also gone; markers with more than one element type use the parallel MIXED API (cgp_poly_*). Without MPI the serial CGNS API is used, as before. The chunked MPI transfers of the old gather path are no longer needed and were removed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PARAVIEW_ASCII, RESTART_ASCII and TECPLOT_ASCII wrote their files by letting the ranks take turns: each rank opened the file, appended its part and closed it, with a barrier after every rank and for every block of data (points, connectivity, cell types, each field). Only one rank wrote at any moment, and the number of barriers grows with the number of ranks, which makes these formats very slow at scale. Each rank now formats its own data into a string, and all ranks write their strings at the same time with the collective MPI-IO call of CFileWriter, at the offset that follows the text of the ranks before it (new helper CFileWriter::WriteMPIStringAll). The headers are still written by the master node only. The files are byte-identical to those of the previous implementation. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
CSU2MeshFileWriter and CSU2MeshBinaryFileWriter also let the ranks take turns: each rank opened the file, appended its elements or points and closed it, with a collective call after every rank. Each rank now builds its own part of the file in memory and all ranks write it at the same time with the collective MPI-IO call of CFileWriter, as the other output files do. The global element and point indices come from the number of elements and points of the ranks before, gathered once instead of once per rank. Multizone meshes are written into a single file, so CFileWriter:: OpenMPIFile takes an append flag that continues at the end of an existing file (and creates it when it does not exist, e.g. for the first zone of a file of its own). The files are byte-identical to those of the previous implementation, also for multizone cases. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The CGNS writer never set the bandwidth, so the file writing summary always printed 0 MB/s for CGNS and SURFACE_CGNS. Time the writing and read the file size after closing it, as the Tecplot binary writer does. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
The ASCII writers (RESTART_ASCII, PARAVIEW_ASCII, TECPLOT_ASCII and the SU2 mesh file) now write in parallel. Write the solution of the vandv/rans/flatplate case on 2 ranks without iterating, with SU2_CFD, SU2_SOL and SU2_DEF, and compare the files with reference files written by the previous writers. The reference files are in su2code/TestCases, folder output_writers. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Temporary, for the reference files of su2code/TestCases#205. To be reverted to develop before merging. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Use unsigned long for the field counter of the Tecplot ASCII writer, which is compared with the size of the field names, and rename the offset of the boundary elements of the SU2 binary mesh writer, which hid the one of the volume elements. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
|
wasn't this an issue that was raised at a conference a looong time ago? :-) |
…ker names unique - Appending (zones after the first of a multizone SU2 mesh file) opened the file without MPI_MODE_CREATE and closed the handle of a failed open. The file is now opened with MPI_MODE_CREATE|MPI_MODE_WRONLY, an open error is reported, and no invalid handle is closed after a failed exclusive open. - CGNS names have at most 32 characters. Two marker tags with the same first 32 characters gave duplicate section, BC and family names. A number is now appended to the truncated name to make it unique, and the renaming is reported on screen. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
|
@EvertBunschoten @joshkellyjak Can I put you guys on this PR? |
The CGNS writer combines the element types of all ranks with MPI_BOR, but CMediMPIWrapper::convertOp only converted MPI_SUM, MPI_PROD, MPI_MIN and MPI_MAX, so SU2_CFD_AD stopped with "Conversion not implemented" when it wrote a CGNS file. MeDiPack provides AMPI_BOR. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
|
Fixed in e42adb1: Test in 591c180: no AD regression test wrote a CGNS file, so |
No AD regression test wrote a CGNS file, so the missing MPI_BOR in the MPI wrapper of the AD builds was not caught. The reference values do not change. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Copy CCGNSFileWriter from the fixCGNSOutput branch of su2code#2924 (4f6729d): named boundary sections with BCs and families per marker (SetBoundaryMarkers), one zone per marker in surface files (SetSurfaceMarkers), coordinates always in double precision, sections split above 2^31-1 entries, parallel cgp_* output and the serial path. With it the MPI_BOR placeholder of the build without MPI and the two calls in COutput::WriteToFile. The WRT_OUTPUT_DOUBLE_PRECISION option, the Paraview, ASCII and data sorter changes of the PR are not ported; the fields stay in single precision. Drop this commit when su2code#2924 is merged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GK1QTdfVPmjXvNwwwRN6gX
The CGNS boundary sections of the ported PR su2code#2924 writer reduce a bitmask of element types with MPI_BOR. In AD builds SU2_MPI is the MeDiPack wrapper, whose convertOp only knew SUM/PROD/MIN/MAX and stopped with "Conversion not implemented", so SU2_CFD_AD could not write CGNS meshes (adapted mesh export or volume CGNS output). Map MPI_BOR to medi::AMPI_BOR. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GK1QTdfVPmjXvNwwwRN6gX
…s (also needed in PR su2code#2924) The writer ported from PR su2code#2924 names the boundary sections, BCs and families of a volume file after the markers, but only made them unique among themselves. A marker named like a volume section ("Triangles", "Tetrahedra", ...), the solution node ("Fields") or the zone ("Zone") gave a second child node with the same name (Elements_t / FlowSolution_t in the zone, Family_t / Zone_t in the base): cgnslib refuses it in CG_MODE_WRITE ("Duplicate child name found: Triangles") and the run stops with the file half written. This affects any CGNS output, also without adaptation. All names are now decided before the file is opened (PrepareNames): - boundary sections, BCs and families keep the marker names, which the SU2 CGNS reader takes as marker tags (at most 32 characters; a number is appended to a truncated name equal to another one, or to the standard names GridCoordinates, ZoneBC, ZoneType, with a message); - the internal names (volume sections, also when split into "<name>_k", the solution "Fields", the zone "Zone") get the prefix "SU2 " when a marker has the same name; marker names cannot contain spaces (the readers remove them), so the prefixed names cannot collide. Files without such markers are unchanged. - the zones of a surface file (one per marker) are made unique as well. Test: mesh output from memory in CGNS with markers named Triangles, Quadrilaterals, Zone, Fields, Lines (2D) and Tetrahedra, Triangles, Zone, Fields, Prisms, Hexahedra, Pyramids (3D), read back with the same marker names (before: the run stopped with "Duplicate child name found: Triangles"). Runs: NACA0012 with markers Triangles and Zone, ordinary CGNS volume and surface output and the adaptation loop's CGNS volume and adapted-mesh output; both reload in SU2_CFD. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GK1QTdfVPmjXvNwwwRN6gX
…al nodes
The boundary sections, BCs and families of a CGNS volume file are named after the markers, but these
names were only made unique among themselves. A marker named as a volume section (Triangles,
Quadrilaterals, ...), as the solution (Fields) or as the zone (Zone), or as a node with a standard name
(GridCoordinates, ZoneBC, ZoneType), gave two children with the same name, and the CGNS library stopped
the run with "Duplicate child name found".
All the names are now decided before the file is opened. The marker names are kept, because the SU2
CGNS reader takes them as marker tags:
- an internal node with the name of a marker gets the prefix "SU2 " (e.g. "SU2 Triangles");
- a marker named GridCoordinates, ZoneBC or ZoneType, nodes that cannot be renamed, is written with a
space ("Grid Coordinates", "Zone BC", "Zone Type"); the reader removes the spaces from the section
names, so it reads the original tag back and a restart with the same configuration works.
Files without such markers are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
|
Another bug, fixed in 375864c: the boundary sections of a CGNS volume file are named after the markers, but these names were only checked against each other. A marker with the name of another node of the zone (a volume section such as The marker names are kept, because SU2 reads them back as marker tags. Instead:
Files without such markers are unchanged. Checked on the flat plate with its five markers renamed |
EvertBunschoten
left a comment
There was a problem hiding this comment.
Very nice additions! Try to keep things a bit more consistent with regard to the declaration of index variables in for-loops and refactor some of the longer functions.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Only the finite volume data sorter knows which boundary elements are halo copies. Say so with a check of femOutput instead of a dynamic_cast. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
…heir loops Also remove the empty blocks left in the Tecplot writer by the removed loop over the ranks. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
Points, cells (classic and VTK 9 layouts), cell types and point data are written by separate private functions. The written files are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
…iew writers Fixes the CodeQL warnings about comparing an unsigned short with a size_t. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
… in PR su2code#2924) The CGNS writer names the boundary sections, BCs and families of a volume file after the markers, but a section cannot be called GridCoordinates, ZoneBC or ZoneType (standard children of the zone) and names have at most 32 characters, so such markers were written as GridCoordinates_1, ..., or truncated. The SU2 reader takes the marker tags from the section names, so the file could not be read with the config that wrote it (unknown marker): an adapted mesh saved as a CGNS checkpoint was not restartable. A boundary section whose name is not the marker name now holds the marker name in a Descriptor_t "SU2MarkerName" (CGNS_MARKER_DESCRIPTOR), and CCGNSMeshReaderBase takes the marker tag from it (FVM and FEM readers). Files whose markers are valid section names are unchanged. Test: CGNS mesh output with markers GridCoordinates, GridCoordinates_1, ZoneBC, ZoneType, Zone and two names of 43 characters equal in their first 32 (2D and 3D), read back with the unchanged config: same markers with the same elements. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GK1QTdfVPmjXvNwwwRN6gX
… | DOUBLE It is easier to tell apart from OUTPUT_PRECISION, whose description now says that it only sets the digits of the text outputs. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
EvertBunschoten
left a comment
There was a problem hiding this comment.
Everything looks fine, the only remaining change I would like to see is a different name for the option in the configuration file in order to prevent confusion with the WRT_DOUBLE_PRECISION option.
|
Thanks everyone for the reviews and the great suggestions, they made the PR much better! No more work is planned from my side. The last point (the option name) is done: it is now |
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LEL91DW5WPbPwgFtCvHga6
|
ok, great, when Josh is satisfied, we merge. |
joshkellyjak
left a comment
There was a problem hiding this comment.
Sorry for delayed response, LGTM, thank you for the changes
Proposed Changes
This PR fixes the output of large meshes and makes the output writing faster. The problems appeared on a mesh with 273M points and 496M cells (mostly prisms) run on 576 ranks: some output files were broken, the CGNS file could not be opened in ParaView, and some formats took very long to write. Each commit is self-contained, and its message gives the details.
1. Integer overflows
Several writers stored sizes, offsets or MPI counts in a 32-bit
int. All of them hit the same limit, 2^31 - 1 = 2 147 483 647: above it, the run crashed or, worse, wrote a broken file without any error. The limit is reached at different mesh sizes only because each writer counts something different: the data of one rank or of the whole file, in bytes or in values, with a different number of values per point or per element.intThe fixes (in all cases, the files of meshes below the limits are unchanged):
size_t.2. CGNS: sections above 2^31-1 connectivity entries
The CGNS file of the large mesh is valid, but ParaView crashed when opening it. The VTK CGNS reader stores the connectivity size of each section in an
int(reported in https://gitlab.kitware.com/vtk/vtk/-/work_items/20170). Until the reader is fixed, an element type above this limit is written as several sections with consecutive element ranges (<name>_1,<name>_2, ...). Smaller meshes still get one section per element type.3. CGNS: named boundaries and double precision coordinates
ZoneBCand aFamilythat carries the BC type. ParaView shows each marker as a named patch, and SU2 can read the file back as a mesh with the same markers. SURFACE_CGNS writes one zone per marker.VOLUME_OUTPUT_PRECISION(defaultSINGLE):DOUBLEwrites the fields of CGNS, PARAVIEW and PARAVIEW_MULTIBLOCK in double precision. The default keeps single precision, so file sizes do not change.4. Parallel writing
cgp_*, parallel HDF5, already built by SU2 with MPI). Without MPI, the serial API is used as before. The write bandwidth of the CGNS files is now reported in the output summary (it always showed 0 MB/s).Testing
The large mesh cannot be shared, and the overflow limits cannot be reached in CI (a single section above 2^31-1 entries needs more than 20 GB of memory). The tests therefore check that the parallel writers give the same files as before:
TestCases/output_writers/ascii_output.cfg, parallel regression on 2 ranks): the solution ofvandv/rans/flatplateis written without iterating by SU2_CFD (RESTART_ASCII), SU2_SOL (PARAVIEW_ASCII, TECPLOT_ASCII and their surface versions) and SU2_DEF (SU2 mesh file). Each file is compared with a reference file written by the previous writers. The new files are byte-identical to the old ones, on 2 and 4 ranks. The reference files are in Reference files for the parallel ASCII output writer tests TestCases#205.oneram6parallel regression test (CGNS and SURFACE_CGNS output).Temporary:
.github/workflows/regression.ymluses thefixCGNSOutputbranch of TestCases until su2code/TestCases#205 is merged. It will be set back todevelopbefore merging.Question for the reviewers
With MPI, the new writers always write in parallel; the old serial path (gather on the master rank, or ranks writing one after the other) is removed. Parallel HDF5 can be slow on some file systems (e.g. NFS). Should we add a config option to write serially (for example only for CGNS), or is the parallel writing enough?
Related Work
PR Checklist
pre-commit run --allto format old commits.🤖 Generated with Claude Code