Skip to content

fix(ShaderConverter): read subprograms from their referenced segments - #385

Open
nichinichisou0609 wants to merge 2 commits into
K0lb3:masterfrom
nichinichisou0609:codex/shader-segments
Open

nichinichisou0609 wants to merge 2 commits into
K0lb3:masterfrom
nichinichisou0609:codex/shader-segments

Conversation

@nichinichisou0609

Copy link
Copy Markdown
Contributor

Shader export currently decompresses only the first segment for each platform and reads every subprogram offset from that segment. For example, an entry with segment = 1 and offset = 0 is read from the index table at the start of segment 0. The change reads the program at the start of segment 1 instead. In the regression fixtures, valid multi-segment exports previously fail during subprogram parsing; after the fix, the exported text contains the expected payloads in blob-index order.

The patch decompresses all segments described by each platform's offsets and length arrays. ShaderProgram accepts optional segment readers while preserving its existing two-argument constructor. It reads the entry's segment field for Unity 2019.3 and later; earlier entries use segment 0. Flat arrays and a single segment remain supported. Inconsistent segment-array lengths and references to unavailable segments raise ValueError; multiple entries may still reference the same record.

The public evidence for this scope is:

  • At the base revision, ConvertSerializedShader selects item[0] from nested arrays, and ShaderProgram reads offsets using one reader. These are the two places that lose segment selection.
  • PR #280 established scalar/list compatibility while retaining the first list element. This patch extends that handling to the remaining segments.
  • PR #137 added guards for platform/program count mismatches. Those guards remain unchanged; they address a different dimension from the per-platform segment lists.
  • AssetStudio's segment loop decompresses each segment separately. Its entry reader and dispatch read the segment field from 2019.3 onward and seek the offset within the matching segment. This supplies a public implementation reference for the layout and offset semantics.
  • The maintained AssetStudioMod fork retains the same segment dispatch, including independent seeks that allow repeated offsets. This is corroborating code from the same lineage, not a separate Unity format specification.

The 19 new regression cases construct complete subprogram records in memory. The full-export cases use real LZ4 compression and the normal subprogram parser, covering:

  • Distinct code payloads in multiple segments, with offsets relative to each segment and entries ordered independently of their segment numbers.
  • Flat legacy arrays, nested single-segment arrays, and both pre-2019.3 and 2019.3-or-later entry layouts.
  • The existing two-argument ShaderProgram constructor.
  • Exported shader text containing the expected subprogram payloads, exercising decompression, record parsing, and export together.
  • Records sharing one offset, separate segment lists for GLES3 and OpenGLCore, unavailable segments, and inconsistent metadata arrays.
  • Repeated references into a nonzero segment with both memory-backed and stream-backed readers.

Validation on Linux x86_64 with CPython 3.9.25 / pytest 8.4.2 and CPython 3.13.16 / pytest 9.1.1, using lz4 4.4.5. Both the compiled UnityPyBoost path and the Python fallback were exercised:

Check Result
New regression module against unmodified 417d595 10 failed, 9 passed
Complete suite, CPython 3.9.25 with extension / Python fallback 67 passed / 67 passed
Complete suite, CPython 3.13.16 with extension / Python fallback 67 passed / 67 passed
ruff check and ruff format --check Passed

The ten baseline failures comprise three valid multi-segment export cases, two calls to the new optional segment-reader argument, and five error-reporting cases for malformed metadata. The nine passing baseline cases cover explicit single-reader parsing and single-segment export. To reproduce the checks from the repository root after installing .[tests] and Ruff:

python -m pytest -q tests/test_shader_segments.py
python -m pytest -q tests
ruff check
ruff format --check

The scope is segment selection for subprogram records that the existing parser understands. It does not add player-subprogram or parameter parsing, new GPU program decoders, or reconstruction of a compilable shader. The tests select layout branches using 5.3, 2018.4, 2019.2, 2019.3, and 2022.3 version values; these are synthetic fixtures, not exports from those Editors. Windows/macOS CI and an Editor-generated asset matrix remain unverified.

The 5.3 constructor fixture supplies an explicit little-endian reader; it does not validate the separate legacy m_SubProgramBlob export entry point. Additional checks of that entry point reproduce an existing byte-order failure on both 417d595 and this change: the entry point constructs a reader without specifying its endianness. That path is unchanged by this patch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant