Skip to content

[Feature] M2: source-independent cooked scene loading and complete first-frame rendering #873

Open

Description

@JeanPhilippeKernel

Type

  • Bug
  • Feature
  • Refactor / cleanup
  • Task / chore

Overview

Deliver milestone 2: launch and render a cooked scene without authoring sources, importers, or source asset-registry lookup. Extend the M1 engine-owned transaction rather than introducing another scene loader. Publication requires a complete rendered first frame, not just successful binary decoding or mesh upload.

Load intent is explicitly RendererOnly. This milestone does not implement gameplay, Play, actor restore, or live physics/audio reconstruction.

Context

What needs to be done

Implement in reviewable slices; each slice needs negative tests before integration:

Feature checkboxes reflect the implementation in PR #879; they do not claim CI or platform qualification. Strict performance and calibrated-resource qualification are tracked separately in #880.

1. Retained package snapshot and strict preflight

  • Capture one immutable manifest/package snapshot and retain allocator, schema, and format-registry lifetimes for the whole request and dependent resources.
  • Validate the current DesktopVulkanLE64 target, current supported versions, selected scene, complete typed dependency closure, renderer/device capabilities, counts, sizes, and root-relative paths before opening/allocating object payloads.
  • Reject traversal, aliases/conflicting identities, invalid lengths/offsets, truncated/missing/corrupt objects, unknown schemas/formats, and known-but-unsupported required capabilities with stable diagnostics. A manifest Rebuild result is runtime rejection, not permission to recook.
  • Verify every consumed object and decode exactly the verified bytes. Use guaranteed immutable retained storage or verified private copies; reopening a mutable path or holding a root handle alone is insufficient. Replacing a package must not retarget an admitted load.

2. Cooked asset lookup and version ownership

  • Add engine-asset-layer lookup by typed manifest identity plus content/version, not UUID alone; reuse CookManifestReader/CookObjectReader primitives where appropriate.
  • Decode cooked meshes, materials, textures, and environments through existing shared codecs. Resolve transitive dependencies without editor registry/source/import access.
  • Retain exact immutable asset versions through scene/resource leases so the outgoing scene can use an older version with the same UUID.
  • Keep CPU scene/document/scratch/cache, GPU residency, staging, queues, and retiring versions separately bounded/accounted. Count shared versions once and overlapping versions separately; do not adopt the cooker's 768 MiB workspace as a runtime budget.

3. Complete first-frame preparation

  • Feed strict BinarySceneCodec output into the shared unpublished scene population/hierarchy/UUID-resolution path.
  • Prepare complete mesh/index/instance bounds, materials and blend/depth variants, required sampled mips/textures, descriptors, lights, and enabled environment-derived lighting. Missing required resources fail; only schema-defined optional defaults are allowed.
  • Use compatible versioned shipped host shaders, prepare required pipeline variants, and wait asynchronously for renderer completion/visibility/queue ownership—not merely upload submission or CPU cache presence.
  • Choose the launch camera by explicit camera UUID, otherwise require exactly one IsMain camera. Validate transform/projection/aspect; missing/ambiguous cameras fail deterministically. Preserve the editor's existing camera policy for source loading.
  • Reserve a complete immutable initial frame and all publication/result/retirement capacity before guarded commit. No load-induced device-idle, main-thread fence wait, unbounded first-draw compilation, or fallible work inside commit.

4. Source-free startup and lifecycle integration

  • Adapt GameApplication/existing startup to mount and select a cooked package without an authoring project scan. An existing host is sufficient; a new executable/ABI/installer is not required.
  • Keep request admission/cancellation/supersession/shutdown semantics and GPU cleanup from M1. Post-submit cancellation records its terminal outcome while retaining in-flight resources until real completion.
  • Report RendererOnly intent and inactive simulation data explicitly; reject requested unsupported execution. Preserve outgoing owner/bindings/generation on every precommit failure; a no-active-scene host reports failure without pretending startup succeeded.
  • Route device loss through host fatal/recovery policy, never through a fabricated successful completion or a promise of renderable rollback.

5. Qualification and handoff

  • Extend existing lifecycle/capture evidence tools and add a fresh-process cooked runtime smoke helper registered in CTest/CI. Fixtures must use production cooker/codecs, not hand-written alternate encoders.
  • Freeze fixture/package/shader identities, declared supported capabilities, finite budgets and timing limits before final runs. Attach reproducible commands and evidence to the PR; keep benchmark/build history out of design contracts.

Non-goals: Play/gameplay activation, actor factory design, source fallback/runtime migration/recooking, additive loading/streaming, hot reload, signing, shipping installers, and device recovery implementation.

Follow-up issues

Resources

  • Approved runtime contract, especially §§4–7 and 9.1–9.3/9.5.
  • Cooker wire, package, ownership and resource contracts.
  • Starting points: ZEngine/ZEngine/{ECS,Cook,Managers,Rendering}, Managers/AssetManager.*, Applications/GameApplication.*, SceneInstance, SceneLoadCoordinator, AppRenderPipeline, RenderResourceManager, existing cook/process fixtures, and Scripts/{validate_scene_lifecycle,compare_scene_capture}.py. Verify current locations/APIs after rebasing onto M1.
  • Visual source fixture: SampleProject/project.json; generate and pin a cooked copy from that source.

Testing

  • Unit tests: manifest/typed-closure selection, exact-cap/cap-plus-one admission, verified-byte lifetime, package mutation/replacement, unsupported target/version/capability, deterministic camera selection, immutable-version sharing and budget rollback.
  • Deterministic integration barriers at read/decode/allocation/bind/upload/preflight and before/after commit: cancellation, supersession, owner destruction, shutdown, delayed upload/render completion, failed pipeline preparation, device loss, and successful retry. No sleeps as proof of race correctness.
  • Fresh-process source-free test: deny source directories, source registry and import paths, audit VFS accesses, and render a real cooked first frame using shipped host resources. Register schemas in a harmless different process-local order to expose leaked runtime IDs.
  • Compare source/cooked opaque and transparent materials, textures, hierarchy, camera, lights and environment at fixed viewport/exposure/timestep/history settlement. Semantic assertions must catch missing content independently of pixel tolerance.
  • Performance: 30 fresh-process loads and 30 replacements, separate first-use/warm states and filesystem-cache conditions; measure admission through first rendered frame, main/render stage spans, p50/p95/p99/max and retained outliers.
  • Stress: 100 warm-up plus 1,000 measured alternating load/cancel/switch cycles; zero-cache reference reclamation returns tracked live allocations/pins exactly to warmed baseline after readers/uploads drain. RSS is trend evidence, not the allocation ledger.
  • Run engine/editor regression targets and process/evidence tests on Linux, Windows and macOS. Rendered qualification is claimed only for devices/platforms with actual evidence; use supported sanitizers and Vulkan validation with recorded limitations.

Example existing local entry points (configure the matching platform preset first; add the new helper/CTest commands to the implementing PR):

cmake --build --preset Darwin_arm64_Debug --target ZEngineTests TetragramaTests
./Result.Darwin.arm64.Debug/ZEngine/tests/Debug/ZEngineTests --gtest_brief=1
./Result.Darwin.arm64.Debug/Tetragrama/tests/Debug/TetragramaTests --gtest_brief=1
python3 -m unittest discover -s Scripts/tests -v

Acceptance criteria

  • A fresh process renders the complete selected cooked scene with zero authoring/source/import-registry lookups and no required-resource placeholder fallback.
  • All consumed bytes/versions are verified and retained coherently; target, integrity, capability and path failures are explicit and preserve the outgoing scene.
  • Required GPU resources, initial snapshot and pipeline variants are ready before atomic publication; cancellation and retirement remain completion-safe and bounded.
  • Frozen-profile limits pass: aggregate loader main-thread work ≤4 ms/frame, guarded commit ≤2 ms within that allowance, and loader-attributable additional frame stalls ≤8 ms over matched steady rendering; render-critical preparation is also measured. No unexplained outliers or retroactive budget increases.
  • Source/cooked visual equivalence meets the reviewed profile (initial contract: ≤0.5% pixels differing by >4/255 in declared RGBA8 capture space, predeclared UI masks only), and all semantic checks pass. Calibrate/freeze repeat-capture noise before qualification, not after a failure.
  • Finite memory/staging/queue/retirement/load-time/cancellation limits, exact-boundary tests, stress and reference reclamation evidence pass.
  • Builds without new warnings in Debug and Release; all relevant unit/process tests pass; no editor/source-load regression.
  • Cross-platform CI and the declared rendered-profile evidence are reviewed; PR links the exact reproducible evidence and remaining platform limitations.

Estimated effort

Large, multi-PR milestone. Deliver slices 1–5 incrementally; estimate each after M1 lands and the cooked format/host resource integration seams are audited. No fixed completion date is implied.

Metadata

Metadata

Labels

P1Critical path — blocks other workarea-renderingenhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions