Skip to content

Add restricted native MLX GRPO implementation candidate - #276

Closed
ProfSynapse wants to merge 5 commits into
mainfrom
t3/native-mlx-grpo-integration
Closed

ProfSynapse wants to merge 5 commits into
mainfrom
t3/native-mlx-grpo-integration

Conversation

@ProfSynapse

@ProfSynapse ProfSynapse commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Adds a native MLX GRPO implementation candidate through the existing public TrainingAPI/native family. The host owns exact source/runtime/model/input/reward selection, grants, catalogs and durable stores; submission uses the existing synchronous owned process and guardian. Independent review and fresh Apple qualification remain required before any hardware claim.

The source audit selects an attributed MIT objective subset from inductiveML/mlxrl@5ee8a6bd7764b0585094d78c530ac87d26364034. A separate named profile and 54-package closure preserve the qualified SFT environment. Native GRPO recompiles or rejects public controls, retains sampled IDs and generated-only loss, bounds rollout/reward/evidence work, and requires grouped reward variation plus an independently recomputed saved adapter delta for five-role verification.

Evidence above 1 MiB uses the exact admitted profile limit for exclusive publication and same-tree readback. The generic writer retains its 1 MiB default; larger writes require an explicit validated finite limit. Other SFT/control writes keep their default, and profile/artifact/guardian bounds remain in force. Evidence over the admitted limit rejects before any control document is created. The existing restricted host consumer admits only the exact reviewed GRPO selection; no new public provider verb or permissive qualification issuer is added.

Validation:

  • Native CPU regression: 279 passed, four explicit Apple numerical skips. Includes ten new real-filesystem writer/evidence controls.
  • Exact independently reviewed 64-update/batch-four/128-token fixture: 1,386,552-byte evidence publishes and reads back at both 16 MiB and exact-fit admission, then passes full five-role verification with 128 varied groups/32,768 tokens/256 reward calls. One-byte-under admission rejects before document creation; unchanged default, bounded readback, collisions and invalid limits are checked.
  • Raw and prompt/completion SFT canonical bytes equal the independently reviewed merged-main baseline exactly. Original SFT locks and fixed runtime inventories/pins are unchanged; all six maintainers report CURRENT without writes.
  • Independent scalar loss/analytic derivative checks, Ruff, diff checks and skill sync pass. Fresh sdist-to-wheel archive checks match 18 native/public source modules without package installation. The real MLX autograd test remains unexecuted on Linux.

Mac execution, GPU fit, real grouped updates, fresh runtime/source capture and public artifact readback still require root-owned review and fresh finite admissions. Public logs/cancel/reconcile remain unavailable. See docs/architecture/native-mlx-grpo.md for upstream pins, supported controls, the bounded-publication correction and qualification proposal.

The new source-only example under examples/training_qualification/native_grpo_preparation/ provides native-only Linux input preparation through the existing service/normalizer/verifier, without a Modal profile or ML effects. Its reviewed JSON configuration binds exact seed bytes, prompt column, canonical public request and fresh private output. It retains failures and emits actual prepared identity/publication/request outputs. The two color-choice rows and optimizer/layer/budget proposal are explicitly controlled mechanics data for root review, with no guaranteed reward variation or admission authority. The wrapper plus existing preparation-service regression passes 44 controls, including private collisions, retained partial stages and ML/network guards. Existing runtime/worker/native_host/P2/SFT code and locks are unchanged by this addition.

Closed native GRPO phase diagnosis

The first retained source197 model attempt failed at the old unclassified run_request boundary (122), before an initial output file. No stderr text was retained; neither byte counts nor the later ordinary tokenizer-only pass establish a cause. Source/CPU audit found no established corrective config change.

Commit 24d94530225a057825a0c0d52a479555b75ae232 adds trusted child-local GRPO phase exits130–148 for compilation/tokenizer/import/model/adapter/rollout/objective/update/save/reload/publication/custody. Request/schema fields stay closed, public failure stays native_worker_failed, SFT120/121/122 and the success inventory remain unchanged. No raw diagnostic content or new output file. Historical attempts cannot be reclassified or replayed. See the canonical native-local-training reference for the exact static map.

Validation:289 native CPU tests passed,4 Apple-only skips;26new diagnostics controls inject actual operation failures and round-trip synthetic success through five-role verification. SFT raw/messages canonical hashes unchanged; all7maintainer invocations CURRENT; Ruff/skills/diff pass;15native/public modules match both fresh source archives, with no package install. No Mac/MLX/model/training/provider/admission effect. CPU doubles do not qualify hardware. Root owns integration, fresh independent review, final package/capture refresh and any separately admitted new attempt.

Explicit prompt-ID contract and reusable tokenizer-only profile

Commit 983069ad34dff1ac4a96a054fa447f55906ae294 fixes a concrete pinned Transformers5.20.2 API mismatch: apply_chat_template defaults to a mapping, so the old tuple conversion iterates string keys. GRPO now explicitly requests return_dict=False through a shared prompt-tokenization helper, preserving generation scaffold/configured kwargs and rejecting reserved-control collisions or malformed IDs before ML imports (phase135). SFT, runtime pins, diagnostics and public failure/success inventories remain unchanged. This static defect does not establish historical122's cause or permit replay.

The existing config-first preparation example gains profile-tokenizer for Linux/Darwin, reusing that exact helper with an actual recompiled workload, verified GRPO publication and private complete tokenizer snapshot, exact installed version and finite byte budgets. It retains create-only private token evidence and closed counts/hash summaries, no model/MLX/training/admission. See the checked-in tokenizer-profile.example.json and README. Root owns actual paths/source review and any Mac action.

Validation:304nativeCPU pass/4Apple-only skips,29wrapper pass;25new mapping-default/scaffold/collision/malformed-ID/private-publication/custody/version/bound/CLI controls. All7maintainer invocations CURRENT without pin/inventory changes; SFTcanonical hashes unchanged; Ruff/diff/skill sync pass;15native/public modules match fresh wheel/sdist without installation. Root reports diagnostic-only d88 R4accepted; this new change needs final integrated R5 and fresh source/capture review before any separately admitted model attempt.

Integrated into merged PR #277, merge 5c9a5cf9a4b19e5ce193c3c09a46df22c86296a7. Its reviewed/tested tree includes this work and the final Mac qualification fixes. This draft is superseded; branch and retained evidence are preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant