Apple Silicon training: measure PyTorch-MPS against MLX and record the backend decision (spike and decision record) #1032

Embeddable engine · EMB-05 · Kind: technology + docs (spike and decision record) · Size M · background: #1027

Summary

ForgeLM's training stack is PyTorch, TRL, PEFT and bitsandbytes, and in practice it is Linux plus CUDA. On a Mac it trains on the CPU, and on Windows the qlora and unsloth extras do not install. Before anyone writes a second training backend we need numbers. This issue is a bounded measurement spike and a decision record: should Apple Silicon be served by the existing PyTorch stack on MPS (cheap, same code path, all six trainers in principle), by an MLX backend (a second trainer path), or not at all? It deliberately does not implement a backend.

Current state, verified (2059b59)

Upstream facts that change the picture (PyPI and vendor documentation, 2026-10-20; the bitsandbytes details are documented with sources in #488):

• bitsandbytes 0.50.2 ships wheels for macosx_14_0_arm64, win_amd64 and win_arm64; 0.50.0 reports 4-bit and int8 working on MPS with torch >= 2.9. The "bitsandbytes does not support Metal" statement in ForgeLM's docs is out of date; the limit is ForgeLM's own marker and its CUDA-only 4-bit gate.
• transformers selects MPS automatically unless use_cpu is set (Apple MPS is never used: training and model loading are pinned to CPU on any non-CUDA host (CR-A02-085) #541).
• mlx-lm 0.32.2 (2026-10-16, Python ≥ 3.11): mlx_lm.lora supports LoRA, DoRA and full fine-tuning, and QLoRA through a quantized model; datasets in chat, completions, text and tools form; adapters are saved as MLX safetensors; mlx_lm.fuse merges them, and its GGUF export is documented as limited to fp16 Mistral/Mixtral/Llama-style models. It does not document DPO, ORPO or GRPO; the community package mlx-lm-lora 5.8.6 adds them. ForgeLM supports Python ≥ 3.10.
• I found no controlled LoRA benchmark of mlx-lm against transformers + PEFT on MPS. General framework benchmarks favour MLX (one public benchmark: 426 s against 913 s for transformer training on an M3 Max; a 2025 profiling paper reports a modest gap), which is exactly why this has to be measured.

What a second backend would cost (the reason to measure first)

• Adapter format. MLX adapters are not PEFT adapters. ForgeLM's merge, export, safety evaluation and compliance code read PEFT output (adapter_config.json, safetensors keys), so an MLX run needs a conversion step or MLX-native replacements for each of those consumers.
• Trainer coverage. The MLX route covers SFT; preference methods need a community package.
• Architecture. ForgeLM wraps TRL; an MLX path is a second trainer implementation with its own config mapping, audit events, manifests (versions recorded in Annex IV) and tests, which needs macOS ARM runners (macos-latest is Apple Silicon, so this is feasible).
• Python range. An [mlx] extra needs python_version >= '3.11' and sys_platform == 'darwin' markers.

Spike plan (no repository changes except the decision record)

1. Prerequisites, tracked elsewhere: Apple MPS is never used: training and model loading are pinned to CPU on any non-CUDA host (CR-A02-085) #541 (an explicit device policy so MPS is selectable) and PEFT, quantization and memory methods: bitsandbytes platform scope: sys_platform=='linux' marker, CUDA-only 4-bit gate, 'bitsandbytes does not support Metal/MPS' docs (incorrect-claim) (TECH-T03-007) #488 (widen or justify the bitsandbytes marker and gate). The spike may use local patches; it must not land them here.
2. Measure LoRA SFT on three model sizes that fit a 16 GB, a 32 GB and a 64 GB Mac (the exact models are chosen at the start and recorded), same dataset, same effective batch size, same sequence length, three runs each: tokens/s, peak memory, wall time to a fixed eval loss, and loss-curve parity between the stacks. Include QLoRA on MPS (bitsandbytes ≥ 0.50, torch ≥ 2.9) and on MLX.
3. Measure conversion fidelity for MLX → PEFT: generation agreement before and after conversion on a fixed prompt set.
4. Windows + CUDA: confirm that widening the qlora marker installs and imports on a Windows machine, and record what cannot be tested in CI (no GPU runners; only install, import and gate logic can run there).
5. Write the decision record (a dated row in docs/roadmap/risks-and-decisions.md) choosing one of: (a) MPS on the existing stack; (b) an MLX backend for SFT, with the conversion and the preference-method plan; (c) no training on macOS, documented as such, with export, chat, audit and ingest remaining supported. A proposal for the spike to confirm or refute, not a pre-decision: choose (a) if MPS reaches within 1.5× of MLX wall-clock at equal loss, because it keeps one code path; otherwise take (b) only if the conversion is faithful.

Out of scope

Implementing any backend (the decision record opens those issues); other accelerators (ROCm, XPU, Vulkan); distributed training on macOS or Windows (#522).

Acceptance criteria

• A measurement table (models, hardware, versions, three runs each) is committed with the decision record, with raw logs linked from the record.
• The record answers: MPS, MLX or neither; which trainers each choice covers; conversion fidelity; Python and platform markers; CI implications; and what the project promises users on each platform (a support-tier statement: tested in CI, best effort, or unsupported).
• If (a) or (b) is chosen, the implementation issues exist, are linked here and carry their own acceptance criteria; if (c), the installation, troubleshooting and GPU-cost documentation (English and Turkish) state the support tier.
• The statements corrected by Apple MPS is never used: training and model loading are pinned to CPU on any non-CUDA host (CR-A02-085) #541 and PEFT, quantization and memory methods: bitsandbytes platform scope: sys_platform=='linux' marker, CUDA-only 4-bit gate, 'bitsandbytes does not support Metal/MPS' docs (incorrect-claim) (TECH-T03-007) #488 stay consistent with the record.

Sequencing and dependencies

The spike can start now. Do not begin implementation before #541 and #488 have landed. Stage D of #1027: the decision gates any backend work.

Related

#541 (MPS never used), #488 (bitsandbytes platform scope), #521 (VRAM probe on MPS), #522 (DeepSpeed platform claims), #821 (technology watch list), #962 (cross-OS tests).

_{Raised on 2026-10-20 as part of the embeddable-engine decision. Code links point at 2059b59; line numbers may have moved. Package facts come from PyPI and the vendor documentation read on 2026-10-20. Tracking ID: EMB-05.}