[charlesmartin14]

The existing DistilBERT experiment fine-tunes pretrained weights with PyTorch/XLA. This independent folder adds random-initialization MLM pretraining on native TorchTPU so we can measure spectra that emerge during training.

The experiment is labeled BERT-6 MLM from scratch: the stock six-layer DistilBERT architecture (66,985,530 parameters), trained without a teacher or distillation loss. It uses TorchTPU 0.2.0, tpu_dist, eight workers, AdamW, pinned WikiText-103 data/tokenizer, checkpoint/restart, and CPU WeightWatcher fits on all 36 encoder matrices. Full best-validation weights are retained separately, and evaluation masks remain fixed when training masking changes.

A second complete code audit found additional launch and recovery failures and fixes them here:

• Use the supported accelerator RNG getter: the released TorchTPU wheel rejects direct torch.tpu.get_rng_state calls from user code on PyTorch >=2.13. Every TPU rank now probes RNG save/restore before data preparation.
• Commit training state before publishing measurements or best weights; recover a missing scheduled measurement after interruption.
• Replace unpublished checkpoint generations during replay. Partial and future unpublished directories cannot displace the last two committed checkpoints.
• Drain the final spectral measurement even when training completes during the worker's scan; compare matching step filenames when reporting completeness.
• Escalate cleanup to SIGKILL and reap owned processes when SIGTERM is ignored. Status discovery remains available near lease expiry, while new launches retain the lease gate.
• Reject nonfinite optimizer scalars, fractional sizes, invalid seeds and nonboolean mode flags.

The Cloud Shell launcher reuses the existing idle leased v5e-8 in tpu-builders-504820, checks occupancy and persistent storage, creates a separate Python 3.12+ environment, probes eight ranks, executes two real updates, then resumes the same model. It caps execution at the existing lease and the 5.5-hour default budget. Each invocation runs one committed configuration; the six-point LR/warmup grid is available without an automatic sweep.

Validation: 19 CPU engineering tests passed; one live Gloo test was skipped because this environment previously rejected socket/interface initialization. Coverage includes exact parameters, optimizer state and evaluation metrics after restart for AdamW, Adam and SGD; compiled aot_eager BF16 restart with dropout; global masked-token gradient arithmetic; full architecture; fixed evaluation masks; and injected interruption/cleanup scenarios. After the final retention-filter extension, both focused checkpoint regression tests passed again. VALIDATION.json records the tested source hashes and exact scope. Complete real WikiText preparation was previously verified: 913,341 training, 1,913 validation and 2,191 test sequences, with every row and hash checked; preparation code is unchanged.

Hardware status: no TPU model has been launched or trained from this session. API-selection tests use mocks; native TPU kernel coverage, distributed execution, RNG replay, restart, throughput, convergence and alpha outcomes remain pending. The 10,000-update pilot covers about 2.803 corpus passes. It and the six-layer architecture are defensible starting points, not established optima; AUDIT.md records the primary literature and calibration requirements. Select recipes using validation quality, with alpha remaining an observed outcome.