Training progress events: a versioned, machine-readable JSONL stream for front-ends, dashboards and CI #1028

Embeddable engine · EMB-01 · Kind: code + docs · Size L · background: #1027

Summary

A program that starts a ForgeLM training run cannot find out how the run is going except by scraping human-readable log output. The library call blocks until the whole pipeline has finished, the audit log records only the start and the terminal outcome, webhooks announce lifecycle transitions and nothing else, and none of this is a documented, versioned contract.

This issue adds a versioned, machine-readable progress stream: step, epoch, loss, learning rate, elapsed time, ETA, phase transitions and exactly one terminal event, readable the same way by a front-end, a dashboard or a CI job.

What exists today (2059b59)

Two implementation facts constrain the design:

• GRPO discards every callback. _build_grpo_trainer sets trainer_kwargs["callbacks"] = [] (L1077-L1085) to drop the eval-dependent EarlyStoppingCallback. A progress callback appended to the user list would silently vanish for GRPO.
• OOM recovery rebuilds the trainer and restarts optimisation from step 0 at a smaller batch size (L1203-L1285). A reader must be told that the counter reset, not watch the step number jump backwards.

Prior art: LLaMA-Factory writes a trainer_log.jsonl with current_steps, total_steps, percentage, elapsed_time, remaining_time, loss, learning rate and epoch per line. The approach is proven. This design uses numeric seconds instead of H:MM:SS strings and adds a schema version.

Proposal (the open decisions are listed after it)

Sink. An append-only JSON Lines file, <output_dir>/progress.jsonl, switched on by a new validated config field (default off; see decision 1). Stdout is reserved for the final JSON envelope (logging-observability.md, rule 1) and stderr is for logs, so neither can carry events. A file also survives a front-end restart: the reader re-attaches by byte offset.

Envelope on every line: schema_version (integer, starts at 1), seq (strictly increasing within the file, starts at 1), ts (ISO-8601 UTC), run_id (the same id as the audit log), event.

Events, schema version 1

Writer rules

• Serialise with dumps_strict, so a non-finite number becomes null (project rule; tools/check_strict_json_writers.py enforces it).
• One line per event, flushed per line, not fsync'd: the stream is advisory, unlike the audit log.
• A write failure logs one WARNING and disables the sink. It never raises into training, which is the opposite of AuditLogger, whose failures must surface.
• Rank 0 only (state.is_world_process_zero; see No rank-0 gating: every distributed rank runs the whole post-train pipeline with its own random run_id, writing final_path, model_integrity.json, compliance files, audit events an… (CR-A07-018) #470).
• No secrets, and no path outside output_dir.
• The stream is not compliance evidence: it is not hash-chained and not part of the Annex IV bundle. The audit log stays the system of record, and the docs say so.

Reader rules (documented)

• Tolerate a partial last line; read by byte offset and re-read from the last newline.
• Filter by run_id; a resumed run appends and a new run in the same directory appends, so the file is never rewritten.
• The absence of a run.finished line together with a stale modification time means the process was killed or crashed. Nothing can be written after a hard kill.

Where the code lives. The schema, the writer and the TrainerCallback go into a new module, forgelm/_progress.py. trainer.py is at its no-growth budget (#904), so it gains only the few lines that attach the reporter, including re-attaching it after an OOM rebuild and after the GRPO callback reset. The model_load and data_prepare phases happen in forgelm/cli/_training.py before a ForgeTrainer exists, so the reporter is created at the CLI seam and handed down.

Library surface. Ship the Python hook as Experimental tier first (library API tiers) and promote it to Stable after one minor release in which a real consumer has used it. The file schema is the primary contract; the Python hook is secondary.

Open decisions (record as a dated row in docs/roadmap/risks-and-decisions.md before coding)

1. Config field and default. Proposal: training.progress_events: bool = false next to report_to. monitoring: is not suitable: it is a declarative Art. 72 plan "without runtime hooks". Extending report_to is not possible without a breaking type change, because it is a single-valued Literal that is passed to HF.
2. Who writes the terminal event. The process exit code is known only at the CLI seam, so either the CLI emits run.finished (with exit_code) or ForgeTrainer emits it without one.
3. Throttle. Reuse HF logging_steps (1–50 depending on dataset size) or add a minimum interval (for example 1 s) so a tiny dataset cannot flood the file.
4. Pipelines. One root-level file with a stage field, or one file per stage directory plus stage markers in a root file.
5. Tier. Experimental first (proposed) or Stable at once.

Alternatives considered

Out of scope

Any UI; stopping a run (EMB-02); changing the --output-format json envelope (#412); GPU and system metrics such as VRAM or temperature (can be added later as optional fields); any change to the audit chain.

Acceptance criteria

• The five open decisions are answered in a dated decision row.
• forgelm/_progress.py defines the schema, the dumps_strict writer (flush per line, rank-0 only, never raises into training) and the TrainerCallback.
• SFT, DPO, SimPO, KTO, ORPO and GRPO each emit train.step; a test proves the GRPO path keeps the callback.
• An OOM-recovery retry emits train.restarted and seq stays strictly increasing.
• Phase events cover evaluate, save, benchmark, safety, judge, artifacts and approval, and exactly one terminal event is written for success, failure, auto-revert and awaiting approval.
• Multi-stage pipelines carry the stage fields according to decision 4.
• With the stream enabled, --output-format json stdout is byte-identical to a run without it (guards against regressing Training --output-format json stdout is not a JSON document: print_trainable_parameters and HF Trainer log dicts precede the envelope (CR-A03-008) #412).
• Non-finite values are written as null; a test covers it.
• A reader test covers a torn last line and a gap-free seq.
• Documentation in English and Turkish: a new docs/reference/progress_events.md (and -tr.md), the config reference, config_template.yaml, the library API reference tier row, a cross-link from the JSON output page, and a CHANGELOG Added entry. A test keeps the documented event list equal to the emitted one.
• Tests are CPU-only and run on Linux, macOS and Windows.

Sequencing and dependencies

Stage B of #1027. Start after Phase 16 S8 (#894) has settled trainer.py; coordinate with the module split (#904) and land together with, or after, the stdout fix (#412). EMB-02 builds on the callback created here.

Related

#412 (stdout pollution), #470 (rank-0 gating), #569 (interrupted runs), #904 (trainer split), #894 (S8).

_{Raised on 2026-10-20 as part of the embeddable-engine decision. Code links point at 2059b59; line numbers may have moved. Tracking ID: EMB-01.}