Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,9 @@ in the same binary by 9.3; its documentation (9.4) and runtime tests (9.5) are s
- **`runtime-tests/`** β€” tests that run inside a real Max (plan 6.1): Cycling '74's max-test harness
(submodule), patchers generated by `make_patchers.py` (edit the generator, never the patchers),
fixtures (`python/maxtest_*.py`, copied into `python/` for a run) and `run.py`, which installs the
harness into Max's Packages, launches Max, drives it over OSC and reports. macOS, local only β€”
harness into Max's Packages, launches Max, drives it over OSC and reports β€” `tap.python~`'s tests
and `tap.python`'s (`tap.python.*`, plan 9.5; `tap.python.load` opens first, in a Max with Restore
Windows on Launch turned off for the run, so that `init/` must find the object). macOS, local only β€”
`runtime-tests/README.md` has what the first runs taught about writing one. Two sessions run only
when asked for: `soak` (plan 6.2, an hour) and `perf` (plan 6.3). `runtime-tests/spike/` is the
record of the 9.0 spike for `tap.python` β€” its patchers, runners and Max's logs; the spike's class
Expand Down
2 changes: 1 addition & 1 deletion docs/PRODUCTION-PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -828,7 +828,7 @@ row shipped what only a host platform could show.
generator); a vignette and three tutorials with patchers for the Documentation window; an
Extras overview named as the landing patcher; help patchers with tabs for both objects; CI
checks for the hand-made XML and JSON.
- [ ] **9.5 Runtime tests:** the `tap.python.*` patchers in `make_patchers.py`; `run.py` aware of
- [x] **9.5 Runtime tests** (#50)**:** the `tap.python.*` patchers in `make_patchers.py`; `run.py` aware of
the second object; the soak session gains a `metro`-driven `tap.python` under saves.
- [ ] **9.6 The release session, Mac and Windows:** the whole suite on the Mac, the help patcher
checked and re-saved, Max's page for `tap.python~` committed, the hand checks; the package by
Expand Down
21 changes: 18 additions & 3 deletions docs/TAP-PYTHON-PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ change β€” and the plan to build it (Phase 9 of the production plan, which point
Drafted 2026-10-02 against 1.0.0 and audited the same day (`AUDIT-TAP-PYTHON-PLAN.md`: two
blockers, six major findings, nine minor; its addendum of 2026-10-09 records what 1.0.1 and 1.0.2
changed). **Revised 2026-10-09 against 1.0.2** for every finding; the [revision record](#revision-record)
at the end says what each changed. 9.0 to 9.4 are built β€” the spike ran in Max on a Mac
at the end says what each changed. 9.0 to 9.5 are built β€” the spike ran in Max on a Mac
and on Windows: its answers are under [9.0](#what-90-found), and the design below follows them. Tick the
items below (with the PR) as they land, and keep the design current where a PR decides differently.

Expand Down Expand Up @@ -450,15 +450,30 @@ third was Windows-only.
9.5's), which also shows that the wrapper passes a class's attributes and messages on only through
`setvalue` and `applyvalues`. The ReadMe's message sentence is the perf script's, written when
the tables are next measured (9.6). Checked in Max 9.1.5 on the Mac; Windows in 9.6.)*
- [ ] **9.5 Runtime tests.** The patchers above in `make_patchers.py` (9.4 added `tap.python~.mc`,
- [x] **9.5 Runtime tests** (#50). The patchers above in `make_patchers.py` (9.4 added `tap.python~.mc`,
for the `init/` mapping of `mc.tap.python~`); `run.py` aware of the
second object (the `init/` mapping in the installed package, the `--package` mode, the
reference-page rule for `tap.python~`'s page only). What 9.0's runner learned: a fresh Max
needs Restore Windows on Launch off (a restored window loads the binary first); patchers are
opened with an "open document" event, never `max openfile` through the harness's OSC (it crashes
Max); and Max's log drops a `[print]`'s name. The soak session (6.2) gains a
`tap.python` driven by a `metro` while its file is saved every second: every output checked,
memory flat, the console counts exact.
memory flat, the console counts exact. *(As built: seven `tap.python.*` patchers β€” load (opened
first, in a Max with Restore Windows on Launch off for the run, so `init/` must find the object),
outputs (every row of the table), messages (attributes, `get<attr>` from the dumpout, the
forwarder), reload (an outlet added before the dumpout takes a scripted cord and carries; one
removed takes its cord; a broken save reported once), shared (one file for both kinds: both follow
a save, each kind's reserved name said once β€” `mode` by `tap.python~`, `dumpout` by `tap.python`,
as Max answers `dumpout` on neither but the class that registers it), threads (a `metro` every 5
ms, every output against its input, on the main, scheduler and audio threads) and faults. Outputs
are checked as `[tosymbol]` writes them β€” floats with four decimals, `symbol` kept before a
string β€” which a probe in Max found; the README records it. The `[print]`-free checks needed no
`max openfile`: run.py opens patchers through the harness's `test.master run`, which 6.1 already
used. `run.py` requires the package's `init/` lines, warns if the hand-written page changes, and
puts Max's preferences back. All 21 runtime tests pass in Max 9.1.5 on macOS; the hour-long soak
passed all 18 of its checks, the `tap.python` answering every message of a 10 ms metro correctly
for 60 minutes of saves: Max's memory 658 MB a minute in, 663 MB at the end (peak 675), 3601
executions of the file, Python's object count between 18,700 and 22,000 throughout.)*
- [ ] **9.6 The release session, Mac and Windows.** Build, run the whole runtime suite on the Mac,
check the help patcher and re-save it, commit the page Max writes for `tap.python~`, the hand
checks the tests cannot make (`get<attr>` through the dumpout in a patcher, attrui on a
Expand Down
37 changes: 29 additions & 8 deletions runtime-tests/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Runtime tests in Max

The unit tests (`core/tests/`, and the external's `_test.cpp` against min's mock kernel) never load
the object into Max. These do: each is a patcher that Max runs, driven by Cycling '74's
the objects into Max. These do β€” `tap.python~` and, since plan 9.5, `tap.python` (the
`tap.python.*` patchers): each is a patcher that Max runs, driven by Cycling '74's
[max-test](https://github.com/Cycling74/max-test) harness (MIT, the `max-test` submodule), checking
what only a real Max shows β€” the file watcher, `getattr` on the Python-defined attributes, audio
through the real signal chain, `poly~`, the console, and loading without a runtime. (Plan 6.1 and
Expand Down Expand Up @@ -41,26 +42,32 @@ runtime-tests/

## What the runner does

1. Builds the harness's `oscar` extension if needed, and installs the harness into `Packages` as
1. Checks the package: the external built from the current sources, a runtime, and the lines of
`init/tap.python.txt` without which Max finds neither `tap.python` nor `mc.tap.python~`. Then
backs up Max's preference files, turns off **Restore Windows on Launch** and moves aside the
workspaces Max would reopen (into `logs/`) β€” a window reopened at launch would load the binary
before the first test β€” and puts the preferences back when the run ends.
2. Builds the harness's `oscar` extension if needed, and installs the harness into `Packages` as
`max-test` β€” a **copy**, not a link, because Max loads a package's extensions only from a real
folder. The copy is marked `INSTALLED-BY-PYTHONTAP.txt`, replaced on every run, and left
installed afterwards (Max then loads `oscar` at every launch; delete the folder to uninstall).
Another `max-test` already in `Packages` is refused, not replaced.
2. Links `runtime-tests/` into `Packages` as `PythonTap-runtime-tests` (so Max finds the patchers
3. Links `runtime-tests/` into `Packages` as `PythonTap-runtime-tests` (so Max finds the patchers
by name), and copies `python/maxtest_*.py` into the package's `python/` folder (the object loads
classes from there only). Both are removed afterwards.
3. **Session `without-runtime`**: moves `support/` aside and starts Max. The object must load and
4. **Session `without-runtime`**: moves `support/` aside and starts Max. The object must load and
say the runtime is missing; then, with `support/` put back while Max runs, a new object must
say to restart Max (on macOS the weak binding to libpython is fixed when the external loads).
`support/` is always put back β€” and a run killed midway leaves it as `support.maxtest-aside/`,
which the next run restores.
4. **Session `main`**: starts Max again and runs every `*.maxtest.maxpat`.
5. Only when asked for: **session `soak`** (plan 6.2) runs `soak/tap.python~.soak.maxpat` for an
5. **Session `main`**: starts Max again and runs every `*.maxtest.maxpat` β€” `tap.python.load` first,
so that the fresh Max must find `tap.python` through the package's `init/` mapping (plan 9.0).
6. Only when asked for: **session `soak`** (plan 6.2) runs `soak/tap.python~.soak.maxpat` for an
hour, sampling Max's memory every minute, and writes a summary β€” memory, reloads, and what the
patcher logged each minute β€” to `logs/soak.summary`; **session `perf`** (plan 6.3) runs
`perf/tap.python~.perf.maxpat` and averages Max's CPU-meter readings into `logs/perf.json`,
which `scripts/update-perf-docs.py --max` turns into the ReadMe's table.
6. Reads the results from the harness's SQLite database (in the installed `max-test`) and prints
7. Reads the results from the harness's SQLite database (in the installed `max-test`) and prints
them. Max's standard output goes to `runtime-tests/logs/` β€” only min's own lines appear there;
the console errors seen during a failing test are printed from its log instead.

Expand Down Expand Up @@ -97,13 +104,27 @@ first runs taught, built into the helpers:
rate is not like for like, so the perf patcher measures at the device's own rate.
- **A check that audio did not change passes vacuously if no audio ran**: pair each with a sample
check (the soak's `phase-N-…-output`).
- **`tap.python`'s output is checked as `[tosymbol]` writes it** (`Test.output_is()`): one symbol for
the whole message β€” `symbol C` (the word `symbol` kept), `1 0 1`, `note 60 100`, and floats with
four decimals, `0.5000` β€” held in a `[zl.reg]` until the check compares it with a quoted
`[sel "…"]`. A `bang` makes `[tosymbol]` repeat its last symbol, so a bang output is not checked
this way. An output that never came fails the check: the harness records an assertion that never
answers as a failure ("never returned results"). `Test.output_count()` and `Test.count_is()` check
how many messages an outlet sent β€” nothing, for `None`.
- **Text logged with spaces breaks the harness's SQL** (its log line goes between double quotes, and
Max quotes a symbol with a space): log codes, not text β€” `[atoi]`, as the probe that found the
formats above did.
- **Overdrive is `; max preempt 1`, Scheduler in Audio Interrupt `; dsp takeover 1`**; the threads
test reads both through `[adstatus]` first and puts them back at the end.

The fixtures' docstrings say what each is for. Open a patcher in Max to watch a test run; its
`test.*` objects only record anything when the harness opened it.

## Not covered here

Some of the runbook's checks still need a person: that a `bool` attribute shows as a toggle in the
`tap.python`'s once-per-session notice that Scheduler in Audio Interrupt runs Python on the audio
thread is a warning, which `[error]` does not see: the glue test checks it, and the threads test only
that the messages run there correctly. Some of the runbook's checks still need a person: that a `bool` attribute shows as a toggle in the
inspector, that an `attrui` displaying a removed attribute looks right (the tests check only that
nothing breaks), and whether the audio *device* drops out while a reload holds the GIL (the tests
check every sample the object outputs, not the driver). Changing the sample rate in Audio Status is
Expand Down
Loading
Loading