Skip to content

pir: update to pir-engine 3.10.0 - #234

Open
Qazi-pk wants to merge 5 commits into
cavalab:masterfrom
Qazi-pk:pir-3.6.0
Open

Qazi-pk wants to merge 5 commits into
cavalab:masterfrom
Qazi-pk:pir-3.6.0

Conversation

@Qazi-pk

@Qazi-pk Qazi-pk commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

This updates the pir method to pir-engine 3.6.0, which adds optional fallback "gates" that run after the classical engine. The engine itself and its configuration are unchanged.

Changes

  • algorithms/pir/install.sh: install tag v3.6.0 (was v3.5).
  • experiment/methods/pir/regressor.py: pass gates="all" to the engine. gates is an explicit constructor argument, so it is recorded by get_params().
  • algorithms/pir/metadata.yml: describe the gates, add the software DOI, and remove an outdated benchmark figure from the description.

What the gates do (tried in this order, only when the engine's model is not already exact)

  • rational P(x)/Q(x): accepted only if it reproduces the data exactly (max relative error < 1e-6) and beats the engine
  • exponential C*exp(sum of monomials): same exactness rule
  • noise-tolerant power law C*prod x_j**(p/q): exponents snapped to simple fractions; accepted only if its training MSE is no worse than the engine's

Any error inside a gate keeps the engine's model. All thresholds are fixed in the engine.

Checks

  • ruff check, ruff format --check and scripts/typecheck.py pass locally.
  • Release: https://github.com/Qazi-pk/physics-engine/releases/tag/v3.6.0 (DOI 10.5281/zenodo.23460311), with tests and CI.
  • I ran the 119 Feynman datasets with exactly this configuration (fitted with SRBench's evaluate_model.py, scored with assess_symbolic_model.py from current master 2aabbb6, seeds 23654 and 78901). These are my own runs, not official results:
target noise v3.5 3.6.0 (23654 / 78901)
0 35 / 35 50 / 50
0.001 35 / 35 36 / 36
0.01 30 / 29 35 / 35
0.1 2 / 2 23 / 22

(symbolic solution counts out of 119; no dataset solved by v3.5 was lost at any level)

Happy to adjust anything.

🤖 Generated with Claude Code

https://claude.ai/code/session_019mo31AVugYdfXietM73iWZ

- algorithms/pir/install.sh: install tag v3.6.0 (was v3.5).
- experiment/methods/pir/regressor.py: pass gates="all" to the engine;
  gates is an explicit constructor argument so get_params() records it.
  The vanilla engine configuration is unchanged.
- algorithms/pir/metadata.yml: describe the gates; add the software DOI;
  drop the outdated Tier A figure from the description.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019mo31AVugYdfXietM73iWZ
@Qazi-pk Qazi-pk changed the title pir: update to pir-engine 3.6.0 with built-in fallback gates pir: update to pir-engine 3.7.0 Oct 25, 2026
@Qazi-pk

Qazi-pk commented Oct 25, 2026

Copy link
Copy Markdown
Contributor Author

I've updated this PR from 3.6.0 to pir-engine 3.7.0 (released today, DOI 10.5281/zenodo.23631107), so that the next benchmark run uses the current version. One new commit:

  • install.sh: tag v3.7.0.
  • regressor.py: new explicit constructor argument safeguard=True (recorded by get_params()), passed to the engine with gates; docstring updated.
  • metadata.yml: one sentence on the safeguard; software DOI updated.

What 3.7.0 adds. After the engine and the gates, an optional safeguard compares the final formula with a ridge-regression linear model and the mean. A formula with training R² ≥ 0.999 is always kept. Otherwise the linear model replaces it only if its 5-fold cross-validated R² is higher by more than 0.05, and the mean is returned if both are below 0. The rules were fixed before any benchmark run. Also: a memory cap in the exponential gate (it ran out of memory on wide datasets), and safe symbol names for column names that aren't valid identifiers (e.g. inv-nodes in 1201_BNG_breastTumor crashed the harness's simplicity() parse).

Checks. ruff check, ruff format --check (0.17.0) and scripts/typecheck.py pass locally. My own runs with SRBench's evaluate_model.py and assess_symbolic_model.py (current master, fixed scorer), not official results:

Feynman-119, seeds 23654 / 78901 v3.5 3.6.0 3.7.0
noise 0 35 / 35 50 / 50 50 / 50
noise 0.001 35 / 35 36 / 36 36 / 36
noise 0.01 30 / 29 35 / 35 35 / 35
noise 0.1 2 / 2 23 / 22 23 / 22

No dataset solved by an earlier version is lost at any noise level. On the 120 black-box datasets I could download (one seed, no tuning), median test R² goes from −0.02 (3.6.0) to 0.44 (3.7.0), with no out-of-memory failures and no dataset worse.

No rush. Happy to adjust anything.

@Qazi-pk

Qazi-pk commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Small correction to my last comment: with three seeds (23654, 15795, 860) the black-box median test R² for 3.7.0 is 0.36–0.44 per seed, 0.38 as the median over seeds — about the level of linear regression in your published results, not above it. 3.6.0 on the same datasets: −0.02. The Feynman numbers are unchanged.

@Qazi-pk Qazi-pk changed the title pir: update to pir-engine 3.7.0 pir: update to pir-engine 3.8.0 Oct 31, 2026
@Qazi-pk

Qazi-pk commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

I've updated this PR to pir-engine 3.8.0 (released 31 Oct, DOI 10.5281/zenodo.23827328), so that the next benchmark run uses the current version. One new commit:

  • install.sh: tag v3.8.0.
  • regressor.py: new explicit constructor argument separability=True (recorded by get_params()), passed to the engine with gates and safeguard; docstring updated.
  • metadata.yml: one sentence on separability; software DOI updated.

What 3.8.0 adds. A separability stage that runs after the engine and before the gates, only when the engine's model is not exact. A deterministic cross-term test (quadratic fit of log|y| on the log-inputs) looks for a split of the inputs with y = g(A)·h(B); the parts are solved separately, with at most 4 attempts, each on a fixed 5,000-row subsample, and the product is accepted only if it reproduces all training rows exactly (max relative error < 1e-6). There is no wall-clock budget inside the stage, so results do not depend on machine speed. The rules were fixed before any benchmark run.

Checks. ruff check, ruff format --check (0.17.0), scripts/typecheck.py and scripts/check_method_layout.py pass locally. My own runs with SRBench's evaluate_model.py and assess_symbolic_model.py (current master, fixed scorer), not official results:

Feynman-119, seeds 23654 / 78901 v3.5 3.7.0 3.8.0
noise 0 35 / 35 50 / 50 58 / 58
noise 0.001 35 / 35 36 / 36 36 / 36
noise 0.01 30 / 29 35 / 35 35 / 35
noise 0.1 2 / 2 23 / 22 23 / 22

No dataset solved by an earlier version is lost at any noise level. The 8 additions at noise 0 are I.9.18, I.13.4, I.24.6, II.6.15a, II.11.3, II.11.27, II.37.1 and III.10.19. On the black-box datasets (seed 23654, no tuning), 3.8.0 gives the same results as 3.7.0: 120/120 results, median test R² 0.436 for both, no dataset better or worse by more than 0.01; separability did not change any final model there.

No rush. Happy to adjust anything.

@Qazi-pk Qazi-pk changed the title pir: update to pir-engine 3.8.0 pir: update to pir-engine 3.9.0 Nov 3, 2026
@Qazi-pk

Qazi-pk commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Updated to pir-engine 3.9.0 (DOI 10.5281/zenodo.23926418). The method's code and arguments are unchanged; only the pinned version moves.

3.9.0 extends the separability stage to targets that change sign and, in a late pass after the gates, to factors that are roots or sin/cos terms. A law is still accepted only if it is exact on all training rows. The rules were fixed in a commit before any benchmark run.

Feynman-119, two seeds (23654 / 78901), fixed scorer (#231):

noise 3.8.0 3.9.0
0 58 / 58 74 / 74
0.001 36 / 36 36 / 36
0.01 35 / 35 35 / 35
0.1 23 / 22 23 / 22

No law solved by 3.8.0 was lost. At noise 0, 16 laws were gained: 10 from the development half and 6 from the blind half, which was not looked at while the rules were made. Black-box (seed 23654): median test R² 0.436, the same as 3.8.0, and no dataset is worse by more than 0.01.

🤖 Generated with Claude Code

https://claude.ai/code/session_019mo31AVugYdfXietM73iWZ

@Qazi-pk Qazi-pk changed the title pir: update to pir-engine 3.9.0 pir: update to pir-engine 3.10.0 Nov 12, 2026
@Qazi-pk

Qazi-pk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Updated the pir method to pir-engine 3.10.0 (tag v3.10.0, DOI 10.5281/zenodo.24156912):

  • install.sh: @v3.10.0
  • regressor.py: new explicit argument noise_tolerant=True (recorded by get_params()), docstring updated
  • metadata.yml: description and DOI

What 3.10 adds: an opt-in noise-tolerant stage. It runs only when no model so far is exact and there are at least 2,000 training rows. It fits separable, power-law-times-exp and short rational forms on 75% of the rows, snaps exponents and coefficient ratios to simple fractions within 3 standard errors, and accepts a form only if it predicts the other 25% within 5% of a flexible reference model. The rules were fixed before benchmarking.

My own runs with this harness (Feynman-119, seeds 23654 / 78901, scored with the fixed scorer from #231), symbolic solutions 3.9.0 → 3.10.0:

  • noise 0: 74 → 75 / 75
  • noise 0.001: 36 → 73 / 74
  • noise 0.01: 35 → 73 / 74
  • noise 0.1: 23 / 22 → 55 / 55

No law lost. Black-box (120 datasets, seed 23654): same as 3.9.0, median test R² 0.436. These are my own runs, not official results.

Disclosure on output forms: where two forms are numerically equivalent, 3.10 writes the one the scorer can verify without assuming positive variables (for example k*x**2 rather than sqrt(k**2*x**4); see #235). Each rewritten form is checked numerically against the fitted model. If #235 is adopted, I will rescore; no refit is needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_019mo31AVugYdfXietM73iWZ

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant