[FFroehlich]

Fixes #77

Change

Only subspace_solver='scg' is affected. The Steihaug CG residual tolerance changes from min(0.5, sqrt(|sg|)) * |sg| to min(0.5, sqrt(|sg| / |sg0|)) * |sg|. Here sg0 is the scaled gradient at the first iteration. This is option 2 in the issue.

• Optimizer.minimize stores |sg0| at iteration 1 as self.sgnorm0. It passes the value through a new optional sgnorm0 argument of trust_region() to TRStepSteihaug.
• CGStep / TRStepSteihaug take an optional sgnorm0. Without it, the step counts as the first step, so eta = 0.5.
• The norm is taken with scipy.linalg.norm, which does not overflow for the huge gradients covered by test_huge_gradient_and_hessian.

Why option 2

I measured options 1, 2 and 3 on the reproducer, on the 300-problem fuzz set from #75 and on larger-n problems.

• It is the only option that removes all FTOL-above-optimum cases in the fuzz set. It also has the fewest false convergences and the fewest stalls on quadratics.
• Early steps stay cheap. eta starts at 0.5, which is what the old term gives for |g| >= 0.25, so the first steps still take one CG iteration. It tightens only as |sg| falls relative to |sg0|, and it still goes to zero like sqrt(|sg|), so the asymptotic rate stays superlinear.
• It is invariant to the scale of f, because eta depends only on |g| / |g0|.
• It costs no extra CG iterations where the old term already worked. On the runs that converge with both main and this PR, CG iterations stay the same (-1%) and outer iterations drop by 23%.
• Option 1 with eta = 1e-2 leaves 5 of the issue's cases. With eta = 1e-6 it fixes them but costs +17% CG iterations on those same runs (median ×1.30), and it is a blanket tighter tolerance.
• Option 3 (at least 2 CG iterations) leaves 20 of the issue's cases and loses the optimum in 29 runs.
• I also tried the running maximum of |sg| as the reference, to guard against a tiny |sg0| when x0 sits next to a bound. It gave identical results on all 3600 fuzz runs, and |sg| never rose above 1.8 |sg0|. I kept the simpler fixed reference.

Reproducer from the issue

	main	this PR
default options	FTOL, 6 it, f = 46.4256 (+8.4e-3)	FTOL, 10 it, f = 46.0371115 (−3e-9 vs. bvls)
fatol=gatol=0, f ×1e-4 / ×1 / ×1e4	FTOL, 6 it, gap 7.2e-6 / 8.4e-3 / 8.4e-3	FTOL, 7 / 10 / 10 it, gap ≤ 3.2e-9
48 starts (corners, 1e-10 from bounds, random), f ×1	45/48 end above the optimum (28 FTOL, 17 DELTA_TOO_SMALL)	0/48 above the optimum (27 FTOL, 10 GTOL, 11 DELTA_TOO_SMALL at the optimum)
same, f ×1e4	45/48 above the optimum	0/48

Other options on the reproducer with default options: eta = 1e-2 gives FTOL after 9 iterations at a gap of 4e-8. eta = 1e-6 and option 3 both give GTOL after 12 iterations.

Fuzz set

300 problems from #75 (bounded quadratics and Rosenbrock, n ≤ 12), each run with scg × 4 step-back strategies × exact/BFGS/SR1, for 3600 runs per variant. "Optimum" means f − fref ≤ 1e-6 + 1e-4 |fref|.

	main	opt 1, η=0.1	opt 1, η=1e-2	opt 1, η=1e-6	opt 2 (this PR)	opt 3, ≥2 CG it
FTOL above an optimum that 2D and full reach, reflect, exact/BFGS/SR1	6/17/11	6/10/6	2/2/1	0/0/1	0/0/0	4/11/5
same, all step-back strategies	125	83	25	3	0	74
false convergence (any flag > 0 above the optimum)	306	253	185	133	106	185
quadratics not converged (flag ≤ 0)	149	133	125	57	53	124
runs reaching the optimum	3126	3204	3277	3326	3327	3197
runs that converge with both: outer iterations vs. main		−19%	−22%	−25%	−23%	−4%
runs that converge with both: CG iterations vs. main		−6%	+2%	+17%	−1%	+3%

By problem type, this PR changes:

• Quadratics: false convergence 66 → 0, runs reaching the optimum 2619 → 2759 of 2796.
• Rosenbrock: false convergence 240 → 106, runs reaching the optimum 507 → 568 of 804.

Total CG iterations over all 3600 runs go from 455k to 880k. Almost all of the increase comes from the 306 runs where main stops with a false convergence:

• 131 of them now reach the optimum.
• 93 still stop early. All of these are Rosenbrock.
• 82 now run to MAXITER instead of stopping with FTOL. These are Rosenbrock problems in ±1e10 or mixed-scale boxes, where 2D and full mostly fail as well. Together they add +246k CG iterations.

8 runs reach the optimum on main but not here, against 209 gained. Seed 164 (quadratic, BFGS, all 4 step-backs) gets caught in the bound crawl of #78, with steps of ~3e-5 next to two bounds. Seed 213 is a Rosenbrock problem in a ±1e10 box.

Larger n

Exact Hessian (except the last row), scg/reflect, default options. Gap is relative to the best value over all variants. Each cell shows outer iterations / CG iterations (gap).

problem	main	this PR	opt 1, η=1e-2	opt 1, η=1e-6	opt 3
bounded logistic regression, n=300, m=2000	39 / 9537 (1e-8)	27 / 10736 (4e-9)	25 / 11060	23 / 12743	38 / 10716
same, f ×1e-3	19 / 5479	18 / 5192	17 / 6201	15 / 7864	19 / 5493
quadratic n=500, cond 1e3	22 / 619	16 / 1624	16 / 1313	16 / 3164	23 / 624
quadratic n=500, cond 1e6	77 / 1589 (3.9e-7)	85 / 10617 (5e-10)	78 / 12987	75 / 33357	89 / 1818 (4.6e-7)
same, f ×1e4	83 / 1597 (5.3e-7)	87 / 10938 (0)	90 / 13884	89 / 40440	88 / 1985 (1.4e-7)
same, f ×1e-4	84 / 4032 (0)	98 / 11895 (5e-9)	81 / 12543	85 / 36719	79 / 3615
quadratic n=1000, cond 1e4	36 / 971	31 / 2395	31 / 2788	29 / 8325	37 / 948
bounded Rosenbrock n=1000	43 / 102	23 / 120 (other local min, +9e-4)	28 / 139	23 / 188	40 / 133
bounded Rosenbrock n=200, BFGS	224 / 453	262 / 566	212 / 480	212 / 480	196 / 425

Where this PR costs more. The ill-conditioned n=500 quadratic needs 6.7× the CG iterations. Main stops there with FTOL at a gap of 4e-7 to 5e-7, while this PR converges to below 1e-9. The last 4 steps take about 4000 CG iterations, capped at 2n+10. Option 1 costs as much or more here. Option 3 stays cheap but also stops early.

Cost of a CG iteration in fides. Measured with single-threaded BLAS, taking the best of 200 runs:

n	one Hessian-vector product	one scg iteration of minimize
500	0.015 ms	14.9 ms (≈1000 products)
1000	0.049 ms	80.4 ms (≈1650 products)

Building the dense scaled Hessian and the diagonal scaling matrices for every candidate step dominates the per-iteration cost. So the worst case above, about +9000 CG iterations at n=500, costs roughly as much as 9 outer iterations, or 0.14 s. Elsewhere the reduction in outer iterations outweighs the extra CG iterations. I'm not reporting whole-run CPU times, because they were too noisy on the loaded machine. For example, two runs with the same iteration count differed by 7×.

Behaviour change

scg users will see different iteration counts and results. 2D and full are unchanged. The new sgnorm0 arguments of trust_region(), CGStep and TRStepSteihaug are optional.

Tests

• test_minimize.py::test_steihaug_ill_conditioned_no_premature_ftol[1e-4, 1, 1e4] uses the issue's reproducer at three scales of f. It fails on main (FTOL at f = 46.4256 for scale 1) and passes here.
• test_steps.py::test_steihaug_tolerance_invariant_to_objective_scale checks that the CG step is identical for f scaled by 2^-40, 1 and 2^40. On main the step is the Newton step for 2^-40 and a steepest-descent step for 1 and 2^40.
• pytest tests reports 1206 passed and 3 skipped. The skips are the existing scg + refine combination.

Not addressed here: DELTA_TOO_SMALL at the optimum, which 11 of 48 starts on the reproducer hit regardless of the forcing term; and the bound crawl in seed 164 (#78).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.16%. Comparing base (a1ece4d) to head (d72cca7).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #89      +/-   ##
==========================================
+ Coverage   96.98%   97.16%   +0.18%     
==========================================
  Files          10       10              
  Lines        1159     1445     +286     
==========================================
+ Hits         1124     1404     +280     
- Misses         35       41       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.