sequential-testing

Runs ~20 options strategies against live market data in shadow mode, records every hypothetical fill under worst/base/optimistic assumptions, and grades each with anytime-valid e-processes. Places no orders.

Which model is really behind your API relay or agent IDE? Behavioral fingerprinting + anytime-valid sequential tests (FPR<=1%). LLMs can't be random - measured on 9 frontier models.

Sequential Hypothesis Testing with e-Values and p-Values

Sequential testing for efficient and reliable comparison of stochastic algorithms.

Fixed Sequence Multiple Testing Procedures

Walds Sequential Probability Ratio Testing (SPRT) statistical evaluator for LLM and AI agent comparison in Ruby.

Bayesian A/B Testing for Proportions

Measure your agent harness, find where it wastes the model, and prove the fix worked. Harness-agnostic, agent-agnostic, zero dependencies. Reference implementation of HTP-1.

A/B testing and causal inference scored against known answers: simulations where I set the effect, and a randomised benchmark the observational methods have to recover.

Sequential A/B Testing Framework in Python

Bayesian multi-armed bandits for continuous prompt experimentation: Thompson sampling routes traffic to the best prompt variant and a stopping rule promotes a winner without a fixed-N A/B test. Zero dependencies, TypeScript-first.

A/B testing engine: frequentist + Bayesian + CUPED variance reduction + peeking-safe sequential testing

A/B test analysis that is correct by default, informative by design, and composable by construction. 88 classes, 60+ paper references.

Ships ML models in stages - shadow, then 1/5/25/50% of traffic - and rolls a bad one back automatically. The guardrails stay valid under constant checking: 0.6% false rollback with two identical models, where a repeatedly-read A/B test acts wrongly 36.7% of the time.

A solution for simulating realistic exoplanet transit light curves and training a classifier to detect exoplanet transits

Ad campaign performance evaluation using AB Testing

A/B testing toolkit: power analysis, SRM, CUPED, BH-FDR, and a Monte Carlo calibration harness that checks whether naive peeking rules actually control the false-positive rate they claim to.

Typed toolkit for A/B experiment analysis: power analysis, CUPED variance reduction, always-valid sequential testing (mSPRT), Welch/z analysis with CIs - property-based tests (hypothesis) and a worked forecasting-rollout case study.

A/B testing platform with frozen designs, deterministic assignment, health checks and rendered decision docs

A/B testing framework with Frequentist, Bayesian, and Sequential testing — includes a peeking problem simulator that visually proves why standard testing inflates false positive rates by 3-4x.