GPU batch evaluation of the released TeleopIT checkpoint on LAFAN and BONES-SEED.
git clone --recurse-submodules https://github.com/MotionChallenge-Project/MotionChallenge-teleopit.git
cd MotionChallenge-teleopit
uv sync --locked
uv run evaluate.py --dataset lafan --data /path/to/lafan --device cuda:0 --output /path/to/results/teleopit-lafan
uv run evaluate.py --dataset seed --data /path/to/g1.tar.gz --device cuda:0 --output /path/to/results/teleopit-seedDownload original LAFAN or BONES-SEED data and accept the upstream terms. The LAFAN directory contains g1/*.csv. The SEED archive is selected by Core's single public inclusion list. Motion data stays local.
The command downloads and verifies the original model weights, converts them to safetensors plus a PyTorch execution graph, evaluates the complete selected dataset, and independently verifies the recorded trajectories. Use a fresh output directory for each run.
uv run evaluate.py --dataset lafan --data /path/to/lafan --weights /path/to/original-checkpoint --offline --device cuda:0 --output /path/to/results/teleopit-lafanLAFAN uses 40 simulation slots and SEED uses 4096. Each control step runs one full-batch network forward. Finished slots are refilled in longest-motion-first order. --slots selects a different capacity. --cache sets the local preparation and model conversion cache.
BONES-SEED evaluation uses a feasibility-filtered subset of the original G1 motion data. Download the original archive; Core applies one public inclusion list of 129,561 motions shared by every model.
The control period is 20 ms with four 5 ms physics substeps. Sliding friction is 1.0 and the Newton solver uses at most 50 iterations. Core precomputes scoring reference FK before rollout and reads actual body poses from MJWarp.
Policies receive joint positions and velocities, root orientation, angular velocity, and root/joint reference motion. Actual global position and actual linear velocity are unavailable. Model-specific reference features and policy history are maintained by this repository.
The original checkpoint SHA256 is 1ebd341d9193e1c49a986450f6043ba1a9473ad46636ce0bcb1c7755c856e0de. runtime.mjwarp.json specifies its source, input configuration, observation contract and conversion entrypoint.
validation/mjwarp-network-parity.json records original ONNX versus PyTorch batch parity. validation/full-batch-parity.json records a 4096-sample forward and original ONNX row checks. validation/tensor-observation-parity.json records original observation/history/reset and PD-target parity.
Each evaluation writes result.json, trace files, input hashes and verification.json. Dataset scores require full motion coverage and independent trace verification.
validation/full-batch-adapter.json verifies 4096 active adapter states, a full forward and partial resets. validation/onnxruntime-parity.json checks the original ONNX in an independent ONNX Runtime environment.
LAFAN1 full evaluation · BONES-SEED full evaluation
MPKPE (Mean Per-Keypoint Position Error) measures the 3D Euclidean distance between actual and reference keypoint positions. The protocol defines these keypoints at robot body-frame origins: 14 points for G1. Errors are averaged over keypoints and scoring samples within each motion, then motions are weighted equally. Results report full-motion and first-failure-prefix scores with no alignment, XY alignment, and XY + yaw alignment. XY alignment removes pelvis horizontal translation while retaining height error. Leaderboard values are reported in millimetres (mm).