Skip to content

About

ProtoMotions G1 BONES checkpoint adapter for reproducible MotionChallenge evaluation.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

43 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ProtoMotions G1

GPU batch evaluation of the released ProtoMotions G1 checkpoint on LAFAN and BONES-SEED.

Run

git clone --recurse-submodules https://github.com/MotionChallenge-Project/MotionChallenge-protomotions.git
cd MotionChallenge-protomotions
uv sync --locked
uv run evaluate.py --dataset lafan --data /path/to/lafan --device cuda:0 --output /path/to/results/protomotions-lafan
uv run evaluate.py --dataset seed --data /path/to/g1.tar.gz --device cuda:0 --output /path/to/results/protomotions-seed

Download original LAFAN or BONES-SEED data and accept the upstream terms. The LAFAN directory contains g1/*.csv. The SEED archive is selected by Core's single public inclusion list. Motion data stays local.

The command downloads and verifies the original model weights, converts them to safetensors plus a PyTorch execution graph, evaluates the complete selected dataset, and independently verifies the recorded trajectories. Use a fresh output directory for each run.

uv run evaluate.py --dataset lafan --data /path/to/lafan --weights /path/to/original-checkpoint --offline --device cuda:0 --output /path/to/results/protomotions-lafan

LAFAN uses 40 simulation slots and SEED uses 4096. Each control step runs one full-batch network forward. Finished slots are refilled in longest-motion-first order. --slots selects a different capacity. --cache sets the local preparation and model conversion cache.

BONES-SEED evaluation uses a feasibility-filtered subset of the original G1 motion data. Download the original archive; Core applies one public inclusion list of 129,561 motions shared by every model.

Evaluation contract

The control period is 20 ms with four 5 ms physics substeps. Sliding friction is 1.0 and the Newton solver uses at most 50 iterations. Core precomputes scoring reference FK before rollout and reads actual body poses from MJWarp.

Policies receive joint positions and velocities, root orientation, angular velocity, and root/joint reference motion. Actual global position and actual linear velocity are unavailable. Model-specific reference features and policy history are maintained by this repository.

The original checkpoint SHA256 is a59baa3e04a951e5cf0b4cc68f24ebaafa9272714226618b99a5017dfc805b4c. runtime.mjwarp.json specifies its source, input configuration, observation contract and conversion entrypoint.

Validation

validation/mjwarp-network-parity.json records original ONNX versus PyTorch batch parity. validation/full-batch-parity.json records a 4096-sample forward and original ONNX row checks. validation/tensor-observation-parity.json records original observation/history/reset and PD-target parity.

Each evaluation writes result.json, trace files, input hashes and verification.json. Dataset scores require full motion coverage and independent trace verification.

validation/full-batch-adapter.json verifies 4096 active adapter states, a full forward and partial resets. validation/onnxruntime-parity.json checks outputs against the original ONNX in an independent ONNX Runtime environment. The unified network produces the final joint position, stiffness and damping targets.

Current evaluation evidence

LAFAN1 full evaluation · BONES-SEED full evaluation

Metric

MPKPE (Mean Per-Keypoint Position Error) measures the 3D Euclidean distance between actual and reference keypoint positions. The protocol defines these keypoints at robot body-frame origins: 14 points for G1. Errors are averaged over keypoints and scoring samples within each motion, then motions are weighted equally. Results report full-motion and first-failure-prefix scores with no alignment, XY alignment, and XY + yaw alignment. XY alignment removes pelvis horizontal translation while retaining height error. Leaderboard values are reported in millimetres (mm).

About

ProtoMotions G1 BONES checkpoint adapter for reproducible MotionChallenge evaluation.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages