Implementation and task specifications for G1, G2, and G4.
A benchmark for evaluating mobile agents as everyday smartphone assistants.
MobileLife-Bench evaluates whether an agent can complete practical tasks on a real Android phone, use information across applications, and turn persistent memory into correct actions. It combines natural-language tasks, controlled device preparation, observable outcome checks, and execution traces to measure both task performance and where failures occur.
The benchmark is built on evaluation infrastructure developed alongside ClawMobile. MobileLife-Bench is the benchmark; ClawMobile is its reference agent runtime.
| Group | Capability | Representative tasks |
|---|---|---|
| G1 — Local phone state and data | Device settings, local files, contacts, and calendar operations | Change brightness, manage Downloads files, update a contact, or create an event. |
| G2 — App operations and workflows | Single-app tasks, cross-app workflows, and combined shell/GUI execution | Complete an app workflow and carry information between tools and applications. |
| G4 — Memory and personalization | Cross-session retention, retrieval, and resolution of evidence from multiple sources | Recall a user preference, combine complementary facts, or resolve conflicting historical evidence before acting. |
These groups describe capabilities rather than increasing difficulty. Self-improvement and skill learning are extension directions; they are not presented as a completed evaluation suite.
The reference setup runs the agent's control loop and mobile tools on the phone. Model inference may run on a server or through a cloud API; a phone-side runtime does not require phone-local inference.
Host evaluation controller
-> phone-side ClawMobile / OpenClaw runtime
-> agent tools
-> Android device and applications
Host fixture and verification harness
-> deterministic preparation, state inspection, and cleanup
Each task follows a defined lifecycle:
Preflight -> Setup -> Agent execution -> Outcome checks -> Teardown
The evaluated agent receives the task instruction and the tools allowed by the experimental condition. The host separately manages fixtures and verification. In comparisons with and without agent shell access, the host may still use ADB for preparation and checking; the experimental distinction concerns the agent's available tools and context.
A task definition specifies its instruction, parameter variants, initial state, expected outcome, relevant side-effect constraints, and cleanup procedure. A task template, a parameterized case, and an execution attempt are separate units.
Verification prioritizes observable state over an agent's claim that it has finished. Depending on the task, evidence can include Android settings, provider records, file contents and hashes, application UI state, or a controlled external receiver. The available oracle and its limitations are part of the task protocol.
Scoring is group-specific. G1 distinguishes positive goals, invariants, and prohibited side effects. G2 uses positive goal checks under its current evaluation contract. G4 combines the applicable execution and memory-provenance requirements. Infrastructure failures and unverified outcomes are recorded separately from ordinary task failures.
G4 studies three evidence sources:
- User: facts, preferences, or constraints expressed by the user.
- Agent: evidence from earlier agent actions or decisions.
- Environment: evidence observed from the phone or application state.
The G4 task specifications define 56 core tasks (19 SI, 14 CP, and 23 CF) spanning Clock, Calendar, Documents, Settings, Photos, YouTube, and Maps. It includes single-source, complementary-source, and conflict-resolution tasks.
A core task includes source-capture sessions, two intervening distractor turns, and a fresh probe session:
Capture -> Distractor 1 -> Distractor 2 -> Probe
Each core task contributes one task-level result. Individual capture and distractor turns are not counted as additional benchmark tasks. Memory conditions include full-memory evaluation and diagnostic controls such as empty memory and source-specific ablations. The persistent-memory integration uses MemOS.
Memory analysis distinguishes whether a necessary fact was captured, persisted, retrieved, delivered to the probe, and used in the final action. Successful operation alone does not establish that memory caused the success.
The framework records task outcomes, action steps, tool calls, latency, token usage, and execution traces. Timing analysis separates session startup, model requests, observation, UI actions, deterministic tools, other tools, and runtime overhead. Reports must state their aggregation and timeout policy.
Reproducible results should identify:
- The task catalog, checker revision, and parameterized task set.
- The model, runtime, tool policy, memory backend, and experimental condition.
- Device and app versions, timeouts, and environment qualification.
- The selected execution attempt, its outcome, and supporting traces.
- Missing or invalid cases, rerun selection rules, and metric coverage.
Different task or oracle revisions must remain identifiable. A collection of the latest available records can contain mixed revisions and should not be described as a single matched experiment.
src/ Benchmark runners, checkers, harnesses, and analysis
task_specs/ G1, G2, and G4 task definitions
tests/ Offline regression tests
test_data/ Test fixtures
config/ Tool category mappings
docs/ Protocols and execution guides
lib/ClawMobile/ Validated reference runtime source and plugins
lib/MemOS/ MemOS submodule (v2.0.36)
G1 contains 36 templates and 112 parameter instances. G2 contains 60 CN and
85 INTL instances, matching the selected final result records. G4 contains
56 instances. The Python package and project-facing runtime names use
mobilelife-bench; inside OpenClaw, the integration is exposed as the generic
benchmark channel (channels.benchmark).
git submodule update --init --recursive
conda create -n mobilelife-bench python=3.12 -y
conda activate mobilelife-bench
python -m pip install -e '.[dev,report]'
python -m pytest -p no:cacheprovider -qAll commands below run from this repository root. Local results, generated exports, credentials, and runtime caches are ignored by Git.
Configure the device and ordered model list in config.yaml, then use exactly
one entry point for each group:
python -m scripts.run_g1
python -m scripts.run_g2
python -m scripts.run_g4Each entry point performs a basic host/device preflight, configures the required
host routes and services, starts the phone agent loop, waits for its health
endpoint, runs every configured model sequentially, and collects results. G1
and G2 rerun only INFRA_FAILURE cases up to three times. G4 applies the same
limit inside its phase-aware runner. Exhausted cases are written to a
manual_intervention.json report with their last recorded reason.
Use --dry-run to inspect the model, phone-start, and benchmark commands
without changing the device. Additional unknown arguments are forwarded to the
underlying group runner; for example:
python -m scripts.run_g1 --model deepseek/deepseek-flash --dry-run
python -m scripts.run_g2 --model deepseek/deepseek-flash --task-ids CN/A-1
python -m scripts.run_g4 --model deepseek/deepseek-flash --task-ids SI-CLK-UThe phone control bridge expects Termux to be at an idle shell prompt the first
time it is started. G4 also expects the generated runtime bundle under
exports/ and a valid MemOS Docker configuration.
Start the configured ClawMobile service on the phone with clawmobile run.
The host needs ADB access and the phone agent needs working mobile tools.
export DEVICE_SERIAL='<adb-device-serial>'
export MODEL_LABEL='deepseek/deepseek-flash'
adb -s "$DEVICE_SERIAL" forward tcp:18765 tcp:8765
export MOBILELIFE_BENCH_AGENT_BASE_URL=http://127.0.0.1:18765
bash scripts/check_environment.sh --help
# G1: the currently configured phone model, without a model matrix
python -m scripts.run_task_set --task-group g1 \
--device-serial "$DEVICE_SERIAL" --model-label "$MODEL_LABEL"
# G2: CN and INTL final task sets
python -m scripts.run_task_set --task-group g2 --version CN INTL \
--device-serial "$DEVICE_SERIAL" --model-label "$MODEL_LABEL"--model-label records the configured phone model; configure that model on
the phone before running. Use --task-ids to run a subset. See
the execution guide for environment checks and recovery.
MemOS runs on the host; its lifecycle plugin and the agent run on the phone.
Initialize lib/MemOS, configure its .env according to its documentation,
and start the service from lib/MemOS/docker with docker compose up -d.
The benchmark expects a reachable MemOS API, normally at port 8000.
Build the phone runtime bundle on the host:
npm ci --prefix lib/ClawMobile/openclaw-plugin-mobile-ui
python -m scripts.build_l8_runtime_bundle
# Route the phone runtime to the host MemOS service and ADB server
adb -s "$DEVICE_SERIAL" reverse tcp:18000 tcp:8000
adb -s "$DEVICE_SERIAL" reverse tcp:15037 tcp:5037This generates exports/MobileLife-Bench-L8-runtime-current/ and its ZIP/checksum.
Copy the generated directory to exports/ in the phone's MobileLife-Bench
checkout, then run in Termux:
sh scripts/android/run_g4_memos.shThe phone must already have ClawMobile installed and a model configured.
The wrapper stages the generated runtime and starts it; run it after stopping
an existing foreground ClawMobile service. See
the G4 protocol and
python -m scripts.run_g4_memory_v2 --help for host evaluation options.
lib/ClawMobile preserves the tested ClawMobile-Dev runtime, including the
mobile UI and MemOS plugins. It is vendored source; MemOS is a submodule.
Runtime provenance and
migration provenance record the source revisions.
MOBILELIFE_CLAWMOBILE_ROOT can select another compatible runtime checkout.