Skip to content

Fix MiniWoB reward parsing: int() crashes on fractional reward strings - #390

Open
aashish254 wants to merge 2 commits into
google-research:mainfrom
aashish254:fix/miniwob-float-reward
Open

aashish254 wants to merge 2 commits into
google-research:mainfrom
aashish254:fix/miniwob-float-reward

Conversation

@aashish254

Copy link
Copy Markdown

What changed

miniwob_base.get_episode_reward() did float(int(reward)), but the MiniWoB app reports partial-credit rewards as float strings — e.g. MiniWobBisectAngle returns "0.8680475871297262". int() raises ValueError: invalid literal for int() with base 10 on any decimal string, so the exception propagates through is_episode_terminated() → episode_runner.run_episode() → suite_utils._run_task(), which logs "SKIPPING MiniWobBisectAngle" and discards the entire episode. The agent had actually earned a reward; the task just crashes on it. Decimal success strings like "1.0" crash identically.

The function's signature already declares -> float, so the fix is to parse as float(reward) directly. Downstream behavior is unchanged: is_episode_terminated() compares != 0.0 and is_successful() compares == 1.0, both of which behave exactly as before for integral rewards and now also handle partial credit correctly (terminated but not successful).

This is the error reported verbatim in #326.

Tests

Added test_get_episode_reward_fractional (the exact string from #326) and test_get_episode_reward_decimal_string ("1.0"), alongside the existing integral-reward tests.

How verified

  • Mutation check: both new tests fail on pristine main with ValueError: invalid literal for int() with base 10: '0.8680475871297262' / '1.0' at miniwob_base.py:59; pass with the fix.
  • pytest android_world/task_evals/miniwob/ — 9 passed.
  • Full suite — 514 passed; the 2 failures (interface_test timeout flake, receipt_generator_test) fail identically on a pristine base worktree without this change and are unrelated.
  • Only open PR touching these files is check home button success in emulator reset #359 (copyright header, line 1); git merge-tree with its head is clean.

get_episode_reward wrapped the reward in int(), which raises ValueError
for fractional reward strings such as "0.8680475871297262" returned by
partial-credit tasks (e.g. MiniWobBisectAngle), aborting the whole task
in suite_utils. Parse as float, which the function already declares it
returns.
… marker, not just data=

- Updated extract_broadcast_data to use .find() on full marker
- This prevents truncation when earlier lines contain 'data=uri'
- Added test_preamble_with_data_uri_in_intent_not_truncated regression test
- Mutation check: new test fails on base (truncates), passes with fix

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant