Skip to content

CT-2322 Run Maestro E2E tests on Sauce Labs #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Jameson13B wants to merge 3 commits into from
Closed

CT-2322 Run Maestro E2E tests on Sauce Labs #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c90d1e9
CT-2322 Make the example app's proxy URL configurable at build time
Jameson13B Oct 21, 2026
8fc5142
CT-2322 Run the Maestro flows on Sauce Labs in GitHub Actions
Jameson13B Oct 21, 2026
a7e98f6
CT-2322 Harden the Sauce Labs workflow after review
Jameson13B Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
300 changes: 300 additions & 0 deletions .github/workflows/e2e-sauce-labs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,300 @@
name: E2E tests (Sauce Labs)

# Builds the example app as a release APK and runs the Maestro flows in
# example/maestro on a Sauce Labs device with maestro-runner. The SSO proxy
# runs on this runner, and the device reaches it through a Sauce Connect
# tunnel. See E2E_TESTING.md for the required secrets and for debugging.

on:
pull_request:
paths: &paths
- "src/**"
- "example/**"
- "package.json"
- "package-lock.json"
- "vite.config.build.ts"
- "tsconfig.json"
- "babel.config.js"
- ".github/workflows/e2e-sauce-labs.yml"
# Runs on the base branches warm the npm and Gradle caches that PR runs can
# read (PR-scoped caches aren't shared between PRs).
push:
branches: [v4, master]
paths: *paths
workflow_dispatch:
inputs:
target:
description: Sauce Labs device type
type: choice
options:
- android-emulator
- android-real-device
default: android-emulator

concurrency:
group: e2e-sauce-labs-${{ github.ref }}-${{ inputs.target || 'default' }}
cancel-in-progress: true

permissions:
contents: read

env:
# Sauce Connect can't proxy "localhost" to real devices, so the app talks to
# this alias, which the test job points at 127.0.0.1 in /etc/hosts.
PROXY_HOST: sdk-e2e-proxy
PROXY_PORT: "8089"

jobs:
check-secrets:
runs-on: ubuntu-latest
outputs:
run: ${{ steps.check.outputs.run }}
steps:
# PRs from forks and Dependabot don't get repository secrets, so skip
# the tests instead of failing them.
- id: check
env:
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
MX_CLIENT_ID: ${{ secrets.MX_CLIENT_ID }}
MX_API_KEY: ${{ secrets.MX_API_KEY }}
run: |
missing=()
for name in SAUCE_USERNAME SAUCE_ACCESS_KEY MX_CLIENT_ID MX_API_KEY; do
[ -n "${!name}" ] || missing+=("$name")
done
if [ ${#missing[@]} -gt 0 ]; then
echo "::notice::Skipping Sauce Labs E2E tests, missing secrets: ${missing[*]}"
echo "run=false" >> "$GITHUB_OUTPUT"
else
echo "run=true" >> "$GITHUB_OUTPUT"
fi

# Doesn't need secrets, so it runs even when the Sauce Labs tests are
# skipped and still catches build breaks.
build-android:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false

- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
cache-dependency-path: |
package-lock.json
example/package-lock.json

- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "17"

# ubuntu-latest already has the Android SDK (ANDROID_HOME). Gradle installs
# anything missing, like the NDK version React Native pins.
- uses: actions/cache@v4
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: gradle-${{ runner.os }}-${{ hashFiles('example/package-lock.json', 'example/app.json') }}
restore-keys: gradle-${{ runner.os }}-

- name: Install SDK dependencies
run: npm ci

- name: Build SDK
run: npm run build

- name: Install example app dependencies
working-directory: example
run: npm ci

- name: Build release APK
working-directory: example
env:
EXPO_PUBLIC_PROXY_BASE_URL: http://${{ env.PROXY_HOST }}:${{ env.PROXY_PORT }}
run: |
npx expo prebuild --platform android --no-install
cd android
# Sauce emulators are x86_64 and real devices arm64; skip the other ABIs.
./gradlew assembleRelease --no-daemon -PreactNativeArchitectures=x86_64,arm64-v8a

- uses: actions/upload-artifact@v4
with:
name: example-app-release-apk
path: example/android/app/build/outputs/apk/release/app-release.apk
retention-days: 7
if-no-files-found: error

test-android:
needs: [check-secrets, build-android]
if: needs.check-secrets.outputs.run == 'true'
runs-on: ubuntu-latest
timeout-minutes: 45

# The Sauce and MX secrets are passed only to the steps that use them.
env:
TARGET: ${{ inputs.target || 'android-emulator' }}
SAUCE_REGION: ${{ vars.SAUCE_REGION || 'us-west-1' }}
RUN_NAME: rn-sdk-e2e-${{ github.run_id }}-${{ github.run_attempt }}
TOOLS: example/sauce/node_modules/.bin

steps:
- uses: actions/checkout@v4
with:
persist-credentials: false

- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
cache-dependency-path: example/sauce/package-lock.json

# The SSO proxy and maestro-runner, pinned in example/sauce/package-lock.json.
# Neither needs install scripts.
- name: Install E2E tools
run: npm ci --ignore-scripts --prefix example/sauce

- uses: actions/download-artifact@v4
with:
name: example-app-release-apk
path: ${{ runner.temp }}/apk

- name: Start SSO proxy
env:
MX_CLIENT_ID: ${{ secrets.MX_CLIENT_ID }}
MX_API_KEY: ${{ secrets.MX_API_KEY }}
MX_API_HOST: ${{ vars.MX_API_HOST || 'https://int-api.mx.com' }}
MX_DEFAULT_USER_GUID: ${{ vars.MX_DEFAULT_USER_GUID || 'USR-e7353471-51d7-447f-b621-db255148a10c' }}
run: |
echo "127.0.0.1 $PROXY_HOST" | sudo tee -a /etc/hosts

# The proxy (1.3.4) exits on the first error from the MX API, so keep
# restarting it. </dev/null: if a setting is missing it prompts for
# it; exit instead of hanging.
nohup bash -c 'while :; do
"$TOOLS/mx-sso-proxy" run --port "$PROXY_PORT" </dev/null
echo "sso-proxy exited with $?, restarting"
sleep 1
done' >"$RUNNER_TEMP/sso-proxy.log" 2>&1 &

# Wait until the proxy can create a widget URL, which also checks the credentials.
for _ in $(seq 1 20); do
if curl -sf --max-time 20 -o /dev/null -X POST "http://$PROXY_HOST:$PROXY_PORT/user/widget_urls" \
-H "Content-Type: application/json" \
-d '{"widget_url":{"widget_type":"connect_widget","is_mobile_webview":true}}'; then
echo "SSO proxy is up"
exit 0
fi
sleep 3
done
echo "::error::SSO proxy didn't return a widget URL. Check MX_CLIENT_ID, MX_API_KEY and MX_API_HOST; the 'SSO proxy log' step shows the responses."
exit 1

- name: Start Sauce Connect tunnel
# v3; pinned because it receives the access key.
uses: saucelabs/sauce-connect-action@cb88b508c6f9ff4d84490093733315dbd55de022
with:
username: ${{ secrets.SAUCE_USERNAME }}
accessKey: ${{ secrets.SAUCE_ACCESS_KEY }}
region: ${{ env.SAUCE_REGION }}
tunnelName: ${{ env.RUN_NAME }}
# Only the proxy goes through the tunnel; the device reaches everything
# else (widgets, OAuth pages) directly.
tunnelDomains: ${{ env.PROXY_HOST }}
# The proxy alias resolves to 127.0.0.1, which Sauce Connect 5 blocks by default.
proxyLocalhost: allow

- name: Upload APK to Sauce Labs
id: upload
env:
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
run: |
curl -fsS -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \
-X POST "https://api.$SAUCE_REGION.saucelabs.com/v1/storage/upload" \
-F "payload=@$RUNNER_TEMP/apk/app-release.apk" \
-F "name=$RUN_NAME.apk" \
-o "$RUNNER_TEMP/upload.json"
file_id=$(jq -er '.item.id' "$RUNNER_TEMP/upload.json")
echo "file_id=$file_id" >> "$GITHUB_OUTPUT"

# Never put credentials in the caps: maestro-runner logs them, and they're printed here.
- name: Write Appium capabilities
run: |
jq --arg run "$RUN_NAME" \
'. + {"appium:app": ("storage:filename=" + $run + ".apk")} | ."sauce:options" += {build: $run, tunnelName: $run}' \
"example/sauce/$TARGET.json" > "$RUNNER_TEMP/caps.json"
cat "$RUNNER_TEMP/caps.json"

- name: Run Maestro flows on Sauce Labs
timeout-minutes: 30
env:
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
MAESTRO_RUNNER_NO_TELEMETRY: "1"
run: |
# Real devices share the account's concurrency with the mobile team, so use one.
parallel=2
if [ "$TARGET" = "android-real-device" ]; then
parallel=1
fi

"$TOOLS/maestro-runner" \
--driver appium \
--appium-url "https://$SAUCE_USERNAME:$SAUCE_ACCESS_KEY@ondemand.$SAUCE_REGION.saucelabs.com/wd/hub" \
--caps "$RUNNER_TEMP/caps.json" \
test --parallel "$parallel" --output maestro-report --flatten example/maestro/

# maestro-runner 1.1.29 writes the Appium URL, access key included, into
# maestro-runner.log. The report is a public artifact, so redact the key
# and refuse to upload if any copy is left.
- name: Redact credentials in the report
id: redact
if: always()
env:
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
run: |
set -euo pipefail
[ -d maestro-report ] || exit 0
# grep exits 1 when nothing matches, which is fine here.
grep -rlF --null -- "$SAUCE_ACCESS_KEY" maestro-report >"$RUNNER_TEMP/files-with-key" || [ $? -eq 1 ]
xargs -0 -r perl -pi -e 's/\Q$ENV{SAUCE_ACCESS_KEY}\E/***/g' <"$RUNNER_TEMP/files-with-key"

# Only exit 1 (no match) is clean; 0 (found) and 2 (error) are not.
rc=0
grep -rqF -- "$SAUCE_ACCESS_KEY" maestro-report || rc=$?
if [ "$rc" -ne 1 ]; then
echo "::error::Couldn't confirm the Sauce access key is gone from the report; not uploading it"
exit 1
fi

- name: Upload Maestro report
if: always() && steps.redact.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: maestro-report-${{ env.TARGET }}
path: maestro-report/
retention-days: 14
if-no-files-found: ignore

# Only request lines and status codes: the proxy's crash output includes the client credentials.
- name: SSO proxy log
if: failure()
run: |
grep -E '^(GET|POST|OPTIONS|HEAD) |Running server|restarting|status code [0-9]+' \
"$RUNNER_TEMP/sso-proxy.log" | tail -n 100 || true

- name: Delete APK from Sauce Labs
if: always() && steps.upload.outputs.file_id != ''
env:
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
FILE_ID: ${{ steps.upload.outputs.file_id }}
run: |
curl -fsS -o /dev/null -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" \
-X DELETE "https://api.$SAUCE_REGION.saucelabs.com/v1/storage/files/$FILE_ID" || true
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ TestRnApp_*
build
coverage
dist
.mx-sso-proxy-rc.yaml
# Local SSO proxy config (real credentials). The example file is committed.
.mx-sso-proxy-rc*
!.mx-sso-proxy-rc.example.yml
node_modules
npm-debug.log
84 changes: 84 additions & 0 deletions E2E_TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,87 @@ We use Maestro for E2E testing.
# Run from the example folder
npm run e2e
```

The example app talks to the SSO proxy started by `npm run dev:dependencies` (see
[DEVELOPMENT.md](DEVELOPMENT.md)). To point it somewhere else, set
`EXPO_PUBLIC_PROXY_BASE_URL` when you build or start the app, for example
`EXPO_PUBLIC_PROXY_BASE_URL=http://192.168.1.20:8089 npm run android:e2e`. The value is compiled into the
JS bundle, so after changing it for a release build, clear the caches (`./gradlew clean`, `npx expo start --clear`).

## CI (Sauce Labs)

[`.github/workflows/e2e-sauce-labs.yml`](.github/workflows/e2e-sauce-labs.yml) runs the same flows on a
Sauce Labs Android device. It runs on pull requests and pushes to `v4` and `master` that touch the SDK
(`src/`, its build config, the package files), `example/`, or the workflow itself.

What it does:

1. **`build-android`** builds the SDK and the example app as a release APK, with the JS bundled so no Metro
server is needed, and with `EXPO_PUBLIC_PROXY_BASE_URL=http://sdk-e2e-proxy:8089`. The example app allows
cleartext http (`usesCleartextTraffic` in `example/app.json`) so the release build can reach the proxy.
The APK is signed with the debug keystore; it's for testing only. This job needs no secrets, so it runs
even when the tests are skipped.
2. **`test-android`** starts the SSO proxy on the runner, with `sdk-e2e-proxy` pointed at `127.0.0.1` in
`/etc/hosts`.
3. It opens a [Sauce Connect](https://docs.saucelabs.com/secure-connections/sauce-connect-5/) tunnel that
carries only `sdk-e2e-proxy` traffic from the device to the runner. Sauce Connect can't proxy `localhost`
to real devices, which is why the alias is needed.
4. It uploads the APK to Sauce Labs App Storage and runs `example/maestro/` with
[maestro-runner](https://docs.saucelabs.com/basics/community-frameworks/maestro/), a community tool that
runs Maestro flows through Appium. Device capabilities are in `example/sauce/*.json`.
5. It uploads the HTML, JUnit and Allure report as the `maestro-report-<target>` artifact, with the Sauce
access key redacted, and deletes the APK from App Storage.

The SSO proxy and maestro-runner versions are pinned in `example/sauce/package.json` (and its lockfile). To
upgrade maestro-runner, change the version there and run `npm install` in `example/sauce`.

### Running it manually

The workflow can be started by hand, which is also the only way to run it on a real device. GitHub only
offers this once the workflow file is on the default branch (`master`). From then on, use **Actions > E2E
tests (Sauce Labs) > Run workflow**, or:

```
gh workflow run e2e-sauce-labs.yml --ref <branch> -f target=android-real-device
```

### Required configuration

Repository secrets (Settings > Secrets and variables > Actions):

| Secret | Value |
| ------------------ | ----------------------------------------------------------------------------------------------------- |
| `SAUCE_USERNAME` | Sauce Labs username. Prefer a service user over a personal account. |
| `SAUCE_ACCESS_KEY` | That user's access key |
| `MX_CLIENT_ID` | Client ID of the E2E test client (see [`.mx-sso-proxy-rc.example.yml`](.mx-sso-proxy-rc.example.yml)) |
| `MX_API_KEY` | API key of the same client |

If any of them is missing, the workflow still builds the APK but skips the Sauce Labs tests and passes, with
a notice saying which secrets are missing. That is also what happens on pull requests from forks and from
Dependabot, which don't get repository secrets.

Optional repository variables:

| Variable | Default | Notes |
| ---------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| `SAUCE_REGION` | `us-west-1` | The data center of the Sauce account: `us-west-1`, `eu-central-1` or `us-east-4`. Use the full name; `us` or `eu-central` won't work. |
| `MX_API_HOST` | `https://int-api.mx.com` | |
| `MX_DEFAULT_USER_GUID` | `USR-e7353471-51d7-447f-b621-db255148a10c` | The user the proxy creates widget URLs for |

### When it fails

- **Which flow failed, and why:** download the `maestro-report-<target>` artifact from the run and open the
HTML report, which has each flow's steps and screenshots. `maestro-runner.log` in the artifact has the
full run.
- **Video and device logs:** in Sauce Labs, open **Automated > Test Results** (emulators) or **Real
Devices**, and filter by the build `rn-sdk-e2e-<run id>-<attempt>`. maestro-runner reuses a session for
several flows, so a Sauce job can contain more than one flow; it's named after the first one. Use the
report's timestamps to find a flow in the video.
- **"SSO proxy didn't return a widget URL":** the proxy couldn't get a widget URL from the MX API. Usually
`MX_CLIENT_ID` / `MX_API_KEY` are wrong or rotated, or `MX_API_HOST` is wrong. The **SSO proxy log** step
shows the requests and status codes (the full log isn't printed because it can contain the credentials).
- **Every flow fails at its first widget screen:** the device probably couldn't reach the proxy. Check the
**SSO proxy log** step: if it has no `POST /user/widget_urls` lines from the test run, the requests never
reached the runner (tunnel, alias or cleartext problem).
- **The Maestro step fails before any flow runs:** Sauce couldn't start a session (no free concurrency, an
emulator that didn't boot, a Sauce outage). Re-run the failed jobs.
Loading
Loading