ci(flutter): add lint, build and BrowserStack UI tests by teodorciuraru · Pull Request #173 · getditto/quickstart

Teodor Ciuraru (teodorciuraru) · 2025-09-24T12:49:37Z

Summary

Adds comprehensive CI/CD pipeline for Flutter app with lint, multi-platform builds, and BrowserStack integration testing for Android and iOS.

Key Changes

CI Workflow

Dedicated Flutter CI workflow with parallel jobs for faster execution
Lint & Analysis: Code formatting checks and static analysis
Multi-platform Builds: Android (debug), iOS (release), and Web
BrowserStack Integration Tests: Automated UI tests on real devices
- Android: Google Pixel 7 (Android 13.0)
- iOS: iPhone 13/14/12 (iOS 15/16/17)

Build Configuration

Android: Upgraded to AGP 8.7.3 and Kotlin 2.1.0 for Flutter compatibility
Gradle Caching: ~30-50% faster Android builds with dependency caching
Debug Builds for Testing: Android uses debug builds for BrowserStack (release builds don't support assembleAndroidTest)

Test Infrastructure

Ditto Cloud Test Seeding: Seeds test document with inverted timestamp for each CI run
Integration Test: Verifies app syncs test document from Ditto Cloud
Required Environment Variable: TASK_TO_FIND must be set via --dart-define, throws error if missing

Decisions & Trade-offs

Web build only (no BrowserStack tests): BrowserStack doesn't support Flutter Web integration tests
iOS may have occasional device flakes: BrowserStack infrastructure issues on specific device/OS combinations (typically 67% pass rate)
Removed pr-checks.yml Flutter job: Each SDK now has dedicated workflow

Test Plan

✅ Local builds: Android, iOS, Web all build successfully
✅ Lint and analyze pass
✅ BrowserStack Android tests pass
✅ BrowserStack iOS tests pass (2/3 devices, 1 infrastructure flake)
✅ Test seeding works correctly with inverted timestamps

github-actions · 2025-09-29T13:13:07Z

🤖 Flutter BrowserStack Test Results

❌ Tests failed

Test Document: 8240848432_flutter_ios_ci_test_18098115621_5
Android Build ID: N/A
iOS Build ID: N/A

github-actions · 2025-09-29T13:20:13Z

🤖 Flutter BrowserStack Test Results

❌ Tests failed

Test Document: 8240848001_flutter_ios_ci_test_18098312094_6
Android Build ID: N/A
iOS Build ID: N/A

github-actions · 2025-09-29T13:36:38Z

🤖 Flutter BrowserStack Test Results

❌ Tests failed

Test Document: 8240847016_flutter_ios_ci_test_18098741121_7
Android Build ID: N/A
iOS Build ID: N/A

github-actions · 2025-09-29T15:31:38Z

🤖 Flutter BrowserStack Test Results

❌ Tests failed

Test Document: 8240840201_flutter_ios_ci_test_18102069401_9
Android Build ID: N/A
iOS Build ID: N/A

github-actions · 2025-09-29T16:09:49Z

🤖 Flutter BrowserStack Test Results

❌ Tests failed

Test Document: 8240838479_flutter_ios_ci_test_18102857898_11
Android Build ID: N/A
iOS Build ID: 47f0cbe3d4ab393add0135e0adc9de2908d1f2f5
iOS Dashboard: View Results

github-actions · 2025-09-29T16:38:38Z

🤖 Flutter BrowserStack Test Results

❌ Tests failed

Test Document: undefined
Android Build ID: N/A
iOS Build ID: N/A

Cameron Mcloughlin (cameron1024)

Looking good, thanks for getting this over the line 🙏

I think for the actual test script, we can take two paths:

be satisfied with not-well-reviewed AI-generated scripts
treat them like production code and thoroughly review them

Personally, I'm OK with the first option, given that it's
for testing (as long as we're sure that a failing test does
actually result in a test failure). If you'd like a more
thorough review though, I think splitting them out and
getting shellcheck involved would be a good first step.
Up to you though

- Create flutter-ci.yml with lint, build (iOS/Android), and test jobs - Remove Flutter job from pr-checks.yml workflow - Format Flutter code to pass linting checks - Build jobs run in parallel after lint completes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Add comprehensive integration test suite for Flutter app - Configure BrowserStack for cloud device testing - Add CI/CD test that verifies document sync via environment variables - Fix permission handling for Android tests with INTEGRATION_TEST_MODE flag - Add mounted checks to prevent setState after dispose errors - Update dependencies including Ditto SDK to 4.12.1 - Configure network security for Android cleartext traffic The integration tests verify that the app can: 1. Load and display initial UI correctly 2. Add, toggle, and clear tasks 3. Sync data with Ditto cloud 4. Find specific documents for CI/CD verification Tests work on both iOS and Android platforms, with Android requiring the INTEGRATION_TEST_MODE=true flag to skip permission dialogs. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

…Ditto instance - Fixed multiple Ditto instance issue by initializing app only once - Simplified test suite to single comprehensive test - Added environment variable support to skip permissions in test mode - Documented working BrowserStack configuration in BS_WORKING.md - Verified tests can both pass (correct task) and fail (incorrect task) Test results: - PASS: bs://572570ea0e83853297ea3484084add8fa0c3134b - FAIL: bs://19a000afac63bc5be6d0799a2cdcc5ce7e22f8fa (with _bollocks suffix) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Added TASK_TO_FIND environment variable support to integration tests - Task name can now be configured at build time (defaults to GitHub test task) - Updated BS_WORKING.md with encoding instructions for custom task names - Verified both pass and fail scenarios with different task names 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Added RunnerTests.m for iOS test runner - Created BS_WORKING_iOS.md documentation - Configured Podfile for RunnerTests target - Same environment variables work for iOS (INTEGRATION_TEST_MODE, TASK_TO_FIND) Note: iOS builds require Apple Developer account with proper code signing 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

…ests - Create dedicated CI workflow for Flutter Android BrowserStack tests - Seed unique test documents to Ditto Cloud before each test run - Pass seeded task title to integration tests via base64-encoded environment variable - Execute tests on multiple Android devices (Pixel 7, S23, Pixel 6) - Add PR commenting with test results and dashboard links 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

…sting - Consolidate separate flutter-browserstack.yml into unified flutter-ci.yml - Add comprehensive BrowserStack testing for both iOS and Android platforms - Implement proper task seeding pattern using Flutter task model (title, done, deleted) - Add dynamic environment variable handling with --dart-define and base64 encoding - Support for local integration testing and BrowserStack real device testing - Add automated PR commenting with test results and dashboard links - Include proper device coverage (Android: Pixel 7, Galaxy S23; iOS: iPhone 13-14) - Add test monitoring with 20-minute timeout and status polling - Remove hardcoded TASK_TO_FIND from .env, use dynamic seeding instead 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Add CODE_SIGN_IDENTITY="" and CODE_SIGNING_REQUIRED=NO to xcodebuild - Enables CI to build iOS test packages without requiring development team - Verified working with BrowserStack upload (bs://5d19936adb0533a8432548a61c58d89116d6ced6) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Format integration_test/app_test.dart - Format integration_test/simple_test.dart - Format lib/main.dart - Format test_driver/integration_test.dart 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Change build-ios job to build release IPA for device - Remove simulator requirement, build with --no-codesign - Create unsigned IPA package for archival - Upload IPA artifact instead of simulator build 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

BrowserStack jobs now properly depend on both lint and their respective build jobs to ensure correct execution order 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Rename 'Test' job to 'Run Unit Tests' for clarity - Move unit tests to run before build jobs for earlier failure detection - Remove local 'Integration Test' job (using BrowserStack for all integration testing) - Fix iOS BrowserStack signing issue by adding --no-codesign flag - Update job dependencies to ensure proper execution order 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

…tack-android job

- Add ORDER BY title ASC to ensure seeded test task appears first - Fix Android BrowserStack build by setting correct working directory - Inverted timestamp tasks will now sort to the top for reliable test discovery 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Changed working-directory back to flutter_app with explicit cd android - Added chmod +x gradlew before executing to ensure permissions - Applied fix to both assembleDebug and assembleAndroidTest steps 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Changed iPhone 12 from iOS 14.7 to iOS 15 - Fixes "_backtrace_async" symbol not found error - This symbol is only available in iOS 15+ but tests were running on iOS 14.7 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- iPhone 13 with iOS 15 - iPhone 14 with iOS 16 - iPhone 12 with iOS 17 All versions are iOS 15+ to avoid symbol compatibility issues 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Android fixes: - Build test APK wrapper FIRST, then app APK (critical for BrowserStack) - Follows documented working process from BS_WORKING.md - Test wrapper must be built before app APK for proper pairing iOS documentation: - Added comment explaining iOS 15+ requirement - Flutter testing framework needs _backtrace_async symbol (iOS 15+) - Reference: https://developer.apple.com/documentation/os/backtrace_async 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Add ls command to verify gradlew exists - Make chmod conditional on file existence - This helps debug why gradlew is not found in CI 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Android fixes: - Remove gradlew and gradlew.bat from .gitignore - Commit gradle wrapper files (gradlew, gradlew.bat, gradle-wrapper.jar) - Gradle wrapper should be committed for CI/CD (standard practice) - Fixes "gradlew: No such file or directory" error in CI iOS fixes: - Add automatic "OK" button tap after app initialization - Handles Bluetooth permission dialog that may appear on iOS - Uses conditional tap to avoid errors if dialog doesn't appear 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

…l setup - Combine assembleDebug and assembleDebugAndroidTest into single gradle command - Build both APKs together in browserstack-android job with seeded task title - Follow proven working pattern from BS_WORKING.md - Keep build-android job as sanity check only 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Change build-android sanity check to release - Use assembleRelease and assembleReleaseAndroidTest for BrowserStack - Update APK paths to release artifacts - Matches iOS which already uses release builds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

SECURITY: Remove hardcoded BrowserStack credentials from upload_and_run_ios_tests.sh Cleanup: - Remove BS_WORKING*.md documentation files (no longer needed) - Remove simple_test.dart (only app_test.dart is used) - Remove browserstack_config.json (unused) - Remove build_ios_test_package.sh (replaced by CI workflow) - Remove session_response.json and upload_response.json artifacts Improvements: - Fix Android APK artifact path (app-release.apk not app-debug.apk) - Add Gradle caching for faster Android builds (~30-50% improvement) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Throw error if TASK_TO_FIND is not set instead of using default. This ensures tests fail fast if misconfigured. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Add comprehensive build coverage for all Flutter platforms: - Web (ubuntu-latest) - Windows (windows-latest) - macOS Desktop (macos-latest) All builds: - Run in parallel after lint/tests pass - Use release mode for production fidelity - Cache Flutter dependencies for speed - Upload build artifacts with 1-day retention - BrowserStack tests wait for all builds to succeed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Fix dart format issues in app_test.dart and dql_builder.dart 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Flutter app only supports Android, iOS, and Web platforms. Desktop support (Windows/macOS) is not configured. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Fixes Android build failure in CI caused by outdated Gradle versions. Changes: - Upgrade Android Gradle Plugin: 8.2.2 → 8.7.3 - Upgrade Kotlin: 1.9.20 → 2.1.0 - Set NDK version to 27.0.12077973 (required by ditto_live) Tested locally: ✅ Build successful 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

assembleReleaseAndroidTest task doesn't exist in the project configuration. Android test builds typically only work with Debug variant. Switching to assembleDebug and assembleDebugAndroidTest which are standard tasks. Tested locally: - ./gradlew assembleDebug assembleDebugAndroidTest succeeds - APKs generated at expected debug paths 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Use flutter.ndkVersion instead of hardcoded "27.0.12077973" The hardcoded version was added during troubleshooting but is not needed. Flutter's default NDK version works correctly. - Restore dql_builder.dart to main (no changes needed in this PR) Tested locally: - ./gradlew assembleDebug succeeds - ./gradlew assembleDebugAndroidTest succeeds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Addressed Copilot PR review feedback. androidx.test dependencies must remain at 1.2.0/3.2.0 due to strict version constraints from Flutter's integration_test package. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

The false positive test worked as expected - BrowserStack tests failed when looking for non-existent task. Reverting to correct test selector. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Co-authored-by: Claude <[email protected]>

Update permissions check to only request Bluetooth and WiFi permissions on Android and iOS platforms. Desktop platforms (macOS, Windows, Linux) don't require these permissions and will fail if we attempt to request them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Apply dart format to ensure consistent code formatting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

The seed-data job now waits for all platform builds (Android, iOS, Web, macOS, Windows) to complete before seeding test data. Previously it only waited for Android, iOS, and Web builds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Teodor Ciuraru (teodorciuraru) self-assigned this Sep 24, 2025

Teodor Ciuraru (teodorciuraru) mentioned this pull request Sep 25, 2025

ci(flutter): add pipelines with BrowserStack integration #153

Closed

5 tasks done

Teodor Ciuraru (teodorciuraru) added the enhancement New feature or request label Sep 26, 2025

Teodor Ciuraru (teodorciuraru) changed the title ~~ci(flutter): add dedicated CI workflow with lint and build jobs~~ ci(flutter): add lint, build and BrowserStack UI tests Sep 26, 2025

Teodor Ciuraru (teodorciuraru) marked this pull request as ready for review October 1, 2025 09:57

Teodor Ciuraru (teodorciuraru) requested review from a team and Cameron Mcloughlin (cameron1024) as code owners October 1, 2025 09:57

Copilot AI review requested due to automatic review settings October 1, 2025 09:57

This comment was marked as resolved.

Sign in to view

Teodor Ciuraru (teodorciuraru) mentioned this pull request Oct 1, 2025

feat(flutter): add macOS and Windows desktop platform support #179

Merged

Cameron Mcloughlin (cameron1024) approved these changes Oct 2, 2025

View reviewed changes

Comment thread .github/workflows/flutter-ci.yml

Comment thread .github/workflows/flutter-ci.yml

Comment thread flutter_app/lib/main.dart Outdated

Comment thread flutter_app/pubspec.lock

Teodor Ciuraru and others added 13 commits October 2, 2025 12:14

fix(ci): add proper job dependencies for BrowserStack tests

8751303

BrowserStack jobs now properly depend on both lint and their respective build jobs to ensure correct execution order 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

fix(ci): fix workflow syntax error - missing indentation for browsers…

e8c7645

…tack-android job

Teodor Ciuraru and others added 25 commits October 2, 2025 12:15

chore: remove BROWSERSTACK_WORKING_SOLUTION.md

3ae9917

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

test: require TASK_TO_FIND env var, no default value

99203d8

Throw error if TASK_TO_FIND is not set instead of using default. This ensures tests fail fast if misconfigured. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

style: format Flutter code

54cfd86

Fix dart format issues in app_test.dart and dql_builder.dart 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

fix(ci): remove unsupported Windows and macOS desktop builds

a3f9c0b

Flutter app only supports Android, iOS, and Web platforms. Desktop support (Windows/macOS) is not configured. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

style: format Flutter code

e402789

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

test: add false positive check with _randomsuffix

3de7e94

docs(android): add comment explaining androidx.test version constraints

184762b

feat(flutter): add macOS and Windows desktop platform support (#179)

e981820

Co-authored-by: Claude <[email protected]>

Teodor Ciuraru (teodorciuraru) force-pushed the teodorciuraru/sdks-1611-flutter_app-add-integration-test-support branch from f1f033f to 03e08f7 Compare October 2, 2025 09:15

Teodor Ciuraru and others added 2 commits October 2, 2025 12:21

style(flutter): format code with dart format

8330e57

Apply dart format to ensure consistent code formatting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Teodor Ciuraru (teodorciuraru) merged commit 2d21e22 into main Oct 2, 2025
7 checks passed

Teodor Ciuraru (teodorciuraru) deleted the teodorciuraru/sdks-1611-flutter_app-add-integration-test-support branch October 2, 2025 09:53

Navigation Menu

Repository navigation

ci(flutter): add lint, build and BrowserStack UI tests - #173

ci(flutter): add lint, build and BrowserStack UI tests#173
Teodor Ciuraru (teodorciuraru) merged 47 commits into
mainfrom
teodorciuraru/sdks-1611-flutter_app-add-integration-test-support

Teodor Ciuraru (teodorciuraru) commented Sep 24, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 29, 2025

Uh oh!

github-actions Bot commented Sep 29, 2025

Uh oh!

github-actions Bot commented Sep 29, 2025

Uh oh!

github-actions Bot commented Sep 29, 2025

Uh oh!

github-actions Bot commented Sep 29, 2025

Uh oh!

github-actions Bot commented Sep 29, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

Cameron Mcloughlin (cameron1024) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Teodor Ciuraru (teodorciuraru) commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

CI Workflow

Build Configuration

Test Infrastructure

Decisions & Trade-offs

Test Plan

Uh oh!

github-actions Bot commented Sep 29, 2025

🤖 Flutter BrowserStack Test Results

Uh oh!

github-actions Bot commented Sep 29, 2025

🤖 Flutter BrowserStack Test Results

Uh oh!

github-actions Bot commented Sep 29, 2025

🤖 Flutter BrowserStack Test Results

Uh oh!

github-actions Bot commented Sep 29, 2025

🤖 Flutter BrowserStack Test Results

Uh oh!

github-actions Bot commented Sep 29, 2025

🤖 Flutter BrowserStack Test Results

Uh oh!

github-actions Bot commented Sep 29, 2025

🤖 Flutter BrowserStack Test Results

Uh oh!

This comment was marked as resolved.

Uh oh!

Cameron Mcloughlin (cameron1024) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Teodor Ciuraru (teodorciuraru) commented Sep 24, 2025 •

edited

Loading