Publish the snode list and translations to every repo that consumes them

README.md

@@ -12,8 +12,8 @@ share for translations. One package, `session_ops`, deployed to one self-hosted
 | `zendesk-relay` | Drafts and sends Zendesk replies from `claude:` private notes | [zendesk-relay](docs/jobs/zendesk-relay.md) |
 | `github-prs-digest` | Weekday Discord digest of open pull requests from outside contributors | [github-prs-digest](docs/jobs/github-prs-digest.md) |
 | `crowdin-duplicates` | Crowdin string slots holding more than one translation, as they open and close | [crowdin-duplicates](docs/jobs/crowdin-duplicates.md) |
-| `crowdin-sync` | Weekday: Crowdin translations into pull requests on iOS, Android and the localization module | [crowdin-sync](docs/jobs/crowdin-sync.md) |
-| `snode-list` | Weekday: the fallback service node list into session-ios | [snode-list](docs/jobs/snode-list.md) |
+| `crowdin-sync` | Weekday: Crowdin translations into iOS, Android and the localization module, and its submodule bumped in each client | [crowdin-sync](docs/jobs/crowdin-sync.md) |
+| `snode-list` | Weekday: the fallback service node list from the seed nodes into dynamic-assets, Desktop and iOS | [snode-list](docs/jobs/snode-list.md) |
 | `release-stats` | On demand: download counts of the latest releases | [release-stats](docs/jobs/release-stats.md) |
 | `session-ops-silence` | Discord alerts for a job that failed, or stopped running | [session-ops-silence](docs/jobs/session-ops-silence.md) |
 

deploy/README.md

@@ -84,7 +84,8 @@ for link in /etc/systemd/system/timers.target.wants/session-ops@*.timer; do
   [ -L "$link" ] && systemctl disable --now "${link##*/}"
 done
 systemctl disable --now session-ops-queue.timer zendesk-relay.service
-for repo in session-android session-ios session-localization; do
+for repo in session-android session-ios session-localization session-desktop-dynamic-assets \
+    session-desktop session-app session-website session-appium session-playwright; do
   gh pr list -R "session-foundation/$repo" --state open --json number,headRefName \
     -q '.[] | select(.headRefName | startswith("rehearsal/")) | .number' |
     xargs -r -I{} gh pr close -R "session-foundation/$repo" {} --delete-branch

docs/jobs/crowdin-sync.md

@@ -3,13 +3,23 @@
 Downloads the approved translations from Crowdin, validates them, and publishes each
 platform's strings: a pull request on session-android and session-ios, and a commit
 straight onto session-localization's `main`, the TypeScript module Desktop and QA use.
+Each client holding that module as a submodule then gets a pull request moving it to the
+commit just pushed:
+
+| Target | Repo | Base | Submodule |
+| --- | --- | --- | --- |
+| `desktop` | session-desktop | `dev` | `ts/localization` |
+| `app` | session-app | `main` | `packages/localization/src/localization-src` |
+| `website` | session-website | `main` | `lib/app_localization` |
+| `appium` | session-appium | `main` | `run/localizer/lib` |
+| `playwright` | session-playwright | `main` | `tests/localization/lib` |
 
 | | |
 | --- | --- |
 | Runs | `session-ops-queue.timer`, Mon–Fri from 10:00 Australia/Melbourne, after `zendesk-digest` |
 | Secrets | `/etc/session-ops/crowdin.env`: a read-only `CROWDIN_API_TOKEN`; `/etc/session-ops/publish.env` and the GitHub App key, to publish |
 | Dry run | `session-ops run crowdin-sync --dry-run`: everything but the push, with each platform's diff in the journal |
-| Re-run | `systemctl start [email protected]`; one platform with `session-ops run crowdin-sync -- --only ios` |
+| Re-run | `systemctl start [email protected]`; one target with `session-ops run crowdin-sync -- --only ios` |
 | Logs | `journalctl -u session-ops@crowdin-sync -n 100 --no-pager`; the run's downloads, parsed JSON and validation report under `/var/lib/session-ops/crowdin-sync/runs/`, for 14 days |
 
 One process, in order:
@@ -21,20 +31,31 @@ One process, in order:
 3. **For each platform**, independently, so one failing does not stop the others:
    a shallow, sparse checkout of just the paths its generator writes, the generator,
    then publishing. The alert says which platform failed and at which step.
+4. **Bump each client's submodule** to the commit localization published, cloning only
+   `.gitmodules` and moving the gitlink. A bump is skipped, and reported, when
+   localization fails; with `--only` leaving localization out, it moves to `main`'s tip.
 
 The pull requests come from `feature/update-crowdin-translations`, rebuilt from `dev`
 each run and force-pushed: the branch is the job's, and nobody else commits to it. An
 unchanged tree is not pushed again, and a run with nothing to change closes the pull
 request and deletes the branch. Android's own CI validates its pull request, so no
-Gradle build runs here.
+Gradle build runs here. The bumps follow the same rules from `update-localization`,
+rebuilt from each client's base; appium's and playwright's assertions follow the
+strings, so their pull request may need test fixes before it merges.
 
 ## Publishing
 
-Publishing authenticates as the GitHub App: `GITHUB_APP_ID` in `publish.env` and its
+Publishing authenticates as the GitHub App, which must be installed on every repo above
+with contents and pull requests write: `GITHUB_APP_ID` in `publish.env` and its
 key in `/etc/session-ops/github-app.pem`, and a run missing either exits naming it (a
 dry run needs neither); see [publish.env.example](../../deploy/env/publish.env.example).
 Commits are authored as `PUBLISH_GIT_AUTHOR`.
 
+Whatever the App's installation allows, session-ops writes only the branches listed in
+`PUBLISHABLE` in `shared/github.py`: a push, a pull request or a branch deletion
+anywhere else is refused before it reaches GitHub, as is any API call publishing does
+not make (merging, reviewing, settings). A job writing a new branch adds it there.
+
 ### Crowdin token scopes
 
 Crowdin scopes personal access tokens per endpoint family, and each scope can be

docs/jobs/snode-list.md

@@ -1,24 +1,32 @@
 # Static Snode List
 
-Copies `service-nodes-cache.json` from session-desktop-dynamic-assets into session-ios
-as `Session/Meta/service-nodes-cache.json`: the list a new client falls back on when it
-cannot reach the seed nodes. A change opens, or updates, a pull request from
-`feature/update-static-snode-list`.
+Fetches the service node list from the seed nodes and publishes it wherever a client
+bundles it, as the list a new client falls back on when it cannot reach them:
+
+| Target | Repo | What it gets |
+| --- | --- | --- |
+| `dynamic-assets` | session-desktop-dynamic-assets | `service-nodes-cache.json`, committed straight onto `main` |
+| `desktop` | session-desktop | a pull request from `update-dynamic-assets` moving its `dynamic_assets` submodule to that commit |
+| `ios` | session-ios | `Session/Meta/service-nodes-cache.json`, in a pull request from `feature/update-static-snode-list` |
 
 | | |
 | --- | --- |
-| Runs | `session-ops-queue.timer`, Mon–Fri from 10:00 Australia/Melbourne, after `crowdin-sync`; the source updates at 10:00 UTC |
-| Secrets | `/etc/session-ops/publish.env` and the GitHub App key; `CROWDIN_DISCORD_WEBHOOK_URL` from `crowdin.env` for the summary |
+| Runs | `session-ops-queue.timer`, Mon–Fri from 10:00 Australia/Melbourne, after `crowdin-sync` |
+| Secrets | `/etc/session-ops/publish.env` and the GitHub App key, installed on the three repos above; `CROWDIN_DISCORD_WEBHOOK_URL` from `crowdin.env` for the summary |
 | Dry run | `session-ops run snode-list --dry-run` |
 | Re-run | `systemctl start [email protected]` |
 | Logs | `journalctl -u session-ops@snode-list -n 50 --no-pager` |
 
-The file is committed exactly as fetched, but only once it holds service nodes, each
-with an IP and a key: an error page or an empty list published as the fallback would
-strand exactly the clients it exists for. The
-branch follows the same rules as [the translation sync's](crowdin-sync.md): rebuilt
-from `dev`, not pushed when unchanged, retired when `dev` already has the list.
+The seeds are asked in turn until one answers with at least 20 active nodes, each
+carrying every field the clients read; anything less publishes nothing, since an empty
+or broken fallback would strand exactly the clients it exists for. Nodes without a
+public IP are dropped, and the rest sorted by `pubkey_ed25519` so a run's diff is only
+what changed on the network.
+
+The targets are independent, except that Desktop's bump needs dynamic-assets to have
+published. The pull requests follow [the translation sync's](crowdin-sync.md) rules:
+rebuilt from `dev`, not pushed when unchanged, retired when `dev` already has it.
 
-Each run posts one line to the translations channel: how many nodes in the list have
-`requested_unlock_height` set, meaning they asked to exit, and what happened to the pull
-request. A failed run reports there too.
+Each run posts one message to the translations channel: how many nodes have
+`requested_unlock_height` set, meaning they asked to exit, then what happened in each
+repo. A failed target is reported there too.

src/session_ops/crowdin/sync.py

@@ -5,10 +5,14 @@
 - session-android and session-ios get a pull request from
   feature/update-crowdin-translations, rebuilt from `dev` each run.
 - session-localization gets a commit straight onto `main`.
+- The clients holding session-localization as a submodule (session-desktop,
+  session-app, session-website, session-appium, session-playwright) each get a pull request from
+  update-localization moving it to that commit, or to `main`'s tip when this run
+  leaves localization out.
 
 Each platform is checked out shallow and sparse, only the paths its generator writes.
-The three are independent targets: one failing to publish does not stop the others,
-and the alert says which. Android's own CI validates its pull request, so no Gradle
+The targets are independent: one failing to publish does not stop the others, and the
+alert says which; only the submodule bumps wait on localization, when it runs. Android's own CI validates its pull request, so no Gradle
 build runs here.
 
 The run's inputs, outputs and validation report are kept under the job's
@@ -19,7 +23,7 @@
     PUBLISH_GIT_AUTHOR    "Name <email>" the commits are authored as
     plus what shared/github.py reads to publish (not needed with --dry-run)
 
-    session-ops run crowdin-sync [--dry-run] [-- --only android]
+    session-ops run crowdin-sync [--dry-run] [-- --only android desktop]
 """
 import argparse
 import os
@@ -31,7 +35,7 @@
                                  generate_android_strings, generate_ios_strings,
                                  generate_language_list, parse_xliff)
 from session_ops.ops.runner import Outcome, call_target, step
-from session_ops.platforms import publish
+from session_ops.platforms import publish, submodules
 from session_ops.shared import github
 from session_ops.shared.git import Repo
 
@@ -50,9 +54,25 @@
 IOS_TRANSLATIONS = "Session/Meta/Translations"
 IOS_CONSTANTS = "SessionUIKit/Style Guide/Constants.swift"
 
-TARGETS = ("android", "ios", "localization")
+SUBMODULE_BRANCH = "update-localization"
+SUBMODULE_TITLE = "chore: Update localization submodule"
+SUBMODULE_BODY = """[Automated]
+Moves the `session-localization` submodule to the latest translations from Crowdin.
+"""
+CLIENTS = {
+    "desktop": submodules.Submodule("session-desktop", "dev", "ts/localization"),
+    "app": submodules.Submodule("session-app", "main",
+                                "packages/localization/src/localization-src"),
+    "website": submodules.Submodule("session-website", "main", "lib/app_localization"),
+    "appium": submodules.Submodule("session-appium", "main", "run/localizer/lib"),
+    "playwright": submodules.Submodule("session-playwright", "main",
+                                       "tests/localization/lib"),
+}
+
+TARGETS = ("android", "ios", "localization", *CLIENTS)
 REPOS = {"android": "session-android", "ios": "session-ios",
-         "localization": "session-localization"}
+         "localization": "session-localization",
+         **{target: client.repo for target, client in CLIENTS.items()}}
 
 
 def checkout(target, work, token):
@@ -94,13 +114,31 @@ def publish_target(target, repo, api, author, dry_run):
                                 dry_run)
 
 
-def sync_target(target, work, parsed, token, api, author, dry_run):
+def sync_target(target, work, parsed, token, api, author, dry_run, published):
     step(f"{target}: checkout")
     repo = checkout(target, work, token)
     step(f"{target}: generate")
     generate(target, repo, parsed)
     step(f"{target}: publish")
     print(publish_target(target, repo, api, author, dry_run))
+    published[target] = repo.head()
+
+
+def localization_tip(work, token):
+    url = f"{publish.GITHUB}/{publish.ORG}/{REPOS['localization']}"
+    listed = Repo(work, token).git("ls-remote", url, "refs/heads/main").stdout.split()
+    if not listed:
+        raise RuntimeError(f"{url} has no main branch")
+    return listed[0]
+
+
+def bump_target(target, work, token, api, author, dry_run, published, localization_ran):
+    step(f"{target}: publish")
+    if localization_ran and "localization" not in published:
+        raise RuntimeError("localization did not publish, so there is nothing to bump to")
+    sha = published.get("localization") or localization_tip(work, token)
+    print(submodules.bump(CLIENTS[target], sha, work, token, api, SUBMODULE_BRANCH,
+                          SUBMODULE_TITLE, SUBMODULE_BODY, author, dry_run))
 
 
 def keep(work, runs_dir, names):
@@ -147,8 +185,17 @@ def main(argv=None):
 
     token = None if args.dry_run else github.publish_token(
         publish.ORG, [REPOS[t] for t in args.only])
-    api = github.session(token) if token else None
-    results = {target: call_target(lambda _, target=target: sync_target(
-                   target, work, parsed, token, api, author, args.dry_run), [])
-               for target in args.only}
+    api = github.publish_session(token) if token else None
+    published, localization_ran = {}, "localization" in args.only
+
+    def run_target(target):
+        if target in CLIENTS:
+            bump_target(target, work, token, api, author, args.dry_run, published,
+                        localization_ran)
+        else:
+            sync_target(target, work, parsed, token, api, author, args.dry_run, published)
+
+    # TARGETS' order, whatever --only's: the bumps follow localization.
+    results = {target: call_target(lambda _, target=target: run_target(target), [])
+               for target in TARGETS if target in args.only}
     return Outcome(targets=results)

src/session_ops/jobs.toml

@@ -100,7 +100,7 @@ unit = ["LoadCredential=github-app.pem:/etc/session-ops/github-app.pem"]
 
 [[job]]
 name = "snode-list"
-description = "Fallback service node list into session-ios"
+description = "Fallback service node list into dynamic-assets, Desktop and iOS"
 entry = "session_ops.platforms.snode_list:main"
 user = "publisher"
 env_files = ["/etc/session-ops/crowdin.env", "/etc/session-ops/publish.env"]

src/session_ops/platforms/publish.py

@@ -10,8 +10,8 @@
 from session_ops.shared.env import rehearsing
 
 GITHUB = "https://github.com"
-ORG = "session-foundation"
-REHEARSAL_PREFIX = "rehearsal/"
+ORG = github.ORG
+REHEARSAL_PREFIX = github.REHEARSAL_PREFIX
 REHEARSAL_NOTE = ("**Rehearsal of session-ops: do not merge.** Close it and delete the branch "
                   "once reviewed; production publishes to the branch without the "
                   f"`{REHEARSAL_PREFIX}` prefix.\n\n")
@@ -21,6 +21,8 @@ def pull_request(repo, api, name, base, branch, title, body, author, dry_run):
     """Publish `repo`'s uncommitted changes to `branch`. Returns a one-line result."""
     if rehearsing():
         branch, title, body = REHEARSAL_PREFIX + branch, f"[Rehearsal] {title}", REHEARSAL_NOTE + body
+    # Before anything else, so a dry run catches a branch the App may not write.
+    github.require_publishable(name, branch, force=True)
     if not repo.changed():
         if not dry_run:
             github.retire_branch(api, name, branch)
@@ -42,6 +44,7 @@ def direct_push(repo, api, name, branch, message, body, author, dry_run):
     if rehearsing():
         return pull_request(repo, api, name, branch, f"direct-push-to-{branch}", message, body,
                             author, dry_run)
+    github.require_publishable(name, branch)
     if not repo.changed():
         return f"{name}: no changes on {branch}"
     repo.commit(message, author)