diff --git a/learn/developers/deploying-from-ci.mdx b/learn/developers/deploying-from-ci.mdx index bdf48cf1d..9aa115c72 100644 --- a/learn/developers/deploying-from-ci.mdx +++ b/learn/developers/deploying-from-ci.mdx @@ -19,7 +19,8 @@ Both work with private sources: `deploy_component` accepts a [`credentials`](/re - A GitHub Actions workflow that deploys a tagged release straight from a repository - A GitHub Actions workflow that builds, publishes to GitHub Packages, and deploys the published artifact - How to wait until every node has taken the release, and how to go back to a previous one -- How to deploy isolated applications and keep one preview per PR, with shared databases and cleanup on closure +- How to deploy isolated applications +- How to give each pull request its own preview, on its own branch of the data, with one setup command and the `HarperFast/deploy-action` GitHub Action ## Prerequisites @@ -190,6 +191,23 @@ jobs: For a private repository, add `credential=true` to the deploy command. It attaches the `deploy.my-app.git.github.com` reference you created with `harper deploy setup=true`. +:::tip + +On a v5.4.0 cluster, the [`HarperFast/deploy-action`](https://github.com/HarperFast/deploy-action) GitHub Action can replace the job's steps, the wait included, since a v5.4.0 `harper deploy restart=rolling` waits for every node itself: + +```yaml +steps: + - uses: HarperFast/deploy-action@v1 + with: + target: ${{ vars.HARPER_CLI_TARGET }} + app: my-app + by_ref: true +``` + +It installs the CLI, deploys with `restart=rolling` unless you set `restart`, and authenticates with the job's OIDC token alone. A private repository also needs `credential: true`. The job still needs `environment: production` and `id-token: write` for the trust policy; it needs no checkout. + +::: + A few details worth noting: - **No secrets.** `HARPER_CLI_TARGET` is the only thing the workflow is given. `id-token: write` lets the CLI request an identity token, and `environment: production` puts the `environment` claim the trust policy requires into it. Any other step in the job could request one too, so the job runs none of the repository's code. The CLI prints which policy authenticated it, and as whom, to stderr. @@ -197,7 +215,7 @@ A few details worth noting: - **`harper@^5.3`** pins the CLI's major version. 5.3 is the first CLI that authenticates with the identity token, and rerunning an old tag after a new major release still installs a 5.x CLI. - **`by_ref=true`** deploys `git+https://github.com//.git#`, where the SHA is the commit the tag points to (`GITHUB_SHA`). Every node clones that exact commit, even if the tag is moved later. Pass `ref=`, with the full commit SHA, to deploy a different commit. In a private repository, a tag resolves only if the checkout fetched it, as `fetch-depth: 0` does: the CLI looks up anything else with `git ls-remote`, which has no credential once `persist-credentials: false` keeps the token out of `.git/config`. - **`replicated` isn't needed.** On Harper Fabric and Harper Pro, a deploy goes to every node in the cluster unless you pass `replicated=false`. Each node fetches and installs the package itself, resolving the credential from the replicated secrets store. Harper core on its own does not replicate. -- **`restart=rolling`** restarts nodes one at a time, so they never all restart at once. Each node stays in rotation while it restarts. `restart=true` deploys to every node at once; each node replaces its workers a few at a time while the others keep serving. [Choose a restart mode](../administration/safe-deployments-and-rollback.mdx#choose-a-restart-mode) compares the two. The [preview recipe](#per-pr-previews-with-shared-databases) also uses inline restarts on clusters, with an explicit allowance, before v5.4.0, for concurrent initial restarts of the shared workers. On Harper core alone, a rolling job that only restarts — before v5.4.0, or when `certification` is `unavailable` — fails with `Replication not implemented`, which fails the wait step. +- **`restart=rolling`** restarts nodes one at a time, so they never all restart at once. Each node stays in rotation while it restarts. `restart=true` deploys to every node at once; each node replaces its workers a few at a time while the others keep serving. [Choose a restart mode](../administration/safe-deployments-and-rollback.mdx#choose-a-restart-mode) compares the two. [Per-PR previews](#per-pr-previews) use `restart=true`, which restarts only each preview's own worker. On Harper core alone, a rolling job that only restarts — before v5.4.0, or when `certification` is `unavailable` — fails with `Replication not implemented`, which fails the wait step. - **`json=true`** prints the deploy's result as JSON on stdout. Progress and authentication messages go to stderr, so `deploy.json` holds only the result. A deploy that fails exits non-zero, which fails the step. - **Waiting for every node** . With `restart=rolling` on v5.4.0 and later, unless the deploy's `certification` is `unavailable`, the other nodes take the release in a job, one at a time, and `harper deploy` [waits for them](/reference/v5/cli/commands#waiting-for-a-rolling-deploy) on the deployment record before it exits: the deploy step fails, naming each node that did not take the release, or passes once every node has. Its result then carries `rollout`, and the wait step does nothing. Otherwise, as before v5.4.0, `harper deploy` exits `0` once the node you called has taken the release and started the rolling job: every node has installed the release, and the job only restarts them; on v5.4.0 and later, a node that did not restart fails the job, and its `result.replicated` says which (see [how a rolling restart reports each node](/reference/v5/operations-api/operations#restart_service-replicated)). The wait step then polls the job's `restartJobId` with `get_job` until it ends `COMPLETE` or `ERROR`, and retries a poll that fails, such as one that reaches a node while it restarts. Each `harper get_job` authenticates through the trust policy the same way. It gives up after 60 polls, about ten minutes. Nodes restart one after another, so on a cluster with many nodes raise the count. With `restart=true` there is no job, and the deploy step itself fails when a node rejects the release, unless you pass `ignore_replication_errors=true`. - **The job lives on one node.** The deployment record that `harper deploy` waits on replicates, so its wait works through a load-balanced cluster URL. Harper does not replicate jobs, so only the node that received the deploy knows the rolling job, and the wait step's `get_job` polls have that limit. Any other node answers `get_job` with an empty list, which the wait step treats as not finished and polls again. So through a load-balanced cluster URL, at least one poll has to reach the node that received the deploy. Routing that keeps the runner on one node sends its polls where the deploy went, and routing that spreads requests across nodes reaches it within a few polls. Routing that sends the polls to a different node than the deploy never does: the step fails, saying the job was not on the node that answered and its outcome is unknown, even when the rollout succeeded. If your load balancer routes that way, use one node's own URL for both the deploy and the wait, and as the trust policy's `audience`. @@ -343,168 +361,302 @@ Keep the workflow's wait step when using `restart=rolling`. For a single-node in Turning isolation off also restarts the shared workers, as does turning it on for an application they had loaded, or, before v5.4.0, creating an isolated application. From v5.4.0, a new isolated application's first `restart=true` deploy starts only its own worker. Later redeploys that keep the application isolated restart only its worker, except an activation retry whose release is already live. A package redeploy that omits `isolated` preserves it; pass `isolated=false` to return the application to the shared pool. -## Per-PR previews with shared databases +## Per-PR previews - + + +A preview runs a pull request's code at its own URL, beside production, so a reviewer can try the change before it merges. PR `42`, from the branch `fix/cart`, deploys as `my-app-pr-42`, an [isolated application](/reference/v5/components/applications#isolated-applications) served at `https://fix-cart.app./`. Each push to the PR updates it. Closing or merging the PR drops it, and the cluster drops it on its own seven days after its last deploy, so a missed close event leaves nothing behind for long. By default, each preview runs on its own [branch of the cluster's databases](/reference/v5/components/applications#branched-databases), so what it writes stays out of production's data. + +It takes one workflow file and one command, and stores no secret in GitHub: -A preview can run each PR's code at its own hostname while sharing the cluster's databases. PR `42`, for example, deploys as `my-app-pr-42` at `pr-42.preview.example.com`; new commits replace that copy, and closing or merging the PR triggers cleanup. +- The workflow, `.github/workflows/preview.yaml`, runs the [`HarperFast/deploy-action`](https://github.com/HarperFast/deploy-action) GitHub Action on `pull_request_target`, from your default branch. When a PR opens or changes, the action deploys its head commit as the PR's preview and keeps one comment on the PR with the preview's URL. When the PR closes, it drops the preview. +- [`harper preview setup`](/reference/v5/cli/commands#harper-preview-setup) checks that workflow and enables previews of the application on the cluster. It creates a user whose role can deploy and drop only that application's previews, and trust policies that let that one workflow file act as that user. -**The preview shares data with every application using those databases, including production if it runs on the same cluster.** Its writes, schema changes, and application startup work affect that shared data. Instance-wide roles declared by the PR also apply, and background work can run once per preview. Isolation separates worker state and exported routes; it does not sandbox code or isolate database access. This recipe is for trusted PRs from branches in the same repository. +Read [How previews stay safe](#how-previews-stay-safe) before you turn previews on for a cluster that also serves production. -### Prepare routing and preview authentication +:::note Coming soon -Choose a hostname suffix such as `preview.example.com`. For each preview, provision DNS, TLS, and routing for `pr-.`: +Harper Studio's Previews panel, not released yet, will turn previews of an application on and off and show its limits from the browser, using the same cluster-side setup as `harper preview setup`. -- **Fabric:** add, verify, and bind each hostname as a custom domain. The workflow below deploys the application; it does not register domains. Wildcard-domain automation is still tracked under [PR preview deployments](https://github.com/HarperFast/harper/issues/3042), so a wildcard DNS record alone is not sufficient. -- **Self-managed:** configure your proxy to route each hostname to the corresponding application's socket. For `my-app-pr-42` and secure port `9927`, the HTTP socket is `/sockets/app-my%2Dapp%2Dpr%2D42-9927.sock`. That socket serves plain HTTP; the proxy terminates TLS. +::: + +### Before you start + +- **Every node runs v5.4.0 or later.** [On v5.3](#previews-on-v53) has what differs before that. +- **The cluster can run isolated applications**: a nonzero `threads.count`, `http.securePort`, and `tls.unixDomainSockets: true`. Each running preview takes a slot of [`threads.maxIsolated`](/reference/v5/configuration/options#threads), `8` by default, on each node, and a worker thread with its memory. See [Requirements and refusals](/reference/v5/components/applications#requirements-and-refusals). +- **Branched data needs RocksDB storage**, and a module loader other than `native`. On a cluster that cannot branch, or for previews that should use the cluster's own data, set previews up with `dataMode=shared`, and read [Shared or branched data](#shared-or-branched-data) first. +- **Preview hosts reach the cluster.** Every Colocated and Dedicated cluster on Harper Fabric serves `*.app.` with no setup. A cluster created before preview hosts gains them one instance at a time, as each instance's certificate renews, within about seven weeks, and until every instance has renewed, a preview answers only some of the time. See [Preview Hosts](/fabric/cluster-creation-management#preview-hosts). On a self-managed cluster, point a wildcard DNS record and certificate for a preview domain at a proxy that routes each host to its application's socket, as in [Reaching an isolated application](/reference/v5/components/applications#reaching-an-isolated-application), and pass that domain to setup with `domain=` and to the action with `domain:`. +- **On your machine:** the Harper CLI, `super_user` access to the cluster, and the [`gh` CLI](https://cli.github.com/) signed in to an account that can manage the repository's Actions variables. + +### Add the preview workflow + +In a checkout of the repository, on its default branch, add `.github/workflows/preview.yaml`, with your application's component name as `app` and your default branch under `branches`: + +```yaml +name: Preview +on: + pull_request_target: + types: [opened, reopened, synchronize, closed] + branches: [main] +permissions: {} +jobs: + deploy: + if: github.event.action != 'closed' && github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + environment: preview + permissions: + id-token: write + pull-requests: write + deployments: write + concurrency: + group: harper-preview-${{ github.event.pull_request.number }} + steps: + - uses: HarperFast/deploy-action@v1 + with: + target: ${{ vars.HARPER_PREVIEW_TARGET }} + app: my-app + preview: true + cleanup: + if: github.event.action == 'closed' && github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + environment: preview-cleanup + permissions: + id-token: write + pull-requests: write + deployments: write + steps: + - uses: HarperFast/deploy-action@v1 + with: + target: ${{ vars.HARPER_PREVIEW_TARGET }} + app: my-app + preview: true +``` -On GitHub.com, create a separate `preview` GitHub environment and OIDC policy instead of reusing the production policy. This workflow uses `pull_request_target`, runs from the trusted default branch, and does not check out or execute PR code on the runner. In **Settings > Environments > preview**, allow `main` under **Deployment branches and tags**, and, when sharing a production cluster, require deployment reviewers if your repository's GitHub plan supports them. Without that approval gate, every same-repository writer and automation that opens a PR can run code with access to production data. Required reviewers also gate closure and retargeting cleanup: approve those runs, or the preview remains running. Every same-repository PR closure can request that approval, even if the PR never had a preview. Before closing a previewed PR, finish or cancel any older run waiting for approval so per-PR concurrency can admit cleanup. Replace `main` throughout this example if your default branch has another name. See [GitHub environment rules](https://docs.github.com/en/actions/reference/workflows-and-actions/deployments-and-environments) and [the `pull_request_target` event](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request_target). +- **It never checks out the PR.** The action has the cluster fetch the PR's head commit itself, by reference, so no step of a job holding the identity token runs the PR's code. It authenticates with the job's OIDC token alone, against the `HARPER_PREVIEW_TARGET` variable setup sets, and refuses any other Harper credential in its environment. +- **Fork PRs are skipped.** Each job runs only when the PR's head repository is this repository, and the action skips a fork's PR itself too. +- **Only PRs into the default branch get a preview**, and retargeting a PR is not one of the events the workflow runs on. A PR retargeted to the default branch gets its preview at its next push. One retargeted away keeps the preview it had, unchanged, and closing it does not drop it: it lasts until it expires, a [sweep](#sweeping-missed-previews) drops it, or you run `harper preview drop`. +- **A deploy that waited goes ahead only while it is still current.** The action deploys only while the PR is open at the commit the event named, since an approval on `preview` can hold the job for a while. If the PR closes while it deploys, it drops what it deployed. The concurrency group runs one deploy per PR at a time. +- **The PR shows the preview.** The action keeps one comment on the PR with the preview's URL, updated in place, and records each preview as a GitHub Deployment in the `my-app-preview` environment, kept apart from the `preview` environment that gates the job. Both need the `pull-requests: write` and `deployments: write` permissions, and the deploy still counts if either fails. Turn them off with `comment: false` and `github_deployment: false`. +- **Cleanup runs in its own environment**, with a token that can only drop, so it never waits for the `preview` environment's reviewers. While a deploy of the same PR is still activating, the drop answers `409`, so the action retries it for about two minutes. A PR reopened before cleanup drops keeps its preview. One reopened in the moment between cleanup's check and its drop can lose the preview its reopen just deployed; push again, or rerun the deploy job, to bring it back. -Ensure your Actions execution policy allows `pull_request_target` for `.github/workflows/preview.yml`. GitHub's default protection will block that event for affected public repositories starting November 2, 2026; private and internal repositories are unaffected. See [GitHub's execution-protection rollout](https://github.blog/changelog/2026-09-17-workflow-execution-protections-in-github-actions-generally-available/). +The [action's README](https://github.com/HarperFast/deploy-action#inputs) lists every input. Keep the workflow's rules when you edit it later, since `harper preview setup` checks the file only when it runs: no checkout or fetch of the PR's code in a job with `id-token: write`, and the same-repository condition on every job that holds the token. The cluster still enforces the role's limits, whatever the workflow does. -Cleanup needs a **direct Operations API URL for every node**, rather than a balanced cluster URL. Put these URLs in the `HARPER_PREVIEW_NODE_TARGETS` repository variable as a JSON array, including the port and trailing slash. On a single-node instance, the array contains its one URL. On Fabric, arrange direct instance access with your administrator or support before enabling this recipe; a cluster URL alone cannot safely check each node. Keep the list current when cluster membership changes. +### Enable previews -From your machine, logged in as a `super_user`, create a role that can deploy and clean up previews: +From that checkout, logged in to the cluster as a `super_user`: ```bash -harper add_role role=ci_preview permission='{"operations":["deploy_component","drop_component","get_components","system_information"]}' -harper add_user username=ci-preview role=ci_preview active=true password="$(openssl rand -base64 32)" -harper add_oidc_trust \ - id=my-app-preview \ - issuer=https://token.actions.githubusercontent.com \ - audience=https://my-cluster.example.com:9925/ \ - user=ci-preview \ - claims='{"repository_id":"67890","workflow_ref":"my-org/my-app/.github/workflows/preview.yml@refs/heads/main","environment":"preview","event_name":"pull_request_target"}' -gh variable set HARPER_PREVIEW_DOMAIN --repo my-org/my-app --body preview.example.com -gh variable set HARPER_PREVIEW_NODE_TARGETS --repo my-org/my-app \ - --body '["https://node-a.example.com:9925/","https://node-b.example.com:9925/"]' +harper login https://my-cluster.example.com:9925 +harper preview setup app=my-app ``` -Use your repository id, workflow path, and cluster URL, as in the [release policy](#add-a-trust-policy). Add a trust policy for each node URL as well, with a unique `id`, that exact URL as `audience`, and the same user and claims. On Pro and Fabric, policies, users, and roles replicate, so you can create these policies from any node. On independent instances, authenticate as a `super_user` at each node and repeat the setup with `target=`. Ensure each node has the preview user, role, and matching policy. The CLI requests a token for the target it calls, so a policy with only the cluster URL as its audience does not authorize node URLs. The workflow uses the existing `HARPER_CLI_TARGET` variable. The preview role's operations apply to the whole cluster, not just projects whose names begin `my-app-pr-`; trust the people allowed to run this workflow accordingly. `get_components` can expose component configuration and files, so keep its response out of logs and artifacts. +`app` is the application's component name: the `project` it deploys as. Setup checks the GitHub side, then enables previews on the cluster, then writes the GitHub side: + +1. It looks the repository up for its id and default branch. +2. It reads `.github/workflows/preview.yaml` from the checkout, and refuses a workflow that could give the preview identity to a fork or to a PR's code. If the file doesn't exist, it offers to write [a starter workflow of its own](#the-workflow-setup-writes). +3. It calls [`enable_previews`](/reference/v5/operations-api/operations#enable_previews) on the cluster, which creates: + - the role and user `my-app-ci-preview`, whose role can deploy, drop and read only `my-app-pr-` previews, and run the version check `harper` makes before it deploys; + - the trust policy `github-actions-my-app-preview`, for jobs in the `preview` environment; + - the trust policy `github-actions-my-app-preview-cleanup`, for jobs in the `preview-cleanup` environment, whose token can only drop. +4. It sets the repository variable `HARPER_PREVIEW_TARGET` to the cluster's URL. + +Commit the workflow to the default branch. GitHub runs a `pull_request_target` workflow from there, not from the PR. + +A few options change what setup enables. [`harper preview setup`](/reference/v5/cli/commands#harper-preview-setup) lists them all, and three need the matching action input in both jobs: + +| `harper preview setup` | What it does | Action input | +| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | +| `credential=true` | Seals a GitHub token for the cluster to fetch each PR's commit with, for a private repository; setup offers it when it sees one. See [Private repositories](#private-repositories). | `credential: true` | +| `dataMode=shared` | Lets previews use the cluster's databases rather than requiring a branch of them. | `branched_databases: false` | +| `domain=` | Sets the preview domain, for a self-managed cluster or one you reach through a custom domain. | `domain: ` | +| `maxPreviews=` | Caps how many previews of the application each node keeps live. A deploy past the cap is refused, so close PRs you are done with. | none | +| `maxTtlDays=` | The longest a preview lives after its last deploy, `7` by default. | none | + +The cluster holds these limits in the preview role, and enforces them on every deploy, so a change to the workflow cannot raise them. + +### Create the GitHub environments + +In the repository's **Settings > Environments**, create `preview` and `preview-cleanup`. On each, set **Deployment branches and tags** to the default branch only. GitHub creates an environment a workflow names with no rules at all, so create them before the first PR. + +- On **`preview`**, add required reviewers if the cluster also serves production and your GitHub plan supports them. Every deploy then waits for someone to approve the PR's code before it runs on the cluster. +- Leave **`preview-cleanup`** without reviewers. Its token can only drop previews, and a reviewer there would keep a closed PR's preview running until someone approves. + +### Public repositories: allow `pull_request_target` -### Deploy on PR updates and remove on closure +From December 2, 2026, GitHub blocks `pull_request_target` by default in public repositories that have no Actions event policy of their own; it already did so during brownouts on November 9 and 16, 2026. Private and internal repositories are not affected. See [GitHub's announcement](https://github.blog/changelog/2026-11-02-pull_request_target-blocked-by-default-december-2). -This example is Path A for an application that runs from source. Add `.github/workflows/preview.yml` to your repository: +In a public repository, allow the event for the preview workflow alone. In **Settings > Actions > Policies**, create a policy: + +1. Set its enforcement status to **Active**. +2. Under its target workflows, add `.github/workflows/preview.yaml`. +3. Turn on **Restrict events**, and allow `pull_request_target`. + +An organization or enterprise policy can still block the event. See [Control workflow execution](https://docs.github.com/en/actions/how-tos/administer/control-workflow-execution). + +### The workflow setup writes + +When `.github/workflows/preview.yaml` doesn't exist, `harper preview setup` offers to write one, which runs the same lifecycle with `harper preview` commands instead of the action. It posts no PR comment and records no GitHub Deployment. For `app=my-app` and the default branch `main`: + +
+The starter workflow ```yaml -name: Preview on Harper +# Written by `harper preview setup`. +name: Preview on: pull_request_target: - types: [opened, reopened, synchronize, edited, closed] - + types: [opened, reopened, synchronize, closed] + branches: ['main'] +permissions: {} +env: + HARPER_CLI_TARGET: ${{ vars.HARPER_PREVIEW_TARGET }} + PR: ${{ github.event.pull_request.number }} jobs: - preview: - if: >- - github.event.pull_request.head.repo.full_name == github.repository && - (github.event.action == 'closed' || - (github.event.action == 'edited' && github.event.changes.base) || - (github.event.action != 'edited' && github.event.pull_request.base.ref == 'main')) + deploy: + if: github.event.action != 'closed' && github.event.pull_request.head.repo.full_name == github.repository runs-on: ubuntu-latest environment: preview - timeout-minutes: 20 permissions: - contents: read - pull-requests: read id-token: write + pull-requests: read concurrency: - group: harper-preview-${{ github.repository }}-${{ github.event.number }} - cancel-in-progress: false + group: harper-preview-${{ github.event.pull_request.number }} env: - HARPER_CLI_TARGET: ${{ vars.HARPER_CLI_TARGET }} - PREVIEW_DOMAIN: ${{ vars.HARPER_PREVIEW_DOMAIN }} - PREVIEW_NODE_TARGETS: ${{ vars.HARPER_PREVIEW_NODE_TARGETS }} - PR_NUMBER: ${{ github.event.number }} + BRANCH: ${{ github.event.pull_request.head.ref }} + SHA: ${{ github.event.pull_request.head.sha }} + GH_TOKEN: ${{ github.token }} steps: - uses: actions/setup-node@v7 with: - node-version: 26 - - run: npm install --global harper@^5.3 - - name: Update or remove the preview - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - set -euo pipefail - [[ "$PR_NUMBER" =~ ^[0-9]+$ ]] - PROJECT="my-app-pr-$PR_NUMBER" - PR=$(gh api "repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER") - jq -e 'type == "array" and length > 0 and all(.[]; type == "string" and test("^https?://"))' <<< "$PREVIEW_NODE_TARGETS" > /dev/null - if [ "$(jq -r .state <<< "$PR")" = closed ] || - [ "$(jq -r .base.ref <<< "$PR")" != main ]; then - cleanup_node() { - export HARPER_CLI_TARGET="$1" - harper get_components json=true > components.json || return 1 - harper system_information attributes='["threads"]' json=true > system.json || return 1 - jq -e '.error == null and (.entries | type == "array")' components.json > /dev/null || return 1 - jq -e '.threads | type == "array"' system.json > /dev/null || return 1 - SHOULD_RESTART=$(jq -r --arg project "$PROJECT" 'any(.threads[]; .application == $project)' system.json) || return 1 - if jq -e --arg project "$PROJECT" 'any(.entries[]; .name == $project)' components.json > /dev/null || - [ "$SHOULD_RESTART" = true ]; then - harper drop_component project="$PROJECT" restart="$SHOULD_RESTART" replicated=false json=true > drop.json || return 1 - jq -e 'type == "object" and .error == null' drop.json > /dev/null || return 1 - harper get_components json=true > after-components.json || return 1 - harper system_information attributes='["threads"]' json=true > after-system.json || return 1 - jq -e --arg project "$PROJECT" '.error == null and (.entries | type == "array") and (any(.entries[]; .name == $project) | not)' after-components.json > /dev/null || return 1 - jq -e --arg project "$PROJECT" '(.threads | type == "array") and (any(.threads[]; .application == $project) | not)' after-system.json > /dev/null || return 1 - fi - } - FAILED=0 - while IFS= read -r NODE_TARGET; do - if ! cleanup_node "$NODE_TARGET"; then - echo "Preview cleanup failed: $NODE_TARGET" >> "$GITHUB_STEP_SUMMARY" - FAILED=1 - fi - done < <(jq -r '.[]' <<< "$PREVIEW_NODE_TARGETS") - exit "$FAILED" + node-version: 24 + - run: npm install --global harper@^5.4 + - run: | + pr() { gh api "repos/$GITHUB_REPOSITORY/pulls/$PR" --jq '.state + " " + .head.sha'; } + before=$(pr) + if [ "$before" = "open $SHA" ]; then + harper preview deploy app=my-app pr="$PR" branch="$BRANCH" by_ref=true ref="$SHA" || deployed=$? + # Closed while this deployed: its cleanup may have run first. + after=$(pr) + if [ "${after%% *}" = closed ]; then harper preview drop app=my-app pr="$PR"; fi + exit "${deployed:-0}" fi - SHA=$(jq -r .head.sha <<< "$PR") - HOST="pr-$PR_NUMBER.$PREVIEW_DOMAIN" - harper deploy project="$PROJECT" by_ref=true ref="$SHA" \ - isolated=true host="$HOST" restart=true json=true > deploy.json - echo "Preview: https://$HOST" >> "$GITHUB_STEP_SUMMARY" + cleanup: + if: github.event.action == 'closed' && github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + environment: preview-cleanup + permissions: + id-token: write + pull-requests: read + env: + GH_TOKEN: ${{ github.token }} + steps: + - uses: actions/setup-node@v7 + with: + node-version: 24 + - run: npm install --global harper@^5.4 + - run: | + # A deploy still holding the preview answers 409 until it finishes; a reopened PR keeps its preview. + for attempt in 1 2 3 4 5; do + if state=$(gh api "repos/$GITHUB_REPOSITORY/pulls/$PR" --jq .state); then + [ "$state" = closed ] || exit 0 + harper preview drop app=my-app pr="$PR" && exit 0 + fi + sleep 30 + done + exit 1 ``` -The workflow reads the PR's current state, base branch, and head SHA from GitHub, rather than deploying the default-branch commit in `GITHUB_SHA`. Its concurrency group serializes updates and cleanup for that PR without canceling a deploy midway. A delayed run sees a closed or retargeted PR and cleans up instead of recreating the preview. The job filter ignores title/body edits and unrelated updates on other base branches, while admitting closures and base-branch edits. The fresh API read remains authoritative. +
-Cleanup checks component files and running workers separately on **each listed node**, so it also handles a worker left running after files were removed. An absent node is skipped: before v5.4.0, dropping an absent preview with `restart=true` can restart its shared workers. Each necessary drop uses `replicated=false`, and requests a restart only when that node has the preview worker. Files without a worker are dropped with `restart=false`; the node's shared workers stay running. It then verifies the component and worker are both absent; a successful drop response alone is insufficient. The loop attempts the remaining nodes after an error and fails the job with each failed target in the summary. After a failed, timed-out, or interrupted run, rerun cleanup and check those nodes directly with an administrator account; do not reuse the project name until cleanup has completed on every node. An omitted node cannot be checked or cleaned by this workflow. +It reads the PR's state and head commit with `gh` before it deploys, and again afterward, and passes `ref="$SHA"` with `by_ref=true`: without it, `by_ref` deploys `GITHUB_SHA`, which for `pull_request_target` is the base branch's commit. With `credential=true`, `dataMode=shared` or a `domain=` other than the default, setup adds the matching `harper preview deploy` argument. The rules in [Add the preview workflow](#add-the-preview-workflow) apply to it too. -The deploy still propagates to peers by default on Fabric and Pro, and a failed peer install fails the CLI unless `ignore_replication_errors` is set. Do not enable that option here. Cleanup uses direct node calls specifically to recover from asymmetric deploy or drop state. +### Managing previews -`restart=true` waits for the restart on the answering node, but a successful response does not establish that every peer loaded the application: a peer restart can be slow or fail, and a node can refuse isolation. This workflow prints the URL but does not probe preview health. Check [application status](/reference/v5/operations-api/operations#set_status--get_status--clear_status) on each node with an administrator account and verify the routed preview endpoint. This recipe uses inline restarts instead of the production workflows' rolling job. **Before v5.4.0, the first deploy of each preview restarts the shared workers on all nodes concurrently**, disrupting existing connections; from v5.4.0, it starts only the preview's own worker. On earlier versions, use a dedicated preview cluster if that interruption is unacceptable; adapting the [rolling workflow](#path-a-deploy-a-tagged-release-from-your-repository) also requires granting `get_deployment` and `get_job` to the preview role and retaining its wait step. +As a `super_user`, or as the preview user: -Fork PRs are skipped; keep that same-repository condition. The policy pins `workflow_ref` so a PR cannot change the trusted workflow to mint a token with different commands; `event_name` explicitly authorizes `pull_request_target`, which Harper otherwise refuses unless constrained. Provision the hostname before opening the preview URL; successful deployment alone does not make an unconfigured hostname reachable. Close unused previews to release worker slots. Dropping a running isolated preview stops its worker and normally reclaims retained installs; inspect node logs if storage cleanup fails. Removing an application does not undo its shared-database writes or remove DNS and Fabric custom-domain registrations; clean up that routing separately. +```bash +harper preview list app=my-app # each preview: PR, branch, commit, URL, expiry +harper preview drop app=my-app pr=42 +``` -#### Cleanup in one call on v5.4.0 +- **Expiry.** Each deploy sets the preview to expire `ttl` days later, 7 by default and never more than the application's `maxTtlDays`. One node drops each expired preview on every node, checking every [`deployment.previews.sweepInterval`](/reference/v5/configuration/options#deployment). +- **How it is set up.** [`get_preview_policy`](/reference/v5/operations-api/operations#get_preview_policy) returns the application's identity, policies and limits, and in `problems`, anything on the cluster that differs from what `enable_previews` wrote. +- **A failed preview.** A preview that fails to load in its [canary worker](/reference/v5/operations-api/operations#certifying-a-release-in-a-canary-worker) fails the deploy, and the job's log says why. +- **Turning previews off.** `harper preview teardown app=my-app` revokes the identity, drops every preview of the application on every node, and removes the user, role and trust policies. It leaves the workflow file and the repository variable for you to remove. - +#### Sweeping missed previews + +The cluster expires every preview on its own, so a missed close event already costs no more than the preview's remaining days. A scheduled sweep removes those previews sooner: the action, run on `schedule` or `workflow_dispatch`, drops each of this repository's previews whose PR has closed. The trust policies setup writes match only `pull_request_target`, and the sweep has to list what is deployed, so it needs its own workflow file and its own policy. Commit `.github/workflows/preview-sweep.yaml`: -When every node runs v5.4.0 or later, cleanup is one replicated drop through `HARPER_CLI_TARGET`. The per-node targets, their trust policies, and the role's `get_components` and `system_information` grants are then unnecessary. In the workflow, delete the `PREVIEW_NODE_TARGETS` entry under `env:` and the `jq` line that validates it, and replace the cleanup branch, from `if [ "$(jq -r .state <<< "$PR")" = closed ]` through its `fi`, with: +```yaml +name: Preview sweep +on: + schedule: + - cron: '17 3 * * *' + workflow_dispatch: +permissions: {} +jobs: + sweep: + runs-on: ubuntu-latest + environment: preview-cleanup + permissions: + id-token: write + pull-requests: write + deployments: write + steps: + - uses: HarperFast/deploy-action@v1 + with: + target: ${{ vars.HARPER_PREVIEW_TARGET }} + app: my-app + preview: true +``` + +Then, as a `super_user`, add its policy, with your repository's id and name, your default branch, and the cluster's URL: ```bash -if [ "$(jq -r .state <<< "$PR")" = closed ] || - [ "$(jq -r .base.ref <<< "$PR")" != main ]; then - harper drop_component project="$PROJECT" json=true > drop.json - jq -e '[., .replicated[]?] | all(.error == null and (.componentRemoved | type == "boolean") and (.workerStopped | type == "boolean"))' drop.json > /dev/null - exit 0 -fi +harper add_oidc_trust id=github-actions-my-app-preview-sweep \ + issuer=https://token.actions.githubusercontent.com \ + audience=https://my-cluster.example.com:9925/ \ + user=my-app-ci-preview \ + operations='["get_components","drop_component"]' \ + claims='{"repository_id":"67890","workflow_ref":"my-org/my-app/.github/workflows/preview-sweep.yaml@refs/heads/main","event_name":["schedule","workflow_dispatch"],"environment":"preview-cleanup"}' ``` -The check fails the job when a node reports an error, or reports no outcome, as a node on an earlier version does. The drop stops the preview's worker on every node without restarting the shared workers, with no `restart` needed, and each node answers only once that worker has stopped. A node that no longer has the preview's files only stops a worker left running there, and restarts nothing, so rerunning a failed cleanup is safe. The response's `replicated` results give each node's outcome; see [`drop_component`](/reference/v5/components/applications#drop_component). A node on an earlier version keeps the worker running until its next restart, so keep the per-node cleanup while any node runs one. +`harper preview teardown` doesn't know about this policy, so drop it yourself when you turn previews off: `harper drop_oidc_trust id=github-actions-my-app-preview-sweep`. -### Private sources and built artifacts +### Private repositories -For a private repository, each preview project needs its own grant to a durable source credential. Once you know the PR number, run setup from your machine for that project's name, using a dedicated read-only preview token: +The cluster fetches each PR's commit itself, so for a private repository it needs a GitHub token that lasts longer than the workflow run. `harper preview setup credential=true` asks for one, seals it on your machine with the cluster's public key, and grants it to every preview of the application at once, as the secret `deploy.my-app-pr-_.git.github.com`. Pass `credential: true` to the action in both jobs to deploy with it; the starter workflow setup writes passes `credential=true` already. -```bash -harper deploy setup=true provider=github project=my-app-pr-42 -``` +Use a token for this alone: a [fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) with **Contents: read-only** on this repository. Not the token production deploys with. Harper uses it only to install each preview; the previews' own code cannot read it. To rotate it, run `harper deploy setup=true provider=github project='my-app-pr-*'` with the new token. + +### Applications that need a build + +A preview deploys by reference, so every node installs the PR's commit as it is in the repository. That suits an application that runs from source. Don't add a build step to the preview workflow: any step in a job that holds the identity token could use it, and a build runs the PR's code and its dependencies. + +To preview an application that needs a build, build and publish each PR's commit from a separate `pull_request` workflow, which gets no identity token, and deploy the published package from the preview job, with the action's `args` input (`args: package=npm:@my-org/my-app@pr-${{ github.event.pull_request.number }}`). GitHub starts both workflows at once, and the trust policies match only `pull_request_target`, so the preview job has to wait for that build itself. + +### How previews stay safe + +- **The workflow is trusted, not the PR.** The workflow runs on `pull_request_target`, from the default branch, and never runs the PR's code on the runner. The trust policies pin its `workflow_ref` to the default branch, `event_name` to `pull_request_target`, and its environment, so a PR that edits the workflow cannot change what runs with the identity: its edits take effect only once it merges. +- **Forks get nothing.** Each job runs only for a PR from a branch of the same repository. A fork's PR never gets the identity, and no preview. +- **The PR's code still runs on the cluster.** The preview role can deploy and drop only `my-app-pr-`, as an isolated application under the preview domain, within the application's limits, with no table permissions. Harper does not sandbox an application's code, though: a preview runs in the Harper process like any application, and can reach what any application can. So anyone who can open a PR from a branch of the repository can run code on the cluster. Required reviewers on the `preview` environment are the gate for that. +- **Previews share a site with production.** A preview's host is under the cluster's own host name, so a browser treats it as the same site as anything production serves from that host name. A preview can set cookies production receives, and `SameSite` does not keep them apart: see [Previews share a site with production](/fabric/cluster-creation-management#previews-share-a-site-with-production) for `__Host-` cookie names and CSRF or `Origin` checks. What production serves only from a custom domain is a different site. +- **Revoking.** `harper preview teardown` revokes the identity at once: it drops the trust policies and deactivates the user, which also stops a token already issued, before it drops the previews. To stop CI without dropping the previews, deactivate the user alone: `harper alter_user username=my-app-ci-preview active=false`. + +#### Shared or branched data + +With the default `dataMode=branched`, the cluster refuses a preview deploy that does not branch, and each preview gets a private copy of every database except `system`, taken when it first loads. Its writes, and the tables and schema changes it makes, stay in its copy, which is removed when the preview is dropped. Some things are still shared: -Then add `credential=true` to the workflow's deploy command. It derives `deploy.my-app-pr-42.git.github.com` for PR `42`. A secret granted only to `my-app` does not authorize `my-app-pr-42`; the production setup cannot be reused just by changing `project`. The `opened` run may happen before this setup and fail; provision the credential, then rerun the workflow. Preview code can read secrets granted to its component, so do not grant it the production source token. Rotation must update each active preview project's secret row. Dropping the component does not delete that row: after cleanup, an administrator should remove `deploy.my-app-pr-42.git.github.com` with [`delete_secret`](/reference/v5/operations-api/operations#delete_secret), including its grants. Reopening after secret deletion needs setup again. Credential provisioning, rotation, and deletion remain outside this workflow. +- **The `system` database**, users and roles, cannot be branched. Roles a preview declares apply to the whole cluster. +- **A database created after the preview loads** is not branched. +- **Work outside the database**, such as calls to other services, or files written outside the application's own directory, happens once per preview. -For applications that need a build, build and publish an immutable package version for each PR commit using Path B's delivery model, and trigger the trusted deploy only after publication finishes. Keep PR builds in a separate workflow; do not check out or run PR code in this `pull_request_target` job. Replace `by_ref=true ref="$SHA"` with `package="@my-org/my-app@"`, retaining `project`, `host`, `isolated=true`, and cleanup. Set up the registry credential for each preview project and pass its `credentials` reference, as in Path B. A payload upload cannot set `isolated=true`. +With `dataMode=shared`, previews read and write the cluster's databases, the ones production uses if it runs on the same cluster. A preview's writes, schema changes and startup work then change production's data. Use it only for an application whose data a preview may change, or on a cluster for previews alone. -### Adding private database forks later +### Previews on v5.3 -The recipe deliberately omits `branchedDatabases`. On v5.3.0 and v5.3.1, a package deployment ignores that key when loading the application ([harper#3071](https://github.com/HarperFast/harper/issues/3071)). Follow [Branched databases](/reference/v5/components/applications#branched-databases) for current behavior and limitations; database forks can be added to the recipe once that deployment path is fixed. Revisit cleanup then as well: before v5.4.0, a drop with `restart=false` leaves fork storage in place. From v5.4.0, dropping an isolated preview removes its forks with or without `restart`. +None of the above exists before v5.4.0. On v5.3, a preview needs a role, user, trust policy and workflow written by hand, and that role can deploy and drop any component on the cluster. On Fabric each preview host is a custom domain to add, verify and bind; a payload deploy cannot be isolated or branched; and cleanup has to call every node itself. The v5.3 recipe is in [the previous version of this guide](https://github.com/HarperFast/documentation/blob/62139d54/learn/developers/deploying-from-ci.mdx#per-pr-previews-with-shared-databases). Run v5.3 previews on a cluster for previews alone, or behind required reviewers when they share a cluster with production. ## Other CI systems diff --git a/learn/developers/multiple-applications.mdx b/learn/developers/multiple-applications.mdx index 17e12bcf5..6127bd778 100644 --- a/learn/developers/multiple-applications.mdx +++ b/learn/developers/multiple-applications.mdx @@ -24,7 +24,7 @@ Harper isolates each application's module context automatically (see [What co-lo Harper runs as a single process. Every co-located application shares that process, and, unless it is isolated, its worker threads, so it is worth being precise about what is isolated between applications and what is not. - **Application source is isolated; some dependencies are not.** Harper loads each application's JavaScript in its own module context using Node.js's VM module loader, giving every application a distinct module cache. One application's own modules, imports, and module-scoped state are not visible to another, so two applications can depend on different packages—or different versions of the same package—without colliding. The exception is dependencies: under the default `dependencyLoader: auto`, packages that do not declare `harper` as a dependency load through Node's loader and share its cache within that worker, so two applications in the same worker resolving the same package file get the same instance and the same singleton state. Different workers have separate caches. Both the loader and the dependency policy are configurable—see [Module Loading](/reference/v5/components/module-loading). -- **Database data is shared by default.** The `tables` and `databases` APIs normally reach the same base databases, whether imported from `harper` or used as globals. An isolated worker has its own API objects but still reaches that shared data. The exception is a database an application [forks](#a-private-fork-of-a-database): its `harper` import of `databases` resolves that name to the fork, while the bare globals still reach the base. A record written to a base database is visible to applications using that same base; writes to a fork stay in that fork. Any application can still read or write unbranched tables in-process. This is what makes co-location efficient, and it is why separate databases are a [namespacing convention](#namespacing-data-by-database) rather than an enforced boundary. +- **Database data is shared by default.** The `tables` and `databases` APIs normally reach the same base databases, whether imported from `harper` or used as globals. An isolated worker has its own API objects but still reaches that shared data. The exception is a database an application [forks](#a-private-fork-of-a-database): its `harper` import of `databases` resolves that name to the fork. The bare globals still reach the base in the shared workers under the default module loader; from v5.4.0 they reach the fork in an isolated application, and under the `vm` and `compartment` loaders. A record written to a base database is visible to applications using that same base; writes to a fork stay in that fork. Any application can still read or write unbranched tables in-process. This is what makes co-location efficient, and it is why separate databases are a [namespacing convention](#namespacing-data-by-database) rather than an enforced boundary. - **Users, roles, and sessions are instance-wide.** Harper's RBAC belongs to the instance, not to an application. Every application's `roles.yaml` reconciles into the same instance-wide role registry, and a user authenticates against the instance as a whole. See [Access control is instance-wide](#access-control-is-instance-wide). - **Worker threads are shared, unless an application is isolated.** By default every worker thread loads every application, so applications share each thread's globals and are restarted together. As of v5.3.0, an application whose root-config entry sets `isolated: true` runs instead in a worker thread of its own that loads no other application, and a redeploy that keeps it isolated restarts only that thread. Its worker listens only on its own Unix domain socket, not on Harper's ports, so it needs a proxy in front that routes its own `host` to that socket. On Harper Fabric the platform does this for a host the cluster claims. See [Isolated applications](/reference/v5/components/applications#isolated-applications) for the requirements and the `threads.maxIsolated` cap. - **The process is shared.** Because every application runs in one process, operational actions apply to all of them: restarting the instance restarts every co-located application, and applications cannot change the process working directory. Plan restarts and deployments with the whole instance in mind. @@ -103,7 +103,7 @@ An application can also declare `host` on an individual plugin, but a `host` on A mount is a routing prefix, not an isolation boundary. Two limits matter when several applications share an instance: - **A mount does not namespace resources.** REST endpoint paths come from the resources and tables an application exports, and those exports land in the worker's shared registry for the applications it loads. Mounting namespaces the external URL an application answers on; it does not make two applications' exports independent. Two applications that both export a `User` resource still conflict, wherever each is mounted. Give each application uniquely named resources or, preferably, its own database. An [isolated](/reference/v5/components/applications#isolated-applications) application is the exception: its exports are registered only in its own worker thread. -- **A mount does not host-constrain Fastify routes.** [`fastifyRoutes`](/reference/v5/fastify-routes/overview) registers as a global fallback outside the routed middleware chain, so those routes answer on every hostname. A `urlPath` mount does apply—it becomes the Fastify route prefix—but a `host` mount does not, and Harper refuses to load a host-mounted application that declares `fastifyRoutes` rather than silently serving it unconstrained. Port those routes to [custom resources](/reference/v5/resources/overview) or `server.http()` before mounting the application by host. +- **A mount does not host-constrain Fastify routes.** [`fastifyRoutes`](/reference/v5/fastify-routes/overview) registers as a global fallback outside the routed middleware chain, so those routes answer on every hostname. A `urlPath` mount does apply—it becomes the Fastify route prefix—but a `host` mount does not, and Harper refuses to load a host-mounted application that declares `fastifyRoutes` rather than silently serving it unconstrained. Port those routes to [custom resources](/reference/v5/resources/overview) or `server.http()` before mounting the application by host. From v5.4.0, an [isolated](/reference/v5/components/applications#isolated-applications) application with a `host` loads its `fastifyRoutes` as they are, since its worker serves no other application. ### Custom dispatch in the middleware chain @@ -152,7 +152,7 @@ listings-api-staging: The first time the application loads, Harper takes a snapshot of each named database as the application's fork. The `databases` the application imports from `harper` then resolves those names to its fork, so its code is unchanged, and every other application keeps using the base. The `tables` import follows only a fork of the default `data` database, so reach any other branched database, like `listings` here, through `databases`. The fork is durable and is never refreshed from the base. It is not replicated: each node of a cluster keeps its own. `drop_component` with `restart=true` removes it. Without `restart`, the shared workers still have it loaded, so it stays until they restart; from v5.4.0 Harper then removes it on its own, and before v5.4.0 you drop the application again with `restart=true`. Dropping an isolated application removes its fork with or without `restart` (v5.4.0), and `keepBranches` keeps it. See [Removing a fork](/reference/v5/components/applications#removing-a-fork). -A fork is still not an enforced boundary. The bare `databases` and `tables` globals reach the base. The application can also still open any database that it did not branch. On v5.3.0 and v5.3.1, an application deployed by `package` or `by_ref` gets no fork at all ([harper#3071](https://github.com/HarperFast/harper/issues/3071)). See [Branched databases](/reference/v5/components/applications#branched-databases) for the requirements and the manual steps that work in the meantime. +A fork is still not an enforced boundary. In the shared workers, under the default module loader, the bare `databases` and `tables` globals reach the base; from v5.4.0 they reach the fork in an isolated application, and under the `vm` and `compartment` loaders. The application can also still open any database that it did not branch. On v5.3.0 and v5.3.1, an application deployed by `package` or `by_ref` gets no fork at all ([harper#3071](https://github.com/HarperFast/harper/issues/3071)). See [Branched databases](/reference/v5/components/applications#branched-databases) for the requirements and the manual steps that work in the meantime. When a boundary has to hold against code you do not fully trust, or against a security or compliance requirement, put the application on a separate instance or cluster. That is the only boundary Harper enforces for data. diff --git a/reference/cli/commands.md b/reference/cli/commands.md index cb20eec41..c9cf4023b 100644 --- a/reference/cli/commands.md +++ b/reference/cli/commands.md @@ -154,8 +154,10 @@ All parameters are passed as `key=value` arguments. Every parameter is optional. - `deployment_timeout=` - How long, in milliseconds, a peer waits to receive the replicated payload before failing (default: `120000`). (Added in: v5.1.4) - `ignore_replication_errors=true` - Treat a peer that fails to receive the deploy as non-fatal instead of failing the whole operation. (Added in: v5.1.4) - `force=true` - Allow deploying over a protected core component name. -- `urlPath=` - HTTP path the component is mounted at (e.g. `/api/v2`). Requires `package`. -- `host=` - Virtual hostname the component is served on (e.g. `api.example.com`). Requires `package`. (Added in: v5.2.0) +- `urlPath=` - HTTP path the component is mounted at (e.g. `/api/v2`). Before v5.4.0, requires `package`. +- `host=` - Virtual hostname the component is served on (e.g. `api.example.com`). Before v5.4.0, requires `package`. (Added in: v5.2.0) +- `isolated=true` - Run the application in a worker thread of its own. See [Isolated applications](../components/applications.md#isolated-applications). Before v5.4.0, requires `package`. (Added in: v5.3.0) +- `branchedDatabases=''` - Give the application a private branch of these databases, as a JSON list, or `true` for all of them. See [Branched databases](../components/applications.md#branched-databases). (Added in: v5.3.0) - `credentials=''` - JSON array of credential objects for installing from a private npm registry or git repository. See [Private deploy sources](#private-deploy-sources). (Added in: v5.2.0) - `activate=false` - Build and install the release on each node without making it live. The result's `deployment_id` names the staged build. Cannot be combined with `restart`. See [Staging a build and activating it later](../operations-api/operations.md#staging-a-build-and-activating-it-later). (Added in: v5.3.0) - `deployment_id=` - Make a staged build live, or go back to a release a later deploy replaced, with no rebuild or reinstall. Takes no `package`, `by_ref`, or other build inputs. Pass `restart` here rather than on the stage. See [Going back to a previous release](../operations-api/operations.md#going-back-to-a-previous-release). (Added in: v5.3.0) @@ -266,6 +268,8 @@ This interactive flow prompts for the provider (a private GitHub repository or a See [`deploy_component` credentials](../operations-api/operations.md#deploy-credentials-credentials) for the full credential-entry shape, including passing a literal `token` instead of a sealed `secret` reference. + With `provider=github` or `provider=npm`, `project` can be a pattern ending in `*`, which grants one credential to every matching component, such as every preview of an application: `harper deploy setup=true provider=github project='my-app-pr-*'`. The secret's name has `_` in place of `*`, as in `deploy.my-app-pr-_.git.github.com`, and a deploy references it by that name with `credentials`, as [`harper preview deploy credential=true`](#harper-preview-deploy) does; `credential=true` on `harper deploy` would derive the one component's name instead. + #### GitHub Actions deploys (OIDC) @@ -290,7 +294,7 @@ It needs no secret custody, and it does this: 4. Reads the policy back and compares it, claim by claim, with what it meant to write. 5. Sets the repository's `HARPER_CLI_TARGET` variable with `gh`, if the repository doesn't have one. One that already names this cluster is left as it is. One that names another cluster is left too, since every workflow in the repository that reads it deploys there: setup prints the command to change it and exits non-zero. Without `gh`, or if the variable can't be read or written, it prints the command to run and exits non-zero. -It only creates. Running it again leaves the cluster and the variable unchanged, and a role, user or policy that already exists but differs, or that someone deactivated or disabled, stops it before it writes, naming what differs. To replace a policy, drop it with `harper drop_oidc_trust id=github-actions-` and run setup again. Because `add_oidc_trust` replaces a policy, setup checks again just before writing it, but a policy with the same id that another super user creates in that moment, enabled or not, would still be overwritten. Setup never writes a policy that already exists, so it can't undo a disable or a drop. +It only creates. Running it again leaves the cluster and the variable unchanged, and a role, user or policy that already exists but differs, or that someone deactivated or disabled, stops it before it writes, naming what differs. To replace a policy, drop it with `harper drop_oidc_trust id=github-actions-` and run setup again. Because `add_oidc_trust` replaces a policy, setup checks again just before writing it, but a policy with the same id that another super user creates in that moment, enabled or not, would still be overwritten. Setup never writes a policy that already exists, so it can't undo a disable or a drop. It also stops, before and again just before it writes, when another enabled policy could match the same runs and give them a different user or operations, since [the exchange refuses](../operations-api/operations.md#exchange_oidc_token) a token two policies disagree on: give the project its own workflow (`workflow=`) or environment (`environment=`), or drop the other policy. If the variable wasn't set, the cluster part is already done: run the printed `gh variable set` command, or run setup again, which leaves the cluster as it is and tries the variable again. @@ -298,6 +302,123 @@ The deploy user can deploy any component on the cluster, not only this project, The policy pins one branch, so it suits a workflow that deploys when changes merge. A workflow that deploys on tags needs a policy written by hand: see [Deploying from a CI/CD Pipeline](/learn/developers/deploying-from-ci#add-a-trust-policy). +### `harper preview` + + + +Runs a preview of an application for each pull request. `setup` and `teardown` run once per application, from your machine. `deploy`, `drop` and `list` run for each PR, usually from the workflow `setup` writes. [Per-PR previews](/learn/developers/deploying-from-ci#per-pr-previews) walks through the whole flow. + +A preview of the application `my-app` for PR `42` is the [isolated application](../components/applications.md#isolated-applications) `my-app-pr-42`. It is served at `.`, where the slug comes from the PR's head branch, and on a cluster that supports it, it runs on its own [branch of the databases](../components/applications.md#branched-databases). The cluster drops it when the PR closes, or when it expires. + +#### `harper preview setup` + +Enables previews of one application on the cluster, and sets up the GitHub side. Run it once, signed in to the cluster as a `super_user` (`harper login`), from a checkout of the application's repository: + +```bash +harper preview setup app=my-app target=https://my-cluster.example.com:9925 +``` + +It checks everything it can before it changes anything, so a refusal before step 6 leaves the cluster and the repository as they were. + +1. Takes the GitHub repository from the checkout's `origin` remote, or `repo=/`, and looks it up with the `gh` CLI or GitHub's public API for its numeric id, its default branch, and the spelling of its name that GitHub uses in the token. When neither answers, pass `repository_id=`. +2. Reads the preview workflow, `.github/workflows/preview.yaml`, from the checkout. It refuses a workflow the trust policies could never match, and one that may give the preview identity to a fork or to a PR's own code: see [What setup checks in the workflow](#what-setup-checks-in-the-workflow). If the file doesn't exist, it offers to write the starter workflow (`write_workflow=true` without a prompt). When it can neither check nor write the workflow, such as when it is not run from a checkout of the repository, it stops unless you pass `allow_unverified=true`. +3. For a private repository, offers to seal a credential for the cluster to fetch each PR with (`credential=true`). +4. Reads the repository's `HARPER_PREVIEW_TARGET` variable. One that names another cluster stops setup. One it cannot read, such as when `gh` is not installed, stops it unless you pass `allow_unverified=true`, and you then set the variable yourself afterward. +5. With `credential=true`, asks for the token and checks that the cluster can hold a sealed secret. +6. Calls [`enable_previews`](../operations-api/operations.md#enable_previews), which creates the role, the user and the two trust policies, and stores the application's preview limits. +7. With `credential=true`, seals the token on your machine and grants it to every preview of the application, as the secret `deploy.-pr-_.git.github.com`. +8. Writes the starter workflow, if step 2 said to. Commit it to the default branch: GitHub runs `pull_request_target` workflows from there. +9. Sets `HARPER_PREVIEW_TARGET` to the cluster's URL with `gh`, if the repository doesn't have it. + +`enable_previews` only creates. Running setup again with the same arguments leaves the cluster as it is, and one whose records differ from what is already there stops with a `409` that names each difference. To change the limits, run `harper preview teardown`, then setup again. + +| Parameter | Default | Description | +| --------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `app=` | required | The application to preview. Its previews deploy as `-pr-`. | +| `target=` | your login's target | The cluster. Its URL, with port and trailing slash, is the trust policies' `audience` and the value of `HARPER_PREVIEW_TARGET`. | +| `repo=/` | the checkout's `origin` | The GitHub repository. | +| `repository_id=` | looked up | The repository's numeric id, for when setup can't look it up (`gh api repos// --jq .id`). | +| `branch=` | the repository's default branch, else `main` | The branch the workflow runs from. The trust policies pin the workflow at this branch. | +| `workflow=` | `.github/workflows/preview.yaml` | The workflow file, a `.yml` or `.yaml` file in `.github/workflows/`. | +| `environment=` | `preview` | The GitHub environment of the deploy job. The cleanup job's is always `preview-cleanup`. | +| `domain=` | `app.` | The domain every preview host sits one label under. Required when the target is an IP address or `localhost`. | +| `dataMode=branched\|shared` | `branched` | `branched` requires each preview to run on its own branch of the databases. `shared` lets previews use the cluster's databases, and the starter workflow then deploys with `branchedDatabases=false`. | +| `maxPreviews=` | `deployment.previews.maxPerApplication` | How many previews of the application each node keeps live at once. A deploy past it is refused. Without it, the node's [`deployment.previews.maxPerApplication`](../configuration/options.md#deployment) applies, which has no default. | +| `maxTtlDays=` | `7` | The longest a preview may live after its last deploy. A later expiry is lowered to it. | +| `credential=true` | asked, for a private repository | Seal a GitHub token for the cluster to fetch each PR's commit with. | +| `write_workflow=true` | asked | Write the starter workflow when the file doesn't exist. | +| `allow_unverified=true` | `false` | Go on when setup cannot check the workflow or the repository variable, or when the workflow has a step it cannot verify. Check the workflow yourself first: it is what keeps forks and PR code from the token. | + +##### What setup checks in the workflow + +The trust policies pin the workflow file at the default branch, `pull_request_target`, and an environment, so a PR cannot change what runs with the preview identity. Whatever that workflow does with the identity is up to the file, so setup reads it first. + +It looks only at the jobs that hold the token, those with `permissions: id-token: write`, and at reusable-workflow jobs. It refuses, whatever you pass: + +- a workflow that cannot match the trust policies: no `pull_request_target` trigger, a branch filter that leaves out the default branch, no job in the `preview` environment with `id-token: write`, or a job in `preview-cleanup` without it; +- a job that a fork's PR could run: one with no condition such as `github.event.pull_request.head.repo.full_name == github.repository` on it; +- a job that checks out or fetches the PR's head where the file shows the head outright: `gh pr checkout`, or a `git` checkout, switch, fetch, pull, worktree, clone or reset, or an `actions/checkout` `ref` or `repository`, that names the head on its own line, or through a variable whose `env:` value names it and that no script in the job could change. + +It stops unless you pass `allow_unverified=true` for what it cannot decide: a same-repository condition in a form it doesn't recognize, or one inherited through `needs`; a head reached through a shell variable, and any `run` step with `git` fetch or checkout code that also takes any input (`$`, a backtick, `<`, `|` or `xargs`), even one that fetches only the base branch; an `actions/checkout` `ref` or `repository` given as an expression; and a reusable-workflow job. It also notes, without stopping, a workflow whose cleanup may not run for a closed PR: one with no job in `preview-cleanup`, one whose `types` leave out `closed`, or one whose cleanup job's condition, or that of a job it needs, excludes `closed` or says more than setup can read. + +The check reads the workflow file only. It does not read scripts or local actions the workflow runs, and a head commit fetched without `git`, such as a downloaded tarball, is not caught. + +#### `harper preview teardown` + +```bash +harper preview teardown app=my-app +``` + +Calls [`disable_previews`](../operations-api/operations.md#disable_previews): it revokes the preview identity by dropping its trust policies and deactivating its user, drops every preview of the application on every node, then removes the user and the role, and prints what each node dropped. If a node fails or does not answer, it stops with the identity revoked; run it again to finish, which you have to before setup can enable previews again. It leaves the GitHub side as it is: the workflow file, the `HARPER_PREVIEW_TARGET` variable, and the GitHub environments. A credential setup sealed is left in the secrets store too; remove it with [`delete_secret`](../operations-api/operations.md#delete_secret). + +#### `harper preview deploy` + +Deploys or updates the preview of one PR. This is what [`HarperFast/deploy-action`](https://github.com/HarperFast/deploy-action) and the starter workflow run, as the preview user: + +```bash +harper preview deploy app=my-app pr=42 branch=fix/cart by_ref=true ref= +``` + +`app`, `pr` and `branch` are required. Every argument other than the ones below goes to [`deploy_component`](../operations-api/operations.md#deploy_component) as it does for [`harper deploy`](#harper-deploy), such as `by_ref`, `ref`, `package`, `credentials` and `restart`. With neither `by_ref` nor `package`, it packages and uploads the current directory. With `by_ref=true` on `pull_request_target`, always pass `ref=` the PR's head commit: without it, `by_ref` deploys `GITHUB_SHA`, which for that event is the base branch's commit. + +| Parameter | Default | Description | +| ------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `app=` | required | The application. The preview deploys as `-pr-`. | +| `pr=` | required | The PR number. | +| `branch=` | required | The PR's head branch, which names the preview's host. | +| `domain=` | `app.` | The preview host's domain: the one setup used, or the cluster refuses the deploy. Required when the target is an IP address or `localhost`. | +| `label=