feat(cloudformation): serve BuildBot through API Gateway on the internal load balancer

CHANGELOG.md

@@ -11,6 +11,8 @@ Please ADD ALL Changes to the UNRELEASED SECTION and not a specific release
 ### Security
 - ECS task secrets are no longer passed as plain Environment values; they are injected through the ECS Secrets mechanism from the BuildBot Secrets Manager secret, referenced by the SecretsManagerArn parameter (#419)
 - /github/* webhooks are verified against the X-Hub-Signature-256 HMAC-SHA256 signature (GitHub:WebhookSecret) before the body is parsed. Rejected requests get 401 (413 for bodies over 25 MB, 400 if the body cannot be read) and are logged with the reason, delivery id, event, remote address and X-Forwarded-For, with client-supplied values sanitised. When the secret is empty or whitespace, requests are accepted unverified and a startup warning is logged (#422)
+- BuildBot is served through an API Gateway HTTP API over a VPC Link to the internal load balancer instead of the public testing load balancer, exposing only the six application routes (#421)
+- Rejected webhook logs include the caller's IP address from X-Client-Ip, set by the API Gateway (#421)
 ### Added
 - BuildBot.GitHub - Increased code coverage to 100%
 - [BuildBot.CloudFormation] Increase code coverage to 100%
@@ -30,6 +32,7 @@ Please ADD ALL Changes to the UNRELEASED SECTION and not a specific release
 - Added missing IncludeAssets on NuGet package references required by FunFair.BuildCheck
 - Added missing build;buildTransitive IncludeAssets on Serilog package reference required by FunFair.BuildCheck
 - CloudFormation task settings now live in Mappings so every deploy applies them, reducing ECS task memory from 4 GB to 1 GB
+- On a new stack, ECS tasks no longer fail to start with a 'log group does not exist' error, and their final logs are kept when the stack is deleted, because the ECS application log group is now created before the tasks and removed after them (#421)
 ### Changed
 - Dependencies - Updated NSubstitute.Analyzers.CSharp to 1.0.17
 - Switched to use minimal APIs
@@ -81,6 +84,9 @@ Please ADD ALL Changes to the UNRELEASED SECTION and not a specific release
 - Dependencies - Updated Meziantou.Analyzer to 3.0.299
 - CLOUDFORMATION__AWS__REGION is now set from the stack's AWS region instead of being read from the AWS_REGION secret key (#419)
 - SDK - Updated DotNet SDK to 11.0.100-rc.2.26504.105
+- CloudFormation template restructured into the shared format, with parameters only where the deploy workflow passes them and all other settings in a Settings mapping (#427)
+- ECS application logs in /ecs/<ApplicationName>/ are kept for 30 days instead of never expiring, using the same LogRetentionInDays setting as the API Gateway access logs (#421)
+- The ECS service is now named after the application, so it can be found by name in the DiscordBots cluster (#421)
 ### Deprecated
 ### Removed
 - Removed NuGet audit suppressions for Scriban 6.2.0 vulnerabilities now that the transitive dependency has been dropped
@@ -89,6 +95,10 @@ Please ADD ALL Changes to the UNRELEASED SECTION and not a specific release
 - ECS service deployment circuit breaker with rollback, so tasks that fail to start (e.g. a missing secret key) fail the deploy quickly (#419)
 - ECS tasks now read their secrets at start-up using the ecsTaskExecutionRole execution role, which must have secretsmanager:GetSecretValue on the BuildBot secret (#419)
 - ECS task now gets GITHUB__WEBHOOKSECRET from the Secrets Manager key GITHUB_WEBHOOK_SECRET; the key must exist in the BuildBot secret (an empty value disables verification) or the task will not start (#422)
+- Before the first deploy, delete the existing buildbot.funfair.io Route 53 record; the stack then creates and owns it, with up to about 15 minutes of NXDOMAIN (#421)
+- This deploy replaces the ECS service, target group and listener rule (#421)
+- Payloads over 10 MB are rejected by API Gateway (#421)
+- The deploy user needs logs:PutRetentionPolicy on the /ecs/* log group, because the stack now sets a 30-day retention on it (#421)
 
 <!--
 Releases that have at least been deployed to staging, BUT NOT necessarily released to live.  Changes should be moved from [Unreleased] into here as they are merged into the appropriate release branch

COVERAGE.md

@@ -30,4 +30,4 @@ excluded
 
 ---
 
-Captured at commit `98111e91f39f085f8f99be5b8899fa3d35540c09` on 2026-10-15.
+Captured at commit `5838d6adf7b290fa6688493d7b506ecb2b56ab84` on 2026-10-16.

README.md

@@ -14,6 +14,59 @@ FunFair's Build Discord Bot
 |------------------------|-------------------------|--------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `GitHub:WebhookSecret` | `GITHUB__WEBHOOKSECRET` | string | (empty) | Shared secret configured on the GitHub webhook, used to verify the HMAC-SHA256 `X-Hub-Signature-256` header on `/github/*` requests. Requests with a missing or invalid signature are rejected with 401. When empty, signatures are not verified and a warning is logged at startup. In AWS it is read from the Secrets Manager key `GITHUB_WEBHOOK_SECRET`. |
 
+## Deployment
+
+BuildBot runs behind the internal load balancer. Webhooks (GitHub, Watchtower and CloudFormation SNS notifications) arrive through an API Gateway HTTP API, which forwards them over a VPC Link to the internal load balancer. The CloudFormation stack owns the API Gateway custom domain and the Route 53 `A` alias record for the API host name.
+
+### Request handling through API Gateway
+
+- Only these routes are exposed: `GET /ping`, `POST /github/ping`, `POST /github/push`, `POST /github/status`, `POST /watchtower/deploy` and `POST /cloudformation/deploy`. Any other route gets API Gateway's own 404 and never reaches BuildBot.
+- Payloads over 10 MB are rejected by API Gateway before they reach BuildBot.
+- The `$default` stage throttles every route to a burst of 20 and a steady 10 requests per second, and writes a JSON access log line per request (source IP, route, status, integration error) to the `/apigateway/<ApplicationName>/` log group, kept for 30 days. The ECS application log group (`/ecs/<ApplicationName>/`) is also kept for 30 days.
+- The real client IP address is in the `X-Client-Ip` header, which API Gateway overwrites so a caller cannot spoof it. `X-Forwarded-For` only holds the VPC Link's private address.
+- Authentication for new webhook sources (SNS in #424, Watchtower in #425) must use the `Authorization` header with basic auth. Never use `X-Api-Key`.
+- The VPC Link goes `INACTIVE` after 60 days without traffic. If webhooks stop arriving after a long quiet period, check the VPC Link status in the API Gateway console.
+
+### First deploy: DNS cut-over
+
+CloudFormation cannot import an existing Route 53 record set, so the existing `buildbot.funfair.io` record must be deleted by hand immediately before the first deploy that includes the API Gateway. Expect up to about 15 minutes of NXDOMAIN while resolvers' negative caches expire, and webhooks sent in that window may be lost.
+
+1. Find the hosted zone ID and the current record's exact values (a `DELETE` must match the existing record exactly):
+
+   ```shell
+   aws route53 list-hosted-zones-by-name --dns-name funfair.io. --max-items 1
+   aws route53 list-resource-record-sets --hosted-zone-id <HOSTED_ZONE_ID> --query "ResourceRecordSets[?Name=='buildbot.funfair.io.']"
+   ```
+
+   If records of other types (for example `AAAA`) exist for the name, delete them too, as the stack only creates the `A` record.
+
+2. Delete the record, substituting the values returned above:
+
+   ```shell
+   aws route53 change-resource-record-sets \
+     --hosted-zone-id <HOSTED_ZONE_ID> \
+     --change-batch '{
+       "Changes": [
+         {
+           "Action": "DELETE",
+           "ResourceRecordSet": {
+             "Name": "buildbot.funfair.io.",
+             "Type": "A",
+             "AliasTarget": {
+               "HostedZoneId": "<CURRENT_ALIAS_HOSTED_ZONE_ID>",
+               "DNSName": "<CURRENT_ALIAS_DNS_NAME>",
+               "EvaluateTargetHealth": <CURRENT_EVALUATE_TARGET_HEALTH>
+             }
+           }
+         }
+       ]
+     }'
+   ```
+
+3. Run the deploy straight away. The stack creates the `A` alias record pointing at the API Gateway custom domain. The record has a `Retain` deletion policy, so deleting or replacing the stack leaves it in place. Keep the output of step 1, so the old record can be restored with an `UPSERT` if the change has to be backed out.
+
+4. If the deploy fails and rolls back, the `Retain` policy also keeps the newly created record, which then points at the API Gateway custom domain that the rollback removed, and a retry fails because the record already exists. Repeat steps 1 and 2 (the alias values are now the API Gateway ones) before retrying the deploy.
+
 ## Changelog
 
 View [changelog](CHANGELOG.md)