Skip to content

fix(azure): one admission webhook alert per cluster and identity - #2785

Draft
kryonsx wants to merge 1 commit into
utmstack:v11from
kryonsx:codex/azure-alert-noise-20260929
Draft

kryonsx wants to merge 1 commit into
utmstack:v11from
kryonsx:codex/azure-alert-noise-20260929

Conversation

@kryonsx

@kryonsx kryonsx commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Why

"Azure Kubernetes Admission Webhook Modified" produced 4,389 alerts between 2026-09-28 00:00 and about 17:45 UTC on 2026-09-29 on the one v11 deployment that sends AKS audit logs, 547 of them in its worst hour, and the rule flood guard switches it off. The target is fewer than 10 alerts per hour per rule on any server.

Every alert came from the AKS control plane identity aksService patching six webhooks that AKS manages itself (the AKS node validating and mutating webhooks, the AKS webhook admission controller, and the Azure Policy and Gatekeeper webhooks), every few minutes, from five internal addresses. The rule grouped by six keys including the source address, so each address opened its own parent and every patch stored a child alert.

Noise is reduced here only by de-duplication. The condition is unchanged, and no identity or webhook is excluded by name.

What changes

Before After
Alerts groupBy lastEvent.dataSource, azureScopeType, azureScope, azureOperation, adversary.user, adversary.ip deduplicateBy dataSource, lastEvent.log.azureScope, adversary.user: one alert per data source, cluster and identity, repeats dropped for seven days

An administrator or any other identity that creates, updates or patches a webhook still alerts, once per cluster per seven days; the control plane's reconciliation raises one alert per cluster per seven days instead of one per patch.

Expected volume

The Azure filter is loaded only since v11.2.14, so the deployment has 1.5 days of normalized AKS audit records (4,403 successful webhook patches, all by aksService, one cluster). Replayed with the new keys and seven-day de-duplication: 1 alert, worst hour 1. The engine stores alerts that arrive in the same second in parallel, so the first alert can come with up to 5 copies here (the patches arrive in bursts of up to 23 per second); that engine limit is separate from this change.

Tests

  • plugins/alerts/azure_alert_volume_test.go (new) runs fabricated AKS audit records through the Azure parser model and the pinned go-sdk v1.1.36 CEL. It pins the de-duplication keys and checks that they resolve for a control plane patch and an administrator's create, and that a read, a RequestReceived stage, a denied patch and another resource do not match.
  • go test ./... in plugins/alerts passes on this branch, which is based on current v11 (89cd26c4).
  • Local EventProcessor lab (engine 8a3ade7, go-sdk v1.1.36, the production events and alerts plugins built from v11 dda9d45a, OpenSearch 2.19.1) with fabricated AKS audit records in two runs: a control plane patch, an administrator's create and four records that must not match, then two control plane repeats (another webhook, another internal address) and the control plane in a second cluster. Every run processed all of its events; one run hung after writing its output (the engine's plugin restart under memory pressure on the test machine), with complete events and alerts.
    • With this branch: 5 alerts produced and 3 stored (the control plane in each cluster and the administrator); both repeats were dropped as duplicates. None of the four records that must not match produced an alert.
    • With the rule as shipped: 5 produced and 4 top-level alerts plus 1 child stored; the repeat from another internal address opened a new parent.

🤖 Generated with Claude Code

The AKS control plane (aksService) patches its own admission webhooks
(the AKS node webhooks, Azure Policy and Gatekeeper) every few minutes
from changing internal addresses. "Azure Kubernetes Admission Webhook
Modified" grouped by six keys including the source address, so it kept
opening new parents and stored a child alert for every patch, and the
rule flood guard switched it off.

The rule now raises one alert per data source, cluster (azureScope) and
identity, and drops repeats for seven days (deduplicateBy). Its
condition is unchanged, so an administrator or any other identity that
creates or changes a webhook still alerts.

azure_alert_volume_test.go pins the keys and checks that they resolve on
fabricated AKS audit records.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant