Conversation
The AKS control plane (aksService) patches its own admission webhooks (the AKS node webhooks, Azure Policy and Gatekeeper) every few minutes from changing internal addresses. "Azure Kubernetes Admission Webhook Modified" grouped by six keys including the source address, so it kept opening new parents and stored a child alert for every patch, and the rule flood guard switched it off. The rule now raises one alert per data source, cluster (azureScope) and identity, and drops repeats for seven days (deduplicateBy). Its condition is unchanged, so an administrator or any other identity that creates or changes a webhook still alerts. azure_alert_volume_test.go pins the keys and checks that they resolve on fabricated AKS audit records. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
"Azure Kubernetes Admission Webhook Modified" produced 4,389 alerts between 2026-09-28 00:00 and about 17:45 UTC on 2026-09-29 on the one v11 deployment that sends AKS audit logs, 547 of them in its worst hour, and the rule flood guard switches it off. The target is fewer than 10 alerts per hour per rule on any server.
Every alert came from the AKS control plane identity
aksServicepatching six webhooks that AKS manages itself (the AKS node validating and mutating webhooks, the AKS webhook admission controller, and the Azure Policy and Gatekeeper webhooks), every few minutes, from five internal addresses. The rule grouped by six keys including the source address, so each address opened its own parent and every patch stored a child alert.Noise is reduced here only by de-duplication. The condition is unchanged, and no identity or webhook is excluded by name.
What changes
groupBylastEvent.dataSource, azureScopeType, azureScope, azureOperation, adversary.user, adversary.ipdeduplicateBydataSource, lastEvent.log.azureScope, adversary.user: one alert per data source, cluster and identity, repeats dropped for seven daysAn administrator or any other identity that creates, updates or patches a webhook still alerts, once per cluster per seven days; the control plane's reconciliation raises one alert per cluster per seven days instead of one per patch.
Expected volume
The Azure filter is loaded only since v11.2.14, so the deployment has 1.5 days of normalized AKS audit records (4,403 successful webhook patches, all by
aksService, one cluster). Replayed with the new keys and seven-day de-duplication: 1 alert, worst hour 1. The engine stores alerts that arrive in the same second in parallel, so the first alert can come with up to 5 copies here (the patches arrive in bursts of up to 23 per second); that engine limit is separate from this change.Tests
plugins/alerts/azure_alert_volume_test.go(new) runs fabricated AKS audit records through the Azure parser model and the pinned go-sdk v1.1.36 CEL. It pins the de-duplication keys and checks that they resolve for a control plane patch and an administrator's create, and that a read, aRequestReceivedstage, a denied patch and another resource do not match.go test ./...inplugins/alertspasses on this branch, which is based on currentv11(89cd26c4).8a3ade7, go-sdk v1.1.36, the production events and alerts plugins built fromv11dda9d45a, OpenSearch 2.19.1) with fabricated AKS audit records in two runs: a control plane patch, an administrator's create and four records that must not match, then two control plane repeats (another webhook, another internal address) and the control plane in a second cluster. Every run processed all of its events; one run hung after writing its output (the engine's plugin restart under memory pressure on the test machine), with complete events and alerts.🤖 Generated with Claude Code