[hazdl]

Analysis Platform AP-1370

Add four Prometheus alert rules to monitor CPU and memory usage for containers in the workflows namespace.
The alerts detect when container resource usage approaches configured limits, providing early warning of resource pressure and critical alerts when usage becomes very high.

Proposed alert thresholds
These thresholds are sensible starting points and should be reviewed against actual workload behaviour. They are not confirmed project requirements.

Alert	Threshold	Duration	Severity
Workflow CPU high	Above 80% of CPU limit	10 minutes	Warning
Workflow CPU critical	Above 95% of CPU limit	5 minutes	Critical
Workflow memory high	Above 80% of memory limit	10 minutes	Warning
Workflow memory critical	Above 95% of memory limit	5 minutes	Critical

Implementation
Add the four alert rules to alert-rules/alert-rules.tpl.yaml.

• Use Prometheus container CPU usage metrics and Kubernetes container resource limits to calculate CPU saturation.
• Use container memory working-set metrics and Kubernetes container resource limits to calculate memory saturation.
• Group usage and limits by namespace, pod and container so each alert identifies the affected workload.
• Restrict the rules to the workflows namespace and containers with configured resource limits.
• Retain the existing Helm templating and ConfigMap generation mechanism so the rules can be managed through the monitoring chart and deployed by Argo CD.

Validation
Helm lint: Passed.
Helm template rendering: Passed.
Alert rule presence: All four rules found in rendered manifests.
Threshold verification: Confirmed the configured percentages, durations and severities.
Deployment verification: Report separately after checking the live ConfigMap and Prometheus rule loading.

Notes:
The rendered rules have been validated locally; their presence in the live Prometheus configuration should be verified after the changes are merged and reconciled by Argo CD.

Will re-review once focus of alerts is decided

[JamesDoingStuff]

I'll reiterate that I think the alert info and group should be edited to remove the overloaded word 'workflow', but otherwise I think this is good

[hazdl]

Removed the 'workflow' , I just exclude visit namespaces.

[hazdl]

Updated the alert group name to reflect core service resource saturation.