Activity

1. Sparc0

   added

   bug

   on Jul 10, 2026
2. jansyren commented on Jul 13, 2026

   

   jansyren

   on Jul 13, 2026

   More actions

   PrometheusExporterImpl calls deployment-global recalculateCapacity() once per zone, causing O(zones) redundant full-infrastructure capacity recomputes on every scrape

   Summary

   PrometheusExporterImpl.updateMetrics() invokes alertManager.recalculateCapacity() inside the per-datacenter loop. AlertManagerImpl.recalculateCapacity() is deployment-global — it takes no zone/scope argument and recomputes capacity for all hosts, storage pools, pods and datacenters every time it runs. Calling it once per zone therefore repeats the entire global recompute N times per scrape, where N = number of zones.

   On a multi-zone deployment this makes the /metrics scrape duration scale with zones × (hosts + pools + pods) instead of hosts + pools + pods, and it runs synchronously on the scrape-serving thread.

   Affected code

   plugins/integrations/prometheus/.../PrometheusExporterImpl.java, updateMetrics():

   for (final DataCenterVO dc : dcDao.listAll()) {
       final String zoneName = dc.getName();
       final String zoneUuid = dc.getUuid();
       alertManager.recalculateCapacity();        // <-- deployment-global, called once PER ZONE
       addHostMetrics(latestMetricsItems, dc.getId(), zoneName, zoneUuid);
       addVMMetrics(...);
       ...
   }

   server/.../AlertManagerImpl.java, recalculateCapacity() — takes no zone parameter and iterates the entire deployment:

   public void recalculateCapacity() {
       ...
       recalculateHostCapacities();      // hostDao.listIdsByType(Routing) -> ALL routing hosts
       recalculateStorageCapacities();   // _storagePoolDao.listAllIds()   -> ALL storage pools
       List<DataCenterVO> datacenters = _dcDao.listAll();
       for (DataCenterVO datacenter : datacenters) { ... }   // ALL datacenters
       List<HostPodVO> pods = _podDao.listAll();
       for (HostPodVO pod : pods) { ... }                    // ALL pods
   }

   Because the method is global, the per-zone invocation in the exporter recomputes the same all-zones capacity set once for every zone.

   Impact / measurements

   Environment: CloudStack 4.22.0.0, 7 zones, 30 routing hosts.

   • Observed Com_select delta for a single /metrics scrape: ~20,839 queries.
   • Scrape duration climbs from ~0.9 s to ~4.0 s over ~3 days of uptime, resetting on management-server restart.
   • Thread dumps during a scrape show the serving thread predominantly in PrometheusExporterImpl.updateMetrics() at the recalculateCapacity() call site and in the downstream host-capacity recompute.
   • JVM heap is flat (no leak); MariaDB executes the individual capacity query in <1 ms. The cost is the sheer number of redundant recomputes per scrape, not slow individual queries or GC.

   With a default Prometheus scrape_timeout of 10s, a deployment on this trajectory eventually crosses the timeout and drops scrapes.

   Suggested fix

   recalculateCapacity() is deployment-global and is already run on a timer by the background capacity checker (capacity.check.period, default 300000 ms). The exporter does not need to force a synchronous recompute at all, and certainly not once per zone.

   Options, in order of preference:

   1. Remove the call from the exporter. Let the metrics read whatever the scheduled recalculateCapacity() background pass last wrote. Capacity gauges tolerate being up to capacity.check.period stale, and the scrape becomes a pure read.
   2. If a fresh recompute at scrape time is considered necessary, hoist it out of the loop so it runs at most once per scrape:

   alertManager.recalculateCapacity();                 // once, before the loop
   for (final DataCenterVO dc : dcDao.listAll()) {
       addHostMetrics(...);
       ...
   }

   Either change reduces the recompute work per scrape by a factor of N(zones) with no loss of correctness (single-zone deployments are unaffected).

   Secondary observations (not the primary defect)

   • addHostMetrics() iterates hostDao.listAll() for every zone and continues on hosts not in the current dcId, so all hosts are scanned once per zone. Filtering by datacenter at the DAO level (hostDao.listByDataCenterId(dcId) or equivalent) would avoid the repeated full scan.
   • Within the surviving per-host body, several single-row DAO lookups are issued per host (findByHostId, getHostTags, getHostStatistics, findByHostIdType ×3, listByHostId). These could be batched, though they are a minor cost next to the per-zone global recompute.

   Environment

   • CloudStack: 4.22.0.0
   • Hypervisor mgmt: KVM
   • DB: MariaDB
   • Zones: 7, Routing hosts: 30
   • JDK: OpenJDK 17

   Added some troubleshooting I have done regarding this with Claude Opus, at least it didn't destroy anything and it seems to have found some good pointers. I hope this will help since I am not a JAVA programmer I have to pass the torch to the experts.

   Best of luck and thanks for looking into this

   Jan
3. DaanHoogland

   removed

   bug

   on Jul 20, 2026
4. DaanHoogland

   added the

   Bug

   issue type on Jul 20, 2026
5. DaanHoogland

   added

   component:prometheus

   on Jul 20, 2026
6. DaanHoogland

   added this to the 4.22.2 milestone on Jul 20, 2026
7. DaanHoogland commented on Jul 20, 2026

   

   DaanHoogland

   on Jul 20, 2026

   Contributor

   More actions

   thanks, guys, an AI generated analysis:

   Root cause (code inspection, 4.22.0.0 tag):

   PrometheusExporterImpl.updateMetrics() calls alertManager.recalculateCapacity() once per datacenter inside its scrape loop:

   https://github.com/apache/cloudstack/blob/4.22.0.0/plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java#L486-L504

   AlertManagerImpl.recalculateCapacity() already loops over every zone, host, and storage pool internally:

   https://github.com/apache/cloudstack/blob/4.22.0.0/server/src/main/java/com/cloud/alert/AlertManagerImpl.java#L373-L376

   So with Z zones, a single /metrics scrape triggers Z redundant full-environment recalculations (O(Z²) work). Each call also spins up a brand-new Executors.newFixedThreadPool(...) per host and per storage pool (recalculateHostCapacities() / recalculateStorageCapacities(), lines ~296-368), then tears it down — so the thread-pool churn scales with scrape frequency, not with the intended once-per-capacity.check.period schedule from AlertManagerImpl's own CapacityChecker timer (start(), line ~264).

   On top of that, PrometheusExporterServerImpl.start() creates the JDK HttpServer without a custom executor (no setExecutor(...) call), so scrape requests are served through the server's single default dispatch path — a slow scrape blocks/queues the next one rather than running concurrently.

   Put together: every Prometheus scrape (typically every 15-60s) forces a full, uncoordinated, thread-pool-churning capacity recalculation across the whole environment, running concurrently with — and uncoordinated against — the existing timer-driven recalculation. That matches "duration climbs slowly over 7-10 days, restart clears it" well: no single scrape is catastrophic, but the accumulated thread churn and request queuing degrade the JVM over time until a restart resets it.

   I checked open issues/PRs for recalculateCapacity/prometheus and didn't find this tracked elsewhere, so filing this as new.

   Proposed fix (PR incoming): stop calling recalculateCapacity() from the scrape path — the exporter should read the capacity data the existing timer already maintains, not force a synchronous recompute per request — and have AlertManagerImpl reuse one long-lived executor for capacity calculation instead of creating/tearing one down per call.

   Can you confirm on your end? If you're able to test a build with alertManager.recalculateCapacity() commented out of updateMetrics(), that would confirm this is the whole story before we merge. In the meantime, reducing your Prometheus scrape interval (and/or the periodic restart you're already doing) is the only workaround.
8. DaanHoogland

   modified the milestones: 4.22.2, 4.20.4 on Jul 20, 2026
9. DaanHoogland

   linked a pull request that will close this issueoptimise prometheus scraping workflow #13650on Jul 20, 2026