[11.2.14] Federation Services connectivity failure and tenant alerts not loading

[Andrei-R-HancoGlobal]

Acknowledgements

• I have searched (https://github.com/utmstack/UTMStack/issues) for past instances of this issue
• I have verified that my UTMStack version is up-to-date

Describe the bug

Two issues are affecting SOC monitoring and alert management in UTMStack 11.2.14-enterprise, both still present on 29 September 2026:

1. Federation Services is no longer communicating correctly with connected client instances, preventing normal monitoring through the central panel.
2. One affected tenant cannot display its alerts, even when accessed directly through its individual URL. Alert counters and filter summaries populate, but the alert table fails to load.

Direct tenant access works for some other instances and provides a partial workaround for the Federation Services issue.

Client names and tenant URLs are withheld for confidentiality.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Federation Services should communicate with connected client instances and allow normal monitoring and alert management.

The tenant alert table should display the records matching the selected time range and status, consistently with the displayed counters and filter summaries.

Current Behavior

Federation Services is failing to communicate correctly with connected client instances.

On one affected tenant, direct access also fails to load the alert list under Threat Management → Manage Alert. The table displays "No data found" and the following error appears:

An error occurred while listing the alerts. Please try again later.

We captured these results in separate screenshots:

Selected period	Total alerts	Open	In review	Completed	Alert table
Last 7 days	1,913	1,080	6	827	No data found
Last 12 hours	1,089	1,089	0	0	No data found
Last 15 minutes	5	5	0	0	No data found

The counters and filter summaries change with the selected period, indicating that some data remains available while individual alert records cannot be displayed.

The 1,089 open alerts are counted within the last-12-hours filter. Reducing the range to the last 15 minutes returns only five matching alerts, but the listing still fails.

This demonstrates that the failure also affects small result sets and should not be attributed solely to a large accumulated alert backlog. The loading failure itself prevents us from reviewing and managing these alerts.

Last 12 hours

Last 15 minutes

Reproduction Steps

Federation Services issue:

1. Open the central Federation Services panel.
2. Attempt to access and monitor connected client instances.
3. Observe that communication/access through the federated panel is not working correctly.

Tenant alert-listing issue:

1. Access the affected tenant directly through its individual URL.
2. Navigate to Threat Management → Manage Alert.
3. Select All statuses and the last 12 hours.
4. Observe populated alert counters and filter summaries, while the table displays "No data found."
5. Change the time range to the last 15 minutes.
6. Observe that the counters update, but the alert list still fails and displays the alert-listing error.

At the time of testing, the last-15-minutes selection contained only five alerts.

Possible Solution

No confirmed fix or root cause is available.

Please investigate the Federation Services communication failure and compare the requests that populate alert counters/filter summaries with the request that retrieves the alert list.

Please also confirm whether either issue is related to the deployment of version 11.2.14 and provide a workaround and estimated resolution time.

Additional Information/Context

Centralised monitoring is disrupted, requiring technicians to check client instances individually. For the affected tenant, direct access does not resolve the alert-listing failure, preventing investigation, triage, and status updates.

The screenshots show that counters respond to recent time-range selections. They do not independently establish ingestion timestamps or confirm that all backend services are healthy.

We noticed that the affected instance is running version 11.2.14-enterprise and suspect recent downtime may have coincided with an upgrade or maintenance activity. This connection is not confirmed.

Our SOC was not aware of an announced maintenance window. Please confirm whether maintenance occurred, its timing, and any relationship to these issues.

We have previously requested advance notification of maintenance affecting availability. Please ensure future maintenance windows are communicated beforehand so we can plan monitoring coverage.

UTMStack Version

v11.2.14-enterprise

Operating System and version

v11.2.14-enterprise

Hypervisor and Version | Server Vendor and Model

N/A

Browser and version

Brave/Chome/Edge on latest versions prior to 29.09.2026

[apb1966]

UTMStack data-model drift across the 11.2.x upgrades: events.log.severityLabel used to be a string label (older alerts → mapped text) and newer versions write it as a number (e.g. 0, seen in today's Golden Ticket alerts → mapped long). So it's text in 113 older indices, long in
51 newer ones. When the alert view's child-alert ("echoes") lookup sorts/aggregates on that field, the 113 text shards throw ("text fields can't be sorted") → partial failure → 500 → "unable to list alerts."

[Andrei-R-HancoGlobal]

I forwarded your theory. I hope you got the reason behind the bug and it is going to be fixed soon.

[Klenk-IT]

Title: [11.2.14] Alerts table fails to load ("all shards failed" in getEchoesByParentIds) and 401 on /api/correlation-rule/search-by-filters

Environment

• UTMStack v11.2.14 (all services on v11.2.14, Docker Swarm), edition: [community / enterprise]
• Upgraded from: [v11.2.13] on 2026-09-29
• Host disk usage 47 %, all containers running and healthy after the upgrade (a few containers exited on first start and were restarted by Swarm, typical for start-up ordering)

Problem 1: Alerts table does not load

On the Alerts page the counters work (104 alerts in the last 6 hours), but the table fails with:

An error occured while listing the alerts. Please try again later.

So alerts are being generated and stored; only the listing fails. Logs and data sources are ingested normally and visible in the Log Explorer.

Backend log (repeats while the alerts page is open):

ERROR ElasticsearchResource.search: ElasticsearchService.getEchoesByParentIds: ElasticsearchService.search: OpenSearch.search: Request failed: [search_phase_execution_exception] all shards failed

The backend's own OpenSearch connection check reports "Success". The Health Check page shows 49 HTTP 500 responses with a mean response time of about 51 s, and about 24 s mean for HTTP 200.

Not verified: cluster and shard state, because the OpenSearch API requires credentials.

Looks related to #2772 and #2574.

Problem 2: 401 on /api/correlation-rule/search-by-filters

Every 5 minutes the event processor logs two warnings, for the rules "Microsoft 365 Mailbox Mass Item Access" and "Known Malicious IP Detected":

WARNING search-by-filters call returned error status ... "status":401,"error":"Unauthorized","path":"/api/correlation-rule/search-by-filters"

At the same timestamps the backend logs:

WARN InternalApiKeyFilter.doFilterInternal: Internal key presented for a non-internal path and ignored. method=GET path=/api/correlation-rule/search-by-filters

It looks like the endpoint is called by the event processor with the internal key (rule-flood-guard, cf. #2359), but the backend no longer treats this path as internal, so the request is rejected. The Health Check page shows 344 HTTP 401 responses.

Additional observation

Every 2 minutes:

ERROR UtmAlertTagRuleService.assignAssetGroupsToReviewAlerts: Cannot invoke "String.replace(CharSequence, CharSequence)" because "assetName" is null

Expected behavior

The alerts table loads, and internal calls from the event processor to the backend are accepted.

Happy to provide more logs if needed. (All times in the logs are UTC.)

[osmontero]

Thanks for the detailed report and the extra context in the thread — that helped a lot.

Alert listing failure (issue 2) — root-caused and fixed

We have confirmed this is a regression introduced in v11.2.14. The new alert view added a "child alerts / echoes" enrichment that runs on every alert-list request. That lookup performs an aggregation on the alert parentId field, and because the alert indices are created with OpenSearch dynamic mapping (no explicit field types), parentId is mapped as a text field. OpenSearch does not allow aggregating or sorting on text fields, so the shards holding those alerts fail and the whole listing request returns a 500 — which is why the table shows "No data found" while the counters (a separate query) still populate.

One clarification on the analysis above: the field that is actually failing the lookup is parentId, not events.log.severityLabel. (They are the same class of issue — a string dynamically mapped to text — but the child-alert aggregation runs on parentId.)

The fix (#2782) changes the lookup to aggregate on parentId.keyword. It is merged to v11 and released in v11.2.15.

Instances are updated automatically, so no action is required on your side — your instance will pick up v11.2.15 on its next automatic update and the alert table should load normally once it lands. We will monitor your instance to confirm the update has applied and then confirm here.

Federation Services connectivity (issue 1)

This is a separate issue and is still under investigation. We will follow up in this thread with the cause and fix once confirmed.