From 61da981cd8d05c4e3158390033657e7e8523fc3e Mon Sep 17 00:00:00 2001 From: Seki Xu Date: Wed, 14 Oct 2026 16:36:47 +0800 Subject: [PATCH] fix(syslogd): raise the imjournal rate limit so one noisy service cannot drop events imjournal was loaded with only StateFile, so the default budget of 20000 messages per 600s applied to the whole journal. On cube36 (3.1.20) a keystone traceback flood used it up on every control node and rsyslog dropped everything after it, including hex_log_event events (cubecos#2064). Set Ratelimit.Interval=600 and Ratelimit.Burst=300000. Measured on cube36's three control nodes from 14 days of /var/log/messages and 36h of journal: normal traffic peaked at about 3,100 messages per 600s (cluster boot), and the keystone flood reached about 200,000 per 600s. journald still limits each service on its own; this budget stays as a backstop. Tested on cube36 by editing /etc/rsyslog.conf by hand (rsyslogd -N1 accepts the exact line), then rolled back: - c3 at 300000 under the live keystone flood, c1/c2 at the default as controls. gunicorn lines journald -> /var/log/messages, 17:13-17:25: c3 228,820 -> 228,820 with no drop report; c1 36,521 -> 20,656 with "15992 messages lost". - hex_log_event SYS00001I reached the influx events DB from c3 and c1. Sent again on c1 while its default budget was used up, it was in journald but never reached /var/log/messages or the events DB. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Seki Xu --- src/modules/config_syslogd.cpp | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/modules/config_syslogd.cpp b/src/modules/config_syslogd.cpp index 075da60..ca0aece 100755 --- a/src/modules/config_syslogd.cpp +++ b/src/modules/config_syslogd.cpp @@ -34,7 +34,13 @@ UpdateConfig(const char* filepath) return false; fprintf(fout, "module(load=\"imuxsock\" SysSock.Use=\"off\")\n"); - fprintf(fout, "module(load=\"imjournal\" StateFile=\"imjournal.state\")\n"); + // imjournal applies one budget to every journal entry, so a single noisy service + // used up the default 20000 per 600s and hex_log_event events were dropped with + // the rest (cubecos#2064). journald already rate-limits each service on its own; + // this budget is only a backstop. cube36 (3 control nodes, 14 days): normal + // traffic peaked at ~3,100 per 600s at boot, a keystone traceback flood at + // ~200,000 per 600s. + fprintf(fout, "module(load=\"imjournal\" StateFile=\"imjournal.state\" Ratelimit.Interval=\"600\" Ratelimit.Burst=\"300000\")\n"); fprintf(fout, "\n"); fprintf(fout, "global(workDirectory=\"/var/lib/rsyslog\")\n"); fprintf(fout, "\n");