fix(syslogd): raise the imjournal rate limit so one noisy service cannot drop events

[SekiXu]

What type of PR is this?

/kind bug

What this PR does / why we need it

rsyslog loaded imjournal with only StateFile. One budget, the default 20000 messages per 600 s, then covered every journal entry. On cube36 (3.1.20) a keystone traceback flood used it up on all three control nodes. rsyslog then dropped everything after it, hex_log_event events included (bigstack-oss/cubecos#2064).

This sets Ratelimit.Interval="600" Ratelimit.Burst="300000" on the imjournal line in config_syslogd.cpp. journald still rate-limits each service on its own, so this budget is only a backstop.

How the value was chosen, per 600 s on cube36's three control nodes (14 days of /var/log/messages, 36 h of journal):

traffic	per 600 s
normal, without gunicorn, p50 / p99	703–821 / 876–1017
normal peak (cluster boot)	2819–3120
non-keystone spike (c1, ipmi_exporter)	23,236
keystone traceback flood, journal on c3	201,501

300000 is about 1.5× the largest flood window and about 100× the normal peak. Worst case it writes ~7.6 GB/day to /var/log/messages (~176 B per line). That is under the 5 % hex_trim_syslog cap (16.4 GB on cube36).

Validation

All on cube36 (CUBE_3.1.20_20261007-1922_ac7c3240), 2026-10-14. /etc/rsyslog.conf was edited by hand and restored afterwards.

• Under the live keystone flood. c3 was set to Burst="300000"; c1 and c2 stayed at the default as controls. gunicorn lines, journald vs /var/log/messages:

  window	node	journald	/var/log/messages	drop report
  17:13–17:25	c3 (300000)	228,820	228,820	none
  	c1 (default)	36,521	20,656	15992 messages lost
  17:25–17:35	c3 (300000)	153,884	153,884	none
  	c2 (default)	15,005	14,691	6419 messages lost

• One event end to end. hex_log_event -e SYS00001I from c3 and c1 reached the influx events DB about 5 s later.

• Negative control. The same event was sent on c1 while its default budget was used up. It is in journald, but it never reached /var/log/messages or the events DB.

• Controlled load on c3. 48,000 lines plus 6 probes were sent with the default budget: 19,267 lines and 0 probes arrived. With a raised budget, all of them arrived.

• Build. hex_config was built in the cubecos jail. It links, the changed file has no warnings, and the exact line is in the binary. rsyslogd -N1 accepts the line on rsyslog 8.2604.0-5.el9.

• Not tested: the line as written by hex_config on a real install; compute nodes; 3.2.0; the 2026-10-12 23:29 drop of ~4 M lines on all three control nodes at once (cause unknown; 300000 would not have absorbed it).

Which issue(s) this PR fixes

Refs bigstack-oss/cubecos#2064. This is Suggested Fix item 2 only. Item 1 (events on their own path) and item 3 (a drop raises its own event) remain outside this PR.

Special notes for your reviewer

Undo: easy — revert the commit; the old line comes back on the next commit and rsyslog restart. No data is written.
Affects: rsyslog on every node. During a flood, /var/log/messages grows instead of dropping lines, worst case ~7.6 GB/day, which hex_trim_syslog trims.
Please check: Commit() rewrites /etc/rsyslog.conf on every commit, but restarts rsyslog only when syslog.disk_percentage changes (config_syslogd.cpp:185-192). On an existing node, the new budget applies from the next rsyslog restart or reboot. This PR keeps that behavior.

After this merges, cubecos needs a hex submodule bump. The rgw auth-order change that removes today's flood source is bigstack-oss/cubecos#2089 (Refs bigstack-oss/cubecos#2065).

Additional documentation

Handbook known-issue note: bigstack-oss/bigstack-handbook#1065 (kb/cubecos/known-issues/syslog-flood-drops-cluster-events.md).

🤖 Generated with Claude Code