Skip to content

Retry topic creation in the live Kafka e2e when librdkafka retires a connection - #146

Merged
rodrigobnogueira merged 1 commit into
mainfrom
fix/kafka-e2e-retired-broker
Oct 26, 2026
Merged

rodrigobnogueira merged 1 commit into
mainfrom
fix/kafka-e2e-retired-broker

Conversation

@rodrigobnogueira

Copy link
Copy Markdown
Contributor

Summary

  • The live Kafka e2e retries topic creation when librdkafka retires the connection it bootstrapped through. The race failed main's live-broker job once on 2026-10-22 (run 39716997661) and passed on a rerun.
  • No compatibility or release-process change.

Changes

  • createSubscribedTopics() in test/integration/reliable-messaging.kafka.spec.ts retries createTopics up to five times, with a short backoff, when every per-topic error is ERR__DESTROY_BROKER ("Broker handle destroyed without termination"). Any other error still fails the spec at once.
  • Why that error: once librdkafka learns the broker list from the cluster, it retires the bootstrap connection and fails the requests still queued on it. It retries its own metadata requests on that error (rdkafka_metadata.c), but admin requests surface it, so a CreateTopics sent right after connect can lose the race.
  • CHANGELOG.md entry under Unreleased.

Modules Touched

  • organizations / users / memberships
  • projects / tasks
  • activity (feed read-model, projections)
  • audit-log
  • outbox / inbox (the official outbox: store, handlers, consumers, worker)
  • Kafka profile (consumers, transport)
  • reminders (deferred + scheduled jobs)
  • events-catalog (asyncapi) / assistant (ai-sdk)
  • cache (read caching, tag invalidation)
  • auth / context / lockout
  • trpc (routers, generated schema)
  • database (schema, migrations, wiring)
  • Tooling / CI / docs only

Public Surface (libraries)

  • No use of library internals introduced
  • Roster unchanged, or the roster checklist was followed (README,
    package.json description, docs/, repo About — npm run docs:check)

Security Review

  • No unresolved high-risk security finding remains (test-only change)

Dependency Review

  • No dependency or lockfile changes

Migrations

  • No schema changes

Validation

  • git diff --check
  • npm run typecheck
  • npm run lint
  • npm run complexity:check
  • npm run test:cov
  • npm run security:audit
  • npm run build
  • npm run smoke

Validation Notes

  • The retry only runs against a live broker, so the "Kafka profile e2e (live broker)" job is the check that exercises it.

Release Notes

  • Release impact: CHANGELOG.md updated

…connection

createSubscribedTopics() now retries createTopics up to five times when
every per-topic error is ERR__DESTROY_BROKER ("Broker handle destroyed
without termination"). Once librdkafka learns the broker list from the
cluster it retires the connection it bootstrapped through and fails the
requests still queued on it. It retries its own metadata requests on that
error but not admin requests, so a CreateTopics sent right after connect
can lose the race. It failed main's live-broker job once, on 2026-10-22,
and passed on a rerun. Creating a topic that already exists succeeds, so
retrying is safe.
@rodrigobnogueira
rodrigobnogueira force-pushed the fix/kafka-e2e-retired-broker branch from 7163825 to 8771d8a Compare October 26, 2026 02:15
@rodrigobnogueira
rodrigobnogueira merged commit e4c87f2 into main Oct 26, 2026
3 checks passed
@rodrigobnogueira
rodrigobnogueira deleted the fix/kafka-e2e-retired-broker branch October 26, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant