Skip to content

[DS-3502] Report start-up failures as JSON errors, fix initializing.* logger category - #541

Open
MixalisV wants to merge 3 commits into
masterfrom
fix/DS-3502-startup-errors-json
Open

MixalisV wants to merge 3 commits into
masterfrom
fix/DS-3502-startup-errors-json

Conversation

@MixalisV

@MixalisV MixalisV commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Jira: DS-3502 · Incident: QA-8308 · Replaces the maestro workaround Workable/maestro#1206

Problem

Errors during orka start-up reach Datadog as info under initializing.<name>:

  • Before log4js(config), logs are plain text even with LOG_JSON=true, so Datadog merges them into the preceding start-up output, and the merged entry gets status info.
  • getLogger(name) keeps the temporary initializing.<name> category for the whole process.
  • json-appender only finds the error in the first argument, so logger.error('msg', { reason: err }) logs "undefined - msg" and no stack trace.

Changes

  • Preload: if dd-trace init fails with LOG_JSON=true, write one JSON error line and exit with code 1. dd-trace still loads first.
  • Before configuration: with LOG_JSON=true, logs are JSON from the start. Loggers use their plain name (app, orka).
  • Builder: start-up errors are logged as errors and re-thrown.
  • json-appender: takes the stack trace from an Error passed later or inside a context object.
  • Category cleanup: "Initializing riviere..." logs under orka.riviere, and the unused initializing log category default is removed.

Behaviour changes

  • With LOG_JSON, error logs whose first argument is a string change message from "undefined - msg" to "msg" and now include stack_trace.
  • App loggers taken before orka() follow log.level instead of the initializing category's level. This only matters if LOG_LEVEL is above debug.

Tests

  • New early-logging.test.ts:
    • preload failure, with and without LOG_JSON
    • builder failure
    • an app uncaughtException handler like maestro's
    • a logger taken before configuration
  • New json-appender cases for the error-finding rules.
  • 301 passing locally. The 2 failures are the Redis health examples, which need a running Redis (same on master).

Follow-up

  • maestro runs orka 4.4.0 and needs to upgrade to get this.
  • Then drop maestro's logError workaround.

🤖 Generated with Claude Code

… logger category

- Log a dd-trace preload failure as a single JSON line when LOG_JSON=true, then rethrow.
- Configure log4js with the JSON appender at load time when LOG_JSON=true, so logs
  emitted before log4js(config) are JSON instead of plain text.
- Log errors thrown during orka start-up and rethrow them.
- Move loggers taken before configuration from 'initializing.<name>' to '<name>'.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
MixalisV and others added 2 commits October 16, 2026 12:00
…ted errors in JSON logs

- getLogger(name) uses the plain name from the start instead of 'initializing.<name>', so
  errors logged before log4js(config) (e.g. an app's uncaughtException handler) are filed
  under the app's category. Configuring log4js still resets the early 'trace' level.
- json-appender takes the stack trace from an Error passed after the message or inside a
  context object (e.g. { reason: error }) and no longer prefixes such messages with
  "undefined - ".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ializing category

- With LOG_JSON=true, exit with code 1 after writing the preload failure instead of
  rethrowing, so Node doesn't print the stack again as plain text lines that each become a
  separate entry. Write the line synchronously so it isn't lost on exit.
- Log "Initializing riviere..." under orka.riviere and drop the now unused
  `initializing: 'debug'` log category default.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@klesgidis

Copy link
Copy Markdown
Contributor

Can you give an example of what is happening now? A link to the logs maybe

@MixalisV

Copy link
Copy Markdown
Contributor Author

Can you give an example of what is happening now? A link to the logs maybe

As you can see here,

The v0.317.0 start-up crash was logged as [ERROR] initializing.app - Uncaught Exception…, but Datadog marked it info and filed it under initializing.app.
Screenshot 2026-10-21 at 11 31 48 AM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants