Logger: design for diagnostics in production

Logger Design: Prioritizing Diagnostics and Control in Production

In production systems, loggers aren't judged by API speed but by how quickly they help diagnose incidents without adding extra load. Logging introduces overhead, so it's minimized during normal operations. But when issues arise, skimpy details or chaotic logs make troubleshooting impossible. A good logger ensures diagnosability without sacrificing performance.

The trouble starts when log structure mirrors the code instead of business processes. In multithreaded systems, messages from different tasks get jumbled, making it hard to trace a specific request.

Tying Context to Tasks

The fix is associating logs with tasks, not call sites. Passing context via parameters clutters function signatures. Better to bind it to the execution thread.

In logme, this is done via thread channels:

auto requestCh = Logme::Instance->CreateChannel(Logme::ID("req-42"));
LogmeThreadChannel(requestCh);
HandleRequest();

Internal code logs to the channel automatically:

void ParseHeaders() {
    LogmeW("invalid header");
}

Logs now group by operation. In spdlog and Quill, context revolves around the logger, which is simpler but less ideal for tasks.

Another issue is noise from repeated messages. Retry errors flood logs with duplicates. Event-level limits fix this:

LogmeW(LOGME_ONCE4THIS, "disk is full");

The message appears once per class instance, keeping logs readable.

Flexible Message Routing

Global config isn't enough. In HTTP/2, you need to separate connection and request logs. A request channel links to the protocol channel, but request details shouldn't clutter the main log.

Logme::Override ovr;
ovr.Add.DisableLink = true;
LogmeI(ovr, "request-specific detail");

This isolates the message to the local channel. General events show everywhere; private ones stay local.

Subsystems for Precise Detailing

One dimension (channel) isn't sufficient. Tasks have subsystems: cache, network, parsing. Log selectively:

LOGME_SUBSYSTEM(cacheSid, "cache");
LOGME_SUBSYSTEM(parserSid, "parser");

LogmeD(cacheSid, "cache miss");
LogmeD(parserSid, "header parsed");

At runtime, enable just the needed subsystem, like cache. Others stay silent, keeping logs compact.

Integrating with Libraries

Third-party libraries have their own logging. We intercept and integrate:

void FfmpegLogCallback(void* ptr, int level, const char* fmt, va_list args) {
    char buffer[2048];
    vsnprintf(buffer, sizeof(buffer), fmt, args);

    Logme::Override ovr;
    ovr.Add.DisableLink = true;

    LogmeW(ovr, "[ffmpeg] %s", buffer);
}

Messages follow the same rules: channels, subsystems, deduplication.

Format support eases migration:

LogmeI("value=%i", value);
fLogmeI("value={}", value);
LogmeI() << "value=" << value;

logme blends fmt and stream styles, unlike spdlog (fmt-only) and Quill (async fmt).

Production Control Without Restarts

In always-on systems, logs stay minimal normally. During incidents:

• Suspect cache? Enable the cache subsystem.
• Isolate the task via channel.
• Limit repeats.

Control is dynamic via logmectl: levels, subsystems, no restarts needed.

Post-fix, dial back details. Otherwise, cranking up global logs causes overload.

spdlog emphasizes speed, Quill asynchronicity, logme message flow control while maintaining performance.

Key Takeaways:

• Loggers should group by tasks via channels, not loggers.
• Event-level deduplication cuts noise.
• Subsystems enable targeted runtime detailing.
• Library integration standardizes output.
• Dynamic control without restarts is essential for production.

— Editorial Team