Centralized logging is a good start to improving your log management as it allows collection, storage and analysis from multiple sources in a single repository, making it easier to manage and access your logs for dev, support, product & SRE teams as well as more easily meeting security and compliance requirements. Having centralized your logs, we would suggest you follow the best practices outlined below.

High-quality logs are the foundation of effective observability. Consistent, structured, and well-tagged log data allows teams to quickly identify performance issues, troubleshoot errors, and optimize cost and performance. Also in a world where AIs are starting to automate more and more, having clean, high quality logs opens up the door to further automation and efficiencies and as such becomes even more important as it enables additional benefits and new AI use cases.

If AI is defined as the intersection of where intelligence meets data … data quality is key in an AI world.

This guide outlines Bronto’s recommended best practices for log structure and context enrichment, team ownership, and agent configuration. Following these principles will help you improve data quality, searchability, and performance while building a sustainable, scalable observability culture across your organization.

‍

1. Log Structure and Context

Tags, log metadata, and message attributes are all key–value pairs (KVPs), but they serve different purposes and live at different levels of your event

• Tags – Properties that apply to an entire stream of events (a dataset).
  
  
• Log metadata – Properties added to individual log records typically by the logging agent or its plugins.
  
  
• Message attributes – Properties embedded directly in the log message itself.

‍

Tags: Properties of the Dataset

Tags apply to all entries in a stream of events while not visible as part of the log event.

They are ideal for separating environments at query time (e.g. avoid mixing staging and prod).
Examples of good tags:

• ^{environment=production}
• ^{account_id=12345678}
• ^{region=us-east-1}
  
  

In Bronto, we recommend setting tags via the agent configuration. These tags are applied to all data processed by that agent. Configuration management tools such as Terraform or CloudFormation can set these tags automatically.

Log Metadata: Properties of the Source

Log metadata are key–value pairs associated with a specific log but typically added by the agent (often via plugins), not by the application itself.

Log metadata usually describes:

• The host or node
  
  • e.g. ^{host_name=web-01}, ^os=linux
    
    
• The pod or container
  
  • e.g. ^{pod_name=api-6c8d3f5c2f-wz2vt}, ^{namespace=payments}
    
    
• The service name and version
  
  • e.g. ^{service=checkout-api, version=2.3.1}

A key point: the information in log metadata is not unique per agent. A single agent can process data from multiple hosts, pods, services, or versions, and the metadata will reflect those differences on a per-record basis.

‍

Message Attributes: Properties Inside the Log Message

Message attributes are key–value pairs present inside the log message body itself.

An example would be an application generating structured data, e.g. JSON:

^{{"level":"info","message":"request processed","duration_ms":123}}

Message attributes are:

• Authored by application developers
  
  
• Specific to a single log entry
  
  
• Ideal for capturing fine-grained, per-request context such as:
  
  
  • ^{duration_ms=123}
  • ^{request_id=abc-123}
  • ^{retry_count=2}

With Bronto, there are two supported options by default:

• Ensure the entire message follows the JSON format, or
• Use the key=value format within the log message (values may be quoted if needed; : can be used instead of =).]
• Beyond the default out of the box support, use the Bronto auto parser to convert your unstructured logs into KVP format automatically  - it uses  an LLM to build parsing rules so you don't have to.  

Note: Indexing is automatic in Bronto. Managing and configuring your indexes is a manual process in most logging systems and can be a time consuming and cumbersome task.

Exception and Stack Trace handling

• Use agent-side multiline support (e.g., FluentBit collector) to capture stack traces as single log events.
  
  
• Report exception name and stack trace as attributes:
  
  
  • ^{exception.type}
  • ^{exception.stacktrace}

This makes it easy to query and alert on recurring or unexpected exceptions.
‍

2. Correlation

Trace and Correlation IDs

Add fields like ^trace_id, ^span_id, and/or ^request_id to your logs so you can tie them back to a single user request or workflow across multiple services. In a distributed system a single call can pass through frontends, APIs, queues, and background workers; without a shared ID, the logs from each hop look like isolated events.

With a common ID, you can filter on that value and reconstruct the full timeline of “what happened where and when,” instead of guessing based on timestamps and hosts.

​​In terms of how to add them, it’s usually a combination of code and tooling:

• A tracing library or standard (such as OpenTelemetry) generates and propagates the trace and span context across service boundaries, and most logging frameworks can be configured to automatically include those IDs on every log entry.
  
  
• At the same time, you can use an application-level ^request_id or correlation ID (often taken from or added to an HTTP header at the edge) and pass it through your services. Your logging setup then adds that ID to every log line for that request.

A robust setup does both: use tracing context (^trace_id, ^span_id) and ensure they are consistently present in logs so any logging or observability system can correlate events end-to-end.

‍

3. Agent Configuration & Processing

The OpenTelemetry Collector and similar agents like Fluentbit, Logstash and Vector, can enrich, sanitize, and optimize log data.

Recommended Configurations

• Redact PII (personally identifiable information) before logs ever leave your infrastructure so you’re not streaming secrets, user data, or compliance risks into third-party systems. That usually means masking or dropping fields like emails, full names, IPs, IDs, and tokens at the agent or collector level, so even if logs are leaked or shared, sensitive data isn’t.
• Configure multiline stacktrace handling so that full exceptions are captured as a single log event instead of being split into many noisy lines; this typically means using a multiline rule that continues a record while lines match patterns like ^\s+at or similar. For concrete examples of how such a config can look in practice, you can point readers to the IBM Cloud Logs multiline docs and Instana stacktrace handling docs.
• Normalize log levels (e.g., ^info → ^INFO). Normalize log levels before you ship logs so that every event uses a consistent format. If you don’t, any breakdown by log level in your dashboards or reports will look noisy and fragmented. Instead of seeing a clean split like ^INFO / ^WARN / ^ERROR, you’ll see multiple tiny buckets such as ^info, ^Info, ^INFO, ^error, and ^ERR that all really mean the same thing.
• Use batch and memory limiter processors, for example with OTel:

batch

• Groups spans/logs/metrics into batches
• Improves throughput
• Reduces overhead and can lower end-to-end lag

memory_limiter

• Puts a hard cap on memory usage
• Drops data or throttles when usage exceeds thresholds
• Protects the host from resource exhaustion

Strike a balance: let agents fix inconsistencies from 3rd-party logs, but rely on developers to structure first-party logs correctly.

‍

4. Team Practices & Ownership

Why It Matters

Logging is not just a technical setup, it’s a shared responsibility across teams. Establishing clear ownership early ensures that logs are consistent, searchable, and actionable throughout your organisation’s lifecycle, and also makes it clear who is accountable for volume control (for example, leaving DEBUG on in production). With assigned ownership, you can better attribute value to logs, understanding which teams query which datasets, and which logs are actually driving high-value insights.

Best Practices

• Assign team ownership from day one.
  Each dataset or service should have a defined owning team responsible for log quality, metadata, and alerting setup. This avoids confusion later when troubleshooting or optimizing costs. You can setup and associate teams easily in Bronto, see more here.
  
  
• Tag logs by team.
  Include a team or owner tag in metadata or agent configuration. This enables Bronto to group logs, usage metrics, and cost by a responsible team automatically. Ideally it is done at source but you can also add these in the UI in Bronto afterwards. This is particularly useful when trying to understand log volume and what teams are responsible for spikes in volume increases for example. In such circumstances Bronto will allow you to set up usage alerts so that a given team is notified if their volumes suddenly go off the charts avoiding blowing through your ingestion budget.

‍

• Encourage collaboration through shared queries.
  Make it a habit for teams to share saved queries, dashboards, and monitors within Bronto. Common examples:
  
  
  
  • “Error spikes by environment”
    
    
  • “Token usage per service”
    
    
  • “Slowest response patterns over 24h”
    
    
• Shared queries reduce duplication and foster best-practice discovery internally.
  
  
• Use team-based datasets.
  Group data logically — e.g., by service ownership rather than by underlying infrastructure — so each team can monitor the performance, health, and behavior of their own services without noise from unrelated systems. Logical grouping also makes it easier to isolate issues, identify trends specific to a service, and avoid mixing staging/production or cross-service signals that can distort analysis.
  
  
• Make accountability visible.
  In Bronto, use tags and naming conventions that make it clear who owns what (e.g., ^{team=payments}, ^{service=checkout-api}, ^env=prod). We can edit dataset tags in the UI via settings, tag a team, and then check what teams are ingesting the most data using our usage explorer feature.
  ‍
• Pro Tip: Building a strong observability culture that promotes best practices internally early creates long-term efficiency. Teams that own their data from the start rarely need a cleanup project later.
  ‍

5. Log Types and Strategy

Define what types of logs your organization will send to Bronto and how they’ll be categorized.

‍

Pro Tip: Review overlap between application and infrastructure logs to avoid duplication and unnecessary ingestion usage. An example would be if your app logs ^request_id, ^user_id, ^status, and ^latency, and NGINX/syslog already records ^status and ^latency, you can keep those fields in one layer and use ^request_id to correlate, instead of ingesting the same details twice.

‍