Why is Bronto so fast at searching logs?

As we shared in a recent post, one of the main reactions we got at Kubecon Europe was about how quick our log search is. That post includes some internal benchmarks showing a variety of searches on up to 5 TB of log data completing in under a second.

In this post I wanted to explain why searching logs quickly is a hard problem to solve and share some insights into how Bronto has solved for that.

‍

Why is it hard to build a really fast log platform?

It is hard because it depends on the characteristics of  the log data, the size of the dataset and nature of the query. Logs are time-based, high-volume, varied and may be unstructured, semi-structured or structured. This makes it hard to store log data cost-effectively and enable fast search, especially if the log data has keys with high cardinality and/or the search involves 10s of TBs of log data.

Also, there is no such a thing as a “typical query” – it varies by customer, by team and by user. Sometimes users want to view events in chronological order and at other times view the results of analytics on the log data for a longer historical period. They also want to be able to search using queries with full words, partial words, combinations of words and conditional expressions.

Current approaches to logging platforms

There are currently two broad technical approaches to logging systems:

• Adapt generic technology from other areas, e.g. a text search engine like ElasticSearch or an analytics database like Clickhouse. Adapting these solutions can give quick results for some cases, but creates issues in handling logs, e.g. having to manage the inverted indexes in ElasticSearch to be performant or to reduce storage costs. 

• Use parallel systems built for Big Data - this had been based on systems like Hadoop (which scaled for large volumes, but could be inflexible), but has moved to a class of systems using object storage such as AWS S3 and massive parallelization using serverless functions like AWS Lambda. At one extreme, this may do limited processing at log ingestion, e.g. not create indexes, and rely on compression and large scale compute to give less costly and performant search.

‍

So how does Bronto do it differently?

Bronto was designed specifically to meet the challenges of Log Management outlined above and to be “the logging layer for the AI era”. As such, it is not based on other text search or database technologies. We have introduced specific innovations in storing and searching log data (such as adaptive indexing, custom search algorithms), have taken the best technology approaches available and followed good engineering practice (such as stateless computing) that allow us to both store the data efficiently at ingestion and enable cost-effective fast search. 

At ingestion time, we do this by:

• Time based file partitioning
• Adaptive indexing
  
  • Seamless search
• High performance storage formats
  
  • Columnar data storage 
  • Optimized data encoders/decoders
• Pre-computed log-based metrics for repeated searches, e.g. dashboards

At search time, we do this by:

•  Distinct event view and statistical search algorithms
•  Large scale parallelization  
  
  • Serverless search
  • Stateless service design 
  • Decoupled storage and compute

Time based file partitioning

Files are created to hold the log’s data and metadata for indexing with the filename based on the time when the files were opened. We use time based files as a specific technique to handle the temporal nature of logs which is different to the way that more generic databases store data. This allows Bronto to search only the relevant files for the time range specified in a query.

These log files can also be partitioned by users into collections to hold a related set of logs in a manner that aligns with a company’s system. 

Adaptive indexing 

We create multiple indexes automatically and in real time as each log event is ingested, without users needing to configure anything. The indexes are used when processing a query to determine whether to open a file to search its contents for matches to the query and to skip files where the index tells us there is no data for this query. 

The more files skipped, the faster the search completes, because we avoid processing most of the underlying log data. This is especially important when looking for rare events such as a problematic IP address that appears once or twice in millions of log events.

We use two types of index:

• summary index holds a dictionary for each key, containing the values of keys in the logs and the dictionary is managed according to the cardinality of the keys in the log event. It also holds statistical data for each key to make aggregate searches fast.

• bloom filter is a memory efficient probabilistic technique to determine if an entry may be in its associated data or is not in its associated data. We use a bloom filter associated with the log data, which can then be checked against the keys and values in a search query. It is particularly well suited when searching for rare events.

‍

‍

Our indexes allow us to easily handle the variety of keys in log data and its unstructured data without user configuration, unlike inverted indexes which need to be updated to handle new keys in a log.

‍

‍

Seamless search

We index ALL ingested data for fast search, unlike other solutions that reduce their costs by indexing only a subset of the log data such as the more recent data. Having ALL data in Bronto indexed means there is no need to move from cold to hot storage or re-index or rehydrate non-indexed data, as happens in other products. 

‍

High performance storage formats

Columnar data storage

Data is encoded at ingestion into a format based on its type and if the data is suitable for the columnar encoding, e.g. based on the cardinality of the events. Specialized columnar encodings are used for certain data formats, such as HTTP. A raw encoding is used if the events received did not match one of the defined schemas or were not suitable for columnar encoding. These encoders allow us to optimize storage and so reduce the amount of data transferred, giving improved performance when retrieving the stored files.

This use of column encoding contrasts with the row-based storage often used to store log events, which generally requires scanning a full table rather than the approach here where we only select columns matching the keys in the query.

Optimized encoding to reduce decode and decompress

Our columnar formats partition the data into a column per key, so that only relevant columns matching the keys in the search must be checked, but we also make search faster by only decoding and decompressing the data for keys of interest to the search query.

Pre-computed log-based metrics

Fast dashboards are required for viewing metrics derived from logs and this can be very demanding on a log search infrastructure, given the frequency at which queries will be made. Bronto pre-computes the metrics for dashboards at ingestion time and stores them in a time-series store for fast retrieval. Bronto also analyses non-dashboard search queries to determine if they are worthwhile to pre-compute and give users instant responses when they make the search.

‍

‍

‍

Distinct event view and statistical search algorithms

Users sometimes view log events in time order and sometimes view the results of some statistical function, such as the calculation of the number of specific errors. These two types of usage have different characteristics that can be exploited in search algorithms running on multiple parallel nodes:

• Statistical search does not require the events to be ordered by time, whereas events are viewed in time order and must be sorted

• Events are viewed as a “list of events”  as a single page of results at a time and these views do not usually require a search of all the data in the time range to populate that page, whereas statistical search must search everything in the time range. 

Hence, the statistical search algorithm performs its operations in parallel on time buckets across the time range (and avoids sorting), whereas the event viewing algorithm runs in parallel and stops when it has found enough matching events for the viewer.

‍

Large scale parallelization

Bronto uses on-demand parallel execution to deliver consistent speed when data volumes in a search or the overall load rises.

Serverless search for large searches 

Bronto’s search algorithms run in parallel, where the files in the time range are divided across multiple nodes with their output combined into the final result, similar to map-reduce. For smaller searches, these parallel searches run as multiple threads on a node, whereas we use Serverless computing for large searches.

Serverless computing (e.g., AWS Lambda) allows us to use a very high amount of computation to take advantage of our parallel search algorithms to process large volumes of data, e.g. we could run 10,000 Lambdas on a 10TB search. 

‍

Stateless service design

Bronto’s services use stateless principles allowing servers to scale horizontally as load varies, e.g. using AWS Auto Scaling groups (ASGs)

Decoupled Storage and Compute

Bronto’s architecture allows compute nodes to be independent of the storage used to hold those files. This decoupling gives us the flexibility to choose storage and compute nodes and technology based on the cost/performance of each independently, e.g. we use object storage (e.g. S3) for long-term storage and historical search, independently of whether search runs on nodes or serverless. 

‍

Conclusion

When we designed Bronto, we set out to solve the fundamental problems that have plagued logging platforms for years – not to be another iteration of the same compromised approaches. The combination of our time-based partitioning, adaptive indexing, columnar storage, and parallel processing creates a system that's genuinely built for the unique challenges of log data at scale. This isn't just incremental improvement; it's a ground-up reimagining of what a logging platform should be in the AI era.

What this means for you is simple yet transformative: you can search terabytes of logs in under a second, without compromising on retention or breaking your budget. No more trade-offs between speed, coverage, and cost. No more waiting for rehydration or managing complex retention policies. Just lightning-fast search across all your logs, all the time. Because when searching logs is this fast, teams actually use them more – turning your logs from high-volume, low-value data into a high-value asset for your entire organization.