Streaming Log Analytics with Kafka

Humio is a log analytics system that is optimised to run in on-prem instances. From a software provider’s point of view, fixing issues in an on-prem solution is inherently problematic, and we have strived to make the solution simple.

To achieve this goal, a Humio installation consists only of a single process per node running Humio itself, being dependent on Kafka running nearby. We then use Kafka for two things: buffering ingest and as a sequencer of events among the nodes of a Humio cluster.

We love Kafka. It has made implementing a distributed system significantly easier than if we had to do the distributed queue from scratch. Using Kafka as a building block has allowed us to focusing our attention on building a fast and pleasant search experience.

In this technical talk, Kresten will discuss how and why we use Kafka internally and demo how we utilise it as a straightforward event-sourcing model for distributed deployments. Kresten will then present customer cases on utilising Kafka to manage and buffer massive volumes of data ingest.