You are viewing content from a past/completed QCon -

SESSION + Live Q&A

Streaming Log Analytics with Kafka

Humio is a log analytics system that is optimised to run in on-prem instances. From a software provider’s point of view, fixing issues in an on-prem solution is inherently problematic, and we have strived to make the solution simple.

To achieve this goal, a Humio installation consists only of a single process per node running Humio itself, being dependent on Kafka running nearby. We then use Kafka for two things: buffering ingest and as a sequencer of events among the nodes of a Humio cluster.

We love Kafka. It has made implementing a distributed system significantly easier than if we had to do the distributed queue from scratch. Using Kafka as a building block has allowed us to focusing our attention on building a fast and pleasant search experience.

In this technical talk, Kresten will discuss how and why we use Kafka internally and demo how we utilise it as a straightforward event-sourcing model for distributed deployments. Kresten will then present customer cases on utilising Kafka to manage and buffer massive volumes of data ingest.


Speaker

Kresten Krab Thorup

CTO @MeetHumio (Humio)

Kresten provides technical leadership and vision at Humio. In his previous role as CTO of Trifork, Kresten was responsible for technical strategy and has provided consulting advice to teams on a variety of technologies to include distributed systems and databases, erlang, Java, and mobile...

Read more
Find Kresten Krab Thorup at:

Location

Fleming, 3rd flr.

Track

Architecting for the Cloud / Streaming Architectures

Topics

Apache KafkaCloud Computing

Share

From the same track

SESSION + Live Q&A Apache Kafka

Cloud Native Data Pipelines With Apache Kafka

As microservices, data services and serverless APIs proliferate in a cloud-native world, analysts still need to report on the business as a whole. Data engineers need to collect and standardize data in an increasingly complex and diverse system. Luckily, the problem is also the solution – the...

Gwen Shapira

Principal Data Architect @Confluent, PMC Member @Kafka, & Committer Apache Sqoop

SESSION + Live Q&A Architecture

Crossing the River by Feeling the Stones

Deng Xiaoping once described managing the economy as crossing the river by feeling the stones—in other words have a direction but be adaptive. But in a world of constant change, how do you determine the right thing to do? Which pebble to tread on? How do you understand where you’re going and...

Simon Wardley

Top UK Tech Influencer & Researcher & @lefep (Leading Edge Forum)

SESSION + Live Q&A Architecture

RSocket: Solving Real-World Architectural Challenges

It may seem surprising that a networking protocol can have such wide-ranging implications for applications are architected – but RSocket, with its unique flexibility and set of guarantees, is being used by some of the largest cloud-based companies to achieve order-of-magnitude reductions in...

Robert Roeser

Co-Founder & CEO @netifi_inc

Andy Shi

Developer Advocate @AlibabaGroup

Ondrej Lehecka

Software Engineer @Facebook

SESSION + Live Q&A Cloud Computing

A Guided Journey of Cloud Native Featuring Monzo

The Cloud Native Trail Map provides a path for enterprises to deploy cloud native applications, and the steps they’ll encounter along their journey. However, the needs of each enterprise are rarely the same and these journeys are rarely linear.   In this talk we’ll look at how the...

Cheryl Hung

Director of Ecosystem @CloudNativeFdn

Matt Heath

Distributed Systems Engineer @monzo

View full Schedule