Speaker: Ben Gamble

Product Champion @Ably


Ben Gamble leads Kafka initiatives as Product Champion at Ably, a protocol agnostic, globally distributed data streaming network. He’s spent the last 10 years in startups and high growth companies, bridging the gap between research and product development. Having worked with the cutting edge of Augmented reality, scaling 3D gaming, and wearable tech, he's no stranger to taking on technical challenges, and the commercial realities that entails. Ben now works to make real time data a reality for anyone who needs it, and make as much of it open as possible through Ably’s open data Hub.

Session + Live Q&A

Data Streams at Scale: Meeting Personalized User Demands in Real-Time

As the amount of data in Fintech and cryptocurrency applications has increased, Kafka has become the norm for wrangling this surge at speed and scale. However, the need to send personalised data streams to users has become paramount. Your mobile daytrader only wants what they can use, and neither you nor they want the data bill for everything they are not using. You also don’t want topics per user. This is where it started, but as we’ve encountered more Kafka users, the needs are increasingly similar - whether it's large scale chat systems, betting applications or social media. Data access patterns are not uniform - some streams are in very high demand, others are not, but all need to be handled fairly. In this session, I’ll walk you through designing a scaling consumer system to allow for customised user streams in high cardinality data topics, all while minimising the data burden on user devices. I’ll be demonstrating a few use cases of the system and running through the various tradeoffs that can be made to both improve user latency and add high availability to the system. Follow along to learn about the key tradeoff of user experience vs cost, latency vs flexibility, and the increasing demands of user- facing systems.


Tuesday Apr 5 / 11:50AM BST (50 minutes)


Video is not available

Add to Calendar

Add to calendar


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.