Machine Learning Through Streaming at Lyft

What is the work you are doing today?

I work for a ride sharing company called Lyft. Our mission is to improve people's lives through the world's best transportation. For the last two and a half years, I have been working on the streaming platform team. In the ride sharing world, it is imperative to build the most recent state of the world and take decisions based on this information, things like determining prices based on supply and demand. We also want to fight fraud in real time. We want to determine the most optimum routes, things like that. My work here is aimed at making all that possible and easy. We want to enable streaming use cases across Lyft through Kafka, Kinesis, Apache Beam, Apache Flink. One of the problems we are trying to solve is making streaming available for machine learning use cases. We are building a self-service platform to make it easy for our Research Scientists and our Data Scientists to spin up streaming programs with no code, declaratively and deploy it and get it running in less than one hour and get results as new events come in, all in real time.

What are the goals for your talk?

One of my goals is to convey the power of event-driven programing and how it ties into the whole machine learning workflow. I also want to talk about some of the challenges that we faced when we were building this self serve streaming feature generation service. Things like bootstrapping state. How do you deal with data skew? How do you build a platform and provide reliability guarantees across the board when all the use cases are so different? And also things like schema evolution, managing high throughput computations while maintaining low-latency aspects. And I will be talking about how our platform evolved over the last two years and what our users can learn from our experience.

Can you give us a preview of the challenges and pitfalls you had?

One of the problems that I realized based on talking to people at conferences and meetups is that everyone is trying to solve the problem of bootstrapping state, especially in a streaming world. So, for example, if your state involves counting some numbers over a period of two days, for example, and if you start your program now, do you wait for those two days for it to have enough data to build that state or do you bootstrap it? If you are limited by things like streams retention periods, bootstrapping state can be challenging. This is something that I will be talking about, how we solved that problem at Lyft. That is one of the things. The other is the operational aspects of building a self service streaming system. How do you scale such a big system? How do you manage deploys with the main goal of abstracting away all those complexities from our users? I will be talking about how Kubernetes came to our rescue and how we incorporated that into our deployment workflow.

What are the key takeaways you would like to provide to the audience?

The main thing that I hope people take away from this talk are ways to tackle some common problems in the streaming world, things like bootstrapping state. Why it is required, why it is important to solve. Also, dealing with things like data skew. These are common problems that almost everyone faces. Data skew can bring up these interesting challenges - like exponentially increasing state size among other things. Data recovery and backfills is another common challenge that almost everyone is trying to solve. The problem of maintaining feature freshness as well as correctness when outages inevitably happen. I hope people can learn from our experiences and find some valuable takeaways from our approach.