Modern Data Pipelines in AdTech—Life in the Trenches

What is the focus of your work these days?

I mostly work on Captify's data pipelines, specifically in advertising. I joined the company almost three years ago. One of the main things for me now is, except for maintenance, the creation of new pipelines is adoption of some new tech and modernizing the pipelines, which they already have, and changes in the data infrastructure. so a lot of innovation as well.

And what's the motivation behind this talk?

Give more production level examples of what's happening for the Big Data engineers when they work on a specific use case. I noticed at various events that a lot of talks are more on the side of frameworks and how they can be used, but sometimes they may not give very practical understanding how to apply that to the pipelines other than if you build it, it will work. That's why I wanted to give this information to all the people that work on various pipelines so they would gain knowledge of how pipelines fail in production, what kind of issues can happen in general, what kind of pipelines can be used for specific tasks within the domain.

And how would you describe the persona and level of your target audience for this session?

I would describe this persona as a middle senior level Big Data engineer or an architect who is going to modernize the pipelines or just work in the company trying to understand how to improve the current data infrastructure and ecosystem that they have. Probably, there are issues with existing pipelines and they are trying to understand how to solve those issues. Also, I'd like to share some tips on how to approach various tasks.

What would you like this persona to walk away with after your session?

I would like this persona to walk away with a better understanding on how to approach the issues that they already have in the company, or just a new knowledge about advertising domains and the challenges that are out there and also, an understanding of the importance of being knowledgeable about the product. Not all of the approaches to the problems are a golden standard, but there are many approaches to the same problem, and the ones that I present reflect the experience of many engineers within the company where I work. But all of those things can also be improved in the future, and there are other approaches. So a conference is a great place to share our experiences within the community as well.