Speaker: Dr. Einat Orr

Version control is fundamental when managing code, but what about data? Our data changes over time, first since it accumulates, we have new data points for new points in time. But this is not the only reason. We also have additional data added to past time, since we were able to get additional data sources, or changed past data in light of new information that was late to arrive.

Since our data is mutable, version control of the data will allow us to ensure we can reproduce a set of results, provide us with the lineage between the input and output data sets of a process or a model, allow us to experiment, provide the relevant information for auditing, and assist us in production management. In this talk we will go over several technologies that version large data sets. We will understand the use cases they support and look under the hood at the technology developed to best support those use cases.

The complexity of tools, distributed systems, and the CAP theorem introduce tradeoffs that practitioners cannot avoid or ignore as they embrace the world of modern data pipelines. What strategies can you employ? This is where data products come into play. Understanding the business objectives of data products helps us make informed decisions about tools, architecture, and services. Join this panel to learn from data thought leaders!