What Lies Between: The Challenge of Operationalising Microservices

The biggest challenge in operationalising microservices is managing the space between them. This is the land of distributed systems: uncertainty and non-determinism. I will present practical approaches that you can use to take microservices into production or increase the value provided by existing systems. I will explore how to integrate microservices at scale, including asset management, security considerations, and representing uncertainty in data. I will examine approaches that can be used to debug, monitor, adapt, and control microservices, and I will expand on why I do not like calling this observability. Finally, I will detail tools and models for handling failure, including how to avoid making reliability worse, rather than better. I will conclude with some thoughts on how the space between microservices, and the challenges it introduces, depends on scale and perspective.

What is the work you’re doing today?

I build and operate software platforms for monitoring, aggregating, optimizing, and controlling renewable-energy assets, including energy production, battery storage, grid services, and electric-vehicle charging. These platforms leverage microservices and reactive-systems principles. I am most focused on mentoring and growing teams, and supporting these teams in their technical decisions.

Tell us about the projects you've been working on lately.

One of the most interesting things that I work on is the aggregation and control of batteries; taking thousands of batteries, in people's homes, and making them look like one power plant. It involves low-latency, closed-loop control, at scale, over the Internet, and all the problems that come with it. Using Scala, Akka, and Akka Streams for building the microservices in these systems has been a really great fit for the domain and has resulted in extremely robust services.

What are your goals for the talk?

Designing and building microservices is not easy, but I think the hardest part of operationalising microservices is in effectively managing the space between them. I will present some of the challenges that I have encountered, along with practical techniques that can be used to manage the relationships among microservices, at scale.

What do you want people to leave the talk with?

An appreciation for the operational challenges that are inherent in the relationships among microservices. I want to inspire people to use a systems-engineering approach to tackling these problems.

What do you think is the next big disruption in software?

I am not very good at predicting the future, so I probably shouldn't try. Perhaps the next big disruption is already here, but it remains underutilized, and that's cloud services. A lot of organisations run their services on cloud providers, but they just use the provider for easily provisioning compute and storage. I still see a lot of organisations building infrastructure and managing infrastructure, and not necessarily doing it very well. The opportunity cost can be huge, but is often ignored. Many cloud services are just fundamentally superior technologies for operating in the cloud. They are engineered for the cloud and they are not just service adaptations of something we could otherwise run on VMs in a data centre. I think a lot of organisations could operate with smaller engineering teams that are more focused on the core business, rather than providing infrastructure or tooling, and these teams could be responsible for a larger number of services, yet still operate them very reliably. But to do so, organisations will need to fully embrace service providers and stop worrying so much about vendor lock-in, or the illusion of “lift-and-shift”.


Colin Breck

Sr. Staff Software Engineer @Tesla

Colin Breck has experience developing software infrastructures for the near real-time monitoring and control of industrial applications. At Tesla, he works on distributed systems for the monitoring, aggregation, optimization, and control of distributed-energy assets, including solar generation,...

Read more
Find Colin Breck at:


Fleming, 3rd flr.


Operationalizing Microservices: Design, Deliver, Operate


MicroservicesSilicon ValleyInterview Available


Slides are not available


From the same track

SESSION + Live Q&A Distributed Systems

Complex Event Flows in Distributed Systems

Event-driven architectures enable nicely decoupled microservices and are fundamental for decentralized data management. However, using peer-to-peer event chains to implement complex end-to-end logic crossing service boundaries can accidentally increase coupling. Extracting such business logic...

Bernd Ruecker

Co-founder and chief technologist @Camunda

SESSION + Live Q&A Reactive Programming

Reactive Systems Architecture

Reactive systems architecture promises resilience and scalability, but building and maintaining a globally distributed system introduces considerable challenges. Jan and Matt will share the most important building aspects of systems that spread over multiple data centres as well as multiple AWS...

Jan Machacek

Senior Principal Engineer @waltdisneyco & Founder @muvrhq

Matthew Squire

Technical Team Leader @BamtechMedia

SESSION + Live Q&A Infrastructure

Lessons From 300k+ Lines of Infrastructure Code

This talk is a concise masterclass on how to write infrastructure code. I’ll share key lessons from the “Infrastructure Cookbook” we developed at Gruntwork while creating and maintaining a library of over 300,000 lines of infrastructure code that’s used in production by...

Yevgeniy Brikman

Co-founder @gruntwork_io

SESSION + Live Q&A Microservices

Cultivating Production Excellence - Taming Complex Distributed Systems

Taming the complex distributed systems we're responsible for requires changing not just the tools and technical approaches we use; it also requires changing who is involved in production, how they collaborate, and how we measure success.    In this talk, you'll learn about several...

Liz Fong-Jones

Site Reliability Engineer

UNCONFERENCE + Live Q&A Microservices

Microservices Open Space

Ian Robins

View full Schedule