What Lies Between: The Challenge of Operationalising Microservices

What is the work you’re doing today?

I build and operate software platforms for monitoring, aggregating, optimizing, and controlling renewable-energy assets, including energy production, battery storage, grid services, and electric-vehicle charging. These platforms leverage microservices and reactive-systems principles. I am most focused on mentoring and growing teams, and supporting these teams in their technical decisions.

Tell us about the projects you've been working on lately.

One of the most interesting things that I work on is the aggregation and control of batteries; taking thousands of batteries, in people's homes, and making them look like one power plant. It involves low-latency, closed-loop control, at scale, over the Internet, and all the problems that come with it. Using Scala, Akka, and Akka Streams for building the microservices in these systems has been a really great fit for the domain and has resulted in extremely robust services.

What are your goals for the talk?

Designing and building microservices is not easy, but I think the hardest part of operationalising microservices is in effectively managing the space between them. I will present some of the challenges that I have encountered, along with practical techniques that can be used to manage the relationships among microservices, at scale.

What do you want people to leave the talk with?

An appreciation for the operational challenges that are inherent in the relationships among microservices. I want to inspire people to use a systems-engineering approach to tackling these problems.

What do you think is the next big disruption in software?

I am not very good at predicting the future, so I probably shouldn't try. Perhaps the next big disruption is already here, but it remains underutilized, and that's cloud services. A lot of organisations run their services on cloud providers, but they just use the provider for easily provisioning compute and storage. I still see a lot of organisations building infrastructure and managing infrastructure, and not necessarily doing it very well. The opportunity cost can be huge, but is often ignored. Many cloud services are just fundamentally superior technologies for operating in the cloud. They are engineered for the cloud and they are not just service adaptations of something we could otherwise run on VMs in a data centre. I think a lot of organisations could operate with smaller engineering teams that are more focused on the core business, rather than providing infrastructure or tooling, and these teams could be responsible for a larger number of services, yet still operate them very reliably. But to do so, organisations will need to fully embrace service providers and stop worrying so much about vendor lock-in, or the illusion of “lift-and-shift”.