Monitoring All the Things: Keeping Track of a Mixed Estate

What is the work you're doing today?

I'm a Principal Engineer on the FT's reliability engineering team. Our main goal is to assist the other teams around the business to help them build stuff that is secure and reliable. That involves us building tools and helping them. Also, a lot of talking to people and giving them advice around what approaches to take. Myself, I do a mixture of coding, tech leading and having discussions with other teams about what we build.

Do you work with monitoring, using specific tools there, or is it about coding integrations?

We have a range of tools across the FT, including some older tools like Nagios, which we still support for the older systems within. Newer stuff tends to use things like CloudWatch and Graphite/Grafana and also Pingdom. We also have some internal tools.

What can people expect from this talk?

I've been to talks before about monitoring. And often they focus very much on a single consistent estate: be that running in the same container platform or all using the same programing language. There's lots of nice, neat tricks for monitoring things when they're all consistent. But the problem I've often faced being in an organization that has more than one team, especially where each team has their own autonomy, you end up with vastly different states. We're not a startup. We've been around for a hundred and fifty years or more. We have legacy tech systems that we still need to support, that are still critical to the business. I want to talk about how you bring those different things together so that you can support the old and the new and the variety that you get in a real working organization.

When you say legacy technology, are you talking about mainframes or the older stuff?

We're talking about some stuff that's been deployed to physical racks that are sitting in a data center that we run ourselves. Even up until a few months back, we had stuff running in the office, but a recent office move means we finally migrated all that stuff off. There's older stuff that isn't the best understood throughout the company, but it's still very important to our operations. A variety of different systems in different languages, and I don't really know what the oldest one is, to be honest.

What are some of the challenges that you encounter?

One of the biggest challenges is looking at old monitoring systems and trying to understand the nuances, because people can tend to understand this is working and this is broken, but a lot of things have interesting failure states, and understanding what failure states you should alert on and what you shouldn't. What does it mean whenever something says it's a warning instead of an error, because different systems have a different understanding of those things and trying to bring them all together. You want the same user experience regardless of what monitoring system it came from. And I think that's actually the tricky bits. It's talking to all the different teams to understand what they mean by good and what they mean by a failure.

What do you want people to take away?

I want them to take away ideas about how they can approach these problems. There isn't one solution that's going to fit every organization. But I want them to have an idea of what they need to think about and things that might trip them up and useful techniques that you can apply to multiple systems so you can start to bring these things together and have a meaningful conversation with people around the organization about what they want to do.