Speaker: Jessica Kerr

At Honeycomb, customers send us lots of data and then compose complex, ad-hoc queries. Most are simple, some are not. Some are REALLY not; this load is both complex, spontaneous, and urgent. It would be prohibitively expensive to size a server cluster to handle these big queries quickly, so we took a different approach: farm the work out to Lambda, Amazon's serverless offering. In this model, Lambda becomes an on-demand accelerator for our always-on servers. The benefits are immense, improving response times by an order of magnitude. But the challenges are numerous and often unexpected. In this talk, I'll review the benefits and constraints of serverless-as-accelerator, and give practical advice based on our own hard-won experience.

When we want to go fast, it helps to see what we are doing.

When we design team and departmental processes, we want to know what’s happening in the software teams. We want to see danger points and obstacles we could smooth. It’s tempting to ask people to fill more fields in JIRA, but that adds burden and distorts reality.

There’s another side of the team we can see into without distortion, if we set it up: the software itself.

Look at the software as part of the team, and observability in the software becomes an asset to organizing teams. We can see:

• what’s happening in production
• the flow through CI/CD
• coordination points, connections between software that needs connection between teams
• when it’s time to shift focus between features and reliability

Visibility into production behavior benefits product planning, organization design, and business decisions. With observability, we can get the software to help us change it smoothly and safely.

Twenty years ago Kent Beck coined the term “test-driven development”: write tests first, develop the code later. Today, even if not practising true TDD, the idea of writing code without tests is an immediate warning sign to any developer. Yet, most teams still continue shipping code without adequate instrumentation to observe real system behaviour in production. Is it time we move to observability-driven development: “instrument first, develop later”?