Knowing how the application will behave gives you a place to start when making these decisions. If a backend dependency of your application becomes unstable or completely unavailable, what should your application do? Your team can use the outcomes of chaos experiments to determine whether a feature should be turned off in certain incidents, or if improvements can be made to increase its resiliency. They might increase the network latency in general, or make components unavailable to give teams a view into what the system will do. Chaos tests might include slowing down requests on a backend dependency or on a central service like DNS. When a team is practicing chaos engineering, they are pushing boundaries on the system. Chaos engineering as a practice emerged from Netflix and has reached a point where it is accessible to many teams via commercial tools like Gremlin.

Large-scale distributed systems present unique challenges related to concepts like scalability, reliability and dependency management that have only increased over time. “Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” The definition from the Principles of chaos engineering website is: Open PagerDuty Partner Twitch Stream with Gremlin on YouTube.Ĭhaos engineering is a methodology for inserting error states into your systems and applications. The recording is now on our YouTube channel for folks who couldn’t join us live. Julie Gunderson of Gremlin recently joined me on PagerDuty’s Twitch channel to talk about integrating Gremlin with PagerDuty, and how your team can benefit from regular chaos experiments. We can add chaos engineering to our workflows to test assumptions about how our applications will behave in production, as well as practice our incident response processes.

We can also improve our team response to incidents in production, so that when something does happen, we’re ready to handle it.

Our testing procedures and other tools help reduce the potential for bugs and problems to make their way into customer-facing environments.

While we strive to make our non-production environments as close to production as possible, there’s always a compromise or two that means the integration environment isn’t exactly like production. Launching new software into production can be a tricky experience. She is a regular speaker at technical conferences and is the author of the O'Reilly Media white paper 'Building a DevOps Culture.' She is interested in the emergence of new tools and workflows to make the task of operating large complex computing systems more approachable.

Show sync provider notifications