Chaos Monkey

Inspired by Chaos Engineering at Netflix and Chaos Monkey for Spring Boot.

The Goal of Chaos Monkey

While helping my customers to build large scale distributed systems in light-4j, I constantly receive the following questions:

  • Will the fallback work?
  • How would the application behave with network latency?
  • What will happen if one of the service instances is down?
  • How does the service discovery help during service deployment?

As you can see, there are a lot of questions to be answered to make our customers comfortable to rollout distributed applications.

These questions led me to dive into Chaos Engineering and start this project to share my thoughts and experience.

How does it work?

As we know, the light-4j framework is using middleware handlers to address cross-cutting concerns in the request/response chain in the same service instance or at the network level with light-router or light-proxy.

To simulate some of the Chaos Monkey behaviours, we built several middleware handlers that can be injected into the request/response chain. These middleware handlers simulate network latency, out of memory, runtime exception and application shutdown etc.

Dynamic Configuration at runtime

Each middleware handler has its configuration file with a switch to enable or disable. So you can even deploy it to the production without any negative impact.

A configuration endpoint /chaosmonkey enables these handlers at runtime with different parameters. It allows controlled Chaos Engineering from the pipeline so that it can be rerun for every release. This also gives the team more opportunities to simulate random production behaviours.

Assault Handlers

Assault handlers are the heart of Chaos Monkey and the service makes use of them based on your configuration. The following assault handlers are actually provided.

If Latency Assault is enabled, latency is added to a request. You control the number of requests where this should occur via the level.

You can determine at runtime whether an exception should occur when the method is used.

When the configured methods are called in the application, the Chaos Monkey will shut down the application.

Memory Assaults attack the memory of the Java Virtual Machine.

Config Handlers

Get all the configurations for assault handlers.

Update one particular assault handler’s configuration.

Tutorial

To learn how to use these handlers, please visit the petstore tutorial.

chaosmonkey post endpoint

It is an endpoint that is enabled by default. It allows the user to update the configuration for the Chaos Monkey assault handlers during the runtime. The same config file chaos-monkey.yml is shared with the chaosmonkey get endpoint. # Light Chaos Monkey API handlers Configuration. # Enable the handlers if set to true to allow user to get or post configurations for the assault handlers. enabled: true In most cases, we should disable all the assault middleware handlers and use this API to update the configuration to enable a specific handler for testing. Read More »

chaosmonkey get endpoint

It is an endpoint that is enabled by default. It allows the user to retrieve the configurations for the Chaos Monkey assault handlers during the runtime. The same config file chaos-monkey.yml is shared with the chaosmonkey post endpoint. # Light Chaos Monkey API handlers Configuration. # Enable the handlers if set to true to allow user to get or post configurations for the assault handlers. enabled: true Here is the path defined in the handler. Read More »

Memory Assault

It is a middleware handler that will eat free memory once it is triggered based on the config file below. Config memory-assault.yml # Light Chaos Monkey Memory Assault Handler Configuration. # Enable the handler if set to true so that it will be wired in the handler chain during the startup enabled: true # Bypass the current chaos monkey middleware handler so that attacks won't be triggered. bypass: true # How many requests are to be attacked. Read More »

Killapp Assault

When this middleware handler is enabled, it will kill the server during one of the requests. The level in the config file controls the chance of this attack. Config killapp-assault.yml # Light Chaos Monkey Kill App Assault Handler Configuration. # Enable the handler if set to true so that it will be wired in the handler chain during the startup enabled: true # Bypass the current chaos monkey middleware handler so that attacks won't be triggered. Read More »

Exception Assault

This is a middleware handler to generate an AssaultException, a runtime exception, during the request handling. As this is a runtime exception, it will bubble up and be handled by the exception handler at the beginning of the request/response chain. For example, the following is a possible setup for the default chain. chains: default: - exception - metrics - traceability - correlation - killapp - latency - memory - exchaos - specification - security - body - audit # - dump - sanitizer - validator Config Here is the config file for the handler. Read More »

Latency Assault

This middleware handler will generate a fixed or random delay in the request/response chain once it is enabled. Config The following is the config file. latency-assault.yml # Light Chaos Monkey Latency Assault Handler Configuration. # Enable the handler if set to true so that it will be wired in the handler chain during the startup enabled: true # Bypass the current chaos monkey middleware handler so that attacks won't be triggered. Read More »