Chaos Injection in Java SDK
By their nature, serverless systems are distributed systems. When using serverless architectures, which are highly distributed and event-driven, you should consider being robust in case latencies and errors happen with any component of your system. For example, you should be prepared if you get a connection error while trying to reach your Redis cache, and should any of the components respond with a latency that slows down the flow, you should have a plan to ensure your system doesn’t collapse. Likewise, you should think ahead – for instance, if you are calling a Lambda inside another Lambda, make sure first that the outer one has bigger timeouts.
It can seem trivial to test applications in this sense, but things will become more complicated as your architecture grows. We developed our
SpanListener
and SpanFilterer
APIs for this purpose: to test your system and find weak points. When you want to inject chaos to your distributed system, you can use Thundra’s SpanListener
API to create latency or an error that your system may face by only playing with the environment variables, and not changing your code. A span is essentially an individual unit of work done in your Lambda functions, a concept that Thundra inherited from Opentracing. You can manually create spans using Thundra; however, thanks to the integrations that Thundra has, spans are automatically created for you if you use the Traceable annotation or our integrations. Currently, Thundra has the following integrations:
- AWS services (Lambda, DynamoDB, SQS, SNS, Kinesis, Firehose, S3, API Gateway, ...)
- MySQL
- PostgreSQL
- Microsoft SQL Server
- MariaDB
- Redis
- Elasticsearch
- HTTP
These integrations mean that even if you don’t create a span manually, whenever you make a Redis call, HTTP call, DynamoDB call, etc., Thundra automatically creates a span for that call and adds the necessary tags related to that specific operation.
There are three out-of-the-box
SpanListener
implementations in Thundra’s Java Agent:ErrorInjectorSpanListener
LatencyInjectorSpanListener
FilteringSpanListener
Parameters include:
- errorMessage: The message that displays the error description when using the
ErrorInjectorSpanListener
. The default value is “Error injected by Thundra!”. - errorType: The type of error that will be raised. The name of the thrown error will be set to this value. The default value is
java.lang.RuntimeException
. - injectOnFinish: The Boolean value that decides whether the error will be injected after the span is finished or before it starts. The default value is
false
. - injectPercentage: The integer value that decides the percentage for error injection. For example, if this value is
10
(i.e., 10%), every 10th span event will be an error. The default value is100
.
ErrorInjectorSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
...
// Create span listener
ErrorInjectorSpanListener spanListener =
new ErrorInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
java.lang.IllegalAccessException.class, // type of the exception to be thrown
"No way!", // the error message
50 // inject error at 50% percentage
);
// Register span listener
TraceSupport.registerSpanListener(spanListener);
Parameters include:
- delay: The integer value that decides the amount of latency that will be injected to the spans in milliseconds. The default value is
100
ms. - randomizeDelay: The Boolean value that enables randomized amounts of injected delays. For example, if the delay is
100
ms and randomization is enabled, a random number is generated between1
and100
as the amount of each injected latency in milliseconds. The default value isfalse
. - injectOnFinish: The Boolean value that decides whether the latency will be injected after the span is finished or before it starts. The default value is
false
.
LatencyInjectorSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.LatencyInjectorSpanListener;
...
// Create span listener
LatencyInjectorSpanListener spanListener =
new LatencyInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
50, // amount of delay to be injected in milliseconds
true // randomize amount of injected delays between 1 and 50 (configured delay)
);
// Register span listener
TraceSupport.registerSpanListener(spanListener);
Parameters include:
- listener: If the span is accepted by the
filterer
, thelistener
is then notified. There is no default value as this parameter is mandatory. - filterer: The
filterer
is the main object that decides whether or not the given span will be delegated to thelistener
parameter, meaning it is basically an object that has an accept method. Before delegating the given span to thelistener
parameter,FilteringSpanListener
first passes the given span to thefilterer
object if the filterer’s accept method returnstrue
. If it does, then the given span is passed to thelistener
parameter; if it does not, the given span is not passed to thelistener
. There is no default value as this parameter is mandatory.
For example:
FilteringSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
...
// Create error injector span listener
ErrorInjectorSpanListener listener =
new ErrorInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
java.lang.IllegalAccessException.class, // type of the exception to be thrown
"No way!", // the error message
50 // inject error at 50% percentage
);
// Create filterer to decide which spans will be listened
FilteringSpanListener.SpanFilterer filterer =
new FilteringSpanListener.SpanFilterer() {
@Override
public boolean accept(ThundraSpan span) {
return "Redis".equals(span.getClassName()) // Targeting only Redis operations
&&
"GET".equalsIgnoreCase(span.getTag(RedisTags.COMMAND)); // Targeting only Redis "GET" commands
}
};
// Create filtering span listener
FilteringSpanListener filteringSpanListener =
new FilteringSpanListener(
listener,
filterer
);
// Register span listener
TraceSupport.registerSpanListener(filteringSpanListener);
The same configuration can be done over
SpanFilter
:FilteringSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
import java.util.Arrays;
import java.util.HashMap;
...
// Create error injector span listener
ErrorInjectorSpanListener listener =
new ErrorInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
java.lang.IllegalAccessException.class, // type of the exception to be thrown
"No way!", // the error message
50 // inject error at 50% percentage
);
// Create filterer to decide which spans will be listened
FilteringSpanListener.SpanFilter filter =
new FilteringSpanListener.SpanFilter(
null, // "domainName" is not checked
"Redis", // Targeting only Redis operations
null, // "operationName" is not checked
new HashMap() {{
put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
}}
);
// Create filtering span listener
FilteringSpanListener filteringSpanListener =
new FilteringSpanListener(
listener,
Arrays.asList(filter)
);
// Register span listener
TraceSupport.registerSpanListener(filteringSpanListener);
Configuration using environment variables:
In order to create a span listener using an environment variable, you should set
thundra_agent_lambda_trace_span_listenerConfig
as a prefixed (multiple listener definitions can have a different suffix) environment variable in JSON format. The basic structure is similar to the following:Basic structure of a span listener created using the environment variable
thundra_agent_lambda_trace_span_listenerConfig...: '{"type":"<span-listener-name>","config":{[<arguments>]}'
<span-listener-name>
can be the full classname of one of the following predefinedSpanListeners
(ErrorInjectorSpanListener
,LatencyInjectorSpanListener
, orFilteringSpanListener
).<arguments>
are lists of objects that have key-value pairs.
Here is an example configuration that creates the same filtering span listener that we created programmatically above:
A FilteringSpanListener configured using the environment variable
thundra_agent_lambda_trace_span_listenerConfig:'{"type":"FilteringSpanListener","config":{"listener":{"type":"ErrorInjectorSpanListener","config":{"errorType":"java.lang.IllegalAccessException","errorMessage":"Noway!","injectPercentage":50}},"filters":[{"className":"Redis","tags":{"redis.command":"GET"}}]}}'
There are a couple of points to note here:
- Using the "
type
":"LatencyInjectorSpanListener
", we state the type of the listener thatFilteringSpanListener
is going to use. You can also use LatencyInjectorSpanListener when you need to inject latency. - Key-value pairs under the config property are passed to the wrapped span listener, which is specified by property type (which in this case is
ErrorInjectorSpanListener
) as parameters. - Key-value pairs under the config property are used to define individual filters. For example,
{"className":"Redis","tags": {"redis.command":"GET"}}
corresponds to the following filter:
SpanFilter
FilteringSpanListener.SpanFilter filter =
new FilteringSpanListener.SpanFilter(
null, // "domainName" is not checked
"Redis", // Targeting only "Redis" operations
null, // "operationName" is not checked
new HashMap() {{
put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
}}
);
The resulting filtering span listener would first filter the spans by looking at the
className
and the tags
of the spans. If the given span has a different className
and the Redis
or the given span will not have the tag redis.command
with the value GET
. This means that the span will be rejected and won’t be sent to the listener
. If it does have both properties, then the filterer accepts the span and FilteringSpanListener
propagates the span to its listener parameter (which is a ErrorInjectorSpanListener
).Parameters include:
- domainName: The domain name value to compare with the given span’s domain name. The default value is
null
. - className: The class name value to compare with the given span’s class name. The default value is
null
. - operationName: The operation name value to compare with the given span’s operation name. The default value is
null
. - tags: A dictionary that contains key-value tag pairs.
SpanFilter
checks that each of the key-value pairs in this dictionary also exists in the given span’s tags. If at least one of the key-value pairs does not exist in the span’s tags or exists with a different value, then theSpanFilter
rejects the span. The default value isnull
.
Last modified 2yr ago