Chaos Injection in Java SDK

Span Listeners
By their nature, serverless systems are distributed systems. When using serverless architectures, which are highly distributed and event-driven, you should consider being robust in case latencies and errors happen with any component of your system. For example, you should be prepared if you get a connection error while trying to reach your Redis cache, and should any of the components respond with a latency that slows down the flow, you should have a plan to ensure your system doesn’t collapse. Likewise, you should think ahead – for instance, if you are calling a Lambda inside another Lambda, make sure first that the outer one has bigger timeouts.
It can seem trivial to test applications in this sense, but things will become more complicated as your architecture grows. We developed our SpanListener and SpanFilterer APIs for this purpose: to test your system and find weak points. When you want to inject chaos to your distributed system, you can use Thundra’s SpanListener API to create latency or an error that your system may face by only playing with the environment variables, and not changing your code. 
What is a span?
A span is essentially an individual unit of work done in your Lambda functions, a concept that Thundra inherited from Opentracing. You can manually create spans using Thundra; however, thanks to the integrations that Thundra has, spans are automatically created for you if you use the Traceable annotation or our integrations. Currently, Thundra has the following integrations:
AWS services (Lambda, DynamoDB, SQS, SNS, Kinesis, Firehose, S3, API Gateway, ...)
MySQL
PostgreSQL
Microsoft SQL Server
MariaDB
Redis
Elasticsearch
HTTP
These integrations mean that even if you don’t create a span manually, whenever you make a Redis call, HTTP call, DynamoDB call, etc., Thundra automatically creates a span for that call and adds the necessary tags related to that specific operation.
What is a Span Listener?
There are three out-of-the-box SpanListener implementations in Thundra’s Java Agent:
ErrorInjectorSpanListener
LatencyInjectorSpanListener
FilteringSpanListener
ErrorInjectorSpanListener
Parameters include:
errorMessage: The message that displays the error description when using the ErrorInjectorSpanListener. The default value is “Error injected by Thundra!”.
errorType: The type of error that will be raised. The name of the thrown error will be set to this value. The default value is java.lang.RuntimeException.
injectOnFinish: The Boolean value that decides whether the error will be injected after the span is finished or before it starts. The default value is false.
injectPercentage: The integer value that decides the percentage for error injection. For example, if this value is 10 (i.e., 10%), every 10th span event will be an error. The default value is 100.
ErrorInjectorSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
3
​
4
...
5
  
6
// Create span listener  
7
ErrorInjectorSpanListener spanListener = 
8
    new ErrorInjectorSpanListener(
9
  	        false, // don't inject on finish but on start before the actual operation
10
            java.lang.IllegalAccessException.class, // type of the exception to be thrown
11
            "No way!", // the error message
12
  	        50 // inject error at 50% percentage 
13
	);
14
​
15
// Register span listener
16
TraceSupport.registerSpanListener(spanListener);
Copied!
LatencyInjectorSpanListener
Parameters include:
delay: The integer value that decides the amount of latency that will be injected to the spans in milliseconds. The default value is 100 ms.
randomizeDelay: The Boolean value that enables randomized amounts of injected delays. For example, if the delay is 100 ms and randomization is enabled, a random number is generated between 1 and 100 as the amount of each injected latency in milliseconds. The default value is false.
injectOnFinish: The Boolean value that decides whether the latency will be injected after the span is finished or before it starts. The default value is false.
LatencyInjectorSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.LatencyInjectorSpanListener;
3
​
4
...
5
  
6
// Create span listener  
7
LatencyInjectorSpanListener spanListener = 
8
    new LatencyInjectorSpanListener(
9
  	        false, // don't inject on finish but on start before the actual operation
10
            50, // amount of delay to be injected in milliseconds
11
            true // randomize amount of injected delays between 1 and 50 (configured delay)
12
	);
13
​
14
// Register span listener
15
TraceSupport.registerSpanListener(spanListener);
Copied!
FilteringSpanListener
Parameters include:
listener: If the span is accepted by the filterer, the listener is then notified. There is no default value as this parameter is mandatory.
filterer: The filterer is the main object that decides whether or not the given span will be delegated to the listener parameter, meaning it is basically an object that has an accept method. Before delegating the given span to the listener parameter, FilteringSpanListener first passes the given span to the filterer object if the filterer’s accept method returns true. If it does, then the given span is passed to the listener parameter; if it does not, the given span is not passed to the listener. There is no default value as this parameter is mandatory.
For example:
FilteringSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
3
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
4
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
5
​
6
...
7
  
8
// Create error injector span listener  
9
ErrorInjectorSpanListener listener = 
10
    new ErrorInjectorSpanListener(
11
  	        false, // don't inject on finish but on start before the actual operation
12
            java.lang.IllegalAccessException.class, // type of the exception to be thrown
13
            "No way!", // the error message
14
  	        50 // inject error at 50% percentage 
15
	);
16
​
17
// Create filterer to decide which spans will be listened  
18
FilteringSpanListener.SpanFilterer filterer =
19
	new FilteringSpanListener.SpanFilterer() {
20
		@Override
21
        public boolean accept(ThundraSpan span) {
22
    	    return "Redis".equals(span.getClassName()) // Targeting only Redis operations
23
        		   &&
24
                   "GET".equalsIgnoreCase(span.getTag(RedisTags.COMMAND)); // Targeting only Redis "GET" commands
25
        }
26
	};
27
​
28
// Create filtering span listener  
29
FilteringSpanListener filteringSpanListener =
30
    new FilteringSpanListener(
31
		    listener,
32
            filterer
33
	);
34
​
35
// Register span listener
36
TraceSupport.registerSpanListener(filteringSpanListener);
Copied!
The same configuration can be done over SpanFilter:
FilteringSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
3
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
4
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
5
​
6
import java.util.Arrays;
7
import java.util.HashMap;
8
​
9
...
10
  
11
// Create error injector span listener  
12
ErrorInjectorSpanListener listener = 
13
    new ErrorInjectorSpanListener(
14
  	        false, // don't inject on finish but on start before the actual operation
15
            java.lang.IllegalAccessException.class, // type of the exception to be thrown
16
            "No way!", // the error message
17
  	        50 // inject error at 50% percentage 
18
	);
19
​
20
// Create filterer to decide which spans will be listened  
21
FilteringSpanListener.SpanFilter filter =
22
	new FilteringSpanListener.SpanFilter(
23
  	        null, // "domainName" is not checked
24
            "Redis", // Targeting only Redis operations
25
            null, // "operationName" is not checked
26
            new HashMap() {{
27
    	        put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
28
            }}
29
	);
30
​
31
// Create filtering span listener  
32
FilteringSpanListener filteringSpanListener =
33
    new FilteringSpanListener(
34
		    listener,
35
            Arrays.asList(filter)
36
	);
37
​
38
// Register span listener
39
TraceSupport.registerSpanListener(filteringSpanListener);
Copied!
Configuration using environment variables:
In order to create a span listener using an environment variable, you should set thundra_agent_lambda_trace_span_listenerConfig as a prefixed (multiple listener definitions can have a different suffix) environment variable in JSON format. The basic structure is similar to the following:
Basic structure of a span listener created using the environment variable
1
thundra_agent_lambda_trace_span_listenerConfig...: '{"type":"<span-listener-name>","config":{[<arguments>]}'
Copied!
<span-listener-name> can be the full classname of one of the following predefined SpanListeners (ErrorInjectorSpanListener, LatencyInjectorSpanListener, or FilteringSpanListener).
<arguments> are lists of objects that have key-value pairs.
Here is an example configuration that creates the same filtering span listener that we created programmatically above:
A FilteringSpanListener configured using the environment variable
1
thundra_agent_lambda_trace_span_listenerConfig:'{"type":"FilteringSpanListener","config":{"listener":{"type":"ErrorInjectorSpanListener","config":{"errorType":"java.lang.IllegalAccessException","errorMessage":"Noway!","injectPercentage":50}},"filters":[{"className":"Redis","tags":{"redis.command":"GET"}}]}}'
Copied!
There are a couple of points to note here:
Using the "type":"LatencyInjectorSpanListener", we state the type of the listener that FilteringSpanListener is going to use. You can also use LatencyInjectorSpanListener when you need to inject latency.
Key-value pairs under the config property are passed to the wrapped span listener, which is specified by property type (which in this case is ErrorInjectorSpanListener) as parameters.
Key-value pairs under the config property are used to define individual filters. For example,{"className":"Redis","tags": {"redis.command":"GET"}} corresponds to the following filter:
SpanFilter
1
FilteringSpanListener.SpanFilter filter =
2
	new FilteringSpanListener.SpanFilter(
3
            null, // "domainName" is not checked
4
            "Redis", // Targeting only "Redis" operations
5
            null, // "operationName" is not checked
6
            new HashMap() {{
7
    	        put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
8
            }}
9
	);
Copied!
The resulting filtering span listener would first filter the spans by looking at the className and the tags of the spans. If the given span has a different className and the Redis or the given span will not have the tag redis.command with the value GET. This means that the span will be rejected and won’t be sent to the listener. If it does have both properties, then the filterer accepts the span and FilteringSpanListener propagates the span to its listener parameter (which is a ErrorInjectorSpanListener).
SpanFilter
Parameters include:
domainName: The domain name value to compare with the given span’s domain name. The default value is null.
className: The class name value to compare with the given span’s class name. The default value is null.
operationName: The operation name value to compare with the given span’s operation name. The default value is null.
tags: A dictionary that contains key-value tag pairs. SpanFilter checks that each of the key-value pairs in this dictionary also exists in the given span’s tags. If at least one of the key-value pairs does not exist in the span’s tags or exists with a different value, then the SpanFilter rejects the span. The default value is null.

Configuring Metrics for Java SDK

Sampling in Java SDK

Last modified 1yr ago

Copy link

Contents

Span Listeners

What is a span?

What is a Span Listener?

ErrorInjectorSpanListener

LatencyInjectorSpanListener

FilteringSpanListener

SpanFilter

Span Listeners

What is a span?

What is a Span Listener?

`ErrorInjectorSpanListener`

`LatencyInjectorSpanListener`

`FilteringSpanListener`

`SpanFilter`