Thundra APM
Search…
Thundra Web Console
Node.js
Python
Java
Go
.Net
IDE Integrations
Observability
Performance
Deployment Integrations
Chaos Injection in Java SDK
Span Listeners
By their nature, serverless systems are distributed systems. When using serverless architectures, which are highly distributed and event-driven, you should consider being robust in case latencies and errors happen with any component of your system. For example, you should be prepared if you get a connection error while trying to reach your Redis cache, and should any of the components respond with a latency that slows down the flow, you should have a plan to ensure your system doesn’t collapse. Likewise, you should think ahead – for instance, if you are calling a Lambda inside another Lambda, make sure first that the outer one has bigger timeouts.
It can seem trivial to test applications in this sense, but things will become more complicated as your architecture grows. We developed our
SpanListener and SpanFilterer APIs for this purpose: to test your system and find weak points. When you want to inject chaos to your distributed system, you can use Thundra’s SpanListener API to create latency or an error that your system may face by only playing with the environment variables, and not changing your code. What is a span?
A span is essentially an individual unit of work done in your Lambda functions, a concept that Thundra inherited from Opentracing. You can manually create spans using Thundra; however, thanks to the integrations that Thundra has, spans are automatically created for you if you use the Traceable annotation or our integrations. Currently, Thundra has the following integrations:
- AWS services (Lambda, DynamoDB, SQS, SNS, Kinesis, Firehose, S3, API Gateway, ...)
- MySQL
- PostgreSQL
- Microsoft SQL Server
- MariaDB
- Redis
- Elasticsearch
- HTTP
These integrations mean that even if you don’t create a span manually, whenever you make a Redis call, HTTP call, DynamoDB call, etc., Thundra automatically creates a span for that call and adds the necessary tags related to that specific operation.
What is a Span Listener?
There are three out-of-the-box
SpanListener implementations in Thundra’s Java Agent:ErrorInjectorSpanListenerLatencyInjectorSpanListenerFilteringSpanListener
ErrorInjectorSpanListener
ErrorInjectorSpanListenerParameters include:
- errorMessage: The message that displays the error description when using the
ErrorInjectorSpanListener. The default value is “Error injected by Thundra!”. - errorType: The type of error that will be raised. The name of the thrown error will be set to this value. The default value is
java.lang.RuntimeException. - injectOnFinish: The Boolean value that decides whether the error will be injected after the span is finished or before it starts. The default value is
false. - injectPercentage: The integer value that decides the percentage for error injection. For example, if this value is
10(i.e., 10%), every 10th span event will be an error. The default value is100.
ErrorInjectorSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
3
4
...
5
6
// Create span listener
7
ErrorInjectorSpanListener spanListener =
8
new ErrorInjectorSpanListener(
9
false, // don't inject on finish but on start before the actual operation
10
java.lang.IllegalAccessException.class, // type of the exception to be thrown
11
"No way!", // the error message
12
50 // inject error at 50% percentage
13
);
14
15
// Register span listener
16
TraceSupport.registerSpanListener(spanListener);
Copied!
LatencyInjectorSpanListener
LatencyInjectorSpanListenerParameters include:
- delay: The integer value that decides the amount of latency that will be injected to the spans in milliseconds. The default value is
100ms. - randomizeDelay: The Boolean value that enables randomized amounts of injected delays. For example, if the delay is
100ms and randomization is enabled, a random number is generated between1and100as the amount of each injected latency in milliseconds. The default value isfalse. - injectOnFinish: The Boolean value that decides whether the latency will be injected after the span is finished or before it starts. The default value is
false.
LatencyInjectorSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.LatencyInjectorSpanListener;
3
4
...
5
6
// Create span listener
7
LatencyInjectorSpanListener spanListener =
8
new LatencyInjectorSpanListener(
9
false, // don't inject on finish but on start before the actual operation
10
50, // amount of delay to be injected in milliseconds
11
true // randomize amount of injected delays between 1 and 50 (configured delay)
12
);
13
14
// Register span listener
15
TraceSupport.registerSpanListener(spanListener);
Copied!
FilteringSpanListener
FilteringSpanListenerParameters include:
- listener: If the span is accepted by the
filterer, thelisteneris then notified. There is no default value as this parameter is mandatory. - filterer: The
filtereris the main object that decides whether or not the given span will be delegated to thelistenerparameter, meaning it is basically an object that has an accept method. Before delegating the given span to thelistenerparameter,FilteringSpanListenerfirst passes the given span to thefiltererobject if the filterer’s accept method returnstrue. If it does, then the given span is passed to thelistenerparameter; if it does not, the given span is not passed to thelistener. There is no default value as this parameter is mandatory.
For example:
FilteringSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
3
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
4
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
5
6
...
7
8
// Create error injector span listener
9
ErrorInjectorSpanListener listener =
10
new ErrorInjectorSpanListener(
11
false, // don't inject on finish but on start before the actual operation
12
java.lang.IllegalAccessException.class, // type of the exception to be thrown
13
"No way!", // the error message
14
50 // inject error at 50% percentage
15
);
16
17
// Create filterer to decide which spans will be listened
18
FilteringSpanListener.SpanFilterer filterer =
19
new FilteringSpanListener.SpanFilterer() {
20
@Override
21
public boolean accept(ThundraSpan span) {
22
return "Redis".equals(span.getClassName()) // Targeting only Redis operations
23
&&
24
"GET".equalsIgnoreCase(span.getTag(RedisTags.COMMAND)); // Targeting only Redis "GET" commands
25
}
26
};
27
28
// Create filtering span listener
29
FilteringSpanListener filteringSpanListener =
30
new FilteringSpanListener(
31
listener,
32
filterer
33
);
34
35
// Register span listener
36
TraceSupport.registerSpanListener(filteringSpanListener);
Copied!
The same configuration can be done over
SpanFilter:FilteringSpanListener
1
import io.thundra.agent.trace.TraceSupport;
2
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
3
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
4
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
5
6
import java.util.Arrays;
7
import java.util.HashMap;
8
9
...
10
11
// Create error injector span listener
12
ErrorInjectorSpanListener listener =
13
new ErrorInjectorSpanListener(
14
false, // don't inject on finish but on start before the actual operation
15
java.lang.IllegalAccessException.class, // type of the exception to be thrown
16
"No way!", // the error message
17
50 // inject error at 50% percentage
18
);
19
20
// Create filterer to decide which spans will be listened
21
FilteringSpanListener.SpanFilter filter =
22
new FilteringSpanListener.SpanFilter(
23
null, // "domainName" is not checked
24
"Redis", // Targeting only Redis operations
25
null, // "operationName" is not checked
26
new HashMap() {{
27
put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
28
}}
29
);
30
31
// Create filtering span listener
32
FilteringSpanListener filteringSpanListener =
33
new FilteringSpanListener(
34
listener,
35
Arrays.asList(filter)
36
);
37
38
// Register span listener
39
TraceSupport.registerSpanListener(filteringSpanListener);
Copied!
Configuration using environment variables:
In order to create a span listener using an environment variable, you should set
thundra_agent_lambda_trace_span_listenerConfig as a prefixed (multiple listener definitions can have a different suffix) environment variable in JSON format. The basic structure is similar to the following:Basic structure of a span listener created using the environment variable
1
thundra_agent_lambda_trace_span_listenerConfig...: '{"type":"<span-listener-name>","config":{[<arguments>]}'
Copied!
<span-listener-name>can be the full classname of one of the following predefinedSpanListeners(ErrorInjectorSpanListener,LatencyInjectorSpanListener, orFilteringSpanListener).<arguments>are lists of objects that have key-value pairs.
Here is an example configuration that creates the same filtering span listener that we created programmatically above:
A FilteringSpanListener configured using the environment variable
1
thundra_agent_lambda_trace_span_listenerConfig:'{"type":"FilteringSpanListener","config":{"listener":{"type":"ErrorInjectorSpanListener","config":{"errorType":"java.lang.IllegalAccessException","errorMessage":"Noway!","injectPercentage":50}},"filters":[{"className":"Redis","tags":{"redis.command":"GET"}}]}}'
Copied!
There are a couple of points to note here:
- Using the "
type":"LatencyInjectorSpanListener", we state the type of the listener thatFilteringSpanListeneris going to use. You can also use LatencyInjectorSpanListener when you need to inject latency. - Key-value pairs under the config property are passed to the wrapped span listener, which is specified by property type (which in this case is
ErrorInjectorSpanListener) as parameters. - Key-value pairs under the config property are used to define individual filters. For example,
{"className":"Redis","tags": {"redis.command":"GET"}}corresponds to the following filter:
SpanFilter
1
FilteringSpanListener.SpanFilter filter =
2
new FilteringSpanListener.SpanFilter(
3
null, // "domainName" is not checked
4
"Redis", // Targeting only "Redis" operations
5
null, // "operationName" is not checked
6
new HashMap() {{
7
put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
8
}}
9
);
Copied!
The resulting filtering span listener would first filter the spans by looking at the
className and the tags of the spans. If the given span has a different className and the Redis or the given span will not have the tag redis.command with the value GET. This means that the span will be rejected and won’t be sent to the listener. If it does have both properties, then the filterer accepts the span and FilteringSpanListener propagates the span to its listener parameter (which is a ErrorInjectorSpanListener).SpanFilter
SpanFilterParameters include:
- domainName: The domain name value to compare with the given span’s domain name. The default value is
null. - className: The class name value to compare with the given span’s class name. The default value is
null. - operationName: The operation name value to compare with the given span’s operation name. The default value is
null. - tags: A dictionary that contains key-value tag pairs.
SpanFilterchecks that each of the key-value pairs in this dictionary also exists in the given span’s tags. If at least one of the key-value pairs does not exist in the span’s tags or exists with a different value, then theSpanFilterrejects the span. The default value isnull.
Last modified 1yr ago