Openshift Container Platform 4.6: Jaeger
Openshift Container Platform 4.6: Jaeger
Openshift Container Platform 4.6: Jaeger
Jaeger
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons
Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is
available at
http://creativecommons.org/licenses/by-sa/3.0/
. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must
provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,
Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,
Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States
and other countries.
Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.
XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States
and/or other countries.
MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and
other countries.
Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the
official Joyent Node.js open source or commercial project.
The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marks
or trademarks/service marks of the OpenStack Foundation, in the United States and other
countries and are used with the OpenStack Foundation's permission. We are not affiliated with,
endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
Abstract
This document provides information on how to use Jaeger in OpenShift Container Platform.
Table of Contents
Table of Contents
.CHAPTER
. . . . . . . . . . 1.. .JAEGER
. . . . . . . . . RELEASE
. . . . . . . . . . NOTES
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. . . . . . . . . . . . .
1.1. JAEGER OVERVIEW 3
1.2. MAKING OPEN SOURCE MORE INCLUSIVE 3
1.3. GETTING SUPPORT 3
1.3.1. New features OpenShift Jaeger 1.20.0 3
1.3.2. New features OpenShift Jaeger 1.17.7 3
1.4. TECHNOLOGY PREVIEW 4
1.4.1. OpenShift Jaeger 2.0.0 Technology Preview 1 4
1.5. JAEGER KNOWN ISSUES 4
1.6. JAEGER FIXED ISSUES 4
.CHAPTER
. . . . . . . . . . 2.
. . JAEGER
. . . . . . . . . .ARCHITECTURE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6. . . . . . . . . . . . .
2.1. JAEGER ARCHITECTURE 6
2.1.1. Jaeger overview 6
2.1.2. Jaeger features 6
2.1.3. Jaeger architecture 7
. . . . . . . . . . . 3.
CHAPTER . . JAEGER
. . . . . . . . . .INSTALLATION
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8. . . . . . . . . . . . .
3.1. INSTALLING JAEGER 8
3.1.1. Prerequisites 8
3.1.2. Jaeger installation overview 8
3.1.3. Installing the Elasticsearch Operator 8
3.1.4. Installing the Jaeger Operator 10
3.2. CONFIGURING AND DEPLOYING JAEGER 11
3.2.1. Deploying the default Jaeger strategy from the web console 12
3.2.1.1. Deploying default Jaeger from the CLI 13
. . . . . . . . . . . 4.
CHAPTER . . .DEPLOYING
. . . . . . . . . . . . .THE
. . . . .JAEGER
. . . . . . . . .PRODUCTION
. . . . . . . . . . . . . . .STRATEGY
. . . . . . . . . . . .FROM
. . . . . . .THE
. . . . .WEB
. . . . .CONSOLE
. . . . . . . . . . . . . . . . . . . . . . . 15
..............
4.1. DEPLOYING JAEGER PRODUCTION FROM THE CLI 16
.CHAPTER
. . . . . . . . . . 5.
. . DEPLOYING
. . . . . . . . . . . . . .THE
. . . . .JAEGER
. . . . . . . . .STREAMING
. . . . . . . . . . . . .STRATEGY
. . . . . . . . . . . . FROM
. . . . . . .THE
. . . . .WEB
. . . . . CONSOLE
. . . . . . . . . . . . . . . . . . . . . . . . . 18
..............
5.1. DEPLOYING JAEGER STREAMING FROM THE CLI 19
5.1.1. Customizing Jaeger deployment 20
5.1.1.1. Jaeger default configuration options 20
5.1.1.2. Jaeger Collector configuration options 23
5.1.1.2.1. Configuring the Collector for autoscaling 24
5.1.1.3. Jaeger sampling configuration options 24
5.1.1.4. Jaeger storage configuration options 26
5.1.1.4.1. Auto-provisioning an Elasticsearch instance 27
5.1.1.4.2. Connecting to an existing Elasticsearch instance 30
5.1.1.5. Jaeger Query configuration options 38
5.1.1.6. Jaeger Ingester configuration options 39
5.1.1.6.1. Configuring Ingester for autoscaling 41
5.1.2. Injecting sidecars 41
5.2. AUTOMATICALLY INJECTING SIDECARS 42
5.3. MANUALLY INJECTING SIDECARS 42
5.4. UPGRADING JAEGER 43
5.5. REMOVING JAEGER 43
5.5.1. Removing a Jaeger instance using the web console 44
5.5.2. Removing a Jaeger instance from the CLI 44
5.5.3. Removing the Jaeger Operator 45
1
OpenShift Container Platform 4.6 Jaeger
2
CHAPTER 1. JAEGER RELEASE NOTES
Search or browse through the Red Hat Knowledgebase of articles and solutions relating to Red
Hat products.
To identify issues with your cluster, you can use Insights in Red Hat OpenShift Cluster Manager. Insights
provides details about issues and, if available, information on how to solve a problem.
If you have a suggestion for improving this documentation or have found an error, please submit a
Bugzilla report against the OpenShift Container Platform product for the Documentation component.
Please provide specific details, such as the section name and OpenShift Container Platform version.
This release adds autoscaling support for the Jaeger Collector and Ingester.
3
OpenShift Container Platform 4.6 Jaeger
This release of OpenShift Jaeger addresses Common Vulnerabilities and Exposures (CVEs) and bug
fixes.
IMPORTANT
Technology Preview features are not supported with Red Hat production service level
agreements (SLAs) and might not be functionally complete. Red Hat does not
recommend using them in production. These features provide early access to upcoming
product features, enabling customers to test functionality and provide feedback during
the development process. For more information about the support scope of Red Hat
Technology Preview features, see
https://access.redhat.com/support/offerings/techpreview/.
The OpenTelemetry collector allows developers to instrument their code with vendor agnostic APIs,
avoiding vendor lock-in and opening the door to a growing ecosystem of observability tooling.
Jaeger streaming via AMQ/Kafka is unsupported on IBM Z and IBM Power Systems.
BZ-1918920 The Elasticsearch pods does not get restarted automatically after an update. As a
workaround, restart the pods manually.
TRACING-809 Jaeger Ingester is incompatible with Kafka 2.3. When there are two or more
instances of the Jaeger Ingester and enough traffic it will continuously generate rebalancing
messages in the logs. This is due to a regression in Kafka 2.3 that was fixed in Kafka 2.3.1. For
more information, see Jaegertracing-1819.
TRACING-1631 Multiple Jaeger production instances, using same name but within different
namespaces, causing Elasticsearch certificate issue. When multiple service meshes were
installed, all of the Jaeger Elasticsearch instances had the same Elasticsearch secret instead of
individual secrets, which prevented the Elasticsearch Operator from communicating with all of
the Elasticsearch clusters.
TRACING-1300 Failed connection between Agent and Collector when using Istio sidecar. An
4
CHAPTER 1. JAEGER RELEASE NOTES
TRACING-1300 Failed connection between Agent and Collector when using Istio sidecar. An
update of the Jaeger Operator enabled TLS communication by default between a Jaeger
sidecar agent and the Jaeger Collector.
TRACING-1208 Authentication "500 Internal Error" when accessing Jaeger UI. When trying to
authenticate to the UI using OAuth, I get a 500 error because oauth-proxy sidecar doesn’t trust
the custom CA bundle defined at installation time with the additionalTrustBundle.
TRACING-1166 It is not currently possible to use the Jaeger streaming strategy within a
disconnected environment. When a Kafka cluster is being provisioned, it results in a error: Failed
to pull image registry.redhat.io/amq7/amq-streams-kafka-24-
rhel7@sha256:f9ceca004f1b7dccb3b82d9a8027961f9fe4104e0ed69752c0bdd8078b4a1076.
5
OpenShift Container Platform 4.6 Jaeger
Distributed tracing is a technique that is used to tie the information about different units of work
together — usually executed in different processes or hosts — to understand a whole chain of events in a
distributed transaction. Developers can visualize call flows in large microservice architectures with
distributed tracing. It’s valuable for understanding serialization, parallelism, and sources of latency.
Jaeger records the execution of individual requests across the whole stack of microservices, and
presents them as traces. A trace is a data/execution path through the system. An end-to-end trace is
comprised of one or more spans.
A span represents a logical unit of work in Jaeger that has an operation name, the start time of the
operation, and the duration, as well as potentially tags and logs. Spans may be nested and ordered to
model causal relationships.
Integration with Kiali – When properly configured, you can view Jaeger data from the Kiali
console.
High scalability – The Jaeger backend is designed to have no single points of failure and to scale
with the business needs.
Distributed Context Propagation – Lets you connect data from different components together
to create a complete end-to-end trace.
Backwards compatibility with Zipkin – Jaeger has APIs that enable it to be used as a drop-in
replacement for Zipkin, but Red Hat is not supporting Zipkin compatibility in this release.
6
CHAPTER 2. JAEGER ARCHITECTURE
Jaeger Client (Tracer, Reporter, instrumented application, client libraries)- Jaeger clients are
language specific implementations of the OpenTracing API. They can be used to instrument
applications for distributed tracing either manually or with a variety of existing open source
frameworks, such as Camel (Fuse), Spring Boot (RHOAR), MicroProfile (RHOAR/Thorntail),
Wildfly (EAP), and many more, that are already integrated with OpenTracing.
Jaeger Agent (Server Queue, Processor Workers) - The Jaeger agent is a network daemon
that listens for spans sent over User Datagram Protocol (UDP), which it batches and sends to
the collector. The agent is meant to be placed on the same host as the instrumented
application. This is typically accomplished by having a sidecar in container environments like
Kubernetes.
Jaeger Collector (Queue, Workers) - Similar to the Agent, the Collector is able to receive
spans and place them in an internal queue for processing. This allows the collector to return
immediately to the client/agent instead of waiting for the span to make its way to the storage.
Storage (Data Store) - Collectors require a persistent storage backend. Jaeger has a pluggable
mechanism for span storage. Note that for this release, the only supported storage is
Elasticsearch.
Query (Query Service) - Query is a service that retrieves traces from storage.
Ingester (Ingester Service) - Jaeger can use Apache Kafka as a buffer between the collector
and the actual backing storage (Elasticsearch). Ingester is a service that reads data from Kafka
and writes to another storage backend (Elasticsearch).
Jaeger Console – Jaeger provides a user interface that lets you visualize your distributed
tracing data. On the Search page, you can find traces and explore details of the spans that
make up an individual trace.
7
OpenShift Container Platform 4.6 Jaeger
You can install Jaeger as part of Red Hat OpenShift Service Mesh. Jaeger is included by default
in the Service Mesh installation. To install Jaeger as part of a service mesh, follow the Red Hat
Service Mesh Installation instructions.
If you do not want to install a service mesh, you can use the Jaeger Operator to install the Red
Hat build of Jaeger by itself. To install Jaeger without a service mesh, use the following
instructions.
3.1.1. Prerequisites
Before you can install OpenShift Jaeger, review the installation activities, and ensure that you meet the
prerequisites:
Possess an active OpenShift Container Platform subscription on your Red Hat account. If you
do not have a subscription, contact your sales representative for more information.
Install the version of the OpenShift Container Platform command line utility (the oc client tool)
that matches your OpenShift Container Platform version and add it to your path.
If your deployment strategy requires persistent storage, install the Elasticsearch Operator via
the OperatorHub.
Deploy one or more instances of Jaeger to your OpenShift Container Platform environment.
8
CHAPTER 3. JAEGER INSTALLATION
The default Jaeger deployment uses in-memory storage because it is designed to be installed quickly
for those evaluating Jaeger, giving demonstrations, or using Jaeger in a test environment. If you plan to
use Jaeger in production, you must install and configure a persistent storage option, in this case,
Elasticsearch.
Prerequisites
WARNING
Do not install Community versions of the Operators. Community Operators are not
supported.
NOTE
If you have already installed the Elasticsearch Operator as part of OpenShift cluster
logging, you do not need to install the Elasticsearch Operator again. The Jaeger Operator
will create the Elasticsearch instance using the installed Elasticsearch Operator.
Procedure
1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
3. Type Elasticsearch into the filter box to locate the Elasticsearch Operator.
4. Click the Elasticsearch Operator provided by Red Hat to display information about the
Operator.
5. Click Install.
6. On the Install Operator page, under Installation Mode select All namespaces on the cluster
(default). This makes the Operator available to all projects in the cluster.
NOTE
8. Select the Update Channel that matches your OpenShift Container Platform installation. For
example, if you are installing on OpenShift Container Platform version 4.6, select the 4.6 update
channel.
9
OpenShift Container Platform 4.6 Jaeger
NOTE
11. On the Installed Operators page, select the openshift-operators-redhat project. Wait until
you see that the Elasticsearch Operator shows a status of "InstallSucceeded" before continuing.
Prerequisites
If you require persistent storage, you must also install the Elasticsearch Operator before
installing the Jaeger Operator.
WARNING
Do not install Community versions of the Operators. Community Operators are not
supported.
Procedure
1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
4. Click the Jaeger Operator provided by Red Hat to display information about the Operator.
5. Click Install.
6. On the Install Operator page, select the stable Update Channel. This will automatically update
Jaeger as new versions are released. If you select a maintenance channel, for example, 1.17-
stable, you will receive bug fixes and security patches for the length of the support cycle for
that version.
7. Select All namespaces on the cluster (default). This installs the Operator in the default
10
CHAPTER 3. JAEGER INSTALLATION
7. Select All namespaces on the cluster (default). This installs the Operator in the default
openshift-operators project and makes the Operator available to all projects in the cluster.
Select an Approval Strategy. You can select Automatic or Manual updates. If you choose
Automatic updates for an installed Operator, when a new version of that Operator is
available, the Operator Lifecycle Manager (OLM) automatically upgrades the running
instance of your Operator without human intervention. If you select Manual updates, when a
newer version of an Operator is available, the OLM creates an update request. As a cluster
administrator, you must then manually approve that update request to have the Operator
updated to the new version.
NOTE
8. Click Install.
9. On the Subscription Overview page, select the openshift-operators project. Wait until you
see that the Jaeger Operator shows a status of "InstallSucceeded" before continuing.
Jaeger has predefined deployment strategies. You specify a deployment strategy in the custom
resource file. When you create a Jaeger instance the Operator uses this configuration file to create the
objects necessary for the deployment.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production 1
allInOne (Default) - This strategy is intended for development, testing, and demo
purposes; it is not intended for production use. The main backend components, Agent,
Collector and Query service, are all packaged into a single executable which is configured
(by default) to use in-memory storage.
NOTE
11
OpenShift Container Platform 4.6 Jaeger
production - The production strategy is intended for production environments, where long
term storage of trace data is important, as well as a more scalable and highly available
architecture is required. Each of the backend components is therefore deployed
separately. The Agent can be injected as a sidecar on the instrumented application. The
Query and Collector services are configured with a supported storage type - currently
Elasticsearch. Multiple instances of each of these components can be provisioned as
required for performance and resilience purposes.
NOTE
The streaming strategy requires an additional Red Hat subscription for AMQ
Streams.
NOTE
There are two ways to install and use Jaeger, as part of a service mesh or as a stand alone
component. If you have installed Jaeger as part of Red Hat OpenShift Service Mesh, you
can configure and deploy Jaeger as part of the ServiceMeshControlPlane or configure
Jaeger and then reference your Jaeger configuration in the SMCP .
3.2.1. Deploying the default Jaeger strategy from the web console
The custom resource definition (CRD) defines the configuration used when you deploy an instance of
Jaeger. The default CR for Jaeger is named jaeger-all-in-one-inmemory and it is configured with
minimal resources to ensure that you can successfully install it on a default OpenShift Container
Platform installation. You can use this default configuration to create a Jaeger instance that uses the
AllInOne deployment strategy, or you can define your own custom resource file.
NOTE
In-memory storage is not persistent, which means that if the Jaeger pod shuts down,
restarts, or is replaced, that your trace data will be lost. For persistent storage, you must
use the production or streaming strategies, which use Elasticsearch as the default
storage.
Prerequisites
NOTE
12
CHAPTER 3. JAEGER INSTALLATION
Procedure
1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
d. Click Create.
4. If necessary, select jaeger-system from the Project menu. You may have to wait a few
moments for the Operators to be copied to the new project.
5. Click the OpenShift Jaeger Operator. On the Overview tab, under Provided APIs, the
Operator provides a single link.
7. On the Create Jaeger page, to install using the defaults, click Create to create the Jaeger
instance.
8. On the Jaegers page, click the name of the Jaeger instance, for example, jaeger-all-in-one-
inmemory.
9. On the Jaeger Details page, click the Resources tab. Wait until the Pod has a status of
"Running" before continuing.
Follow this procedure to create an instance of Jaeger from the command line.
Prerequisites
Access to the OpenShift CLI (oc) that matches your OpenShift Container Platform version.
Procedure
1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.
$ oc login https://{HOSTNAME}:8443
$ oc new-project jaeger-system
3. Create a custom resource file named jaeger.yaml that contains the following text:
Example jaeger-all-in-one.yaml
13
OpenShift Container Platform 4.6 Jaeger
Example jaeger-all-in-one.yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger-all-in-one-inmemory
5. Run the following command to watch the progress of the pods during the installation process:
Once the installation process has completed, you should see output similar to the following:
14
CHAPTER 4. DEPLOYING THE JAEGER PRODUCTION STRATEGY FROM THE WEB CONSOLE
Prerequisites
Procedure
1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
d. Click Create.
4. If necessary, select jaeger-system from the Project menu. You may have to wait a few
moments for the Operators to be copied to the new project.
5. Click the Jaeger Operator. On the Overview tab, under Provided APIs, the Operator provides a
single link.
7. On the Create Jaeger page, replace the default all-in-one yaml text with your production
YAML configuration, for example:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger-production
namespace:
spec:
strategy: production
ingress:
security: oauth-proxy
storage:
type: elasticsearch
15
OpenShift Container Platform 4.6 Jaeger
elasticsearch:
nodeCount: 3
redundancyPolicy: SingleRedundancy
esIndexCleaner:
enabled: true
numberOfDays: 7
schedule: 55 23 * * *
esRollover:
schedule: '*/30 * * * *'
9. On the Jaegers page, click the name of the Jaeger instance, for example, jaeger-prod-
elasticsearch.
10. On the Jaeger Details page, click the Resources tab. Wait until all the pods have a status of
"Running" before continuing.
Prerequisites
Procedure
1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.
$ oc login https://{HOSTNAME}:8443
$ oc new-project jaeger-system
3. Create a custom resource file named jaeger-production.yaml that contains the text of the
example file in the previous procedure.
5. Run the following command to watch the progress of the pods during the installation process:
Once the installation process has completed, you should see output similar to the following:
16
CHAPTER 4. DEPLOYING THE JAEGER PRODUCTION STRATEGY FROM THE WEB CONSOLE
17
OpenShift Container Platform 4.6 Jaeger
The streaming strategy provides a streaming capability that sits between the collector and the storage
(Elasticsearch). This reduces the pressure on the storage under high load situations, and enables other
trace post-processing capabilities to tap into the real time span data directly from the streaming
platform (Kafka).
NOTE
The streaming strategy requires an additional Red Hat subscription for AMQ Streams. If
you do not have an AMQ Streams subscription, contact your sales representative for
more information.
Prerequisites
The AMQ Streams Operator must be installed. If using version 1.4.0 or higher you can use self-
provisioning. If otherwise, you need to create the Kafka instance.
NOTE
Procedure
1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
d. Click Create.
4. If necessary, select jaeger-system from the Project menu. You may have to wait a few
moments for the Operators to be copied to the new project.
5. Click the Jaeger Operator. On the Overview tab, under Provided APIs, the Operator provides a
single link.
18
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
7. On the Create Jaeger page, replace the default all-in-one yaml text with your streaming YAML
configuration, for example:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger-streaming
spec:
strategy: streaming
collector:
options:
kafka:
producer:
topic: jaeger-spans
#Note: If brokers are not defined,AMQStreams 1.4.0+ will self-provision Kafka.
brokers: my-cluster-kafka-brokers.kafka:9092
storage:
type: elasticsearch
ingester:
options:
kafka:
consumer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
2. On the Jaegers page, click the name of the Jaeger instance, for example, jaeger-streaming.
3. On the Jaeger Details page, click the Resources tab. Wait until all the pods have a status of
"Running" before continuing.
Prerequisites
Procedure
1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.
$ oc login https://{HOSTNAME}:8443
$ oc new-project jaeger-system
19
OpenShift Container Platform 4.6 Jaeger
3. Create a custom resource file named jaeger-streaming.yaml that contains the text of the
example file in the previous procedure.
5. Run the following command to watch the progress of the pods during the installation process:
Once the installation process has completed, you should see output similar to the following:
The Jaeger custom resource (CR) defines the architecture and settings to be used when creating the
Jaeger resources. You can modify these parameters to customize your Jaeger implementation to your
business needs.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: name
spec:
strategy: <deployment_strategy>
allInOne:
options: {}
resources: {}
agent:
options: {}
resources: {}
collector:
options: {}
resources: {}
20
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
sampling:
options: {}
storage:
type:
options: {}
query:
options: {}
resources: {}
ingester:
options: {}
resources: {}
options: {}
21
OpenShift Container Platform 4.6 Jaeger
The following example YAML is the minimum required to create a Jaeger instance using the default
settings.
apiVersion: jaegertracing.io/v1
kind: Jaeger
22
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
metadata:
name: jaeger-all-in-one-inmemory
The Jaeger Collector is the component responsible for receiving the spans that were captured by the
tracer and writing them to a persistent storage (Elasticsearch) when using the production strategy, or
to AMQ Streams when using the streaming strategy.
The collectors are stateless and thus many instances of Jaeger Collector can be run in parallel.
Collectors require almost no configuration, except for the location of the Elasticsearch cluster.
23
OpenShift Container Platform 4.6 Jaeger
NOTE
You can configure the Collector to autoscale; the Collector will scale up or down based on the CPU
and/or memory consumption. Configuring the Collector to autoscale can help you ensure your Jaeger
environment scales up during times of increased load, and scales down when less resources are needed,
saving on costs. You configure autoscaling by setting the autoscale parameter to true and specifying a
value for .spec.collector.maxReplicas along with a reasonable value for the resources that you expect
the Collector’s pod to consume. If you do not set a value for .spec.collector.maxReplicas the Operator
will set it to 100.
By default, when there is no value provided for .spec.collector.replicas, the Jaeger Operator creates a
horizontal pod autoscaler (HPA) configuration for the Collector. For more information about HPA, refer
to the Kubernetes documentation.
The following is an example autoscaling configuration, setting the Collector’s limits as well as the
maximum number of replicas:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
collector:
maxReplicas: 5
resources:
limits:
cpu: 100m
memory: 128Mi
The Operator can be used to define sampling strategies that will be supplied to tracers that have been
configured to use a remote sampler.
While all traces are generated, only a few are sampled. Sampling a trace marks the trace for further
processing and storage.
NOTE
24
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
NOTE
This is not relevant if a trace was started by the Istio proxy as the sampling decision is
made there. The Jaeger sampling decision is only relevant when the trace is started by an
application using the Jaeger tracer.
When a service receives a request that contains no trace context, the Jaeger tracer will start a new trace,
assign it a random trace ID, and make a sampling decision based on the currently installed sampling
strategy. The sampling decision is propagated to all subsequent requests in the trace, so that other
services are not making the sampling decision again.
Constant - The sampler always makes the same decision for all traces. It either samples all
traces (sampling.param=1) or none of them (sampling.param=0).
Probabilistic - The sampler makes a random sampling decision with the probability of sampling
equal to the value of the sampling.param property. For example, with sampling.param=0.1
approximately 1 in 10 traces will be sampled.
Rate Limiting - The sampler uses a leaky bucket rate limiter to ensure that traces are sampled
with a certain constant rate. For example, when sampling.param=2.0 it will sample requests with
the rate of 2 traces per second.
Remote - The sampler consults the Jaeger agent for the appropriate sampling strategy to use
in the current service. This allows controlling the sampling strategies in the services from a
central configuration in the Jaeger backend.
Configuration options
spec: that define the sampling
sampling: strategies for tracing.
options: {}
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace
instances being sampled.
25
OpenShift Container Platform 4.6 Jaeger
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: with-sampling
spec:
strategy: allInOne
sampling:
options:
default_strategy:
type: probabilistic
param: 50
You configure storage for the Collector, Ingester, and Query services under spec:storage. Multiple
instances of each of these components can be provisioned as required for performance and resilience
purposes.
Configuration options
spec: that define the storage.
storage:
options: {}
26
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
When the storage:type is set to elasticsearch but there is no value set for
spec:storage:options:es:server-urls, the Jaeger Operator uses the Elasticsearch Operator to create
an Elasticsearch cluster based on the configuration provided in the storage section of the custom
resource file.
Restrictions
You cannot share or reuse a OpenShift Jaeger logging Elasticsearch instance with Jaeger. The
Elasticsearch cluster is meant to be dedicated for a single Jaeger instance.
NOTE
If you already have installed Elasticsearch as part of OpenShift logging, the Jaeger
Operator can use the installed Elasticsearch Operator to provision storage.
The following configuration parameters are for a self-provisioned Elasticsearch instance, that is an
instance created by the Jaeger Operator using the Elasticsearch Operator. You specify configuration
options for self-provisioned Elasticsearch under spec:storage:elasticsearch in your configuration file.
27
OpenShift Container Platform 4.6 Jaeger
28
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
*Each Elasticsearch node can operate with a lower memory setting though this is
NOT recommended for production deployments. For production use, you should
have no less than 16Gi allocated to each Pod by default, but preferably allocate as
much as you can, up to 64Gi per Pod.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
elasticsearch:
nodeCount: 3
resources:
requests:
cpu: 1
memory: 16Gi
limits:
memory: 16Gi
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
elasticsearch:
nodeCount: 1
storage: 1
storageClassName: gp2
size: 5Gi
resources:
requests:
cpu: 200m
memory: 4Gi
limits:
memory: 4Gi
redundancyPolicy: ZeroRedundancy
1 Persistent storage configuration. In this case AWS gp2 with 5Gi size. When no value is specified,
Jaeger uses emptyDir. The Elasticsearch Operator provisions PersistentVolumeClaim and
PersistentVolume which are not removed with Jaeger instance. You can mount the same volumes
if you create a Jaeger instance with the same name and namespace.
29
OpenShift Container Platform 4.6 Jaeger
Jaeger also allows you to use an existing (self-provisioned) Elasticsearch cluster for storage. You do this
by specifying the URL of the existing cluster as the spec:storage:options:es:server-urls value in your
configuration.
Restrictions
There can be only one Jaeger with self-provisioned Elasticsearch instance per namespace.
NOTE
Red Hat does not provide support for your external Elasticsearch instance. You can
review the tested integrations matrix on the Customer Portal.
The following configuration parameters are for an external Elasticsearch instance, that is, an instance
that was not created using the Elasticsearch Operator. You specify configuration options for external
Elasticsearch under spec:storage:options:es in your configuration file.
30
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
31
OpenShift Container Platform 4.6 Jaeger
32
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
Path to a TLS
es: Certificate file, used to
tls: identify this process to
cert: the remote server(s).
33
OpenShift Container Platform 4.6 Jaeger
A time.Duration after 0s
es-archive: which bulk requests are
bulk: committed, regardless
flush-interval: of other thresholds. To
disable the bulk
processor flush interval,
set this to zero.
34
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
[Deprecated - Will be 0
es-archive: removed in a future
max-num-spans: release, use es-
archive.max-doc-
count instead.] The
maximum number of
spans to fetch at a time,
per query, in
Elasticsearch.
The comma-separated
es-archive: list of Elasticsearch
server-urls: servers. Must be
specified as fully
qualified URLs, for
example,
http://localhost:9200.
35
OpenShift Container Platform 4.6 Jaeger
Path to a TLS
es-archive: Certificate file, used to
tls: identify this process to
cert: the remote server(s).
36
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: https://quickstart-es-http.default.svc:9200
index-prefix: my-prefix
tls:
ca: /es/certificates/ca.crt
secretName: jaeger-secret
volumeMounts:
- name: certificates
mountPath: /es/certificates/
readOnly: true
volumes:
- name: certificates
secret:
secretName: quickstart-es-http-certs-public
The following example shows a Jaeger CR using an external Elasticsearch cluster with TLS CA
certificate mounted from a volume and user/password stored in a secret.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
37
OpenShift Container Platform 4.6 Jaeger
type: elasticsearch
options:
es:
server-urls: https://quickstart-es-http.default.svc:9200 1
index-prefix: my-prefix
tls: 2
ca: /es/certificates/ca.crt
secretName: jaeger-secret 3
volumeMounts: 4
- name: certificates
mountPath: /es/certificates/
readOnly: true
volumes:
- name: certificates
secret:
secretName: quickstart-es-http-certs-public
2 TLS configuration. In this case only CA certificate, but it can also contain es.tls.key and es.tls.cert
when using mutual TLS.
4 Volume mounts and volumes which are mounted into all storage components.
Query is a service that retrieves traces from storage and hosts the user interface to display them.
Configuration options
spec: that define the Query
query: service.
options: {}
resources: {}
38
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
apiVersion: jaegertracing.io/v1
kind: "Jaeger"
metadata:
name: "my-jaeger"
spec:
strategy: allInOne
allInOne:
options:
log-level: debug
query:
base-path: /jaeger
Ingester is a service that reads from a Kafka topic and writes to another storage backend (Elasticsearch).
If you are using the allInOne or production deployment strategies, you do not need to configure the
Ingester service.
39
OpenShift Container Platform 4.6 Jaeger
Identifies the Kafka configuration Label for the broker, for example,
kafka: used by the Ingester to consume my-cluster-kafka-
consumer: the messages. brokers.kafka:9092 .
brokers:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-streaming
spec:
strategy: streaming
collector:
options:
kafka:
producer:
topic: jaeger-spans
40
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
options:
kafka:
consumer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
ingester:
deadlockInterval: 5
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
NOTE
You can configure the Ingester to autoscale; the Ingester will scale up or down based on the CPU and/or
memory consumption. Configuring the Ingester to autoscale can help you ensure your Jaeger
environment scales up during times of increased load, and scales down when less resources are needed,
saving on costs. You configure autoscaling by setting the autoscale parameter to true and specifying a
value for .spec.ingester.maxReplicas along with a reasonable value for the resources that you expect
the Ingester’s pod to consume. If you do not set a value for .spec.ingester.maxReplicas the Operator
will set it to 100.
By default, when there is no value provided for .spec.ingester.replicas, the Jaeger Operator creates a
horizontal pod autoscaler (HPA) configuration for the Ingester. For more information about HPA, refer
to the Kubernetes documentation.
The following is an example autoscaling configuration, setting the Ingester’s limits as well as the
maximum number of replicas:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-streaming
spec:
strategy: streaming
ingester:
maxReplicas: 8
resources:
limits:
cpu: 100m
memory: 128Mi
OpenShift Jaeger relies on a proxy sidecar within the application’s pod to provide the agent. The Jaeger
41
OpenShift Container Platform 4.6 Jaeger
OpenShift Jaeger relies on a proxy sidecar within the application’s pod to provide the agent. The Jaeger
Operator can inject Jaeger Agent sidecars into Deployment workloads. You can enable automatic
sidecar injection or manage it manually.
The following snippet shows a simple application that will inject a sidecar, with the Jaeger Agent pointing
to the single Jaeger instance available in the same namespace:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
annotations:
"sidecar.jaegertracing.io/inject": "true" 1
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: acme/myapp:myversion
When the sidecar is injected, the Jaeger Agent can then be accessed at its default location on
localhost.
The following snippet shows the manual definition you can include in your containers section for a
Jaeger Agent sidecar:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: example-statefulset
namespace: example-ns
42
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
labels:
app: example-app
spec:
spec:
containers:
- name: example-app
image: acme/myapp:myversion
ports:
- containerPort: 8080
protocol: TCP
- name: jaeger-agent
image: registry.redhat.io/distributed-tracing/jaeger-agent-rhel7:<version>
# The agent version must match the operator version
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5775
name: zk-compact-trft
protocol: UDP
- containerPort: 5778
name: config-rest
protocol: TCP
- containerPort: 6831
name: jg-compact-trft
protocol: UDP
- containerPort: 6832
name: jg-binary-trft
protocol: UDP
- containerPort: 14271
name: admin-http
protocol: TCP
args:
- --reporter.grpc.host-port=dns:///jaeger-collector-headless.example-ns:14250
- --reporter.type=grpc
The Jaeger Agent can then be accessed at its default location on localhost.
The update approach used by the Jaeger Operator upgrades the managed Jaeger instances to the
version associated with the Operator. Whenever a new version of the Jaeger Operator is installed, all the
Jaeger application instances managed by the Operator will be upgraded to the Operator’s version. For
example, if version 1.10 is installed (both Operator and backend components) and the Operator is
upgraded to version 1.11, then as soon as the Operator upgrade has completed, the Operator will scan for
running Jaeger instances and upgrade them to 1.11 as well.
43
OpenShift Container Platform 4.6 Jaeger
NOTE
When deleting an instance that uses the in-memory storage, all data will be permanently
lost. Data stored in a persistent storage (such as Elasticsearch) will not be deleted when a
Jaeger instance is removed.
Procedure
3. Select the name of the project where the Operators are installed from the Project menu, for
example, jaeger-system.
6. Click the Options menu next to the instance you want to delete and select Delete
Jaeger.
$ oc login
The names of operators have the suffix -operator. The following example shows two Jaeger
Operators and four Jaeger instances:
44
CHAPTER 5. DEPLOYING THE JAEGER STREAMING STRATEGY FROM THE WEB CONSOLE
For example,
For example,
Procedure
After the Jaeger Operator has been removed, if appropriate, remove the Elasticsearch
Operator.
45