Observe metrics, traces, and logs using OpenSearch¶

The sample shop service will be used in this guide. Follow the steps given below to observe BI tracing, metrics, and logs in OpenSearch.

Step 1 - Set up OpenSearch deployment¶

Main components:
- Sample Service: service application that is being observed.
- OpenSearch: stores and indexes logs/traces; exposes API and metrics ports.
- OpenSearch Dashboards: visualizes data from OpenSearch via web UI.
- Data Prepper: receives and processes OpenTelemetry data, sends it to OpenSearch.
- Fluent Bit: collects app logs and forwards them for indexing.
- Setup Container: automates initial setup — creates index templates and imports dashboards.
Download and unzip the opensearch-observability-dashboard.zip in your local machine.
The structure of the opensearch-observability-dashboard directory is as follows.

    .
    |____config
    |       |____dashboards
    |       |       |____opensearch_dashboards.yml
    |       |____data-prepper
    |       |     |____pipelines.yaml
    |       |____fluent-bit
    |       |      |____fluent-bit.conf
    |       |      |____parser.conf
    |       |      |____scripts
    |       |            |____scripts.lua
    |       |____.env
    |
    |____logs
    |         |____ballerina
    |
    |____setup
    |       |____opensearch-dashboards-template.ndjson
    |       |____index-template-request.json
    |
    |____docker-compose.yml

Update OPENSEARCH_INITIAL_ADMIN_PASSWORD in the path/to/opensearch-observability-dashboard/config/.env file.

OPENSEARCH_INITIAL_ADMIN_PASSWORD=<PASSWORD> # Password for the OpenSearch admin user

This password will be used to access the OpenSearch server.

Navigate to the path/to/opensearch-observability-dashboard directory and run the following docker compose command in the terminal to start the OpenSearch deployment along with the BI application.

docker compose -f docker-compose.yml up -d

Step 2 - Set up BI application for observability¶

Create the sample shop service.

Navigate to file explorer view and add the following to main.bal.

import ballerinax/metrics.logs as _;
import ballerinax/jaeger as _;

Navigate to file explorer view and enable observability by adding the following section to the Ballerina.toml.
```
[build-options]
observabilityIncluded=true
```

Navigate to file explorer view and create the Config.toml file in the package directory to set the runtime configurations as follows.

[ballerina.observe]
metricsLogsEnabled = true
tracingEnabled = true
tracingProvider = "jaeger"

[ballerinax.jaeger]
agentHostname = "localhost"
agentPort = 4317
samplerType = "const"
samplerParam = 1.0
reporterFlushInterval = 2000
reporterBufferSize = 1000

[ballerinax.metrics.logs]
logFilePath = "<PATH>/<TO>/opensearch-observability-dashboard/logs/ballerina/<SERVICE_NAME>/app.log"

These configurations enable metrics logs and traces in the BI application and configures the Jaeger exporter.

The table below provides the descriptions of each configuration option and possible values that can be assigned.

Configuration key	Description	Default value	Possible values
ballerinax.jaeger. agentHostname	Hostname of the Jaeger agent	localhost	IP or hostname of the Jaeger agent. Can be localhost if running on same node as Ballerina.
ballerinax.jaeger. agentPort	Port of the Jaeger agent	4317	The port on which the Jaeger agent is listening.
ballerinax.jaeger. samplerType	Type of sampling methods used in Jaeger tracer	const	`const`, `probabilistic`, or `ratelimiting`
ballerinax.jaeger. samplerParam	Floating value parameter for sampler	1.0	const: `0` (no sampling) or `1` (sample all) probabilistic: `0.0` to `1.0` ratelimiting: positive integer (rate/sec)
ballerinax.jaeger. reporterFlushInterval	Interval for sending spans to agent	2000	Any positive integer value
ballerinax.jaeger. reporterBufferSize	Queue size of Jaeger client	1000	Any positive integer value
ballerinax.metrics.logs. logFilePath	Path to application log file	`none`	`PATH/TO/opensearch-observability-dashboard/logs/ballerina/<SERVICE_NAME>/app.log`

Step 3 - Run the BI project¶

When observability is enabled, the BI runtime collects metrics logs and traces.

Start the service in BI.

$ bal run

Compiling source

Running executable

ballerina: started publishing traces to Jaeger on localhost:4317

Step 4 - Send requests¶

Send requests to http://localhost:8090/shop/products.

Example cURL commands:

$ curl -X GET http://localhost:8090/shop/products

$ curl -X POST http://localhost:8090/shop/product \
-H "Content-Type: application/json" \
-d '{
    "id": 4, 
    "name": "Laptop Charger", 
    "price": 50.00
}'

$ curl -X POST http://localhost:8090/shop/order \
-H "Content-Type: application/json" \
-d '{
    "productId": 1, 
    "quantity": 1
}'

$ curl -X GET http://localhost:8090/shop/order/0

Step 5 - View distributed tracing on the OpenSearch Dashboard¶

Open the OpenSearch Dashboard in your browser at http://localhost:5601 and navigate to the Traces tab within the Observability section.

The following image shows a sample of the tracing information available in OpenSearch.
The following image shows the span details in OpenSearch.
The service map shows the relationship between different services in the system.
The following image shows the service details.

Step 6 - View metrics on OpenSearch dashboard¶

Open the OpenSearch Dashboard in your browser at http://localhost:5601 and navigate to the Dashboards tab under OpenSearch Dashboards section.

Then click on the Integration metrics dashboard to view the metrics.

Step 7 - View logs on OpenSearch dashboard¶

Open the OpenSearch Dashboard in your browser at http://localhost:5601 and navigate to the Dashboards tab under OpenSearch Dashboards section.

Then click on the Integration logs dashboard to view the integration logs.