OpenTelemetry | OpenMeter

Non-sampled logs (e.g., access logs) are excellent sources of usage information. Combined with OpenTelemetry open standard for log forwarding, you can easily extract usage information from your logs and forward them to OpenMeter.

#Prerequisites

You will need an OpenTelemetry-compatible log-forwarding solution. The easiest way to get started is to use the OpenTelemetry Collector.

#Configuration

First, create a new YAML file for the collector configuration. You will have to use the otel_log Redpanda Connect input:

input:
  otel_log:
    # Point your log forwarder to this address using the OTLP gRPC protocol.
    address: 127.0.0.1:4317

OpenMeter Collector

This is a custom input plugin that is not part of the official Redpanda Connect distribution. You can find the source code of the plugin here.

Next, you need to configure the mapping from your log schema to CloudEvents using bloblang:

pipeline:
  processors:
    - mapping: |
        root = {
          "id": uuid_v4(),
          "specversion": "1.0",
          "type": "api-calls",
          "source": "otlp-log",
          "time": this.record.attributes.time,
          "subject": this.record.attributes.subject,
          "data": {
            "method": this.record.attributes.method,
            "path": this.record.attributes.path,
            "region": this.record.attributes.region,
            "zone": this.record.attributes.zone,
            "duration_ms": this.record.attributes.duration,
          },
        }

About log attributes

this.record.attributes contains the log attributes extracted by the otel_log input plugin.

Finally, you need to configure the OpenMeter output:

# Send processed events to OpenMeter
output:
  label: 'openmeter'
  drop_on:
    error: false
    error_patterns:
      - Bad Request
    output:
      http_client:
        url: '${OPENMETER_URL:https://openmeter.cloud}/api/v1/events'
        verb: POST
        headers:
          Authorization: 'Bearer ${OPENMETER_TOKEN:}'
          Content-Type: 'application/json'
        timeout: 30s
        retry_period: 15s
        retries: 3
        max_retry_backoff: 1m
        # Maximum number of concurrent requests
        max_in_flight: 64
        batch_as_multipart: false
        drop_on:
          - 400
        # Batch settings for efficient API usage
        batching:
          # Send up to 100 events in a single request
          count: 100
          # Or send after 1 second, whichever comes first
          period: 1s
          processors:
            # Track metrics on sent events
            - metric:
                type: counter
                name: openmeter_events_sent
                value: 1
            # Convert batch to JSON array format
            - archive:
                format: json_array
        dump_request_log_level: DEBUG

Read more about configuring Redpanda Connect in the OpenMeter Collector guide.

Example on GitHub

Check out a fully working example here.

#Installation

Check out the OpenMeter Collector guide for installation instructions.