ClickHouse | OpenMeter

OpenMeter can integrate with your ClickHouse database to collect usage data directly from your database. By leveraging your existing data source for usage metering, you can significantly decrease the upfront cost of changing your data collection strategy.

#Configuration

First, create a new YAML file for the collector configuration. You will have to use the sql_select Redpanda Connect input:

input:
  sql_select:
    driver: clickhouse
    dsn: 'clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&...&paramN=valueN]'
    table: YOUR_TABLE
    columns:
      - id
      - data_column
      - subject_column
      - time
    where: time >= ?
    args_mapping: root = [ now().ts_unix() - 30 ]

The above section will tell Redpanda Connect how to read data from your database.

Next, you need to configure the mapping from your database schema to CloudEvents using bloblang:

pipeline:
  processors:
    - mapping: |
        root = {
          "id": this.id,
          "specversion": "1.0",
          "type": "your-usage-event-type",
          "source": "clickhouse",
          "time": this.time,
          "subject": this.subject_column,
          "data": {
            "data": this.data_column,
          },
        }

Finally, you need to configure the OpenMeter output:

# Send processed events to OpenMeter
output:
  label: 'openmeter'
  drop_on:
    error: false
    error_patterns:
      - Bad Request
    output:
      http_client:
        url: '${OPENMETER_URL:https://openmeter.cloud}/api/v1/events'
        verb: POST
        headers:
          Authorization: 'Bearer ${OPENMETER_TOKEN:}'
          Content-Type: 'application/json'
        timeout: 30s
        retry_period: 15s
        retries: 3
        max_retry_backoff: 1m
        # Maximum number of concurrent requests
        max_in_flight: 64
        batch_as_multipart: false
        drop_on:
          - 400
        # Batch settings for efficient API usage
        batching:
          # Send up to 100 events in a single request
          count: 100
          # Or send after 1 second, whichever comes first
          period: 1s
          processors:
            # Track metrics on sent events
            - metric:
                type: counter
                name: openmeter_events_sent
                value: 1
            # Convert batch to JSON array format
            - archive:
                format: json_array
        dump_request_log_level: DEBUG

Read more about configuring Redpanda Connect in the OpenMeter Collector guide.

#Installation

Check out the OpenMeter Collector guide for installation instructions.