Event Buffering

The OpenMeter Collector can operate in a passthrough mode where it buffers events on a local disk. This allows the collector to continue operating even if the network is down for an extended period of time. When the network is restored, the collector will replay the buffered events to OpenMeter Cloud.

#How Does Event Buffering Work?

OpenMeter offers SDKs for Node.js, Python, Go, and other languages that communicate with the OpenMeter API via HTTP. Installing the OpenMeter Collector with buffer enabled lets you point your SDKs to the OpenMeter Collector instead of sending events directly to OpenMeter Cloud. In this configuration, the OpenMeter SDKs will first send the event to the OpenMeter Collector, which will forward the events to OpenMeter Cloud and buffer them, retrying in the case of network failure.

To increase the resilience of your metering pipeline, the OpenMeter Collector comes with:

• Buffering
• Retries and backoff
• Deduplication
• Otel Metrics and Logging

When buffering is enabled, in the case of connectivity issues, the OpenMeter Collector stores events on an attached persistent volume until the network recovers. When the events are replayed, the OpenMeter Cloud deduplicates events based on their ID and source properties to ensure each event is processed once and exactly once. The OpenMeter Collector also provides visibility into the buffer and processing states by exposing Prometheus metrics. The available space on the attached disk determines the size of the buffer.

#Get Started

You can enable buffering with the OpenMeter Collector in two simple steps:

#1. Install the OpenMeter Collector

# Set your OpenMeter token
export OPENMETER_TOKEN=om_...
 
# Get the latest version or visit https://github.com/openmeterio/openmeter/pkgs/container/helm-charts%2Fbenthos-collector
export LATEST_VERSION=$(curl -s https://api.github.com/repos/openmeterio/openmeter/releases/latest | jq -r '.tag_name' | cut -c 2-)
 
# Install the collector in the openmeter-collector namespace
helm upgrade openmeter-collector oci://ghcr.io/openmeterio/helm-charts/benthos-collector \
  --version=${LATEST_VERSION} \
  --install --wait --create-namespace \
  --namespace openmeter-collector \
  --set fullnameOverride=openmeter-collector \
  --set openmeter.token=${OPENMETER_TOKEN} \
  --set service.enabled=true \
  --set storage.enabled=true \
  --set preset=http-server

#2. Point your SDKs to the installed collector

import { OpenMeter } from '@openmeter/sdk';
 
const openmeter = new OpenMeter({
  // Depends on your Kubernetes cluster configuration
  baseUrl: 'http://openmeter-collector.svc',
});