High Availability
The OpenMeter Collector is designed to be a high availability system. It is built on top of Redpanda Connect, a robust stream processing tool that can connect to a wide range of data sources and sinks. The tool's strong delivery guarantees and capability to retry failed messages make it an excellent choice as a collector for OpenMeter.
Buffering
The OpenMeter Collector uses a persistent queue to buffer events. This allows the collector store events on a persistent disk and retry them later in case of network failures or other issues. Based on the load and disk size, the collector can buffer events for extended periods of time like hours or days and safely replay them to OpenMeter Cloud when the network is restored.
Retries
The OpenMeter Collector will retry failed messages up to 3 times (configurable). If the message still fails after 3 retries, it will be dropped.
Deduplication
The OpenMeter Collector will deduplicate events based on the id and source
field. If a message with the same id is received multiple times, only the
first occurrence will be processed.
SDK Passthrough
OpenMeter SDKs can be pointed to the OpenMeter Collector endpoint to enrich events with additional metadata or increase the reliability of event delivery by leveraging the collector's buffer and retry capabilities.