The Metering Challenge

Building a metering system that's scalable, accurate, and cost-effective is challenging:

• Cost: Database writes at scale can be expensive.
• Accuracy: Metrics systems may not provide the needed accuracy due to sampling.
• Latency: Periodic batch processing in data warehouses can introduce latency.

#Operational and Auditable Data

Auditable data is data where losing any record is intolerable and complete retention is necessary. When using an auditable dataset, it is expected to be comprehensive and complete. Examples of auditable data include transaction logs, replication logs, and billing/finance events.

Operational data, conversely, doesn't require strict completeness. To keep costs manageable, systems often use sampling, and some degree of data loss is acceptable. Tools designed to manage operational data often prioritize efficiency, bypassing retries and costly guarantees of exactly-once delivery. Examples of operational data include telemetry, metrics, and contextual data that describe each request and system component.

OpenMeter is designed to give auditable guarantees required for billing while keeping the scalability and real-time benefits typically seen only with operational data.

Learn more about Auditable or Operational data.

#Inaccuracy of Monitoring Systems

Monitoring systems are tailored to gather, process, and store operational data. However, these systems often lack the level of consistency crucial for auditable use cases like billing, where legally binding contracts require accuracy and complete data.

Metrics systems like Prometheus scale well for many time-series data points, but they require keeping cardinality low, which makes it difficult to track individual user resource consumption at scale.

To learn more about Prometheus accuracy, check out this conversation with contributors and the challenges of adjusting scraping to increase counter accuracy.

#Expensive Databases Writes

Scaling databases to accommodate write-intensive event ingestion and real-time queries can be costly. Not only do you need to store every single record in the database, but you must also scan through them for aggregation in your queries, driving up load and cost.

This is why OpenMeter uses stream processing to pre-aggregate usage data before storing it in a long-term database. As part of the data processing, we send usage events to a message queue first to avoid overwhelming the database and to protect against data loss in case of failures. The long-term aggregates are then stored in an OLAP database designed to store and query large volumes of analytics data efficiently.

#Stale Batch Processing

Storing all records in a data warehouse and processing them in batches may be cost-effective, but it leads to outdated meters. Most companies run daily batch processing, resulting in stale data that is unsuitable for immediate-response product use cases, such as usage gating, customer dashboards, and billing thresholds.

Stale usage aggregation also means slower feedback loops around consumption, which can lead to unexpected charges and customer dissatisfaction.