Overview

OpenMeter leverages the CloudEvents specification incubated under CNCF for a reason. CloudEvents offers a standardized and flexible way to describe event data, making it easier to connect your services and tools seamlessly.

A usage event can be anything you need to track accurately over time for billing or analytics purposes. For example, a CI/CD product can include active or parallel jobs, build minutes, network traffic, storage used, or other product-related actions.

#Event Format

OpenMeter ingests events using the CloudEvents specification. Below is an example of a CloudEvent in JSON format describing the execution duration of a serverless application:

{
  "specversion": "1.0",
  "type": "request",
  "id": "00001",
  "time": "2024-01-01T00:00:00.001Z",
  "source": "service-0",
  "subject": "customer-1",
  "data": {
    "duration_seconds": "12",
    "method": "GET",
    "route": "/hello"
  }
}

OpenMeter currently supports the JSON format of CloudEvents, with plans to extend support for Protobuf and other formats.

It contains the following properties:

specversion: The CloudEvents spec version (currently 1.0).
type: The event type. This is used to match the event to a meter.
id: The event's unique ID.
time: The event's timestamp in RFC3339 format. Defaults to the time the event was received.
source: The event's source (e.g. the service name).
subject: The event's subject (e.g. the customer ID). Meter values are aggregated by subjects.
data is the event's payload.

The data property can contain any valid JSON object, and when configuring meters using JSONPath, individual values can be extracted.

CloudEvents is adopted by many Cloud Native solutions, making it effortless to extract usage data from infrastructure solutions. SDKs are available for common programming languages, simplifying the creation, validation, and reporting of usage events to OpenMeter

#Event Processing

OpenMeter continuously processes usage events, allowing you to update meters in real-time. Once an event is ingested, OpenMeter aggregates the data based on your defined meters. For example, you can define meters called "Parallel jobs", and OpenMeter will aggregate the maximum number of jobs by each customer over a given time period.

Using an example, let’s dive into how OpenMeter’s event processing works. Imagine you want to track serverless execution duration by endpoint and you defined the following meter:

Create Meter in Cloud

Open Source Config:

meters:
  - slug: api_requests_total
    description: API Requests
    eventType: request
    valueProperty: $.duration_seconds
    aggregation: SUM
    groupBy:
      method: $.method
      route: $.route

The meter config above tells OpenMeter that expect CloudEvents with type=request where the usage value is stored in the data.duration_seconds and we need to sum them by data.route. OpenMeter will track the usage value for every time window when least one event was reported and tracks it for every subject and groupBy permutation.

Note that $.duration_seconds is a JSONPath expression to access the data.duration_seconds property, providing powerful capabilities to extract values from nested data properties.

For example, when you send your first event:

{
  "specversion": "1.0",
  "type": "request",
  "id": "00001",
  "time": "2024-01-01T00:00:00.001Z",
  "source": "service-0",
  "subject": "customer-1",
  "data": {
    "duration_seconds": "10",
    "method": "GET",
    "route": "/hello"
  }
}

OpenMeter will track the usage value for the time window and cusomer as:

windowstart   = "2024-01-01T00:00"
windowend     = "2024-01-01T00:01"
subject       = "customer-1"
duration_seconds   = 10
method        = "GET"
route         = "/hello"

When sending the second event (with a different id and duration_seconds value):

{
  "specversion": "1.0",
  "type": "request",
  "id": "00002",
  "time": "2024-01-01T00:00:00.001Z",
  "source": "service-0",
  "subject": "customer-1",
  "data": {
    "duration_seconds": "20",
    "method": "GET",
    "route": "/hello"
  }
}

OpenMeter will increase sum of the duration for the two events for the same time window (windowstart, windowend), method, route and subject:

windowstart   = "2024-01-01T00:00"
windowend     = "2024-01-01T00:01"
subject       = "customer-1"
duration      = 30
method        = "GET"
route         = "/hello"

#Event Deduplication

CloudEvents are unique by id and source, see CloudEvent's specification:

Producers MUST ensure that source + id is unique for each distinct event. If a duplicate event is re-sent (e.g. due to a network error) it MAY have the same id. Consumers MAY assume that Events with identical source and id are duplicates.

OpenMeter deduplicates events by id and source. This ensures that if multiple events with the same id and source are sent, they will be processed only once. This is useful when you want to retry or replay events in your infrastructure.