Parsing Logs into Metered Usage for BillingHow to turn logs into metered usage


In some instances, we need to meter usage of infrastructure components for billing or cloud cost use cases where integrating collectors or making HTTP requests to report usage is challenging. In this case, we can parse logs into billable metered usage events. Using logs for metering is especially powerful to meter usage of low-level infrastructure components like message queues where integrating collectors is impossible or in serverless environments where an extra HTTP request would increase the cloud bill. In this article, we will use Vector to collect logs from our application, transform them into usage, and report to a separate usage metering solution like OpenMeter.
Collecting and Parsing Logs via Vector
As previously mentioned, we will use the open-source Vector project to collect logs from our demo application and transform logs into billable events for usage metering. Vector supports multiple sources to consume logs and metrics from various solutions, including popular solutions like Syslog, Kubernetes, DataDog, and Logstash. Our data will flow as the following:


You can also find the executable version of this example on our GitHub under examples/ingest-logs.
In our example, we have a demo application that produces the following JSON request log every second; this is the log that we are going to turn into usage events and meter via OpenMeter:
{
"level": "info",
"msg": "http request",
"user": "customer-1",
"method": "GET",
"path": "/api/demo",
"response_time": 5
}
Vector has source
, transform
, and sink
components that we can combine to build a log processing pipeline. For example, to consume the sample logs generated by our application running in Docker, we will use the docker_logs
source components. In the next step, we will convert these logs into CloudEvents
format and send them to Openmeter. Our complete processing pipeline will look in Vector as the following:


To configure our Vector source to consume logs, we will define the source
in the vector.yaml
config file as the following:
sources:
# Read logs from Docker container(s)
docker_logs:
type: docker_logs
docker_host: unix:///var/run/docker.sock
include_containers:
- demologs
Note how we filtered for a specific container demologs
via include_containers
. As a next step, we will pipe the output of our source into a transform to parse the log payload using the Vector Remap Language (VRL). The VRL syntax can look unusual initially, but it provides a powerful way to build log processing pipelines.
transforms:
# Parse JSON logs
parse:
type: remap
inputs:
- docker_logs
source: |
msg, err = parse_json(.message)
if err != null { log(err, level: "error") }
.message = msg
Once we JSON parsed the log payload, we will pipe the output of this parse transform into a filter transform which will drop every log where the message is not an HTTP request:
transforms:
# Filter for specific logs
filter_requests:
type: filter
inputs:
- parse
condition: .message.msg == "http request"
In our final transformation, we turn the log message into a CloudEvents format supported by OpenMeter's ingest API. Using the Vector Remap Language, we are adding and renaming properties to convert our parsed log into the CloudEvents format:
transforms:
# Turn log into CloudEvents
cloudevents:
type: remap
inputs:
- filter_requests
source: |
.cloudevent.specversion = "1.0"
.cloudevent.id = .timestamp
.cloudevent.source = .container_id
.cloudevent.type = "api-calls"
.cloudevent.subject = .message.user
.cloudevent.time = .timestamp
response_time, err = to_string(.message.response_time)
if err != null { log(err, level: "error") }
.cloudevent.data.duration_ms = response_time
.cloudevent.data.method = .message.method
.cloudevent.data.path = .message.path
. = .cloudevent
Finally, we send the output of the CloudEvents transform to the OpenMeter HTTP ingest API via a Vector sink as:
sinks:
# Send CloudEvents to OpenMeter ingestion API
openmeter:
type: http
inputs:
- cloudevents
uri: http://openmeter:8888/api/v1/events
method: post
request:
headers:
Content-Type: application/cloudevents-batch+json
encoding:
codec: json
Summary
In scenarios where integrating SDKs or directly reporting usage from infrastructure components isn't feasible, parsing logs and converting them into billable usage events presents an effective metering solution. This approach is particularly valuable for measuring the usage of low-level infrastructure components, legacy systems, or serverless applications. It circumvents the need for integrating collectors or triggering HTTP requests for every transaction, which could inflate cloud costs or be unachievable in some serverless solutions.
Try the executable log to metered usage example on our GitHub.
