Parsing Logs into Metered Usage for BillingHow to turn logs into metered usage

In some instances, we need to meter usage of infrastructure components for billing or cloud cost use cases where integrating collectors or making HTTP requests to report usage is challenging. In this case, we can parse logs into billable metered usage events. Using logs for metering is especially powerful to meter usage of low-level infrastructure components like message queues where integrating collectors is impossible or in serverless environments where an extra HTTP request would increase the cloud bill. In this article, we will use Vector to collect logs from our application, transform them into usage, and report to a separate usage metering solution like OpenMeter.

#Collecting and Parsing Logs via Vector

As previously mentioned, we will use the open-source Vector project to collect logs from our demo application and transform logs into billable events for usage metering. Vector supports multiple sources to consume logs and metrics from various solutions, including popular solutions like Syslog, Kubernetes, DataDog, and Logstash. Our data will flow as the following:

You can also find the executable version of this example on our GitHub under examples/ingest-logs.

In our example, we have a demo application that produces the following JSON request log every second; this is the log that we are going to turn into usage events and meter via OpenMeter:

{
  "level": "info",
  "msg": "http request",
  "user": "customer-1",
  "method": "GET",
  "path": "/api/demo",
  "response_time": 5
}

Vector has source, transform, and sink components that we can combine to build a log processing pipeline. For example, to consume the sample logs generated by our application running in Docker, we will use the docker_logs source components. In the next step, we will convert these logs into CloudEvents format and send them to Openmeter. Our complete processing pipeline will look in Vector as the following:

To configure our Vector source to consume logs, we will define the source in the vector.yaml config file as the following:

sources:
  # Read logs from Docker container(s)
  docker_logs:
    type: docker_logs
    docker_host: unix:///var/run/docker.sock
    include_containers:
      - demologs

Note how we filtered for a specific container demologs via include_containers. As a next step, we will pipe the output of our source into a transform to parse the log payload using the Vector Remap Language (VRL). The VRL syntax can look unusual initially, but it provides a powerful way to build log processing pipelines.

transforms:
  # Parse JSON logs
  parse:
    type: remap
    inputs:
      - docker_logs
    source: |
      msg, err = parse_json(.message)
      if err != null { log(err, level: "error") }
      .message = msg

Once we JSON parsed the log payload, we will pipe the output of this parse transform into a filter transform which will drop every log where the message is not an HTTP request:

transforms:
  # Filter for specific logs
  filter_requests:
    type: filter
    inputs:
      - parse
    condition: .message.msg == "http request"

In our final transformation, we turn the log message into a CloudEvents format supported by OpenMeter's ingest API. Using the Vector Remap Language, we are adding and renaming properties to convert our parsed log into the CloudEvents format:

transforms:
  # Turn log into CloudEvents
  cloudevents:
    type: remap
    inputs:
      - filter_requests
    source: |
      .cloudevent.specversion = "1.0"
      .cloudevent.id = .timestamp
      .cloudevent.source = .container_id
      .cloudevent.type = "api-calls"
      .cloudevent.subject = .message.user
      .cloudevent.time = .timestamp
      response_time, err = to_string(.message.response_time)
      if err != null { log(err, level: "error") }
      .cloudevent.data.duration_seconds = response_time
      .cloudevent.data.method = .message.method
      .cloudevent.data.path = .message.path
      . = .cloudevent

Finally, we send the output of the CloudEvents transform to the OpenMeter HTTP ingest API via a Vector sink as:

sinks:
  # Send CloudEvents to OpenMeter ingestion API
  openmeter:
    type: http
    inputs:
      - cloudevents
    uri: http://openmeter:8888/api/v1/events
    method: post
    request:
      headers:
        Content-Type: application/cloudevents-batch+json
    encoding:
      codec: json

#Summary

In scenarios where integrating SDKs or directly reporting usage from infrastructure components isn't feasible, parsing logs and converting them into billable usage events presents an effective metering solution. This approach is particularly valuable for measuring the usage of low-level infrastructure components, legacy systems, or serverless applications. It circumvents the need for integrating collectors or triggering HTTP requests for every transaction, which could inflate cloud costs or be unachievable in some serverless solutions.

Try the executable log to metered usage example on our GitHub.

Need to parse logs into usage?

Get started with OpenMeter Cloud today!

Need to parse logs into usage?

Join OpenMeter Cloud