In some instances, we need to meter usage of infrastructure components for billing or cloud cost use cases where integrating collectors or making HTTP requests to report usage is challenging. In this case, we can parse logs into billable metered usage events. Using logs for metering is especially powerful to meter usage of low-level infrastructure components like message queues where integrating collectors is impossible or in serverless environments where an extra HTTP request would increase the cloud bill. In this article, we will use Vector to collect logs from our application, transform them into usage, and report to a separate usage metering solution like OpenMeter.
Collecting and Parsing Logs via Vector
As previously mentioned, we will use the open-source Vector project to collect logs from our demo application and transform logs into billable events for usage metering. Vector supports multiple sources to consume logs and metrics from various solutions, including popular solutions like Syslog, Kubernetes, DataDog, and Logstash. Our data will flow as the following:
You can also find the executable version of this example on our GitHub under examples/ingest-logs.
In our example, we have a demo application that produces the following JSON request log every second; this is the log that we are going to turn into usage events and meter via OpenMeter:
Vector has source
, transform
, and sink
components that we can combine to
build a log processing pipeline. For example, to consume the sample logs
generated by our application running in Docker, we will use the docker_logs
source components. In the next step, we will convert these logs into
CloudEvents
format and send them to Openmeter. Our complete processing
pipeline will look in Vector as the following:
To configure our Vector source to consume logs, we will define the source
in
the vector.yaml
config file as the following:
Note how we filtered for a specific container demologs
via
include_containers
. As a next step, we will pipe the output of our source into
a transform to parse the log payload using the
Vector Remap Language (VRL). The VRL
syntax can look unusual initially, but it provides a powerful way to build log
processing pipelines.
Once we JSON parsed the log payload, we will pipe the output of this parse transform into a filter transform which will drop every log where the message is not an HTTP request:
In our final transformation, we turn the log message into a CloudEvents format supported by OpenMeter's ingest API. Using the Vector Remap Language, we are adding and renaming properties to convert our parsed log into the CloudEvents format:
Finally, we send the output of the CloudEvents transform to the OpenMeter HTTP ingest API via a Vector sink as:
Summary
In scenarios where integrating SDKs or directly reporting usage from infrastructure components isn't feasible, parsing logs and converting them into billable usage events presents an effective metering solution. This approach is particularly valuable for measuring the usage of low-level infrastructure components, legacy systems, or serverless applications. It circumvents the need for integrating collectors or triggering HTTP requests for every transaction, which could inflate cloud costs or be unachievable in some serverless solutions.
Try the executable log to metered usage example on our GitHub.