Hi,
I’m trying to use some of the more advanced features of fluent-bit, like processors. Specifically content_modifiers are the issue, is there a way to know what flags the fluent-bit on the base-torizon OS: 6.5.0 quarterly, verdin-imx8mp, torizon-core-docker was built with. From fluent-bit source, the cmake options are on by default, however in all my testing I can not grab any trace or debug logs or get processors to do anything. DEFINE_OPTION(FLB_PROCESSOR_CONTENT_MODIFIER "Enable content modifier processor" ON) which makes me think these are disabled.
Here is an example, this is different from the default fluent-bit.conf, I’ve changed to fluent-bit.yaml. With the fluent-bit.service using ExecStart=/usr/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.yaml. I’ve left in the full content_modifier anyone wants to use this in future, extracting Nginx logs with a regex to a nest of keys; logs.remote, logs.host… etc.
fluent-bit.yaml
pipeline:
inputs:
- name: dummy
dummy: '{"key1": "123.4"}'
processors:
logs:
- name: content_modifier
action: upsert
key: "key1"
value: "5678"
- name: forward
listen: 127.0.0.1
port: 24224
buffer_chunk_size: 1M
buffer_max_size: 6M
tag: container.logs
processors:
logs:
- name: content_modifier
action: extract
key: log
pattern: ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)((?<path>[^\"]*?)(\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"
log
{
"_index": "fluent-bit",
"_id": "wLvcxZAB4UsbSx_KpOgM",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2024-07-18T12:41:28.916Z",
"key1": "123.4",
"device": "verdin-imx8mp-xxxxxxxx"
},
"fields": {
"@timestamp": [
"2024-07-18T12:41:28.916Z"
]
},
"sort": [
1721306488916
]
}
The key hear should have been modified to 5678.
Any points would be great before I rebuild the fluent-bit binary 
Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!
Did you like my answer?
Hand-shake,
To address your concerns regarding Fluent Bit’s advanced features, specifically the content modifier processors, on TorizonCore 6.5.0 with Verdin iMX8MP, it’s important to first verify the build configuration of Fluent Bit included in the OS. While Fluent Bit’s source code may have certain features enabled by default, the actual build options used when compiling Fluent Bit for inclusion in TorizonCore could differ based on the specific requirements or optimizations decided by the Toradex team.
Given your experience with the inability to utilize processors as expected, it seems plausible that the build of Fluent Bit included in TorizonCore might not have all features enabled, including the content modifier processor. Unfortunately, without direct access to the build configuration used by Toradex, it’s challenging to confirm this hypothesis.
However, if you are encountering limitations with the pre-installed Fluent Bit on TorizonCore, you have a few options:
-
Custom Build of Fluent Bit: You could compile Fluent Bit from source with the desired configuration options explicitly enabled. This approach allows you to ensure that the content modifier processor and any other required features are available. You would need to replace the Fluent Bit binary on your device with your custom-built version.
-
Debugging and Logging: To further investigate why processors are not behaving as expected, ensure that you are using Fluent Bit’s debugging options to their fullest. Running Fluent Bit with the -v or --verbose flag can provide more insight into its operations and might highlight any issues with processor configuration or execution.
-
Configuration Syntax: Double-check the syntax and structure of your Fluent Bit configuration, especially since you’ve transitioned to using the YAML format. YAML is sensitive to indentation and formatting, and any discrepancies could lead to unexpected behavior.
-
Fluent Bit Documentation and Community: While direct support from Toradex regarding Fluent Bit’s build configuration might be limited, the Fluent Bit community and documentation can be invaluable resources. They might offer insights into common pitfalls with processors or configuration nuances that are not immediately apparent.
-
Alternative Logging Solutions: If Fluent Bit’s built-in processors cannot meet your needs due to build constraints, consider using external tools or scripts to preprocess logs before they are ingested by Fluent Bit. This approach can offer greater flexibility, albeit at the cost of additional complexity in your logging pipeline.
In summary, while the exact build options of Fluent Bit in TorizonCore might not be readily available, exploring custom builds, leveraging verbose debugging, and consulting Fluent Bit’s
Please, also check if these links can help you solve your problem:
Device Monitoring in Torizon OS | Toradex Developer Center.
Greetings @dk-coligo,
Let me see if I can help clarify what is going on here.
is there a way to know what flags the fluent-bit on the base-torizon OS: 6.5.0 quarterly, verdin-imx8mp, torizon-core-docker was built with.
This is the Yocto recipe we use to add Fluent-bit to our OS: meta-toradex-torizon/recipes-extended/fluent-bit/fluent-bit_git.bb at kirkstone-6.x.y · torizon/meta-toradex-torizon · GitHub
You can see all the cmake build options we explicitly set in this file. As well as any other configurations.
which makes me think these are disabled.
Well the issue is that you’re looking at the latest source for Fluent-bit. Notice in the Yocto recipe I linked we’re building Fluent-bit version 2.2.3. The processor content modifier feature was added with Fluent-bit version 3.0.0: processor_content_modifier: new processor for easy data manipulation · fluent/fluent-bit@1ab1650 · GitHub
So this version of fluent-bit does not support this functionality yet. Now the good news is that in our upcoming Torizon OS 7.X we plan to use at least version 3.0.7 for Fluent-bit. Though This version of the OS is unstable and still a work in progress.
Best Regards,
Jeremias
hi @jeremias.tx thanks for the help, confirmed it’s the version issue. Running v3.1.3 worked as expected:
{
"_index": "fluent-bit",
"_id": "f7zkyZAB4UsbSx_K5Z0D",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2024-07-19T07:28:59.000Z",
"log": "xxxxxxxxxxxx- - [19/Jul/2024:07:28:59 +0000] \"GET /assets/assets/images/xxxxxxxxxxxx.2a17721c4d796ebb3baad80730d6c848.png HTTP/1.1\" 200 15087 \"http://xxxxxxxxxxxx\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0\"",
"container_id": "74a5f59ec875fee42243bb3cdb595051004c82a7da007e2f46fe07d69ec2aa2d",
"container_name": "/xxxxxxxxxxxx",
"source": "stdout",
"remote": "xxxxxxxxxxxx",
"host": "-",
"user": "-",
"time": "19/Jul/2024:07:28:59 +0000",
"method": "GET",
"path": " /assets/assets/images/xxxxxxxxxxxx.2a17721c4d796ebb3baad80730d6c848.png ",
"code": "200",
"size": "15087",
"referer": "http://xxxxxxxxxxxx",
"agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0",
"device": "xxxxxxxxxxxx"
},
"fields": {
"@timestamp": [
"2024-07-19T07:28:59.000Z"
]
},
"highlight": {
"device": [
"@opensearch-dashboards-highlighted-field@dev@/opensearch-dashboards-highlighted-field@-xxxxxxxxxxxx"
]
},
"sort": [
1721374139000
]
}
7.X sounds good
Glad I was able to help clarify. Please keep an eye out for when Torizon OS 7.X is officially released as the version of Fluent-bit used here will be more suitable to your needs. That way it’ll save you a bit of customization work.
Best Regards,
Jeremias
