Offline Cache of Fluent-Bit Metrics

Technical Support
1

I’m trying to add offline caching of fluent-bit metrics, so that the metrics can be uploaded to the torizon.io website to be graphed and plotted even if the device is disconnected from the network for a while. However, I’m getting inconsistent results on the cached data being uploaded. The default thermal data, as well as input from a tcp plugin, are able to be updated once the unit is online, but all the other data is not updated,

For example:
1AM: Unit is disconnected from network
3AM: Unit is reconnected to network
10PM: I check the app.torizon.io website for device metrics.

I see in the device metrics that temperature data from thermal exists from 1AM to 3AM, as expected, as well as my custom database information from tcp plugin. But all the other metrics (cpu, memory, etc) are not available from 1AM to 3AM.

In the fluent-bit logs I see output such as

Mar 07 19:54:03 verdin-imx8mp-06976848 fluent-bit[59658]: [2025/03/07 19:54:03] [error] [output:http:http.0] dgw.torizon.io:443, HTTP status=400
Mar 07 19:54:03 verdin-imx8mp-06976848 fluent-bit[59658]: {"code":"invalid_entity","description":"DecodingFailure at .date: Missing required field","cause":null,"errorId":"c9af9e0b-327c-4c41-8937-d218ff463bfc"}
Mar 07 19:54:03 verdin-imx8mp-06976848 fluent-bit[59658]: [2025/03/07 19:54:03] [ warn] [engine] failed to flush chunk '39431-1741213165.485003260.flb', retry in 76 seconds: task_id=0, input=storage_backlog.5 > output=http.0 (out_id=0)

My fluent-bit.conf is:

[SERVICE]
    flush                     1
    daemon                    Off
    log_level                 info
    parsers_file              parsers.conf
    plugins_file              plugins.conf
    storage.path              /var/log/flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.max_chunks_up     128
    storage.backlog.mem_limit 5M
    scheduler.cap             300

[INPUT]
    name            cpu
    tag             cpu
    interval_sec    300
    storage.type    filesystem

[FILTER]
    Name            nest
    Match           cpu
    Operation       nest
    Wildcard        *
    Nest_under      cpu

[INPUT]
    name            mem
    tag             memory
    interval_sec    300
    storage.type    filesystem

[FILTER]
    Name            nest
    Match           memory
    Operation       nest
    Wildcard        *
    Nest_under      memory

[INPUT]
    name            thermal
    tag             temperature
    name_regex      thermal_zone0
    interval_sec    300
    storage.type    filesystem

[FILTER]
    Name            nest
    Match           temperature
    Operation       nest
    Wildcard        *
    Nest_under      temperature

[INPUT]
    Name            exec
    Tag             filesystem
    Command         /usr/bin/fluent-filesystem
    Parser          json
    interval_sec    300
    storage.type    filesystem

[FILTER]
    Name            nest
    Match           filesystem
    Operation       nest
    Wildcard        *
    Nest_under      custom

[INPUT]
    Name            tcp
    Tag             database
    Listen          127.0.0.1
    Port            5170
    Chunk_Size      32
    Buffer_Size     64
    Format          json
    storage.type    filesystem

[FILTER]
    Name            nest
    Match           database
    Operation       nest
    Wildcard        *
    Nest_under      custom

[OUTPUT]
    name                     http
    match                    *
    host                     dgw.torizon.io
    port                     443
    uri                      monitoring/fluentbit-metrics
    format                   json
    tls                      on
    tls.verify               off
    tls.ca_file              /usr/lib/sota/root.crt
    tls.key_file             /var/sota/import/pkey.pem
    tls.crt_file             /var/sota/import/client.pem
    Retry_Limit              False
    storage.total_limit_size 512M

[OUTPUT]
    name                     stdout
    match                    *

And the fluent-bit version is

torizon@verdin-imx8mp-XXXXXXXX:~$ fluent-bit --version
Fluent Bit v2.2.3
Git commit: be238e162cf97669fa3be3bfb6c7830a8dc6ce9d

I’d like some assistance on getting all the device metrics logged while the device was offline onto app.torizon.io for viewing

2

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:

Hello,

To address the issue of inconsistent metric uploads and HTTP 400 errors with Fluent Bit on your Toradex device running Torizon, please consider the following steps:

  1. Ensure Proper Data Formatting: The HTTP 400 error suggests a missing required field, likely a timestamp. Verify that all metrics include a properly formatted timestamp. Fluent Bit requires certain fields, such as ‘date’ or ‘timestamp’, to be present in the data sent to Torizon Cloud.

  2. Review Configuration: Check your Fluent Bit configuration to ensure all input plugins are set to store data on the filesystem when offline. Confirm that storage.type is set to filesystem for each input.

  3. Check Buffering and Storage Settings: Ensure the storage path (/var/log/flb-storage/) is correctly configured and has enough space to store offline data. Adjust storage.backlog.mem_limit and storage.total_limit_size settings to accommodate the data volume generated while offline.

  4. Analyze Error Logs: Review Fluent Bit logs for additional errors or warnings that might explain why certain metrics are not cached or uploaded. Compare the configuration of input plugins for CPU and memory metrics with those that are working, like thermal and TCP.

  5. Test and Validate: After making changes, test the configuration by disconnecting the device, generating data, and reconnecting to see if all metrics upload. Use the stdout output plugin to verify the data processed by Fluent Bit.

  6. Format Custom Metrics Correctly: If you have custom metrics, ensure they are formatted correctly and include all necessary fields. Use the nest filter to structure the data as required by Torizon Cloud.

By ensuring all metrics are correctly formatted and include necessary fields, and by verifying your Fluent Bit configuration, you should achieve consistent metric uploads to Torizon Cloud after offline periods.

I hope this helps resolve your issue. If you have further questions, feel free to ask.

Best regards,
Toradex AI Assistant

Please, also check if these links can help you solve your problem:

Torizon API returns 403 since today.

3

Hey @CharlieZ,

Can you post more of your fluent-bit logs?

On your error, it says " failed to flush chunk." Which from fluentbit’s Buffering and memory doc’s can be from a mem_buf_limit being reached, which may have occurred.

-Eric

4

The failed to flush is due to the HTTP 400 Bad Request. As you can see, fluent-bit attempted to flush it from the storage backlog into the HTTP output plugin.

Additionally, because I have the storage.type filesystem configured, the mem_buf_limit does not apply, as specified in the fluentbit Buffering and memory docs.

Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [engine] failed to flush chunk '39431-1741372165.327115615.flb', retry in 7 seconds: task_id=1, input=mem.1 > output=http.0 (out_id=0)
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [engine] failed to flush chunk '39431-1741372165.510611528.flb', retry in 11 seconds: task_id=3, input=exec.3 > output=http.0 (out_id=0)
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [engine] failed to flush chunk '39431-1741372165.325370183.flb', retry in 8 seconds: task_id=0, input=cpu.0 > output=http.0 (out_id=0)
Mar 07 18:29:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:26] [ warn] [engine] failed to flush chunk '39431-1741372165.328763551.flb', retry in 9 seconds: task_id=2, input=thermal.2 > output=http.0 (out_id=0)
Mar 07 18:29:33 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:33] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:33 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:33] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:33 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:33] [ warn] [engine] failed to flush chunk '39431-1741372165.327115615.flb', retry in 11 seconds: task_id=1, input=mem.1 > output=http.0 (out_id=0)
Mar 07 18:29:34 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:34] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:34 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:34] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:34 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:34] [ warn] [engine] failed to flush chunk '39431-1741372165.325370183.flb', retry in 17 seconds: task_id=0, input=cpu.0 > output=http.0 (out_id=0)
Mar 07 18:29:35 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:35] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:35 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:35] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:35 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:35] [ warn] [engine] failed to flush chunk '39431-1741372165.328763551.flb', retry in 6 seconds: task_id=2, input=thermal.2 > output=http.0 (out_id=0)
Mar 07 18:29:37 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:37] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:37 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:37] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:37 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:37] [ warn] [engine] failed to flush chunk '39431-1741372165.510611528.flb', retry in 19 seconds: task_id=3, input=exec.3 > output=http.0 (out_id=0)
Mar 07 18:29:41 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:41] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:41 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:41] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:41 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:41] [ warn] [engine] failed to flush chunk '39431-1741372165.328763551.flb', retry in 31 seconds: task_id=2, input=thermal.2 > output=http.0 (out_id=0)
Mar 07 18:29:44 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:44] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:44 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:44] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:44 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:44] [ warn] [engine] failed to flush chunk '39431-1741372165.327115615.flb', retry in 18 seconds: task_id=1, input=mem.1 > output=http.0 (out_id=0)
Mar 07 18:29:51 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:51] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:51 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:51] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:51 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:51] [ warn] [engine] failed to flush chunk '39431-1741372165.325370183.flb', retry in 25 seconds: task_id=0, input=cpu.0 > output=http.0 (out_id=0)
Mar 07 18:29:56 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:56] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:29:56 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:56] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:29:56 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:29:56] [ warn] [engine] failed to flush chunk '39431-1741372165.510611528.flb', retry in 40 seconds: task_id=3, input=exec.3 > output=http.0 (out_id=0)
Mar 07 18:30:02 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:02] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:30:02 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:02] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:30:02 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:02] [ warn] [engine] failed to flush chunk '39431-1741372165.327115615.flb', retry in 36 seconds: task_id=1, input=mem.1 > output=http.0 (out_id=0)
Mar 07 18:30:08 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:08] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:30:08 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:08] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:30:08 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:08] [ warn] [engine] failed to flush chunk '39431-1741213165.324154010.flb', retry in 1794 seconds: task_id=66, input=cpu.0 > output=http.0 (out_id=0)
Mar 07 18:30:12 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:12] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:30:12 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:12] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:30:12 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:12] [ warn] [engine] failed to flush chunk '39431-1741372165.328763551.flb', retry in 72 seconds: task_id=2, input=thermal.2 > output=http.0 (out_id=0)
Mar 07 18:30:16 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:16] [ warn] [net] getaddrinfo(host='dgw.torizon.io', err=11): Could not contact DNS servers
Mar 07 18:30:16 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:16] [error] [output:http:http.0] no upstream connections available to dgw.torizon.io:443
Mar 07 18:30:16 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:16] [ warn] [engine] failed to flush chunk '39431-1741372165.325370183.flb', retry in 33 seconds: task_id=0, input=cpu.0 > output=http.0 (out_id=0)
Mar 07 18:30:36 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:36] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:30:36 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:36] [ info] [engine] flush chunk '39431-1741372165.510611528.flb' succeeded at retry 3: task_id=3, input=exec.3 > output=http.0 (out_id=0)
Mar 07 18:30:38 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:38] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:30:38 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:38] [ info] [engine] flush chunk '39431-1741372165.327115615.flb' succeeded at retry 4: task_id=1, input=mem.1 > output=http.0 (out_id=0)
Mar 07 18:30:49 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:49] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:30:49 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:30:49] [ info] [engine] flush chunk '39431-1741372165.325370183.flb' succeeded at retry 4: task_id=0, input=cpu.0 > output=http.0 (out_id=0)
Mar 07 18:31:11 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis-database: [[1741372270.427946307, {}], {"custom"=>{"events_total"=>929929, "events_published"=>310499, "events_unpublished"=>619430, "images_total"=>162754, "images_published"=>45117, "images_unpublished"=>117637}}]
Mar 07 18:31:11 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:31:11] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:31:24 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:31:24] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:31:24 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:31:24] [ info] [engine] flush chunk '39431-1741372165.328763551.flb' succeeded at retry 4: task_id=2, input=thermal.2 > output=http.0 (out_id=0)
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741372465.324023764, {}], {"cpu"=>{"cpu_p"=>41.215833, "user_p"=>36.140000, "system_p"=>5.075833, "cpu0.p_cpu"=>23.610000, "cpu0.p_user"=>18.113333, "cpu0.p_system"=>5.496667, "cpu1.p_cpu"=>52.040000, "cpu1.p_user"=>46.673333, "cpu1.p_system"=>5.366667, "cpu2.p_cpu"=>39.173333, "cpu2.p_user"=>34.500000, "cpu2.p_system"=>4.673333, "cpu3.p_cpu"=>50.046667, "cpu3.p_user"=>45.280000, "cpu3.p_system"=>4.766667}}]
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741372465.324535748, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3941428, "Mem.free"=>73316, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741372465.324990484, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>50.000000}}]
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741372465.486153410, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4009864", "otaroot_avail"=>"10382748", "sdcard_total"=>"490901416", "sdcard_used"=>"63331364", "sdcard_avail"=>"402560184", "carrier_temp"=>44.687000, "carrier_humidity_temp"=>44.388000, "carrier_humidity"=>100}}]
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:34:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:34:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:34:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:34:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:34:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:34:58 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:34:58] [error] [output:http:http.0] dgw.torizon.io:443, HTTP status=400
Mar 07 18:34:58 verdin-imx8mp-06976848 fluent-bit[39431]: {"code":"invalid_entity","description":"DecodingFailure at .date: Missing required field","cause":null,"errorId":"eaac7c7b-850b-452d-b594-58b43182bc89"}
Mar 07 18:34:58 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:34:58] [ warn] [engine] failed to flush chunk '39431-1741213165.485003260.flb', retry in 1219 seconds: task_id=42, input=exec.3 > output=http.0 (out_id=0)
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741372765.325524419, {}], {"cpu"=>{"cpu_p"=>40.685833, "user_p"=>36.117500, "system_p"=>4.568333, "cpu0.p_cpu"=>22.980000, "cpu0.p_user"=>17.723333, "cpu0.p_system"=>5.256667, "cpu1.p_cpu"=>52.230000, "cpu1.p_user"=>48.476667, "cpu1.p_system"=>3.753333, "cpu2.p_cpu"=>43.053333, "cpu2.p_user"=>38.636667, "cpu2.p_system"=>4.416667, "cpu3.p_cpu"=>44.483333, "cpu3.p_user"=>39.633333, "cpu3.p_system"=>4.850000}}]
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741372765.327590244, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3945648, "Mem.free"=>69096, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741372765.329135581, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>50.000000}}]
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741372765.481903990, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4009864", "otaroot_avail"=>"10382748", "sdcard_total"=>"490901416", "sdcard_used"=>"63331932", "sdcard_avail"=>"402559616", "carrier_temp"=>44.875000, "carrier_humidity_temp"=>44.645000, "carrier_humidity"=>100}}]
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:39:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:39:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:39:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:39:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:39:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741373065.324255906, {}], {"cpu"=>{"cpu_p"=>40.535833, "user_p"=>35.980833, "system_p"=>4.555000, "cpu0.p_cpu"=>22.620000, "cpu0.p_user"=>17.473333, "cpu0.p_system"=>5.146667, "cpu1.p_cpu"=>46.660000, "cpu1.p_user"=>42.853333, "cpu1.p_system"=>3.806667, "cpu2.p_cpu"=>46.793333, "cpu2.p_user"=>42.183333, "cpu2.p_system"=>4.610000, "cpu3.p_cpu"=>46.063333, "cpu3.p_user"=>41.410000, "cpu3.p_system"=>4.653333}}]
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741373065.324928767, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3930516, "Mem.free"=>84228, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741373065.325411757, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>51.000000}}]
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741373065.489606846, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4009864", "otaroot_avail"=>"10382748", "sdcard_total"=>"490901416", "sdcard_used"=>"63332500", "sdcard_avail"=>"402559048", "carrier_temp"=>45.125000, "carrier_humidity_temp"=>44.903000, "carrier_humidity"=>100}}]
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:44:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:44:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:44:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:44:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:44:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741373365.324157014, {}], {"cpu"=>{"cpu_p"=>40.445000, "user_p"=>36.036667, "system_p"=>4.408333, "cpu0.p_cpu"=>22.580000, "cpu0.p_user"=>17.876667, "cpu0.p_system"=>4.703333, "cpu1.p_cpu"=>52.610000, "cpu1.p_user"=>48.703333, "cpu1.p_system"=>3.906667, "cpu2.p_cpu"=>42.766667, "cpu2.p_user"=>38.570000, "cpu2.p_system"=>4.196667, "cpu3.p_cpu"=>43.826667, "cpu3.p_user"=>39.000000, "cpu3.p_system"=>4.826667}}]
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741373365.326916587, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3941108, "Mem.free"=>73636, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741373365.332390734, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>51.000000}}]
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741373365.501810799, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4009864", "otaroot_avail"=>"10382748", "sdcard_total"=>"490901416", "sdcard_used"=>"63333032", "sdcard_avail"=>"402558516", "carrier_temp"=>45.437000, "carrier_humidity_temp"=>45.235000, "carrier_humidity"=>100}}]
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:49:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:49:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:49:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:49:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:49:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741373665.324370130, {}], {"cpu"=>{"cpu_p"=>40.508333, "user_p"=>35.914167, "system_p"=>4.594167, "cpu0.p_cpu"=>22.400000, "cpu0.p_user"=>17.566667, "cpu0.p_system"=>4.833333, "cpu1.p_cpu"=>49.073333, "cpu1.p_user"=>44.503333, "cpu1.p_system"=>4.570000, "cpu2.p_cpu"=>42.063333, "cpu2.p_user"=>37.560000, "cpu2.p_system"=>4.503333, "cpu3.p_cpu"=>48.496667, "cpu3.p_user"=>44.026667, "cpu3.p_system"=>4.470000}}]
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741373665.327891818, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3940048, "Mem.free"=>74696, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741373665.328678554, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>51.000000}}]
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741373665.499386158, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4009864", "otaroot_avail"=>"10382748", "sdcard_total"=>"490901416", "sdcard_used"=>"63333460", "sdcard_avail"=>"402558088", "carrier_temp"=>45.437000, "carrier_humidity_temp"=>45.149000, "carrier_humidity"=>100}}]
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:54:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:54:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:54:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:54:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:54:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:55:18 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:55:18] [error] [output:http:http.0] dgw.torizon.io:443, HTTP status=400
Mar 07 18:55:18 verdin-imx8mp-06976848 fluent-bit[39431]: {"code":"invalid_entity","description":"DecodingFailure at .date: Missing required field","cause":null,"errorId":"a757b042-ff1d-4f11-9add-14e961ecca55"}
Mar 07 18:55:18 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:55:18] [ warn] [engine] failed to flush chunk '39431-1741213165.485003260.flb', retry in 1456 seconds: task_id=42, input=exec.3 > output=http.0 (out_id=0)
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741373965.324754051, {}], {"cpu"=>{"cpu_p"=>40.472500, "user_p"=>36.021667, "system_p"=>4.450833, "cpu0.p_cpu"=>22.846667, "cpu0.p_user"=>18.076667, "cpu0.p_system"=>4.770000, "cpu1.p_cpu"=>51.350000, "cpu1.p_user"=>47.570000, "cpu1.p_system"=>3.780000, "cpu2.p_cpu"=>40.433333, "cpu2.p_user"=>36.166667, "cpu2.p_system"=>4.266667, "cpu3.p_cpu"=>47.263333, "cpu3.p_user"=>42.273333, "cpu3.p_system"=>4.990000}}]
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741373965.325351041, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3947760, "Mem.free"=>66984, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741373965.325830408, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>52.000000}}]
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741373965.521455690, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4015964", "otaroot_avail"=>"10376648", "sdcard_total"=>"490901416", "sdcard_used"=>"63334104", "sdcard_avail"=>"402557444", "carrier_temp"=>45.562000, "carrier_humidity_temp"=>45.310000, "carrier_humidity"=>100}}]
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:59:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:59:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:59:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 18:59:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 18:59:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:00:03 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:00:03] [error] [output:http:http.0] dgw.torizon.io:443, HTTP status=400
Mar 07 19:00:03 verdin-imx8mp-06976848 fluent-bit[39431]: {"code":"invalid_entity","description":"DecodingFailure at .date: Missing required field","cause":null,"errorId":"47b6f07b-7cb7-40ce-a84c-757f6e880c79"}
Mar 07 19:00:03 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:00:03] [ warn] [engine] failed to flush chunk '39431-1741213165.324154010.flb', retry in 956 seconds: task_id=66, input=cpu.0 > output=http.0 (out_id=0)
Mar 07 19:01:10 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis-database: [[1741374070.061027104, {}], {"custom"=>{"events_total"=>933252, "events_published"=>310499, "events_unpublished"=>622753, "images_total"=>163318, "images_published"=>45117, "images_unpublished"=>118201}}]
Mar 07 19:01:10 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:01:10] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741374265.325490291, {}], {"cpu"=>{"cpu_p"=>40.501667, "user_p"=>35.749167, "system_p"=>4.752500, "cpu0.p_cpu"=>22.676667, "cpu0.p_user"=>17.873333, "cpu0.p_system"=>4.803333, "cpu1.p_cpu"=>47.196667, "cpu1.p_user"=>42.483333, "cpu1.p_system"=>4.713333, "cpu2.p_cpu"=>44.490000, "cpu2.p_user"=>39.756667, "cpu2.p_system"=>4.733333, "cpu3.p_cpu"=>47.640000, "cpu3.p_user"=>42.880000, "cpu3.p_system"=>4.760000}}]
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741374265.326295027, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3944932, "Mem.free"=>69812, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741374265.326814643, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>52.000000}}]
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741374265.501994335, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4016480", "otaroot_avail"=>"10376132", "sdcard_total"=>"490901416", "sdcard_used"=>"63334772", "sdcard_avail"=>"402556776", "carrier_temp"=>45.812000, "carrier_humidity_temp"=>45.535000, "carrier_humidity"=>100}}]
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:04:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:04:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:04:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:04:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:04:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741374565.324051013, {}], {"cpu"=>{"cpu_p"=>40.217500, "user_p"=>35.834167, "system_p"=>4.383333, "cpu0.p_cpu"=>22.860000, "cpu0.p_user"=>17.906667, "cpu0.p_system"=>4.953333, "cpu1.p_cpu"=>50.753333, "cpu1.p_user"=>47.063333, "cpu1.p_system"=>3.690000, "cpu2.p_cpu"=>44.263333, "cpu2.p_user"=>39.836667, "cpu2.p_system"=>4.426667, "cpu3.p_cpu"=>42.990000, "cpu3.p_user"=>38.530000, "cpu3.p_system"=>4.460000}}]
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741374565.324534252, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3941128, "Mem.free"=>73616, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741374565.324938242, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>51.000000}}]
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741374565.482205207, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4009864", "otaroot_avail"=>"10382748", "sdcard_total"=>"490901416", "sdcard_used"=>"63335416", "sdcard_avail"=>"402556132", "carrier_temp"=>45.750000, "carrier_humidity_temp"=>45.493000, "carrier_humidity"=>100}}]
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:09:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:09:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:09:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:09:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:09:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] cpu: [[1741374865.325810626, {}], {"cpu"=>{"cpu_p"=>40.771667, "user_p"=>36.260833, "system_p"=>4.510833, "cpu0.p_cpu"=>22.783333, "cpu0.p_user"=>17.860000, "cpu0.p_system"=>4.923333, "cpu1.p_cpu"=>47.966667, "cpu1.p_user"=>43.596667, "cpu1.p_system"=>4.370000, "cpu2.p_cpu"=>43.593333, "cpu2.p_user"=>38.480000, "cpu2.p_system"=>5.113333, "cpu3.p_cpu"=>48.746667, "cpu3.p_user"=>45.106667, "cpu3.p_system"=>3.640000}}]
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] memory: [[1741374865.328410072, {}], {"memory"=>{"Mem.total"=>4014744, "Mem.used"=>3943960, "Mem.free"=>70784, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}}]
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] temperature: [[1741374865.329912541, {}], {"temperature"=>{"name"=>"thermal_zone0", "type"=>"cpu-thermal", "temp"=>51.000000}}]
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [0] adis: [[1741374865.482570874, {}], {"custom"=>{"otaroot_total"=>"15185800", "otaroot_used"=>"4009864", "otaroot_avail"=>"10382748", "sdcard_total"=>"490901416", "sdcard_used"=>"63336028", "sdcard_avail"=>"402555520", "carrier_temp"=>45.625000, "carrier_humidity_temp"=>45.321000, "carrier_humidity"=>100}}]
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:14:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:14:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:14:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:14:26 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:14:26] [ info] [output:http:http.0] dgw.torizon.io:443, HTTP status=204
Mar 07 19:16:00 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:16:00] [error] [output:http:http.0] dgw.torizon.io:443, HTTP status=400
Mar 07 19:16:00 verdin-imx8mp-06976848 fluent-bit[39431]: {"code":"invalid_entity","description":"DecodingFailure at .date: Missing required field","cause":null,"errorId":"cc8c0b0e-3fd6-45f5-a499-2fc2b500be9a"}
Mar 07 19:16:00 verdin-imx8mp-06976848 fluent-bit[39431]: [2025/03/07 19:16:00] [ warn] [engine] failed to flush chunk '39431-1741213165.324154010.flb', retry in 1492 seconds: task_id=66, input=cpu.0 > output=http.0 (out_id=0)

Initially, the chunks fail to flush due to being disconnected from the internet. This is expected, later, once reconnected to the internet, chunks start being able to be sent, such as [engine] flush chunk '39431-1741372165.325370183.flb' succeeded at retry 4: task_id=0, input=cpu.0 > output=http.0 (out_id=0), however, some chunks fail to send due to 400 Bad Request, such as [ warn] [engine] failed to flush chunk '39431-1741213165.324154010.flb', retry in 1492 seconds: task_id=66, input=cpu.0 > output=http.0 (out_id=0). Additionally, even the supposedly sent data is not visible on the torizon website. You can see in this image there is data for the database information, and for the SOM temperature, but, Mem, etc, don’t have data for the March 6 to March 7 time period. There are also issues with the March 8 to March 10 time period, but I don’t have logs for it right now. I will get those logs tomorrow.

image
image2071×1039 168 KB

5

Hey CharlieZ,

I grabbed your config file and set this up on my board with a few alterations: I moved the storage director to /home/torizon/fluent-bit-storage, got rid of the fluent-filesystem input, set log level to debug, and set the reporting interval to 30 on all inputs.

Then I disconnected the network for 30 minutes while watching fluent-bit logs and finally reconnected.

UNHELPFULLY, it worked as expected for me - I was able to receive the data points from the period where the device was offline. For whatever reason they do seem to confuse the charts in our app-v1 for a bit until the new data-points arrive.

Anyways, I know this isn’t helpful so I will keep digging, I just wanted to let you know I am at least looking into this.

I am very suspicious of this error response: "code":"invalid_entity","description":"DecodingFailure at .date: Missing required field" as it implies that some of your buffered (saved to disk) data-points are not being saved with a date, or fluent-bit isn’t sending that field in the request…

You also mentioned that you don’t see any of the metrics from the offline period, so maybe something else is messed up. I will try leaving my device offline longer.

Regards,
Ben

6

Ok, i was able to reproduce this after leaving it for a few hours. I dont have any errors in my fluent-bit logs, but the only metric that shows data for the hours it was off is memory…

It starts to feel like an actual bug on our side, and there is a possible issue with rate-limiting since fluent-bit just slams our server all at once with the buffered data.

more time needed to figure this one out.

7

Thank you for the work investigating this issue. I will get back with any additional information I get. I will try testing on a different device as well.