Stdout, fluentd and logging

I’m trying to do something real simple, but nothing is working.

  1. Trying to see a container log. I have a simple program the just sends a string to stdout. From what I read Docker uses stdout and stderr to send to the logs. I’ll run the program then jump over to a terminal and use docker logs -f . Nothing shows up.

    while (1)
    {
    fprintf( stdout, “Hello all\r\n”);
    fflush(stdout);
    sleep(1);
    }

I did something simular in torizon os 5 and I recall seeing on the log anything I used printf on. is Torison OS 6 different? How can I write to the log from c code?

  1. The next step was to use “Enabling Container Monitoring” from Device Monitoring in Torizon OS | Toradex Developer Center however, the "Docker documentation: in step 2 for compose files is missing when you click on the link.

The whole idea is to do what " Configuring Fluent Bit for Container Monitoring" wants me to do and test it.

First, thing to tackle is the logs not working. Any ideas?

FYI - I’m using VS code with the extension. And I do see my printf in the “Debug Console”

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:


To address your issue with container logging in Torizon OS, let’s break it down into two parts:

1. Viewing Container Logs

Your C code snippet should indeed send output to stdout, which Docker captures and makes available via docker logs. Here are a few steps to troubleshoot why docker logs -f isn’t showing your output:

  1. Ensure the Container is Running: Make sure your container is actually running. You can check this with docker ps.

  2. Check the Container ID: Ensure you are using the correct container ID or name with the docker logs command.

  3. Verify the Logging Driver: By default, Docker uses the json-file logging driver, which should capture stdout and stderr. You can verify this by running:

    docker inspect --format='{{.HostConfig.LogConfig.Type}}' <container_id>
    

    It should return json-file.

  4. Run a Simple Test: Try running a simple container to see if logging works:

    docker run --rm -it busybox sh -c 'while true; do echo "Hello all"; sleep 1; done'
    

    Then, in another terminal, run:

    docker logs -f <container_id>
    

    This should show the “Hello all” messages.

2. Enabling Container Monitoring with Fluent Bit

To enable container monitoring with Fluent Bit in Torizon OS, follow these steps:

  1. Enable Fluent Bit: Ensure Fluent Bit is enabled on your Torizon OS device. This is typically done by provisioning the device to Torizon Cloud, which creates the /etc/fluent-bit/enabled file.

  2. Configure Docker to Use Fluentd Logging Driver:

    • Edit your Docker Compose file to use the Fluentd logging driver. Here is an example:
      version: '3.8'
      services:
        my_service:
          image: my_image
          logging:
            driver: fluentd
            options:
              fluentd-address: localhost:24224
              tag: docker.my_service
      
  3. Configure Fluent Bit:

    • Ensure Fluent Bit is configured to accept logs from Docker. Add the following to your /etc/fluent-bit/fluent-bit.conf:
      [INPUT]
          Name              forward
          Listen            0.0.0.0
          Port              24224
      
      [OUTPUT]
          Name              stdout
          Match             *
      
  4. Restart Fluent Bit:

    systemctl restart fluent-bit
    
  5. Run Your Container:

    • Start your container with the updated Docker Compose file.
  6. Check Fluent Bit Output:

    • Open a terminal and run:
      journalctl -u fluent-bit -f
      
    • This should show the logs being forwarded from Docker to Fluent Bit.

Missing Documentation Link

Regarding the missing link in the “Enabling Container Monitoring” section, it seems there might be an issue with the documentation. You can refer to the official Docker documentation for configuring logging drivers in Docker Compose files: Docker Logging Drivers.

By following these steps, you should be able to see your container logs and configure Fluent Bit for container monitoring in Torizon OS. If you continue to face issues, please provide more details, and we can further troubleshoot.

Please, also check if these links can help you solve your problem:

Device Monitoring in Torizon OS | Toradex Developer Center.

FYI - from AI
I did Run a Simple Test that the AI suggested. That worked.

Writing to stdout from my c program, then looking at the docker logs -f does not work.

Greetings @jeffbelz,

If I run a simple container like so on the device:

docker run --rm -it torizon/debian:3-bookworm

Then in that container I install vim-tiny build-essential. I create a source file with the following contents:

#include <stdio.h>
#include <unistd.h>

int main() {
    while (1)
    {
        fprintf(stdout, "Hello all\r\n");
        fflush(stdout);
        sleep(1);
    }
}

If I compile this and execute it while monitoring the docker logs in a different terminal I can see the output:

torizon@verdin-imx8mp-06849059:~$ docker logs -f 7
Hello all
Hello all
Hello all
Hello all
Hello all

So as shown, it does work.

Now you mentioned you were using the VSCode extension. So I created a fresh project and used the same code as above and started the debug process. Here as you said docker logs is now empty and the output is seen in the debug window on VSCode. I believe this is because when running in debug mode, the stdout gets redirected to the debug console for debugging purposes. Therefore the container logs won’t see it.

If you instead run your container application in release mode, then the output is sent to stdout and seen in the container logs as expected.

Best Regards,
Jeremias

Yeah, that did it. Anyway to tell it not to redirect. I want to debug, but have that stdout on the terminal, not on the debug console.

I tried some of the option in Configure launch.json for C/C++ debugging in Visual Studio Code. but no luck

Yeah, that did it. Anyway to tell it not to redirect. I want to debug, but have that stdout on the terminal, not on the debug console.

After briefly discussing with our IDE extension team, the short answer is that there doesn’t seem to be a way to do this. At least not without also breaking the debug in the process.

When the extension runs in debug mode, it runs first an sshd to connect the remote debugger. The data, including the application output, is under this sshd connection and will be seen only in the output of the VSCode debugger or, gdb in this case.

Theoretically if you were to break this redirection of the application data to gdb. Then, I imagine gdb wouldn’t be able to debug the application properly, which defeats the purpose of the debug mode.

I believe your goal is to parse the output from your application and feed it to fluent-bit correct?

You’ll probably need to do this in two-stages then:

  • The first stage would be to debug your application as normal, until the output is what you want it to be.
  • Then the second stage is to just run your application normally without the debugger, and start developing the data flow from your application to fluent-bit.

Best Regards,
Jeremias

Just checking-in, did you have any further questions on this topic?

Best Regards,
Jeremias

No we are all ser

Glad we were able to help out.