Random number generator hangs for a long time

Technical Support
, ,
1

Hi,

I did a yocto build (morty) (core-image-minimal) using meta-toradex-nxp layers. The kernel used is:

Linux apalis-imx6 4.1.44-2.7.4+gb1555bf

The only difference from the toradex defconfig is:

+ CONFIG_POWER_RESET
+ CONFIG_POWER_RESET_GPIO
+ CONFIG_LOCALVERSION
- CONFIG_MXC_GPU_VIV

All layers comes from the morty branches.

My problem is that when I boot, the boot time is about:

Startup finished in 3.057s (kernel) + 1min 9.620s (userspace) = 1min 12.677s

which is way too long. I’m using connman for the network and rauc. And both systemd services are blocked for about 1min. After the 1min has passed, I can see in the console:

[   72.579956] random: nonblocking pool is initialized
[   72.773527] fec 2188000.ethernet eth0: Freescale FEC PHY driver [Micrel KSZ9031 Gigabit PHY] (mii_bus:phy_addr=2188000.ethernet:07, irq=70)
[   72.786366] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   75.837360] fec 2188000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   75.845422] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

The random pool is finaly initialized, and thus connman and rauc are unblocked and the ethernet link is up.

I was not sure at first if the problem is related to the random pool. But after some more digging I can now say that the problem is indeed caused by the random pool that is being init after a very long time. I did some debugging and found out that the connmand (and rauc too) process is linked to libgnutls, which upon initialization call the kernel random() syscall, and because it’s not initialized yet, just blocks.
Gdb trace:

(gdb) bt
#0  0x76a93dbc in syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
#1  0x768be378 in _rnd_get_system_entropy_getrandom (flags=0, buflen=<optimized out>, buf=<optimized out>)
    at /usr/src/debug/gnutls/3.5.3-r0/gnutls-3.5.3/lib/nettle/rnd-linux.c:80
#2  0x768be378 in _rnd_get_system_entropy_getrandom (_rnd=<optimized out>, size=<optimized out>)
    at /usr/src/debug/gnutls/3.5.3-r0/gnutls-3.5.3/lib/nettle/rnd-linux.c:98
#3  0x768bb3dc in do_device_source (init=init@entry=1, event=event@entry=0x7efffccc, ctx=0x768f5be0 <rnd_ctx>)
    at /usr/src/debug/gnutls/3.5.3-r0/gnutls-3.5.3/lib/nettle/rnd.c:132
#4  0x768bb544 in wrap_nettle_rnd_init (ctx=<optimized out>) at /usr/src/debug/gnutls/3.5.3-r0/gnutls-3.5.3/lib/nettle/rnd.c:234
#5  0x76809964 in _gnutls_rnd_init () at /usr/src/debug/gnutls/3.5.3-r0/gnutls-3.5.3/lib/random.c:49
#6  0x767fbd60 in _gnutls_global_init (constructor=constructor@entry=1) at /usr/src/debug/gnutls/3.5.3-r0/gnutls-3.5.3/lib/global.c:307
#7  0x767d48bc in lib_init () at /usr/src/debug/gnutls/3.5.3-r0/gnutls-3.5.3/lib/global.c:504
#8  0x76fdf34c in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7efffdd4, env=env@entry=0x7efffddc)
    at /usr/src/debug/glibc/2.24-r0/git/elf/dl-init.c:72
#9  0x76fdf4a8 in _dl_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>)
    at /usr/src/debug/glibc/2.24-r0/git/elf/dl-init.c:30
#10 0x76fdf4a8 in _dl_init (main_map=0x76fff958, argc=1, argv=0x7efffdd4, env=0x7efffddc)
    at /usr/src/debug/glibc/2.24-r0/git/elf/dl-init.c:120
#11 0x76fcfac4 in _dl_start_user () at /lib/ld-linux-armhf.so.3
(gdb)

The question now is why is the random pool taking so long to fill? If I try the image from toradex (lxde image) with exact same kernel I don’t have this problem. I also checked and the defconfig contains all the CAAM options to enable the Random-number generator from this hw module.

Has anyone an idea about this problem?

Thanks,

2

Hi

Which Hw are you exactly using? Apalis iMX6Q or i.MX6D? Which lxde image exactly did you try? Can you send me a log of bootup sequence with your own yocto build?

Thanks and best regards, Jaski

3

Hi,
I’m using an apalis-imx6 on an ixora dev board. The Soc is an iMX6Q.

Attached is the log of my own build. As you can see, the random pool init is done at timestamp 65s after the getty is started, which releases connman and the ethernet stack.

Log files

For the lxde image I used this one, flashed using easy installer:

  "name": "Toradex Embedded Linux Demo with LXDE",
  "description": "Toradex Embedded Linux Demo with LXDE.",
  "version": "2.7b4",
  "release_date": "20171004",

Thanks

4

Hello, I have the same problem with random. I’ve installed rng-tools and it has helped, now it starts fast but uses a lot of processor time afterwards.

5

Intersting! I will try to use rng-tools and see.
Thanks

6

What is strange is that the toradex lxde image is not using rng-tools, and still have a fast rng initialization!

7

What is your source of entropy for running rngd?

8

It takes it from urandom

9

Hum… that’s pretty bad. It basically feeds the kernel entropy with itself…

10

Finally I found a working solution.

First CAAM needs to be enabled in the kernel and also CRYPTO_DEV_FSL_CAAM_RNG_API.
It adds /dev/hwrng as a source of entropy.

Next I installed rng-tools and started it at boot using this systemd service file:

[Unit]
Description=Hardware RNG Entropy Gatherer Daemon

[Service]
ExecStart=/usr/sbin/rngd -f --no-tpm=1

[Install]
WantedBy=multi-user.target

Now with that I have a good entropy source coming from the hardware and rng-tools fills the kernel entropy for /dev/random from /dev/hwrng. My system does not block anymore at boot and when I check the entropy available after bootup I get:

root@apalis-imx6:~# cat /proc/sys/kernel/random/entropy_avail
3085

which is quite good.

11

Hi,

good that you have found a working solution. Is the CPU usage with your solution higher than the toradex lxde image?

12

No, it does not seem to use so much CPU. I think the cpu usage problem for @Polarisru is because it feeds the entropy pool from /dev/urandom (the kernel itself).

root@apalis-imx6:~# ps -p $(pidof rngd) -o %cpu,%mem,cmd
%CPU %MEM CMD
 0.0  0.0 /usr/sbin/rngd -f --no-tpm=1

Even when stressing a bit and reading from /dev/random, the cpu usage from rngd is low.

root@apalis-imx6:~# dd if=/dev/random of=/dev/null bs=1024 count=1 iflag=fullblock
1+0 records in
1+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00583959 s, 175 kB/s

Maybe it could be good to consider including rng-tools into toradex images…

13

Hi emcp

Thanks for the information!
We will check if we can include rng-tools into toradex image.

14

Hi,
Can we cross compile the rng-tools for IMX8QXP board.
If so, I am not getting how to do that,can u please guide me?

15

hi @raja: Welcome to the Toradex Community!

Could you ask a new question with all information about your issue, please?

Thanks.

16

rngd (for i.MX) and haved (for Tegra) daemons are included into all Toradex BSP images 3.0b3 since 2019-10-17.