Hi, I switched to BSP 5.7.2 (mainline kernel) and now I am wondering, why only one CPU is running. In previous setup (BSP 5.6.0 - not mainline kernel) two CPUs were used.
Used Setup
Colibri iMX7D 1 GB V1.1A
Kernel 5.4.193-5.7.2-devel+git.f5d73fd6e9f8
Based on Toradex BSP Layers and Reference Images for Yocto Project 5.7.2
This is, what I get with following command:
~$ cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 48.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5
Hardware : Freescale i.MX7 Dual (Device Tree)
Revision : 0000
Serial : 06805514
And this is what dmesg prints out:
~$ dmesg | grep CPU
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
[ 0.003252] CPU: Testing write buffer coherency: ok
[ 0.008055] smp: Bringing up secondary CPUs ...
[ 0.008743] smp: Brought up 1 node, 1 CPU
[ 0.008764] CPU: All CPU(s) started in SVC mode.
[ 1.723740] CPUidle arm: CPU 0 failed to init idle CPU ops
[ 2.051202] imx_thermal tempmon: Extended Commercial CPU temperature grade - max:105C critical:105C passive:95C
When comparing with previous BSP (BSP 5.6.0) following gets printed:
~# dmesg | grep CPU
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
[ 0.003868] CPU: Testing write buffer coherency: ok
[ 0.009074] smp: Bringing up secondary CPUs ...
[ 0.010125] smp: Brought up 1 node, 2 CPUs
[ 0.010170] CPU: All CPU(s) started in HYP mode.
[ 0.010184] CPU: Virtualization extensions available.
[ 2.589169] imx_thermal tempmon: Extended Commercial CPU temperature grade - max:105C critical:105C passive:95C
Can anyone help me setting up my configuration so that both CPUs are activated? Thanks.
I tested it with the original (unchanged) Reference Image from Toradex and the result is as expected: Two CPUs are running. So there must be something maybe within the bootloader? When I compare the dmesg outputs (BSP 5.6.0 vs 5.7.2) I see following difference:
smp: Brought up 1 node, 1 CPU
[ 0.008764] CPU: All CPU(s) started in SVC mode.
smp: Brought up 1 node, 2 CPUs
[ 0.010170] CPU: All CPU(s) started in HYP mode.
I do not know what the difference between “SVC” and “HYP” is but maybe this is the reason for it?
We were using a customized BSP Version 5.6.0 → both CPUs were active (good).
Then we switched to a customized BSP version 5.7.2 → only one CPU is active (not good).
What customizations are you doing on the image?
Since it works with our unmodified reference image, it looks like the root of the cause could be somewhere in your customization.
I agree that it has something to do with our customization.
We added some layers, deactivate not needed packages etc. Nothing special (at least I do not see what could have the effect to deactivate one CPU).
But I think I found a solution. In this thread it is mentioned, to set the U-Boot variable bootm_boot_mode=nonsec. So I tried this and now there are 2 CPUs again.
But one thing still makes me unsure. Is it safe to set bootm_boot_mode=nonsec (it was set to bootm_boot_mode=sec before)? This was not necessary in our previous (5.6.0) customization. Any idea?
Thanks for providing the links. I will read through the links and get back to you.
But I would feel much better if I could switch back to SVC mode (bootm_boot_mode=sec as it was before). Unfortunately I don’t know in which config file I should adjust something to accomplish this (have SVC mode with both active CPUs).
Any help would be appreciated.
Yes, I could share this, but I’m not sure which of those changes matter.
Essentially it’s the addition of a new custom meta layer, changes to the device tree have been made, and changes to the bootloader and kernel config. However, no changes were made to the other meta-xxx layers (meta-freescale, meta-openembedded, meta-toradex-nxp etc.).
I can’t tell which of these changes matters either.
Maybe you can tell me which areas are important?
Okay. I’ll read through the provided links and give you feedback. I’ll also check the differences between the kernel configs. Maybe I’ll find something promising.
I have looked again at the difference of the bootloader between the unchanged BSP 5.7.2 and our BSP. Here I noticed that in the unmodified BSP the variable bootm_boot_mode is also set to ‘nonsec’. This explains why both CPUs are active in the unmodified BSP!
Question: Do you have a reason why in the original BSP this variable is set to ‘nonsec’ instead of ‘sec’?
In this thread it is mentioned that the mainline kernel boots in the non-secure world. It also mentions that if a mainline kernel is used, then a mainline U-Boot should also be used with PSCI support enabled for the mx7. Can you tell me how to enable this? How can a mainline U-Boot be used? Then I could set the variable back to ‘bootm_boot_mode=sec’ and both CPU would run probably…
Here goes quite a lengthy answer written with @rafael.tx’s aid. Hopefully, it helps to understand this boot mode stuff.
About bootm_boot_mode on upstream and downstream kernel
Historically, the downstream kernel boots with bootm_boot_mode=sec (and controls itself the 2nd core), and the upstream kernel boots with bootm_boot_mode=nonsec (and uses PSCI to control the 2nd core).
Unfortunately, I do not have information on why it is like this. One could try to dig into the mailing lists and find out why Upstream chose to do like this (getting this kind of information from NXP will probably be way harder).
One possible explanation could be that having the kernel booting with the processor in a less privileged mode is usually considered a good approach.
About this:
Could you please specify if you are using upstream/downstream on both?
Differences between upstream and downstream kernel for Colibri iMX7
When using upstream, bootm_boot_mode is used as nonsec. Having boot mode as sec doesn’t mean secure boot. It only means that the kernel is booting in a higher privileged state.
From what I could see, that’s actually true! I didn’t know about that
I tried to boot using sec (default) and nonsec (setenv bootm_boot_mode nonsec in U-Boot) on Reference Minimal 6.2.0 Upstream and it loaded both CPUs on COlibri iMX7 1GB (eMMC) V1.1A:
root@colibri-imx7-emmc-06674596:~# cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 48.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5
processor : 1
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 48.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5
Hardware : Freescale i.MX7 Dual (Device Tree)
Revision : 0000
Serial : 06674596
Say Y here to boot in secure mode by default even if non-secure mode
is supported. This option is useful to boot kernels which do not
suppport booting in non-secure mode. Only set this if you need it.
This can be overridden at run-time by setting the bootm_boot_mode env.
variable to “sec” or “nonsec”.
So I would like to ask: What is your use case and why would you need the sec?
I think I misunderstood the meaning of the sec vs nonsec. I agree that it would be better to boot the kernel in less privilege mode. So for now, I will set this variable to nonsec. By the way, will you also switch back to nonsec for newer BSP ( >6.2.0)?
Again, many thanks for your time and clarification on this topic.
Sorry for the late response. I talked to some of my colleagues here.
So the situation is as follows:
The downstream version used a proprietary way of booting the cores of the SoC. There sec was used. Now starting from newer versions Toradex is only building upstream. There the cores are booted through the standard PSCI.
The commit is still there in kirkstone it’s just not used anymore.
