Rootfs not mounting after kernel patch

Hi, using imx7d 512M V1.1D : we have been using kernel 5.4.193 (from git … toradex_5.4-2.3.x-imx …) for some time and applied patches from kernel.org up to 5.4.199 then building and updating with ubiupdatevol /dev/ubi0_0 zImage. After ubiupdate ubinfo looks correct.

Problem is now that when updating with patches above 200 the following error at boot:

ubi0 error: scan_peb: bad sequence number xxxxxx in PEB 4, expected yyyyy

uboot enviroment is default from toradex easy installer
mtdparts:
4: ubi size 0x1fc00000 offset 0x00400000

We have tried different solutions from the community and general googling for hints.

(installing the zImage and dtb from easy installer produces same error)

Any suggestions ?

Best Regards - Paul

Hi @BDSKPEF ,

Can you give more details about your setup:

What patches are you trying to apply? How many? Have you tried to narrow down if one or more patches specifically are causing the issue? Also, which OS are you using?

Best regards,
Lucas Akira

Hi Lucas,
we are starting out with an older version of this from git linux-toradex.git - Linux kernel for Apalis, Colibri and Verdin modules , 5.4.193
The patches are from kernel.org like patch-5.4.193-194 … 201-202
Alle the intermidiate 194 - 201 work

And here i must apologize for a bit of confusion.
The first problem that we have with the 5.4.202 build is
UBIFS error (ubi0:3 pid 1): ubifs_recover_master_node: failed to recover master node
And then a kernel panic for the missing rootfs

The 202 patch has a small change to NAND but i cant see if that could cause the problem.
The u-boot problem shows up later, when we hoped the problem would disappear in a later patch.

Regards - Paul

-------------------------------------- image.json - when installing from easy installer ------------------------
image.json (2.6 KB)
-------------------------------------- console dump ----------------------------------------------------------------------------
ubierror.txt (4.7 KB)

I removed the part of patch-5.4.201-202 that modifies drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c
And the rootfs mounts OK.

{added: actually patch 205-206 reverts 201-202 }

I’ll get back in a new request when i have documented the u-boot problem!

For unknown reasons we now managed to patch all the way to 5.4.260

But the reason we started patching was a kernel error with surprise remove of USB
this is still the case - and a whole different story.

[ 27.815954] ci_hdrc ci_hdrc.1: USB bus 2 deregistered
[ 27.827211] ci_hdrc ci_hdrc.1: switching to host role
[ 27.846671] ci_hdrc ci_hdrc.1: EHCI Host Controller
[ 27.856129] ci_hdrc ci_hdrc.1: new USB bus registered, assigned bus number 2
[ 27.889497] ci_hdrc ci_hdrc.1: USB 2.0 started, EHCI 1.00
[ 27.896288] hub 2-0:1.0: USB hub found
[ 27.903845] hub 2-0:1.0: 1 port detected
[ 27.911409] 8<— cut here —
[ 27.914550] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
[ 27.922785] pgd = 3c5aef42
[ 27.925551] [0000000c] *pgd=00000000
[ 27.929153] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 27.934473] Modules linked in:
[ 27.937541] CPU: 0 PID: 203 Comm: kworker/0:3 Not tainted 5.4.260-42232-g49e4130e2197-dirty #9
[ 27.946171] Hardware name: Freescale i.MX7 Dual (Device Tree)
[ 27.951941] Workqueue: events_power_efficient usb_roleswitch_workqueue
[ 27.958487] PC is at usb_gadget_vbus_disconnect+0x4/0x20
[ 27.963812] LR is at usb_roleswitch_workqueue+0x108/0x168
[ 27.969221] pc : [<806766d4>] lr : [<8066ccdc>] psr: 600f0013
[ 27.975500] sp : 90c09f20 ip : 00000002 fp : 90c08000
[ 27.980735] r10: 00000000 r9 : 00000000 r8 : 00000000
[ 27.985971] r7 : 9f969500 r6 : 90844040 r5 : 9063c840 r4 : 908442e4
[ 27.992510] r3 : 00000000 r2 : 00000000 r1 : 600f0013 r0 : 90844300
[ 27.999052] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 28.006201] Control: 10c5387d Table: 90ba006a DAC: 00000051
[ 28.011960] Process kworker/0:3 (pid: 203, stack limit = 0x7eaad095)
[ 28.018326] Stack: (0x90c09f20 to 0x90c0a000)
[ 28.022699] 9f20: 908442e4 90add300 9f966000 8013dd6c 00000008 9f966018 90add300 90add314
[ 28.030898] 9f40: 9f966000 00000008 9f966018 80f03d00 9f966000 8013e014 90a5de9c 80f6812d
[ 28.039096] 9f60: 90a5de9c 90a5de80 90a400c0 90c08000 00000000 90add300 8013dfc8 9007fed0
[ 28.047295] 9f80: 90a5de9c 801433c4 00000000 90a400c0 80143278 00000000 00000000 00000000
[ 28.055493] 9fa0: 00000000 00000000 00000000 801010e8 00000000 00000000 00000000 00000000
[ 28.063691] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 28.071889] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 28.080096] [<806766d4>] (usb_gadget_vbus_disconnect) from [<8066ccdc>] (usb_roleswitch_workqueue+0x108/0x168)
[ 28.090128] [<8066ccdc>] (usb_roleswitch_workqueue) from [<8013dd6c>] (process_one_work+0x1c8/0x424)
[ 28.099287] [<8013dd6c>] (process_one_work) from [<8013e014>] (worker_thread+0x4c/0x528)
[ 28.107402] [<8013e014>] (worker_thread) from [<801433c4>] (kthread+0x14c/0x18c)
[ 28.114822] [<801433c4>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
[ 28.122060] Exception stack(0x90c09fb0 to 0x90c09ff8)
[ 28.127124] 9fa0: 00000000 00000000 00000000 00000000
[ 28.135322] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 28.143519] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 28.150150] Code: e8bd8070 e3e0005e e12fff1e e5903014 (e593300c)
[ 28.156488] —[ end trace 9215f8d9a74c8700 ]—

By hacking drivers/usb/gadget/udc/core.c - usb_gadget_vbus_disconnect() function to check the pointers the symptom is cured - the background of the problem is still open …

-Paul

Hi @BDSKPEF ,

From what I understood you were able to solve your initial rootfs problem, but now you’re having a problem related to USB. Is this correct?

I also noticed you’re using the 5.4 Downstream kernel from NXP. Is there any particular reason to use it? We have an upstream-based 5.4 we use for our iMX6 and iMX7 images (BSP 5 reference images and Torizon OS 5):

Best regards,
Lucas Akira

Hi Lucas,
the USB was our initial problem and the reason we started patching 5.4.19x

For now we “solved” the USB problem by modifying drivers/usb/gadget/udc/core.c - usb_gadget_vbus_disconnect() function – at least until there may be comes a patch.

We assumed that the downstream was the most stable, and comming from the toradex git repo that it was good for toradex modules :slight_smile: and its longterm support until dec. 2025. On top of that it had good binary compatibility with our previous platform running 4.19.xx on a cortex-A5.

Regards - Paul

Hi @BDSKPEF ,

OK, so you managed to solve the problems you were having, at least for now.

We assumed that the downstream was the most stable, and comming from the toradex git repo that it was good for toradex modules :slight_smile: and its longterm support until dec. 2025. On top of that it had good binary compatibility with our previous platform running 4.19.xx on a cortex-A5.

It’s a valid point. For our iMX6 and iMX7 SoMs we offer both the downstream and the upstream-based kernel 5.4 for our BSP 5 reference images, so we will support those until BSP 5 reaches EOL.

Let us know if you have other questions.

Best regards,
Lucas Akira