We’ve been using Yocto but are desiring to switch to EasyInstaller process with bundled docker containers. If we isolate changes to a directory using --changes-directory and then specify that customization filesystem in the tcbuild.yaml, everything builds fine but the flashing process does not end cleanly (Torizon logo spins for hours) and after power cycle, the console shows the following with lots of failures. The changes isolated can be just a couple new files we added to /etc, it doesn’t seem to matter what we add. If the customization/filesystem/changes directory is removed and the build runs, the problem resolves, but we can’t customize. Any suggestions on what can be causing this problem? We are using TorizonCore 6.6.1+build.14 base image and the very latest tcb-env.sh.
U-Boot SPL 2022.04-6.6.1+git.d262075124dc (Jan 01 1970 - 00:00:00 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 3000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
WDT: Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from MMC1
NOTICE: BL31: v2.6(release):lf_v2.6-g3c1583ba0a
NOTICE: BL31: Built : 00:00:00, Jan 1 1970
CPU: i.MX8MMDL rev1.0 1600 MHz (running at 1200 MHz)
CPU: Industrial temperature grade (-40C to 105C) at 38C
Reset cause: POR
DRAM: 1 GiB
Core: 114 devices, 21 uclasses, devicetree: separate
WDT: Started watchdog@30280000 with servicing (60s timeout)
MMC: FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC… OK
In: serial
Out: serial
Err: serial
Model: Toradex 0060 Verdin iMX8M Mini DualLite 1GB WB IT V1.1B
Serial#: 06944186
get_tdx_eeprom: cannot find EEPROM by node
MISSING TORADEX CARRIER CONFIG BLOCKS
get_tdx_eeprom: cannot find EEPROM by node
flash target is MMC:0
Net: eth0: ethernet@30be0000
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1…
Found U-Boot script /boot.scr
973 bytes read in 1 ms (950.2 KiB/s)
Executing script at 50280000
6659 bytes read in 3 ms (2.1 MiB/s)
66160 bytes read in 3 ms (21 MiB/s)
86 bytes read in 3 ms (27.3 KiB/s)
Applying Overlay: verdin-imx8mm_dsi-to-hdmi_overlay.dtbo
2317 bytes read in 4 ms (565.4 KiB/s)
Applying Overlay: verdin-imx8mm_spidev_overlay.dtbo
561 bytes read in 4 ms (136.7 KiB/s)
13437662 bytes read in 82 ms (156.3 MiB/s)
6120280 bytes read in 40 ms (145.9 MiB/s)
Uncompressing Kernel Image
Flattened Device Tree blob at 50200000
Booting using the fdt blob at 0x50200000
Loading Device Tree to 000000007ded9000, end 000000007df0cfff … OK
Modify /vpu_g1@38300000:status disabled
Modify /vpu_g2@38310000:status disabled
Modify /vpu_h1@38320000:status disabled
Delete node /cpus/cpu@2
Delete node /cpus/cpu@3
Update node /thermal-zones/cpu-thermal/cooling-maps/map0, cooling-device prop
Starting kernel …
[ 1.263968] rtc-ds1307 0-0032: hctosys: unable to read the hardware clock
[ 1.275792] pca953x 3-0021: failed writing register
[ 2.580341] regulator-dummy: Underflow of regulator enable count
[ 2.808864] [drm:drm_bridge_attach] ERROR failed to attach bridge /soc@0/bu s@32c00000/mipi_dsi@32e10000 to encoder DSI-34: -517
[ 2.820700] imx_sec_dsim_drv 32e10000.mipi_dsi: Failed to attach bridge: 32e1 0000.mipi_dsi
[ 2.828982] imx_sec_dsim_drv 32e10000.mipi_dsi: failed to bind sec dsim bridg e: -517
Starting version 250.5+
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[ 7.555125] ina2xx 3-0040: error configuring the device: -6
[ 7.625877] nau8822 3-001a: Failed to issue reset: -6
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
[FAILED] Failed to start set-hostname.service.
[FAILED] Failed to start Network Configuration.
[FAILED] Failed to start set-hostname.service.
[FAILED] Failed to start Network Configuration.
[FAILED] Failed to start set-hostname.service.
[FAILED] Failed to start Network Configuration.
[FAILED] Failed to start set-hostname.service.
[FAILED] Failed to start Network Configuration.
[FAILED] Failed to start set-hostname.service.
[FAILED] Failed to start Network Configuration.
[FAILED] Failed to start set-hostname.service.
[FAILED] Failed to start Network Configura[ 10.011653] mcp251xfd spi2.0 (unnamed net_device) (uninitialized): Failed to detect MCP251xFD (osc=0x00000000).
tion.
[FAILED] Failed to start Network Name Resolution.
[FAILED] Failed to start Network Name Resolution.
[FAILED] Failed to start Network Name Resolution.
[FAILED] Failed to start Network Name Resolution.
[FAILED] Failed to start Network Name Resolution.
[FAILED] Failed to start Network Name Resolution.
[FAILED] Failed to start Avahi mDNS/DNS-SD Stack.
[FAILED] Failed to start RPC Bind.
[FAILED] Failed to start Modem Manager.
[FAILED] Failed to start D-Bus System Message Bus.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start Disk Manager.
[DEPEND] Dependency failed for Disk…vice monitor based on udisks2.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start D-Bus System Message Bus.
[FAILED] Failed to start Network Manager.
[DEPEND] Dependency failed for Network Manager Wait Online.
[FAILED] Failed to start set-hostname.service.
Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!
Did you like my answer?
Hand-shake,
The issue you’re encountering, where the TorizonCore boot process does not complete successfully after applying customizations via the --changes-directory option, could be related to several factors. Here are some steps and considerations to help you troubleshoot and potentially resolve the issue:
Verify Customization Content: Ensure that the changes you’re applying, even if they seem minor (like adding a couple of new files to /etc), do not inadvertently alter system configurations or file permissions in a way that could affect the boot process. It’s crucial to review the content of the changes to ensure they are compatible with the TorizonCore version you’re using.
Check for Overlays and Device Tree Customizations: If you’re applying device tree overlays or making changes to the device tree, ensure these are correctly configured and compatible with your hardware. Incorrect device tree configurations can lead to boot failures or hardware not being correctly initialized.
Review Kernel Modules: If your customizations include external kernel modules, ensure they are built against the exact kernel version and configuration of the TorizonCore image you’re using. Incompatible kernel modules can cause boot issues.
Validate the Build Process: Double-check the steps you’re following to apply the customizations and build the custom TorizonCore image. Ensure that all commands are executed correctly and that there are no errors during the build process.
Debugging Output: Utilize the serial console output to gather more detailed information about where the boot process is failing. The messages you provided indicate issues with hardware initialization, which could be related to device tree or driver issues. Pay close attention to the first error messages that appear during boot, as they can often provide clues about the root cause.
Base Image Verification: Since you mentioned using TorizonCore 6.6.1+build.14, ensure that the base image itself (without any customizations) boots correctly on your hardware. This can help isolate whether the issue is with the base image or the applied customizations.
Incremental Customization Application: If possible, apply your customizations incrementally. Start with a minimal set of changes and gradually add more, testing the boot process at each step. This can help identify the specific change that introduces the boot issue.
TorizonCore Builder Version: Ensure you’re using the latest version of the TorizonCore Builder tool, as updates often include fixes and improvements that could be relevant to your issue.
If after these steps the
Please, also check if these links can help you solve your problem:
It’s not exactly clear to me what happened here. So you took our default Torizon OS 6.6.1 image made some customization with TorizonCore Builder and got a non-functioning image as a result?
Just to be sure, when you create this custom image with TorizonCore Builder, you use Easy Installer to flash the image to the device correct?
Could you please share what customization you are doing exactly. Also please share your entire tcbuild.yml file. I’ll need the full picture on what exactly you’re doing here. A few simple customization shouldn’t result in a broken image like this.
Hi Jeremias, You have it right. I’ve attached the tcbuild.yaml along with the two commands, it isn’t a very complicated picture yet the image is indeed badly broken, not just for me, but another developer on the team using the same process. The only way to fix it is to comment out the customization/filesystem/changes entry in the yaml, then it is fine. No errors during build, successful.
>> NOTES:
>> Lines containing “>>” are simply comments explaining the properties that
Here is the .tcattr file in the changes/usr/etc directory that we have simplified down to. At this point, the trigger for it seems to be any isolated changes dir in the tcbuild.yaml. Easyinstaller never completes cleanly in that case and then you get those missing blocks. We undo the reference to changes in tcbuild.yaml, rebuild, and voila! The config blocks are fine! I will take a little more time if you have never seen this and create a zip file with some very simple changes to reproduce this on your end. Stay tuned.
So your filesystem customization is just 3 files/directories, gateway, gateway/gatewayconfig.json.default, and gateway/torizonOVPNTest.ovpn. These are all under /etc?
How did you create these files specifically? Did you just create them on the device and then use isolate?
Also you said earlier:
it doesn’t seem to matter what we add
So, if you just add a single new blank file like /etc/foo for example, that causes a broken image as well?
We created them on the device, tested them, then ran the isolate command to create the changes dir.
We had more, sshd_config, shadow, group, password and various others. We removed all those to try and simplify the problem and get to resolution.
I’m going to simplify further to just a single directory and single file named foo and with one container that isn’t at all proprietary and will post my results.
Do you have any thoughts on that missing config blocks message on the console and how that can even be possible? I realize right now you are simply thinking it is us doing something wrong, might be, but what could we possibly be doing wrong that would cause this? Any thoughts, or way too broad to even speculate? It just seems very specific that when changes are applied, the blocks are missing (per the error message) and we remove the changes from the yaml, and voila, the blocks are back.
OK, I modified the changes to be a directory named “test” and a text file named “foo”. Exactly the same behavior. I’ve provided a link to two archives, the first and larger one includes everything to build and reproduce on your Verdin SOM including the output_directory that my Ubuntu system built that I flashed with EasyInstaller. You only need to download that one and run “. ./build-torizon-image.sh” to create the image that for us results in a missing config blocks error on boot created in the output_directory. You may want to rename the current output_directory to output_directory_orig or something so you can diff what your build creates in comparison to ours, or simply redownload the archive if necessary. You could also start by just flashing the image in the output_directory first to witness the behavior we are seeing on our Verdin SOM.
Note, you may need to modify the docker tag to include your private repo details and you do have to provide the auth.
The second one does not include the output_dir, so much smaller. Otherwise exactly the same.
I’m very curious what we could possibly be doing to cause this with this simple setup.
Also, when the EasyInstaller UI shows the progress bar at 100%, it then flips to this screen and remains here forever. Is this expected behavior? I don’t know when I can safely power down and reboot. I’ve waited for hours and it always ends with the same result of missing config blocks. But don’t think this screen with a circle that spins forever is expected?
Okay, your files helped me figure this out. In short, your changes directory seems to be malformed. Using your changes directory I was able to reproduce this issue immediately, the resulting image was the same as yours, broken and not booting properly.
I then generated my own changes directory from scratch. I did the same as you on my device, I created /etc/test/foo and then used isolate to generate the changes directory. Here’s what my changes directory looks like:
With my changes the resulting image is fine, with yours it’s broken. Your changes only has the content for /etc/test/foo. I assume you manually created this directory rather than my changes which was made via the isolate command. The extra files and directories in my changes are auto-generated by the OS upon initializing for the first time, that’s why they get picked up by isolate. As I said the only thing I created myself here was /etc/test/foo.
It seems if you try to create a custom image without the rest of this content that you see in my changes directory the result is a broken image. So I guess try to use isolate to generate the initial changes directory so you have all these other files in place as expected by the OS.
OK, I will change that again. After running isolate, I removed the cni/net.d, ipk-postinsts, tab, files. I was not at all clear why those changes needed to be applied back as changes. Is this documented somewhere? I think I was told I could edit the files in here and not use isolate in regards to ssh key authentication vs password auth. So it sounds like isolate is the recommend way to do this without any issues due to some of these cryptic files being required for the TorizonCore builder file to build the image correctly.
Hello @jeremias.tx, thank you, that solved our problem. I have a couple of questions though, out of curiosity mainly and to help others who experience this same issue.
We run the torizoncore-builder isolate command on Ubuntu with WSL2 on the LAN where our dev board and SOM is securely accessible, then we use git to commit the changes dir, as it was put down by the command output. However, git would not allow us to commit the cni/net.d directories, as there is no file inside them. We could have went in and added a dummy file (assuming that wouldn’t break anything in the build tool), but instead those directories were not in SCM, so we had to remove them from the .tcattr to get the CI/CD system to build without failing.
Likewise, git would not let us commit the empty .wh…wh…opq whiteout file. So that wasn’t available to our CI/CD system.
In case it helps anyone who is trying to use SCM and CI/CD for all the benefits those best practices bring, we resolved this by creating a tarball of the output from the isolate command and now commit that to git instead. Then our build script untars it before building on the CI/CD agent. This approach resolves our problem. The negative is that we lose the ability to track differences in git visually for the isolated change files. Diffing a committed tarball is not very helpful in the SCM when doing code reviews. Further, if we wish to tweak these files, such as sshd_config, directly in our workspace using VSCode and git before committing to a PR dev build of without having to do this on the test SOM and run isolate a time, it makes that process harder. We can get by, or we can customize our process further to let most of the changes be modified directly and we’ll add the special files and directories by script to the changes directory passed to the builder tool.
If you know the answer, what is actually required of these directories/files to avoid the issue of the missing config blocks, corrupted image when flashing?
Empty cni/net.d directory structure
ipk-postinsts/.wh…wh…opq empty file
mtab → …/proc/self/mounts link
We can accept that these special files must be there, but it would be beneficial if the builder tool wouldn’t just take the changes directory passed in and say it is successful if any of these special directories or files that must be present are missing. Another request would be to have your tool output a tar instead of the directories and then let the tcbuild.yaml accept the tar. Finally, a sentence or two in the section of the isolate command describing the expectations here at TorizonCore Builder Tool - Commands Manual | Toradex Developer Center would be helpful.
Thank you, @jeremias.tx . I think it would be good to have a chat offline about the number of users of your system so we get a better feel / confidence about things. Could you reach out to me at my email directly.
After further testing and investigation the issue here actually seems to be something else entirely.
I took my changes directory that I generated with isolate and I removed all the other files and folders until it was identical to the changes directory you sent me. I created a custom image with this and it still worked. Meaning I was incorrect and none of those other files are actually needed. Since these files can be removed then that should resolve your issue with regards to checking this into a git repository.
I then did another test where I created an entire changes directory from scratch without using isolate. I made it to be identical to yours. The custom image made form this also still worked.
Now I’m thinking there must be some “invisible” aspect about your changes that make it different to the one I made. Though as far as I can tell they are identical, file permissions are also the same. On my setup I can’t seem to reproduce this phenomenon. Any changes I create either with isolate or manually seem to work fine. It’s only with the changes you provided me where I can even see the issue.
When you generated changes on your setup did you do anything special perhaps? On my setup I’m just using standard Linux CLI tools to create the files, nothing extraordinary. I noticed in the zip files you shared with me there was a _MACOSX folder present. I assume this to mean you’re working on a MAC computer of sorts correct? I can’t imagine this being the issue, but it is a difference between your setup and mine. I’ll need to see if a colleague with a MAC can reproduce what you’re seeing.
No, nothing special. But this is odd, because I have to have at least one those files and empty dirs or it will not work for me. I have proved this with multiple runs, including them solved the problem. But I had to include them in a special way, I tarred them up on the ubuntu WSL bistro that I ran the isolate command on, then untarred them into the changes directory I passed to the torizoncore-builder build command. We now have a very simple changes-base.tar.gz that we commit and untag right before running the command. It looks like this.
If I find some time, I can try and remove these entries one by one till I find the file(s) that are required, but I haven’t taken the time to do that as we are simply glad this approach works and are working on finalizing our image. But this lets us change everything else that is meaningful in the changes dir directly and even augment/remove entires and file contents by hand as we iterate.
Note, we did try checking the files into git using a special .gitignore in the cni/net.d dir and the links and empty files also, but they would not allow the flash to work without the missing config blocks. I was going to compare the rootfs after builder completes between a good and bad output_dir but never got to it. I did notice in the tar, the userid is 1002 for the files, and it might be some expectation on one or more of those files/directories having a specific owner? Just a guess I haven’t validated.
As for the _MACOSX folder, my apologies, before uploading the archive to google drive, I scp’d the tar from the ubuntu CI/CD machine to my Mac and then edited some files to remove proprietary AWS repo information and that must have been added by MacOS during the unzip/rezip. I missed that.
Yet, I only get a broken image when using the changes directory provided by you. Very strange, not sure what conclusion I can even draw from all of this. I guess if you ever uncover any new information with regards to this it could help. Since as of right now I can’t reproduce this behavior from scratch. Nor can I determine what is so special about your changes directory that causes the issue.
