I’m trying to start a torizoncore-builder image “over” after several quarters of development… To condense 4 rolling “changes directory” filesystem modifications into a single new change… all while setting up a new development environment on a new host laptop. My old dev machine still works fine and can create new EasyLoader image directories.
Everything seemed straightforward until I took a “golden” development SOM and ran torizoncore-builder isolate on it. The changes folder AND/OR the docker storage volume DO NOT capture most of the changes on the filesystem. See the differences below: (I’m omitting a bunch of systemd, sysctl, and networking changes for clarity)
From isolate command on the host:
tarform@LAPTOP-ENDURO:/mnt/c/Users/tarfo/Desktop/github/tcbuilder-founders$ ls -lR 20221028/usr/etc/tarform/
20221028/usr/etc/tarform/:
total 0
drwxrwxrwx 1 tarform tarform 512 Oct 28 15:43 startup
-rwxrwxrwx 1 tarform tarform 296 Sep 15 16:44 transmitJournal.sh
20221028/usr/etc/tarform/startup:
total 0
drwxrwxrwx 1 tarform tarform 512 Oct 28 15:43 frames
20221028/usr/etc/tarform/startup/frames:
total 315136
...
Versuses the actual filesystem on the SOM:
torizon@apalis-imx6-10925306:~$ ls -lR /usr/etc/tarform/
/usr/etc/tarform/:
total 8
drwxr-xr-x 3 root root 4096 Jan 1 1970 startup
-rwxr-xr-x 2 root root 286 Jan 1 1970 transmitJournal.sh
/usr/etc/tarform/startup:
total 24
-rwxrwxrwx 2 root root 515 Jan 1 1970 enableCan.sh
drwxrwxrwx 2 torizon torizon 4096 Jan 1 1970 frames
-rwxrwxrwx 2 root root 593 Jan 1 1970 gpioInitDMRELAY.sh
-rwxrwxrwx 2 root root 367 Jan 1 1970 gpioInitTCA6408.sh
-rwxrwxrwx 2 root root 1458 Jan 1 1970 gpioSysFSInitLE910C1.sh
-rwxrwxrwx 2 root root 249 Jan 1 1970 init.sh
/usr/etc/tarform/startup/frames:
total 54084
...
I believe I know what’s going on here and it’s due to a misconception with how the isolate command operates.
TorizonCore the OS uses OSTree in order to version control changes and updates to the filesystem. TorizonCore Builder also leverages OSTree for this purpose. The isolate command captures and records new changes to the /etc directory. However, in order to do this we need to be able to determined what “changed” meaning we need to know the default state of /etc.
That is where /usr/etc comes in. This directory is a read-only immutable record of what the default state of /etc was on initial deployment. Of course on a fresh system /usr/etc and /etc have the same contents, but as time goes on /etc can diverge from /usr/etc. The isolate command captures the difference between the 2 as “changes”. Essentially under the hood we use this command: ostree-admin-config-diff(1) — ostree — Debian testing — Debian Manpages
Now for your issue it looks like you’ve already created a custom TorizonCore with some changes and then flashed it to your “golden” module. I’m assuming this because you showed your custom tarform directory is present in /usr/etc meaning that’s part of the default state on this system.
Therefore isolate won’t capture these changes since they’re the default state and isolate only captures differences from the default state.
If you want isolate to capture every single change of yours in /etc then what you need to do is the following:
Start with a vanilla un-modified TorizonCore image on a device.
Add all your files and changes you want captured to /etc.
Then run isolate on this module.
Since all of the changes are different from the default state as seen in /usr/etc, then isolate should capture all of these.
Thanks for the detailed response, and this clears up what is happening for me.
I suspected ostree but the isolate documentation isn’t clear on this detail. Looks like ostree is configured to only keep one previous FS (boot.1), so I don’t think I can roll back to the commit of the vanilla FS on the SOM? ostree is not a complete history like git, correct?
Is there any way to run an ostree diff of the SOM FS against the vanilla image unpacked into the docker volume, maybe with ostree remote? [not sure what to make of the fact that the only remote on the SOM is… itself, but named after my develop machine’s workdir. torizon@apalis-imx6-10925306:~$ ostree remote show-url tcbuilder http://localhost:8080/
(or do we have to flash it to the SOM first?)
Can I make the changes on the docker image directly then integrate the new volume into a tcbuild.yaml workflow? Looks like torizoncore-builder union could do this but I’m having a hard time finding anything in “storage” that changes at all.
so I don’t think I can roll back to the commit of the vanilla FS on the SOM? ostree is not a complete history like git, correct?
On the device OSTree only keeps a small amount of previous deployments due to space optimization on the device. OSTree can act as a versioning system similar to Git, but this would involve you maintaining an OSTree repo external to the device itself.
Is there any way to run an ostree diff of the SOM FS against the vanilla image unpacked into the docker volume, maybe with ostree remote?
But I’ve never used this command before. Also you’d need to get the two deployments you want on the same system to compare them I assume. Also I imagine this command would note all changes. Meaning even the changes that were done by Toradex between different versions of TorizonCore. Might not be the cleanest way if you’re only interested in your changes, could work though.
Can I make the changes on the docker image directly then integrate the new volume into a tcbuild.yaml workflow? Looks like torizoncore-builder union could do this but I’m having a hard time finding anything in “storage” that changes at all.
I’m not sure if I understand what you’re proposing here. The union command works on the changes picked up by other commands like isolate or splash. I wouldn’t recommend trying to artificially inject changes here that didn’t come from other commands. It may break things.
Do you not have a list or revision controlled history of all the changes you’ve made to TorizonCore since beginning development? I feel it would be easier to just apply the full changes to a fresh TorizonCore image then run isolate on that.
We’ve got the appropriate changes tracked with git but we also have a lot of large media files that I’m deploying to the production filesystem directly using torizoncore. It was untenable to have them in application docker images and they are similarly not appropriate for source control. I was hoping to not have to start fresh, but if that’s the happy path I’ll build a new system by hand.
