Torizoncore-builder and systemd services in /etc

I’m trying to add a custom systemd service and create a new image with torizoncore-builder. My first attempt was to enable the service to test it, and I assumed I could create an image with the service enabled. This failed with the “union” command error below. I think it’s complaining that the service file is not in /usr/etc/systemd/system/multi-user.target.wants (it is in /etc/systemd/system/multi-user.target.wants).

Of course /usr/etc is RO so I can’t just put the file there.

I expect this will work if I try create the image with the service disabled, about to give that a try now…

Here is the error message:

torizoncore-builder union --union-branch=upstream-torizon-5_1-custom
setfacl: systemd/system/multi-user.target.wants/custom.service: No such file or directory
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:

Traceback (most recent call last):
File “/builder/torizoncore-builder”, line 175, in
mainargs.func(mainargs)
File “/builder/tcbuilder/cli/union.py”, line 152, in do_union
union(args.changes_dirs, args.extra_changes_dirs, args.storage_directory,
File “/builder/tcbuilder/cli/union.py”, line 128, in union
set_acl_attributes(changed_dir)
File “/builder/tcbuilder/cli/union.py”, line 107, in set_acl_attributes
apply_tcattr_acl(files_to_apply_tcattr_acl)
File “/builder/tcbuilder/cli/union.py”, line 44, in apply_tcattr_acl
subprocess.run(setfacl_cmd, shell=True, check=True)
File “/usr/lib/python3.9/subprocess.py”, line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ‘cd /storage/changes/usr/etc && setfacl --restore=/storage/changes/usr/etc/.tcattr’ returned non-zero exit status 1.

Indeed, disabling the service works, at least as far as the union command. I’ll test the actual image next.

HI @deltamike how did you create the service initially? Did you do it manually on a booted target and then use “isolate” to bring the changes back into your torizoncore-builder setup?

Drew

I just reproduced this using the isolate command. I’ll submit a bug ticket for the development team.
Drew

Here is the ticket link: https://toradex.atlassian.net/browse/TOR-1859

Please add any details I may have missed.

Yes, I just added the .service to /etc/systemd/system. When you activate it, a copy is placed in /etc/systemd/system/multi-user.target.wants, but not in /usr/etc.

That brings up a possibly related question: If I want this change to always win an OSTree 3-way merge, do I have to mirror changes in /etc with a corresponding change in /usr/etc?

I’m not seeing the file created in /usr/etc. I only see it in /etc/systemd/system/multi-user.target.wants. And the isolate command is supposed to handle that.

Hi @deltamike,

The development team just pushed version 3 of TorizonCore builder which contains the fix for this.

Drew

I’m still seeing a very similar problem in version 3.1.0

$ torizoncore-builder --version
torizoncore-builder 3.1.0

$ torizoncore-builder isolate --remote-host 192.168.137.205 --remote-username torizon --remote-password torizon --changes-directory wwanup
An unexpected Exception occured. Please provide the following stack trace to
the Toradex TorizonCore support team:


Traceback (most recent call last):
  File "/builder/torizoncore-builder", line 204, in <module>
    mainargs.func(mainargs)
  File "/builder/tcbuilder/cli/isolate.py", line 64, in isolate_subcommand
    common.set_output_ownership(changes_dir)
  File "/builder/tcbuilder/backend/common.py", line 537, in set_output_ownership
    apply_workdir_ownership(os.path.join(rootdir, filename),
  File "/builder/tcbuilder/backend/common.py", line 502, in apply_workdir_ownership
    file_uid, file_gid = get_file_ownership(filename)
  File "/builder/tcbuilder/backend/common.py", line 516, in get_file_ownership
    stat = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/workdir/wwanup/usr/etc/systemd/system/multi-user.target.wants/initBIU.service'

And, at least on this torizoncore image, /usr is mounted read-only, any work arounds?

$ sudo touch /usr/test
touch: cannot touch '/usr/test': Read-only file system

Hi @ed-bear

Do you explicitly need the --changes-directory? I think it will work if you remove that since the files will be stored in the docker volume instead of the local filesystem. I’ll reach out to the dev team to see about fixing the base issue.

Drew

Hi @ed-bear

That is intentional and expected. A read-only root filesystem is required for OTA capability using OSTree. This allows the build system to be able to generate the appropriate changeset to deliver for new versions of the base OS. The /var directory should be read/write though.

What are you trying to change in the root filesystem? There may be a better way depending on your specific use case.

Drew

I missed the original posts Read-Only comment, and RO makes tons of sense for an embedded system.
I was trying to add the service file to other systemd directories.

‘isolate’ without an explicit directory did work, but, uhh, how then do i inspect the changes captured or ‘union’ or ‘deploy’, etc?

You can view the actual storage of the docker volume using docker inspect.

ie

$ docker volume inspect storage
[
    {
        "CreatedAt": "2021-09-10T10:04:48-04:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/storage/_data",
        "Name": "storage",
        "Options": null,
        "Scope": "local"
    }
]
$ ls /var/lib/docker/volumes/storage/_data
changes/  ostree-archive/  sysroot/  tezi/

Drew

Judging from the verbose output of isolate this “storage” volume is not as easy to visually inspect.

2021-09-10 20:04:41,499 - paramiko.transport.sftp - INFO - [chan 1] Opened sftp connection (server version 3)
2021-09-10 20:04:41,499 - paramiko.transport.sftp - DEBUG - [chan 1] mkdir(b'/tmp/torizon-builder-2021-09-10_20-04-41.499934', 511)
2021-09-10 20:04:41,501 - paramiko.transport.sftp - DEBUG - [chan 1] listdir(b'/etc/ipk-postinsts/')

...

2021-09-10 20:04:41,786 - paramiko.transport.sftp - DEBUG - [chan 1] stat(b'/tmp/torizon-builder-2021-09-10_20-04-41.499934/isolated_changes.tar')
2021-09-10 20:04:41,787 - paramiko.transport - DEBUG - [chan 3] EOF sent (3)
2021-09-10 20:04:41,788 - paramiko.transport.sftp - DEBUG - [chan 1] open(b'/tmp/torizon-builder-2021-09-10_20-04-41.499934/isolated_changes.tar', 'rb')

But at least there’s mention of it in union --help:

  --changes-directory CHANGES_DIRS
                        Path to the directory containing user changes (can be specified multiple times). If you have changes in the storage, the changes passed in this parameter will be applied on top of them.

deploy's --help or --verbose are not clear if they push from storage or from the last ostree tag… in fact there’s no discussion of deploy's behavior WITH a ‘changes_dir’ provided instead of an ostree ref.

Re-reading these developer.toradex guides, it looks like some of my confusion comes from the differences between dto deploy and deploy

Thanks for the response drew.tx… /var/lib/docker/volumes/storage/_data does not exist on my WSL2 filesystem…

Though, using the VSCode docker plugin, I have success with the right click menu “Explore in development container” - To wit, running isolate without the --changes-directory does not seem to have captured the custom systemd service and bash scripts I added to /etc, even though the command was successful.

At least neither of the filesystems under /sysroot/ostree/ have the changes…

Hi @ed-bear,

I spoke to the dev team and they have a fix queued for this. We expect it to go live next week. You’ll be able to get the fix directly from Docker hub.

Drew

Hi @ed-bear

It looks like the currently published torizoncore-builder image has the fix for this issue. Please give it a try and let me know how it goes.

Drew