Torizoncore-builder platform lockbox fails with "Invalid Refspec"

I am attempting to create the lockbox for an offline update, but it is failing with an “Invalid refspec” error. All of the requisite packages have been uploaded and the lockbox is defined on the web interface, but copy-pasting the command results in the following output:

$ torizoncore-builder platform lockbox --credentials PAT.zip TestUpdate

=>> Handle director-repository metadata
Fetching 'TestUpdate.json'
Fetching 'offline-snapshot.json'
Fetching '1.root.json'
Fetching '2.root.json'
Fetching '3.root.json'
Fetching '3.root.json' (version not available, stopping)

=>> Handle image-repository metadata
Fetching 'snapshot.json'
Fetching 'tdx-quarterly.json'
Fetching 'tdx-monthly.json'
Fetching 'targets.json'
Fetching 'tdx-nightly.json'
Fetching 'tdx-lts.json'
Fetching 'tdx-containers.json'
Fetching '1.root.json'

=>> Process metadata
Loading offline-update targets metadata from 'update/metadata/director/TestUpdate.json'
Loading offline-update snapshot metadata from update/metadata/director/offline-snapshot.json
Offline-update metadata passed basic validation
Loading image-repo targets metadata from 'update/metadata/image-repo/targets.json'
Loading image-repo delegated targets metadata from 'update/metadata/image-repo/tdx-nightly.json'
Loading image-repo delegated targets metadata from 'update/metadata/image-repo/tdx-monthly.json'
Loading image-repo delegated targets metadata from 'update/metadata/image-repo/tdx-quarterly.json'
Loading image-repo delegated targets metadata from 'update/metadata/image-repo/tdx-lts.json'
Loading image-repo delegated targets metadata from 'update/metadata/image-repo/tdx-containers.json'

=>> Handle Uptane targets
Handling OSTree target 'Test OS Image-83e43d79b825341ca12556a76e56d526816ec60873f2877795b6ed9c7ae2cc68'
Commit 83e43d79b825341ca12556a76e56d526816ec60873f2877795b6ed9c7ae2cc68 found on the user's repo
Fetching OSTree commit 83e43d79b825341ca12556a76e56d526816ec60873f2877795b6ed9c7ae2cc68 from https://api.torizon.io/treehub/api/v3...
Uptane info: target 'Test OS Image', version: '83e43d79b825341ca12556a76e56d526816ec60873f2877795b6ed9c7ae2cc68'
1130 metadata, 12761 content objects fetched; 193561 KiB transferred in 321 seconds; 520.6 MB content written
error: Invalid refspec Test OS Image-83e43d79b825341ca12556a76e56d526816ec60873f2877795b6ed9c7ae2cc68
Could not create ref according to Uptane target name (non-fatal)

How can I resolve this?

Greetings @bw908,

From what I can tell it looks like the OS update part of the Lockbox can’t be located and downloaded.

First of all this “TestUpdate” Lockbox of yours does it’s definition contain an OS update package by the name of “Test OS Image”?

If yes, is this package present on your account and can be seen in the web UI? Even more specifically does a package of this name exist on your account with this hash 83e43d79b825341ca12556a76e56d526816ec60873f2877795b6ed9c7ae2cc68?

If a package exists with the name “Test OS Image” but doesn’t have the above hash then that’s not the exact same package as defined in your Lockbox.

Best Regards,
Jeremias

It spends several minutes actually downloading something by that hash prior to the error - and I can see that ‘TestUpdate’ contains “Test OS Image” as one of its packages with the matching hash on the web.

(Note the OS image name has been edited for the purposes of posting it publicly)

That’s strange, perhaps the OS package is corrupted or malformed in some way.

As a sanity test what happens if you try to do an online update with that OS package?

Or alternatively if you upload a completely different OS package, define a lockbox with this new package, and then try and create a lockbox with TorizonCore Builder. Does the same thing happen?

Basically just trying to rule out any abnormalities with this specific OS package.

Best Regards,
Jeremias

Unfortunately I don’t currently have any actual hardware for this image to try an online update, but I have a different version of the OS image I just tried with TestUpdate2 that produces an identical error

I tried creating a custom OS, uploading, and creating a Lockbox with it. It works on my side so I can’t seem to reproduce whatever issue you’re experiencing.

Let’s try this, create a Lockbox using one of the Toradex provided OS packages instead of one of your custom packages. Does the download work with this?

Also how are you creating and uploading your custom OS packages specifically?

Best Regards,
Jeremias

I found the problem - torizoncore-builder will happily let you publish an image name containing spaces, but chokes if you try to create a lockbox with it (I simply renamed my image and it worked).

I clued in to that while I was working on publishing a docker-compose as part of our CI system and it yelled at me for calling the file
[Project] Containers.lock.yml

Which is the “intended” behaviour - supporting spaces, or not?

Strange, I was able to reproduce this behavior as well.

If I push an OS package with package name that has a space (i.e. support test), then that is fine. If I try to do the same with a docker-compose package then an error is produced.

Let me try to get clarification from our team on what our Platforms server is suppose to do. In the meantime please refrain from using package names that contain space characters.

Best Regards,
Jeremias

@jeremias.tx I think there is a similar bug in the lockbox fetching for a docker compose file if the repository names contain hyphens; I get the following error during this process:

Starting DIND container
Using Docker host "tcp://127.0.0.1:22376"
Connecting to Docker Daemon at "tcp://127.0.0.1:22376"
Fetching container image [org]/[project]-app@sha256:a46101f75ef2e3166885d1e4203a324fdba7dfef7b5a358278dcf3665609de62
Stopping DIND container
Removing output directory 'update/' due to errors
Error: container images download failed: 404 Client Error for 
-app: Not Found ("pull access denied for [org]/[project]-app, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

note the -app: Not found at the beginning of the line.

-login was provided in the lockbox command line, and that container with that commit does exist in the registry. I should note that prior to the failure the tool has no problem fetching the manifests for this container from the repository.

After some internal discussion and testing I think I have feedback for all the bugs here reported so far.

First of all spaces in OS package names:

The issue here is that while the server accepts a package name with a space in it, OSTree does not. When a Lockbox is downloaded TorizonCore Builder tries to create a local OSTree repository, where the ref/branch name is the same as the package name on the server. However, OSTree does not accept ref/branch names that have a space. This causes the error you see.

Spaces in docker-compose package names:

This is a bug. As I said above the server should have no issues with package names containing spaces. There seems to be a bug in the server/tooling used to upload packages. Docker-compose packages are uploaded slightly differently than OS packages which is why there’s a difference here.

With that said, it’s probably for the best that you refrain from using any spaces in package names while our team tries to improve/change the behavior here.

Now for this new issue you reported in your latest post:

I think there is a similar bug in the lockbox fetching for a docker compose file if the repository names contain hyphens; I get the following error during this process:

I wasn’t able to reproduce this. I used this docker-compose file as a test:

services:
  foo:
    image: torizon/weston-vivante@sha256:c9c846118538dee213cd740602796ca1a4eb73c5a2d6505659ce37932e40966d
version: '2.4'

As seen above the docker image that is being referenced has a hyphen in the name. When I download a Lockbox containing this, it works.

Are there any other strange characters or such in your docker image name? I assume the [org]/[project]-app you provided was just a placeholder for the actual name.

Best Regards,
Jeremias

Very interesting. You’re right the name is just a placeholder/edited since the product is not public yet, but it follows the same format of companyname/productname-app

There is indeed something else wrong, it was just an unfortunate line-wrap that made it look like it was trying to find a container called “-app”

I’ll dig some more.

I think my torizoncore-builder platform lockbox issue is related to private registries. Despite accepting the “login” argument for building the lockbox, it does not seem to use it to authenticate - I’ve triple checked the username and PAT provided to the command, and even tried different ones, but it always fails with a 404 when trying to fetch the images from dockerhub.

Error: container images download failed: 404 Client Error for https://127.0.0.1:22376/v1.40/images/create?tag=sha256%3Acb646d4952284c0f0db5c0cfbe19d002541f78d0f266beae904699976829bb31&fromImage=[image name removed]: Not Found ("pull access denied for [org]/[repository], repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

torizoncore-builder works just fine with the same credentials using the bundle command with the same docker image - so I’m inclined to think that it is not doing the login

I think my torizoncore-builder platform lockbox issue is related to private registries.

Ok now this I was able to reproduce, the --login switch must be bugged.

However, coincidentally the team has an internal build of TorizonCore builder where this issue seems to be fixed. This change isn’t public yet and is still undergoing code review and polishing.

I’ll try and notify you once this change is public so you can be unblocked on this.

Best Regards,
Jeremias

1 Like

Great, thanks for the responsiveness on this!

Just to follow up on this. The fix I was talking about earlier has now been merged and is available on our public repo for TorizonCore Builder: GitHub - toradex/torizoncore-builder: TorizonCore Builder is a tool that allows the customization of TorizonCore images.

We have not created a new release of TorizonCore Builder yet however. In order to get this fix now you can either use the setup script to pull the early-access tag of TorizonCore Builder. Or just build the latest TorizonCore Builder tool itself from the repo source.

Best Regards,
Jeremias