I downloaded a lockbox for offline update of the application a week ago. But after awhile I get this error when trying to update:
Aug 30 06:36:26 verdin-imx8mp-14772918 aktualizr-torizon[925]: fetchMetaOffUpd() called with source_path: "/media/USB/update"
Aug 30 06:36:26 verdin-imx8mp-14772918 aktualizr-torizon[925]: Invalid role in root.json
Aug 30 06:36:26 verdin-imx8mp-14772918 aktualizr-torizon[925]: Invalid role in root.json
Aug 30 06:36:26 verdin-imx8mp-14772918 aktualizr-torizon[925]: Director metadata update failed: The version of role offline-updates does not match the entry in Snapshot metadata.
Aug 30 06:36:26 verdin-imx8mp-14772918 aktualizr-torizon[925]: Event: UpdateCheckComplete, Result - Error
This might have to do with our use case:
Our Current Setup:
We are testing offline updates in a system that simulates how devices will be updated in the field.
We’re using a lockbox created with Torizon Cloud to update devices to specific versions.
There is 2 types of updates: Formal and non-formal
Formal updates use the lockboxes created by our CI and versioned. Going back and forth between those work. But after a non-formal update, I can’t update it anymore.
Non-formal updates. This is where a dev have built an executable. The test system will take that executable, make it a docker image. Then it will push a docker-compose using that docker image to torizon to make it a lockbox. That lockbox is downloaded, sent to the target device and mounted to simulate USB.
Could you explain what this error mean and if our use-case is to blame for the error?
Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!
Did you like my answer?
The error you’re encountering, specifically “Invalid role in root.json” and “The version of role offline-updates does not match the entry in Snapshot metadata,” indicates a mismatch or inconsistency in the metadata of the lockbox. This can happen due to several reasons, especially in a setup where both formal and non-formal updates are being used.
Here are some potential causes and solutions:
Metadata Mismatch:
When you create a lockbox, it includes metadata that ensures the integrity and authenticity of the update. If the metadata in the lockbox does not match the expected metadata on the device, the update will fail.
Ensure that the metadata for both formal and non-formal updates is consistent. This means that any non-formal updates should be properly versioned and signed to match the expected format.
Snapshot Metadata:
The snapshot metadata keeps track of the versions of all roles (including offline-updates). If a non-formal update modifies the snapshot metadata in an unexpected way, it can cause subsequent updates to fail.
Make sure that when creating non-formal updates, the snapshot metadata is correctly updated and consistent with the formal updates.
Role Definitions:
The error “Invalid role in root.json” suggests that there might be an issue with the role definitions in the root.json file of the lockbox.
Verify that the root.json file in your lockbox correctly defines all necessary roles and that these roles are consistent with the ones expected by the device.
Lockbox Revocation:
If a lockbox is revoked or modified, the device will no longer accept it. Ensure that the lockbox you are using has not been revoked or modified in a way that makes it invalid.
You can check the status of the lockbox in the Torizon Cloud Web Interface to ensure it is still valid.
Testing Environment:
Since you are simulating updates in a test environment, ensure that the environment closely matches the production setup. Any discrepancies in the environment can lead to unexpected issues.
To resolve the issue, you can try the following steps:
Recreate the Lockbox:
Recreate the lockbox for the non-formal update, ensuring that all metadata is correctly generated and consistent with the formal updates.
Check Metadata Consistency:
Verify the consistency of the metadata in the lockbox, especially the root.json and snapshot metadata.
Are you re-using the same lockbox “name” each time for one of your lockboxes?
we used to sporadically see this when we were constantly modifying the definition of the same (“Software_Test”) lockbox with new revisions for internal deploy - updates to newer builds went fine, but we’d see this issue if you tried to downgrade.
My (non-official) understanding is that the update system expects the version information in the lockboxes to only ever increase to support new versions. In our case, if we attempted to “downgrade” to an older Software_Test lockbox we would see this error because the metadata version was both decreasing (older lockbox build) as well as “different” from what the OTA system knew about that lockbox (because the definition had changed). This went away when we progressed past internal development and started making “formal” lockboxes for release which had the build version in the name (“Product_v1.0.0”, “…v1.0.1” and so on).
This means there’s something wrong or outdated with the security metadata that is on the Lockbox you are trying to install. You can try fixing this by re-downloading the Lockbox with TorizonCore Builder. This will re-create the Lockbox, but with up-to-date security metadata.
Let’s say that you have lockbox A intalled on the device and you want to install lockbox B.
If you get that error, this means that:
Lockbox B was downloaded before lockbox A was created and or downloaded
The security metadata was somehow changed between the download of Lockbox B and the download of Lockbox A. This can mean that a new lockbox was created or a lockbox was revoked.
For this reason, the Lockbox B you downloaded before cannot be installed, as the security metadata would need to be downgraded, which would be a security flaw.
You can still install lockbox B if it was not revoked, as long as you download it again from the Torizon Cloud with the updated security metadata.
This topic can be tricky, so please let me know if my explanation was not clear or if you have follow-up questions.
If I understand correctly, this means that every time a lockbox is created or revoked, the security metadata of all other lockboxes is updated.
From this understanding, the correct way of handling for us is:
(Formal versions) Our CI create lockboxes that are tagged V1.0.1, V1.0.2, … Everytime a new one is created, if we want to store them in our network drive, we need to redownload all of them. In consequence, it’s probably better to just store the just name of the lockbox.
(non-formal versions) Our test system creates a lockbox from a developer’s executable of our app. Thus, all lockboxes get their security metadata updated. This means the test system needs to download lockboxes for formal version directly from Torizon cloud.
We can update a device at an older security metadata with a lockbox at a newer security metadata. But not the opposite. This means we can’t really cache update packages with the test system.
For example, I can imagine a few things:
If a customer downloads V1.1.0 at some point and then update his device from V1.0.5 to V1.1.0 with a USB containing the lockbox. There is a bug in the app so he wants to downgrade. He cannot use his older USB containing V1.0.5 lockbox, he needs to re-download a V1.0.5 lockbox.
A customer downloads a V1.1.0 update at some point. It’s a customer with multiple devices. They update some of them but not all. 1 year later they update the ones that weren’t updated. As long as the expiration date of the lockbox hasn’t occurred, the lockbox should still be valid, right? The security metadata does not change other than when an update occurs?
Does the security metadata get updated when a lockbox expires?
In our field, it regularly happens that devices are rarely updated (medical) by customers. So I am trying to estimate the impact of this mechanism. If my understanding is correct, it should be fine as long as every update is downloaded directly from Torizon platform, whether you downgrade or upgrade.
Or are the security metadata of all other lockboxes untouched when one gets created/revoked/recreated?
The general understanding you put here is correct, but I am unsure about the exact triggers of new security metadata.
It is also important to mention that there are a few pitfalls when dealing with lockboxes that may lead to reaching a version limit for the metadata.
I will ask for someone from the Torizon team to comment on this specific use case so we can be sure that such problems will not occur.
I spoke with one of the Torizon OS developers so I can give some more specific answers here. I will try to reply point by point in an order that should help you understand the full picture, but if you have any follow-up questions, please let me know.
To be precise, the security metadata of your whole repository is updated.
And each account on the Torizon Cloud has a unique repository, with all the packages (OS, applications, subsystems, etc) in that account.
This is correct, the security metadata cannot be downgraded to protect against rollback-based attacks.
If a lockbox with known-vulnerable or problematic software was revoked and the system has up to date metadata with this information, it will not be possible to rollback the security metadata to a version where the problematic lockbox was not revoked and could be installed.
Yes, in general that should be correct.
The security metadata has an expiry date as well, so this will also influence the ability to install the lockbox.
As long as the security metadata’s expiry date and the lockbox expiry date have not been reached the customer will be able to install the update.
The security metadata is for the whole repository/account, so changes to other lockboxes will reflect on it.
Your understanding is correct, there should be no problems installing lockboxes which have not been revoked, as longs as they are downloaded from the Torizon Cloud.
This would be a valid approach, but you need to be careful about how this will be implemented to not run into the pitfall I mentioned previously.
There is a limit to the file size of the security metadata, and when you get to hundreds of unique lockboxes this will become a problem.
The way to avoid this is to not create multiple lockboxes, but create multiple versions of lockboxes.
For example, you can have effectively as many versions of the “non-formal” lockbox to test, as long as they are all versions of that lockbox, not unique lockboxes.
For the “formal” lockboxes, you could create one for each release, as long as the releases are planned in such a way that for the lifetime of the product the lockbox limit will not be a problem.
If the release schedule is once every quarter and the product has a lifetime of 20 years, you get 80 different release lockboxes, which should not be a problem.
Please note that even revoked lockboxes are added to the security metadata and therefore increase its file size, so revoking unused lockboxes is not really a solution.
If you have follow-up questions, please reach out as this topic can be quite complex.
Thank you very much for the detailed answers. It seems a lot clearer to me
I still have questions about the last part:
What does it mean “create multiple versions of lockboxes”? I can create multiple versions of a package but for a lockbox I can just define a new one. Is that when you create a lockbox with the same name as an existing one? I was under the impression that it was not possible with the API. Which is why, for non-formal lockboxes, I thought that I needed to revoke the previous one and re-create the new one.
So in my Torizon Cloud Platform, I would have 1 lockbox per release + 1 lockbox for non-formal dev version. For testing, we would just create the non-formal dev lockbox?
You are correct that the lockboxes are not explicitly versioned, but you can update an existing lockbox by using the same name when defining a new lockbox.
This is also possible via the API, as you can check here.
There is no need to revoke an existing lockbox before updating it.
