Provisioning Device

FlosPonum · April 22, 2024, 2:55pm

HI,

I am running into an issue where I can get the device to complete the initial provisioning, but then the device fails to communicate with the server. Starting from the beginning, I had to copy the root certificate from the server by running the following command otherwise the provisioning command would fail due to the certificate not being verified:
openssl s_client -showcerts -connect kc.torizon.io:443 </dev/null 2>/dev/null | sed -n '/BEGIN/,/END/p' > /tmp/torizon.crt

I then copied the certificate into the trusted certificate directory:

sudo cp /tmp/torizon.crt /usr/local/share/ca-certificates/torizon.crt

Then I updated the certificates by using:

sudo update-ca-certificates

This allows the provisioning command from torizon cloud to work. The device is then shown on the dashboard. But the device is unable to communicate with the server and stuck in the state of “This device hasn’t been online for a while.” Please see the picture for reference.

Screenshot 2024-04-22 103146

Looking at the gateway.url located at “/usr/local/sota/gateway.url” it is pointing to: https://dgw.torizon.io
But, when I look at the gateway.url located at “/var/sota/import/gateway.url”, it is pointing to: https://ota-ce.torizon.io

I created a /etc/sota/conf.d/20-sota-device-cred.toml, to point to the new credentials, please see below:

[tls]
server_url_path = "/var/sota/import/gateway.url"

[import]
base_path = "/var/sota/import"
tls_cacert_path = "/var/sota/import/root.crt"
tls_clientcert_path = "client.pem"
tls_pkey_path = "pkey.pem"

Then I restarted aktualizr by running the following commands:

systemctl stop aktualizr

systemctl start aktualizr

This still results in the same behavior.

Thanks for your help,
Jon

jeremias.tx · April 22, 2024, 6:25pm

Greetings @FlosPonum,

This is a odd issue you’ve found. What version of Torizon OS are you running on the module that you were trying to provision?

As far as I know provisioning is currently working fine. What provisioning method did you use? Did you just copy the provisioning command from our web UI? Or did you do something else?

Also from this device could you share the Aktualizr logs. Perhaps something there may provide additional information about what’s going on here.

But, when I look at the gateway.url located at “/var/sota/import/gateway.url”, it is pointing to: https://ota-ce.torizon.io

This detail is strange too, dgw.torizon.io should be the correct URL. Though the other file should take priority over this one anyways.

One final question, if you try to re-provision everything from scratch do you observe the same behavior?

Best Regards,
Jeremias

FlosPonum · April 23, 2024, 1:31pm

OS and kernel version below:

Operating System: TorizonCore Upstream 6.6.1+build.12 (kirkstone)
Kernel: Linux 6.1.84-6.6.1+git.347385861c50

I copied the command from the web UI:

curl -fsSL https://app.torizon.io/statics/scripts/provision-device.sh | sudo bash -s -- -t [token info]  && sudo systemctl restart aktualizr fluent-bit

I reinstalled the OS from scratch from the Toradex easy installer, I will post what I am doing step by step that’s resulting in this behavior.

On boot, with no modifications other than a password change. Run web UI command:

curl -fsSL https://app.torizon.io/statics/scripts/provision-device.sh | sudo bash -s -- -t [token info]    && sudo systemctl restart aktualizr fluent-bit

Result:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Grab and add certificate:

sudo mkdir -p /usr/local/share/ca-certificates/

openssl s_client -showcerts -connect kc.torizon.io:443 </dev/null 2>/dev/null | sed -n '/BEGIN/,/END/p' > /tmp/torizon.crt

sudo cp /tmp/torizon.crt /usr/local/share/ca-certificates/torizon.crt

sudo update-ca-certificates

Result:

Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
rehash: warning: skipping torizon.pem,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

Run web UI provisioning command again with a new token:

curl -fsSL https://app.torizon.io/statics/scripts/provision-device.sh | sudo bash -s -- -t [new token info]    && sudo systemctl restart aktualizr fluent-bit

Result:

Checking dependencies...
== Registering device (deviceID: colibri-imx6-04836707) in system, and downloading credentials.
== Backing up any existing ota device creds
jq: error: Could not open file /var/sota/import/info.json: No such file or directory
jq: error: Could not open file /var/sota/import/info.json: No such file or directory
== Extracing device credentials from archive
Archive:  device.zip
  inflating: client.pem
  inflating: pkey.pem
  inflating: root.crt
  inflating: info.json
  inflating: gateway.url

== Success!
Device has been registered with the system and credentials are in place!

{
  "registeredName": "Firm-Granola",
  "deviceID": "colibri-imx6-04836707-1dfe7b",
  "deviceUuid": "1dfe7ba7-94f5-452b-933c-ea6a5616f3df",
  "createdAt": "2024-04-23T12:45:54Z"
}

== Restarting services...
Restarting aktualizr...
Restarting fluent-bit...
Removing RAC files...

    Aktualizr should automatically connect with the server. For logs run: sudo journalctl -f -u aktualizr*

Check aktualizr logs:

sudo journalctl -f -u aktualizr*

Result:

Apr 23 12:46:19 colibri-imx6-04836707 aktualizr-torizon[5044]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Apr 23 12:46:21 colibri-imx6-04836707 aktualizr-torizon[5044]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Apr 23 12:46:21 colibri-imx6-04836707 aktualizr-torizon[5044]: PROXY: initializing...
Apr 23 12:46:21 colibri-imx6-04836707 aktualizr-torizon[5044]: PROXY: using TCP port 8850.
Apr 23 12:46:21 colibri-imx6-04836707 aktualizr-torizon[5044]: Offline Updates are disabled
Apr 23 12:46:21 colibri-imx6-04836707 aktualizr-torizon[5044]: PROXY: starting thread.
Apr 23 12:46:21 colibri-imx6-04836707 aktualizr-torizon[5044]: PROXY: listening to connections...
Apr 23 12:46:21 colibri-imx6-04836707 aktualizr-torizon[5044]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Apr 23 12:46:22 colibri-imx6-04836707 aktualizr-torizon[5044]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK
Apr 23 12:46:24 colibri-imx6-04836707 aktualizr-torizon[5044]: curl error 60 (http code 0): SSL peer certificate or SSH remote key was not OK

Thanks,
Jon

jon.tx · April 23, 2024, 5:21pm

It’s the initial failure of the provisioning script with unable to get local issuer certificate that’s the most interesting to me here.

I checked our latest releases and our latest nightlies, and they all seem to have the appropriate certs in /etc/ssl/certs/ca-certificates.crt. Did you perhaps replace that file with one of your own, that doesn’t include the DigiCert global CA?

If you can reproduce the issue, I’d appreciate it if you could run curl -vvv https://app.torizon.io and show the output, and also send along the output of your openssl command that grabbed the cert chain for app.torizon.io. My working theory is that perhaps you’re on a corporate network that MitMs your TLS connections.

FlosPonum · April 23, 2024, 8:06pm

Your theory was correct. I was able to contact IT and get it whitelisted. I now have no issues. Thank you guys for your help!

jeremias.tx · April 23, 2024, 8:10pm

Perfect glad we were able to point you in the right direction!