Upgrade python version in TorizonCore with yocto

Technical Support
, , ,
1

Hi,
the current TorizonCore v5.7 comes with python 3.8.13 preinstalled. I’m doing some customization with Yocto, how can I upgrade python to v3.9 during the build?

Regards,
Rocco

2

Hi @RoccoBr ,

Currently there is no straightforward way to do this: The recipe for Python 3.9 is only available starting from Yocto 3.3 Hardknott, and trying to adapt it to Dunfell (which is the version TorizonCore 5.7.0 is based on) means dealing with various dependency errors when building an image.

An easier alternative would be running Python 3.9 on a container. Does your use case allow this?

EDIT: Grammar correction

Best regards,
Lucas Akira

3

hi @lucas_a.tx ,
thank you for your answer. I was suspecting something like this.

we are actually considering moving away from docker, as our container is grown too much with all python libraries that we need, in particular Numpy and Scipy and the OTA has become unmanageable.
We are considering putting everything on the TorizonCore filesystem as OSTree pull seems to work better even in case of poor connectivity over the modem (see my post Torizon OTA: constant TLS handshake timeout error)

Regards,
Rocco

4

In this case another alternative would be waiting for the release of our TorizonCore 6.0 nightly builds, which use Kirkstone as base. This Yocto release has the recipe for Python 3.10.

Our current goal is to release the 6.0 nightly builds in the next one or two weeks if everything goes well. Just keep in mind that they might not boot yet depending on what SoM you’re using.

Best regards,
Lucas Akira

5

thanks @lucas_a.tx ,
but for production we prefer to stick with TorizonCore v5.7 LTS

6

Hi @RoccoBr ,

Right, so the only option would be to adapt a more recent Python 3 recipe to Dunfell.

I can’t promise a solution as Python 3.9 is not officially supported on TorizonCore 5.7.0, but we’ll try adapting the Hardknott recipe in the next few days and I’ll update you if we have any news.

Feel free to also try it on your side in the meantime.

Best regards,
Lucas Akira

7

Hi @RoccoBr ,

We managed to get Python 3.9 into TorizonCore 5.7.0 using Yocto. Version 3.8 is still present in the final image but 3.9.9 is used by default when invoking python3 on the terminal. and it is the default version. To invoke the newer one you have to enter python3.9.

We created a new layer that has an adapted Python recipe from hardknott with a different name (called python3.9 instead of python3). This layer also removes symbolic links that point to 3.8, and adds the newer version as dependency on some recipes that need Python. You can find the layer attached here:

meta-custom-v3.1.zip (63.3 KB)

The steps to add it are:

  • Put the meta-custom directory in layers/;

  • Add an entry in conf/bblayers.conf like this:

[...]
 BBLAYERS = " \
   ${OEROOT}/layers/meta-toradex-torizon \
   ${OEROOT}/layers/meta-toradex-distro \
   ${OEROOT}/layers/meta-toradex-bsp-common \
   ${BASELAYERS} \
   ${BSPLAYERS} \
   ${EXTRALAYERS} \
   ${OEROOT}/layers/openembedded-core/meta \
+  ${OEROOT}/layers/meta-custom \
 "
  • Append this line in conf/local.conf to add the recipe to the image:
CORE_IMAGE_EXTRA_INSTALL += "python3.9"
  • Build the image with:
$ MACHINE=<SoM> bitbake -k torizon-core-docker

Where <SoM> is the module name, like verdin-imx8mm.

Let me know if this helps you.

Best regards,
Lucas Akira

EDIT: Updated meta-custom to v2 as to remove a specific Python 3.8 native file that can conflict with 3.9 native during build.

EDIT 2: Updated meta-custom to v3, as docker-compose didn’t work before due to conflicts between the Python versions. This should be resolved now, as 3.8 was left unaltered and 3.9 was installed with make altinstall, see GitHub - python/cpython: The Python programming language.

1 Like
8

thank you very much @lucas_a.tx !

I have made the changes as you stated and using the torizon/crops container to build the image with the command
docker run --rm -it --name=crops -v ${PWD}:/workdir --workdir=/workdir -e MACHINE=verdin-imx8mm -e IMAGE=torizon-core-docker torizon/crops:dunfell-5.x.y startup-tdx.sh

I get this error:
/workdir/torizon/build-torizon/tmp/work/x86_64-linux/python3.9-native/3.9.9-r0/temp/run.do_install.14508: line 114: /workdir/torizon/build-torizon/tmp/work/x86_64-linux/python3.9-native/3.9.9-r0/check_build_completeness.py: Permission denied
it’s strange that is creating a x86_64-linux folder. I can see there is also a tmp\work\aarch64-tdx-linux\python3.9\3.9.9-r0 folder

9

Hi @RoccoBr ,

I built TorizonCore on Linux using crops as you did, but I couldn’t reproduce your issue. Can you try setting the executable bit in check_build_completeness.py by running:

chmod a+x <your-yocto-dir>/torizon/layers/meta-custom/recipes-python/python3.9/python3.9/check_build_completeness.py

Also, I just updated the meta-custom layer to include a minor correction in python3_3.8%.bbappend when building the image. Although it isn’t related to your issue, I’d recommend you replace the older directory with this one.

Best regards,
Lucas Akira

10

Hi @RoccoBr ,

I ended up updating the layer again after seeing that docker-compose stopped working on the module. This should be solved on the new version. Just make sure to run bitbake python3.9-native -c clean before building the image (if using crops execute the docker run command without the -e IMAGE part to get to a terminal inside the container).

The chmod recommendation from my previous answer stands.

Sorry for the inconvenience.

Best regards,
Lucas Akira

11

hi @lucas_a.tx ,
I have done a clean build from the scratch with the new version of the meta-custom and now I’m getting these errors:

ERROR: libbytesize-2.2-r0 do_package_qa: QA Issue: /usr/bin/bscalc contained in package libbytesize requires /usr/bin/python3, but no providers found in RDEPENDS_libbytesize? [file-rdeps]
ERROR: bluez5-5.55-r0 do_package_qa: QA Issue: /usr/lib/bluez/test/test-sap-server contained in package bluez5-testtools requires /usr/bin/python3, but no providers found in RDEPENDS_bluez5-testtools? [file-rdeps]
ERROR: nfs-utils-2.4.3-r0 do_package_qa: QA Issue: /usr/sbin/nfsiostat contained in package nfs-utils-stats requires /usr/bin/python3, but no providers found in RDEPENDS_nfs-utils-stats? [file-rdeps]
ERROR: usbutils-012-r0 do_package_qa: QA Issue: /usr/bin/lsusb.py contained in package usbutils-python requires /usr/bin/python3, but no providers found in RDEPENDS_usbutils-python? [file-rdeps]

I will let the build to finish and I’ll trigger it again, maybe it will find python3 installed (I have used this “trick” in other situations and it worked)
EDIT: nope it didn’t work

12

These errors occur if the Python 3.9 recipe is the one found in the older meta-custom layers. Did you download the latest meta-custom? It has an altered Python 3.9 recipe compared with the previous layers.

In more detail: before, the 3.9 recipe tried to create symlinks like /usr/bin/python3, but this conflicts with 3.8 as that creates symlinks with the same name. So there were bbappend files that removed these 3.8 files and added python3.9 as RDEPENDS on some packages that need /usr/bin/python3.

The newer 3.9 recipe now doesn’t create any conflicting files, so there is no need for the bbappend files anymore.

I’ve made a minor update to the recipe and now it should work first time on a clean Yocto build.

Try the updated layer and see how this goes for you.

Best regards,
Lucas Akira

13

hi @lucas_a.tx ,
I’m going to try it again with the meta-custom-v3.1.zip
I have a doubt though: if I launch crops with docker run --rm -it --name=crops -v ${PWD}:/workdir --workdir=/workdir -e MACHINE=verdin-imx8mm -e IMAGE=torizon-core-docker torizon/crops:dunfell-5.x.y startup-tdx.sh command, would the script retrieve again meta-custom layer from the repository and restore the previous version?

14

Hi @RoccoBr ,

would the script retrieve again meta-custom layer from the repository and restore the previous version?

I don’t think the script restores the custom layer (or any layer) to a previous version. But it can retain the compilation cache from previous build attempts. Just to be on the safe side I’d recommend a new, different workdir location to start everything from scratch to avoid any cache conflicts from Python or any package that depends on it.

Best regards,
Lucas Akira

15

hi @lucas_a.tx ,
this time I didn’t have any error, I can see in the log recipe for python3.9 being called, but in the final image is reporting the old version of python3:

torizon@verdin-imx8mm-06760613:~$ python3 -V
Python 3.8.13

I can see from the log that recipe for python3.9 has been called. I’ll dig more intro it

16

Hi @RoccoBr ,

Both versions are installed, so running python3/python3.8 calls version 3.8, while python3.9 calls 3.9 instead. I had to do it like this in order to avoid breaking any program in the base image that uses 3.8 e.g. docker-compose.

Best regards,
Lucas Akira

17

thanks @lucas_a.tx ,
I can see python3.9 installed

CB20300084:~$ python3.9 -V
Python 3.9.9
18

Glad I was able to help!

Best regards,
Lucas Akira