Building Torizon Yocto Image offline

[allang1]

We are trying to setup a build server for build the torizon-docker image with our own customizations but are running into some errors most likely due to our company firewall. When we try to build the image we get the following error:

 bitbake torizon-docker
Loading cache: 100% |#####################################################################| Time: 0:00:01
Loaded 2112 entries from dependency cache.
WARNING: /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb: Exception during build_dependencies for fetcher_hashes_dummyfunc
WARNING: /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb: Error during finalise of /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb
ERROR: ExpansionError during parsing /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb
bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export PATH="/home/bamboo/Documents/torizon_test/layers/openembedded-core/scripts:/home/bamboo/Documents/torizon_test/build-torizon/tmp/work/apalis_imx8-tdx-linux/linux-toradex-upstream/mainline/recipe-sysroot-native/usr/bin/aarch64-tdx-linux:/home/bamboo/Documents/torizon_test/build-torizon/tmp/work/apalis_imx8-tdx-linux/linux-toradex-upstream/mainline/recipe-sysroot/usr/bin/crossscripts:/home/bamboo/Documents/torizon_test/build-torizon/tmp/work/apalis_imx8-tdx-linux/linux-toradex-upstream/mainline/recipe-sysroot-native/usr/sbin:/home/bamboo/Documents/torizon_test/build-torizon/tmp/work/apalis_imx8-tdx-linux/linux-toradex-upstream/mainline/recipe-sysroot-native/usr/bin:/home/bamboo/Documents/torizon_test/build-torizon/tmp/work/apalis_imx8-tdx-linux/linux-toradex-upstream/mainline/recipe-sysroot-native/sbin:/home/bamboo/Documents/torizon_test/build-torizon/tmp/work/apalis_imx8-tdx-linux/linux-toradex-upstream/mainline/recipe-sysroot-native/bin:/home/bamboo/Documents/torizon_test/layers/openembedded-core/bitbake/bin:/home/bamboo/Documents/torizon_test/build-torizon/tmp/hosttools"; export HOME="/home/bamboo"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git  failed with exit code 128, output:
fatal: unable to access 'https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/': server certificate verification failed. CAfile: none CRLfile: none

The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

ERROR: Parsing halted due to errors, see error messages above
WARNING: /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-bsp/u-boot/u-boot-toradex_mainline.bb: Exception during build_dependencies for fetcher_hashes_dummyfunc
WARNING: /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-bsp/u-boot/u-boot-toradex_mainline.bb: Error during finalise of /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-bsp/u-boot/u-boot-toradex_mainline.bb

Summary: There were 4 WARNING messages.
Summary: There were 2 ERROR messages, returning a non-zero exit code.

Our network has some issues with the network sanity check as well as fetching recipes that use “http://git.*” as the source

In the past, we have gotten around this by copying over the downloads manually and adding CONNECTIVITY_CHECK_URIS = “” and BB_NO_NETWORK = “1” to the local.conf. However, when we add these options to the torizon build, we get the following error.

$ bitbake torizon-docker
Loading cache: 100% |                                                                    | ETA:  --:--:--
Loaded 0 entries from dependency cache.
WARNING: /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb: Exception during build_dependencies for fetcher_hashes_dummyfunc
WARNING: /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb: Error during finalise of /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb
ERROR: ExpansionError during parsing /home/bamboo/Documents/torizon_test/build-torizon/conf/../../layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-upstream_mainline.bb
bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception NetworkAccess: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY) but access requested with command git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git  (for url https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git)
The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

ERROR: Parsing halted due to errors, see error messages above

Summary: There were 2 WARNING messages.
Summary: There were 2 ERROR messages, returning a non-zero exit code.

Is there a way to setup the torizon yocto build to build offline? Are there a different set of options we can use to get around these network checks?

[ToradexAI]

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

---

Hello,

To build a Torizon Yocto image offline, especially when dealing with network restrictions like a company firewall, you can follow these steps to ensure all necessary resources are available locally:

1. Pre-download Required Resources:

   • Use a machine with internet access to download all necessary source files and dependencies. You can achieve this by running the bitbake command with the -c fetchall option, which will download all sources for your target image. Once downloaded, transfer these files to your offline build server.

2. Configure Local.conf for Offline Use:

   • In your local.conf file, set BB_NO_NETWORK = "1" to prevent any network access during the build process.
   • Set CONNECTIVITY_CHECK_URIS = "" to disable network connectivity checks.
   • Ensure that the DL_DIR variable in local.conf points to the directory where you have stored the pre-downloaded sources.

3. Use a Local Mirror:

   • If feasible, set up a local mirror for the Yocto downloads. This involves configuring the PREMIRRORS and MIRRORS variables in local.conf to point to your local server where the sources are stored.

4. Verify Certificates:

   • The error message you encountered suggests a problem with server certificate verification. Ensure that your build environment has the necessary CA certificates installed. You might need to configure Git to use a specific CA file or, as a last resort, disable SSL verification (though this is not recommended for production environments).

5. Troubleshoot Specific Errors:

   • The error related to fetcher_hashes_dummyfunc indicates a problem with fetching or verifying sources. Double-check that all sources are correctly downloaded and accessible from your offline server.

By following these steps, you should be able to configure your build environment to work offline, thus avoiding the network-related errors you are encountering. If issues persist, ensure that all paths and configurations are correctly set in your local.conf and other relevant configuration files.

I hope this helps you set up your offline build environment successfully.

Best regards,
Toradex AI Assistant.

Please, also check if these links can help you solve your problem:

Problems building Yocto image for iMX7

We are just starting with Yocto / OpenEmbedded. I followed the instructions from Webinar On-Demand: Part 1 Introduction - Building Embedded Linux Images with the Yocto Project - YouTube and High performance, low power Embedded Computing Systems | Toradex Developer Center. I am working inside a VirtualBox VM running Ubuntu 16.04 64 Bit. When trying to build an image, this fails when trying to fetch files required by the RPM recipe. We are not able to fetch required files from www.rpm5.org. rpm5.o…

Several question for torizon-minimal and offline updates

Hello, I have several general questions regarding Yocto build of Torizon and the offline updates: In OSTree | Toradex Developer Center it says “Whenever Torizon OS is built using Yocto/OpenEmbedded’s bitbake, a OSTree repository is automatically initialized and a new OSTree commit is created. For every following build, a new commit is created where only changed files are added to the repository.” What happens to the OSTree repo when the build environment is reset? How is an installation …

Yocto / Torizon + Custom Kernel + Kernel Module + Network Mgmt + Apps [Architecture Help Neded].

[jeremias.tx]

Greetings @allang1,

This is a bit of a tough one. We never had the need to do an “offline” Torizon Yocto build before, so if it is possible the steps aren’t known to us. Is it not possible to modify, or work with your corporate firewall at all?

Even if it is possible to do an “offline” build successfully this is quite a limiting factor for development.

By the way, I also had some related questions, if you don’t mind elaborating:

• What kind of customization are you planning on doing, that require a full Yocto build?
• I noticed from the logs you shared you seem to be building for our Apalis i.MX8, but it’s trying to use the mainline/upstream Linux and U-Boot. Our default i.MX8 build doesn’t use mainline/upstream, did you modify this? And if so, why?

Best Regards,
Jeremias

[allang1]

Also, we were not intentionally trying to use a different version of the kernel. We created the yocto build following the instructions here: Build Torizon OS from Source With Yocto Project/OpenEmbedded | Toradex Developer Center

I don’t believe we would have done anything to change the kernel to track the mainline

[allang1]

Hi Jeremias,

We were able to get things working by installing some missing certificates on our end.

I am, however, still interested if there is a way to build offline. Though acceptable in this application, there are potential application where we would not have continuous network access and will have to bring in any piece of software “by hand”.

The main reason for the full yocto build was that there were some standard linux tools that we would like to add to the image (e.g. tshark, tmux). As I understand, there is no way to add these in via the torizoncore-builder utility but is fairly easy to add into the yocto build.

The other issue I was running to was with adding another user to the system. Using the torizioncore-builder isolate seemed to get most of the way there but it did not capture the new home directory.

Thanks,
Allan

[jeremias.tx]

The main reason for the full yocto build was that there were some standard linux tools that we would like to add to the image (e.g. tshark, tmux). As I understand, there is no way to add these in via the torizoncore-builder utility but is fairly easy to add into the yocto build.

I see, yes that would make sense then to use a Yocto build for.

The other issue I was running to was with adding another user to the system. Using the torizioncore-builder isolate seemed to get most of the way there but it did not capture the new home directory.

One note here. TorizonCore Builder uses OSTree to manage the different filesystem customization. One feature of OSTree is that it does not manage the /var directory at all. This is by design to have a designated location on the filesystem for persistent data that is not affected by OSTree changes.

One consequence of this though is that in OSTree home directories for users are actually symlinked under /var. This means they are not managed by OSTree, which in turn means TorizonCore Builder can’t customize home directories.

Best Regards,
Jeremias