Adding node-sqlite3 to BSP Reference Minimal Image v6.x.y with nodejs

We have built a custom BSP Reference Minimal Image with docker and nodejs packages, that is working on our custom board with no problems.

But now we need to add some nodejs packages into this image.
I’ve created the workspace recipe using devtool as below:

$ devtool add "npm://registry.npmjs.org;package=sqlite3;version=5.1.7"

But I got the following errors trying to build using devtool build sqlite3:

ERROR: sqlite3-5.1.7-r0 do_configure: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:npm_do_configure(d)
     0003:
File: '/home/smar/toradex/oe-bsp/build/../layers/openembedded-core/meta/classes/npm.bbclass', lineno: 220, function: npm_do_configure
     0216:                cached_manifest[depkey] = {}
     0217:            cached_manifest[depkey][name] = version
     0218:
     0219:    if has_shrinkwrap_file:
 *** 0220:        _update_manifest("dependencies")
     0221:
     0222:    if dev:
     0223:        if has_shrinkwrap_file:
     0224:            _update_manifest("devDependencies")
File: '/home/smar/toradex/oe-bsp/build/../layers/openembedded-core/meta/classes/npm.bbclass', lineno: 214, function: _update_manifest
     0210:
     0211:    # Configure the cached manifest file and cached shrinkwrap file
     0212:    def _update_manifest(depkey):
     0213:        for name in orig_manifest.get(depkey, {}):
 *** 0214:            version = cached_shrinkwrap["dependencies"][name]["version"]
     0215:            if depkey not in cached_manifest:
     0216:                cached_manifest[depkey] = {}
     0217:            cached_manifest[depkey][name] = version
     0218:
Exception: KeyError: 'dependencies'

ERROR: Logfile of failure stored in: /home/smar/toradex/oe-bsp/build/tmp/work/armv7at2hf-neon-tdx-linux-gnueabi/sqlite3/5.1.7-r0/temp/log.do_configure.107126
ERROR: Task (/home/smar/toradex/oe-bsp/build/workspace/recipes/sqlite3/sqlite3_5.1.7.bb:do_configure) failed with exit code '1'

I managed that was missing the npm-shrinkwrap.json in the SRC_URI at the sqlite3_5.1.7.bb file, so I put it there as below:

SRC_URI = " \
    npm://registry.npmjs.org/;package=node-sqlite3;version=${PV} \
    npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
    "

And now I got another kind of errors as below:

ERROR: ExpansionError during parsing /home/smar/toradex/oe-bsp/build/workspace/recipes/sqlite3/sqlite3_5.1.7.bb                                                               | ETA:  --:--:--
Traceback (most recent call last):
  File "/home/smar/toradex/oe-bsp/layers/openembedded-core/bitbake/lib/bb/data_smart.py", line 470, in DataSmart.expandWithRefs(s="${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}", varname='PN'):
                     tb = sys.exc_info()[2]
    >                raise ExpansionError(varname, s, exc).with_traceback(tb) from exc

  File "/home/smar/toradex/oe-bsp/layers/openembedded-core/bitbake/lib/bb/data_smart.py", line 454, in DataSmart.expandWithRefs(s="${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}", varname='PN'):
                     try:
    >                    s = __expand_python_regexp__.sub(varparse.python_sub, s)
                     except SyntaxError as e:
  File "/home/smar/toradex/oe-bsp/layers/openembedded-core/bitbake/lib/bb/data_smart.py", line 131, in VariableParse.python_sub(match=<re.Match object; span=(0, 77), match="${@bb.parse.vars_from_file(d.getVar('FILE', False>):
                 if self.varname:
    >                vardeps = self.d.getVarFlag(self.varname, "vardeps")
                     if vardeps is None:
  File "/home/smar/toradex/oe-bsp/layers/openembedded-core/bitbake/lib/bb/data_smart.py", line 852, in DataSmart.getVarFlag(var='PN', flag='vardeps', expand=True, noweakdefault=False, parsing=False, retparser=False):
             if expand or retparser:
    >            parser = self.expandWithRefs(value, cachename)
             if expand:
  File "/home/smar/toradex/oe-bsp/layers/openembedded-core/bitbake/lib/bb/data_smart.py", line 445, in DataSmart.expandWithRefs(s=None, varname='PN[vardeps]'):
             if not isinstance(s, str): # sanity check
    >            return VariableParse(varname, self, s)

bb.data_smart.ExpansionError: Failure expanding variable PN, expression was ${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'} which triggered exception RecursionError: maximum recursion depth exceeded
The variable dependency chain for the failure is: PN -> BPN -> SRC_URI

ERROR: Parsing halted due to errors, see error messages above

NOTE: We cannot use Torizon TCBuilder right now. This will be studied later.

Best regards,
Vilson Balestero

Software summary
------------------------------------------------------------
Bootloader:               U-Boot
Kernel version:           6.1.95-rt34-6.7.0+git.7d6bbd3e0ef0 #1 SMP Wed Jun 26 1                                                                                        5:24:17 UTC 2024
Kernel command line:      enable_wait_mode=off galcore.contiguousSize=50331648 r                                                                                        oot=PARTUUID=4e5d7f3a-02 ro rootwait fec_mac=00:14:2d:ac:5e:30 consoleblank=0 no                                                                                        _console_suspend=1 console=tty1 console=ttymxc0,115200n8 fbmem=8M
Distro name:              NAME="DCN I/O Gateway"
Distro version:           VERSION_ID=6.7.0-build.0
Distro variant:           -
Hostname:                 dcnio
------------------------------------------------------------

Hardware info
------------------------------------------------------------
HW model:                 Toradex Colibri iMX6DL/S on Colibri Evaluation Board V                                                                                        3
Toradex version:          0017 V1.1B
Serial number:            11296304
Processor arch:           armv7l
------------------------------------------------------------

Hi, I’m the Toradex AI Assistant. I’m in early beta, and I would appreciate your feedback.
The Toradex Support team will follow up with you shortly!

Did you like my answer?

:+1: :-1:


Hello,

To address the issues you’re encountering while adding the sqlite3 package to your custom BSP Reference Minimal Image with Node.js, let’s tackle them step by step.

Issue 1: KeyError: ‘dependencies’

The first error you encountered during the do_configure step seems to be related to the handling of the npm-shrinkwrap.json file. Your approach to include the npm-shrinkwrap.json in the SRC_URI was correct. This error typically indicates that the build system expected to find dependency information within the npm-shrinkwrap.json file but did not. Ensure that the npm-shrinkwrap.json file is correctly formatted and includes all necessary dependency information for sqlite3. Sometimes, regenerating the npm-shrinkwrap.json file can resolve such issues.

Issue 2: ExpansionError during parsing

The second error, an ExpansionError, suggests there’s an issue with variable expansion within the BitBake recipe. This could be due to recursive variable references or incorrect syntax within the recipe file. The error message points towards a problem with the PN variable, which is typically automatically derived from the recipe name but seems to be involved in a recursive dependency or an incorrect override.

To troubleshoot this:

  1. Check for Recursive Dependencies: Ensure there are no recursive dependencies in your recipe that could cause an infinite loop during variable expansion.
  2. Recipe Name and File Name Alignment: Verify that the recipe file name (sqlite3_5.1.7.bb) aligns with the expected package name. BitBake uses the filename to derive the PN variable, and discrepancies can lead to unexpected behavior.
  3. Simplify the SRC_URI: Temporarily simplify the SRC_URI to only include the essential source, removing the npm-shrinkwrap.json reference. This can help isolate whether the issue is with the source URI expansion or elsewhere in the recipe.
  4. Manual Variable Setting: As a diagnostic step, try setting the PN variable explicitly at the top of your recipe to see if this bypasses the expansion error. This is not a final solution but can help diagnose the issue.
PN = "sqlite3"

Given the complexity of BitBake recipes and the potential for subtle issues in variable handling, these steps are starting points for troubleshooting. Since you mentioned that using TorizonCore Builder is not an option at this moment, focusing on resolving the BitBake recipe issues is

Please, also check if these links can help you solve your problem:

Workspace - Settings | Toradex Developer Center.

Hello, Vilson.
I apologize for the delay.

I will reproduce your error and when I have more information, I will let you know.

Best regards.
Lucas.

Hello, Lucas.
Thanks for your reply.

After much research on how to compile node-sqlite3 using yocto, I had to use node-sqlite3 from meta-luneos, but I had no luck there either.

So today I went back to devtool and, removing the last workspace created, tried devtool again, and now it works.

Now I can compile node-sqlite3 using devtool build, but I don’t know why it’s working now.

But now I want to add the node-sqlite3 recipe generated by devtool to my final image to be compiled.

I know that using devtool finish recipe <layer> does the job for me.

I tried it, but it just creates a new recipe in my layer and copies the node-sqlite3 folder to it. But inside the workspace more folders were generated like – sources and appends – and those are not copied.

smar@yocto:~/toradex/oe-bsp/build/workspace$ tree -L 3
.
├── appends
│   └── node-sqlite3_5.1.7.bbappend
├── conf
│   └── layer.conf
├── README
├── recipes
│   └── node-sqlite3
│       ├── node-sqlite3
│       └── node-sqlite3_5.1.7.bb
└── sources
    └── node-sqlite3
        ├── binding.gyp
        ├── deps
        ├── lib
        ├── LICENSE
        ├── node_modules
        ├── npm-shrinkwrap.json
        ├── oe-logs -> /home/smar/toradex/oe-bsp/build/tmp/work/armv7at2hf-neon-tdx-linux-gnueabi/node-sqlite3/5.1.7-r0/temp
        ├── oe-workdir -> /home/smar/toradex/oe-bsp/build/tmp/work/armv7at2hf-neon-tdx-linux-gnueabi/node-sqlite3/5.1.7-r0
        ├── package.json
        ├── README.md
        └── src

So when I tried to build my image I got the error:

ERROR: Nothing RPROVIDES 'node-sqlite3' (but /home/smar/toradex/oe-bsp/build/../layers/meta-smar-dcnio/recipes-images/images/dcnio-gateway-image.bb RDEPENDS on or otherwise requires it)
NOTE: Runtime target 'node-sqlite3' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['node-sqlite3']
ERROR: Required build target 'dcnio-gateway-image' has no buildable providers.
Missing or unbuildable dependency chain was: ['dcnio-gateway-image', 'node-sqlite3']

Do you have any idea about doing this?

Best regards.
Vilson Balestero

Hi, Vilson.
This error usually means that the Yocto could not find the image called node-sqlite3.
To make sure that the framework is searching in the right directories, first, you need to add your custom meta-layer in oe-core/build/conf/bblayers.conf. To do this, just add your meta-layer path in the BBLAYERS ?= node, like this:

BBLAYERS ?= " \
  ${BBLAYERS_NXP} \
  ${BBLAYERS_TI} \
  ${TOPDIR}/../layers/meta-toradex-bsp-common \
...
  ${TOPDIR}/../layers/meta-layer \
"

Besides that, it is also important to check the BBFILES += node in layer.conf file inside your meta-layer/conf. This node defines the location for all recipes in the layer, so your .bb file should be inside one of these directories, or you can add your .bb path in this node.

Please, check these files, try to build again, and give me feedback.

Best regards.
Lucas Azeituno

Hi, Lucas.

I didn’t quite understand your answer, because my image is already in the bblayers.conf, and the node-sqlite3 is a package inside the recipes-tools in my layer.

My image is already buidable without node-sqlite3, and by adding the node-sqlite3 in the IMAGE_INSTALL ??= the build fail with that error.

Here is my layer.conf file:

DISTRO_VERSION = "${DCNIO_VERSION}"

# We have a conf and classes directory, add to BBPATH
BBPATH .= ":${LAYERDIR}"

# We have recipes-* directories, add to BBFILES
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
            ${LAYERDIR}/recipes-*/*/*.bbappend"

BBFILE_COLLECTIONS += "smar-dcnio"
BBFILE_PATTERN_smar-dcnio = "^${LAYERDIR}/"
# default priority is 6, 99 give the high priority for our custom meta.
BBFILE_PRIORITY_smar-dcnio = "99"

LAYERDEPENDS_smar-dcnio = " \
    core \
    openembedded-layer \
    networking-layer \
    filesystems-layer \
    "
LAYERSERIES_COMPAT_smar-dcnio = "kirkstone"

Best regards.
Vilson Balestero

Hi, Vilson.

Could you send me how is the structure of your meta-layer?
You can use the tree command in your meta-layer folder and send me the output.

Best regards.
Lucas Azeituno.

Hi, Lucas.

Here our layer.

meta-smar-dcnio
├── conf
│   ├── classes
│   │   └── github-releases.bbclass
│   ├── layer.conf
│   └── machine
│       ├── dcnio.conf
│       └── include
│           └── colibri-imx6.inc
├── README
├── recipes-connectivity
│   ├── smar-rfserver
│   │   ├── files
│   │   └── smar-rfserver_1.0.1.bb
│   └── smar-webapi
│       ├── files
│       │   ├── COPYING
│       │   ├── node_modules.tar.xz
│       │   ├── smar-webapi
│       │   ├── smar-webapi-1.0.7.tar.xz
│       │   ├── smar-webapi.service
│       │   └── start-webapi.sh
│       ├── smar-webapi_1.0.7.bb
│       └── smar-webapi_1.0.7.bkp
├── recipes-containers
│   ├── containerd
│   │   └── containerd-opencontainers_%.bbappend
│   ├── container-host-config
│   │   ├── container-host-config
│   │   │   └── storage.conf
│   │   └── container-host-config.bbappend
│   ├── crun
│   │   └── crun_git.bb
│   ├── docker
│   │   ├── docker-ce
│   │   │   ├── 0001-dockerd-daemon-use-default-system-config-when-none-i.patch
│   │   │   └── 0002-cli-config-support-default-system-config.patch
│   │   ├── docker-ce_git.bbappend
│   │   ├── docker-torizon.inc
│   │   └── files
│   │       ├── chrome.json
│   │       ├── daemon.json
│   │       └── docker.service
│   ├── docker-compose
│   │   ├── docker-compose_%.bbappend
│   │   ├── docker-compose_git.bb
│   │   ├── files
│   │   │   └── modules.txt
│   │   ├── relocation.inc
│   │   └── src_uri.inc
│   ├── docker-compose-up
│   │   ├── docker-compose-up_1.0.bb
│   │   └── files
│   │       └── docker-compose.service
│   ├── docker-integrity-checker
│   │   ├── docker-integrity-checker_1.0.bb
│   │   └── files
│   │       ├── docker-integrity-checker.service
│   │       └── docker-integrity-checker.sh
│   ├── docker-watchdog
│   │   ├── docker-watchdog_1.0.bb
│   │   └── files
│   │       ├── docker-watchdog.service
│   │       └── docker-watchdog.sh
│   ├── fuse-overlayfs
│   │   └── fuse-overlayfs_git.bb
│   ├── podman
│   │   ├── files
│   │   │   └── 0001-cmd-support-config-option-to-locate-authentication-f.patch
│   │   ├── podman
│   │   │   ├── 50-podman-rootless.conf
│   │   │   └── run-ptest
│   │   ├── podman_%.bbappend
│   │   └── podman_git.bb
│   └── skopeo
│       ├── files
│       │   └── 0001-docker-config-support-default-system-config.patch
│       └── skopeo_git.bbappend
├── recipes-core
│   ├── base-files
│   │   ├── base-files
│   │   │   ├── issue
│   │   │   ├── issue.net
│   │   │   ├── release
│   │   │   └── share
│   │   └── base-files_%.bbappend
│   ├── os-release
│   │   └── os-release.bbappend
│   └── systemd
│       ├── systemd
│       │   ├── 0001-rules-whitelist-hd-devices.patch
│       │   ├── 0001-tmpfiles-tmp.conf-reduce-cleanup-age-to-half.patch
│       │   ├── 0002-systemd-networkd-wait-online.service.in-use-any-by-d.patch
│       │   ├── eth0.network
│       │   ├── eth1.network
│       │   ├── static-libsystemd-pkgconfig.patch
│       │   └── systemd-timesyncd-update.service
│       ├── systemd_%.bbappend
│       ├── systemd-conf
│       │   ├── system.conf-docker
│       │   └── system.conf-torizon
│       └── systemd-conf_%.bbappend
├── recipes-devtools
│   ├── nodejs
│   │   ├── node-gyp-native_git.bb
│   │   ├── node-gyp-packages-0.12.2
│   │   │   └── registry-0.12.2.tar.gz
│   │   ├── node-gyp-packages-native_0.12.2.bb
│   │   └── node-gyp-packages-native.inc
│   ├── node-sqlite3
│   │   ├── node-sqlite3
│   │   │   └── npm-shrinkwrap.json
│   │   └── node-sqlite3_5.1.7.bb
│   └── python
├── recipes-extended
│   ├── root-snmp
│   │   ├── files
│   │   │   ├── DF63-MIB.txt
│   │   │   ├── README_snmp
│   │   │   ├── snmp.conf
│   │   │   └── snmpd.conf
│   │   └── root-snmp_1.0.0.bb
│   └── smar-monitor
│       ├── files
│       │   ├── COPYING
│       │   ├── DF63-MIB.txt
│       │   ├── monitor-install-target-package
│       │   ├── monitor-package-2.0.0.3.tar.xz
│       │   ├── README_snmp
│       │   ├── snmp.conf
│       │   └── snmpd.conf
│       └── smar-monitor_2.0.0.3.bb
├── recipes-images
│   └── images
│       └── dcnio-gateway-image.bb
├── recipes-kernel
│   └── linux
│       ├── files
│       │   ├── dcnio-features.cfg
│       │   ├── dcnio-features.scc
│       │   ├── kernel-config-bsp-9
│       │   └── kernel-config-BSP-dcnio
│       ├── linux-toradex_%.bbappend
│       ├── linux-toradex-mainline_%.bbappend
│       └── linux-yocto_%.bbappend
├── recipes-protocols
│   └── net-snmp
│       ├── files
│       │   └── snmpd.conf
│       └── net-snmp_%.bbappend
└── recipes-support
    ├── dbus
    │   └── dbus_%.bbappend
    ├── docker-auto-prune
    │   ├── docker-auto-prune
    │   │   ├── docker-auto-prune.service
    │   │   └── docker-auto-prune.timer.in
    │   └── docker-auto-prune_0.1.bb
    ├── icu
    │   ├── icu
    │   │   ├── 0001-Disable-LDFLAGSICUDT-for-Linux.patch
    │   │   ├── 0001-icu-Added-armeb-support.patch
    │   │   ├── filter.json
    │   │   └── fix-install-manx.patch
    │   └── icu_72-1.bb
    └── libevent
        └── libevent_%.bbappend

Best regards.
Vilson Balestero

Hi, Lucas.

I changed my recipe node-sqlite3 by merging the contents from ../build/workspace/appends/node-sqlite3_5.1.7.bbappend

Then now when building my custom image, the node-sqlite3 is fetched, configured, builded and installed with my image.

So, thanks for your help.

Best regards.
Vilson Balestero

Hi, Vilson.

I’m glad to know that you were able to solve this issue.

Feel free to contact us if you need any other help.

Best regards.
Lucas Azeituno.