SIGILL crash on Torizon (linux 5.4), worked with (pre-Torizon 4.9)

I need to run applications packaged as Ubuntu snap files on a Toradex Colibri IMX6 based board (“mainboard” is custom).

For that, I need the snapd system service (for managing/running) and assorted paraphernalia (like kernel support for squashfs, some scripts/binaries in the rootfs, etc.)

I’m using the torizon yocto setup (linux kernel 5.4) with only some minor modifications;
The snapd package is added as a separate yocto layer (GitHub - morphis/meta-snappy: Snappy for Yocto/OpenEmbedded).

Everything builds, boots without errors and the snapd service is up and running, but as soon as I install snap files the service crashes with a SIGILL errors. From what I can tell this happens when Go code (snapd is written in C + Go) tries to call a C function (getpwnam_r).

On an older (linux 4.9) pre-torizon build everything works.

  • Any advice on how to fix this, or where to even start looking?
  • Was there some change in the toolchain change that could be affecting Go inbetween the working (kernel 4.9) and nonworking (5.4) versions?
  • Are there any steps that I could take to help narrow the problem down?

Full error message below.

Versions:

colibri-imx6-10614561:~$ sudo snap version
snap     2.38
snapd    2.38
series   16
torizon  0.0.0-devel-20200428160200-build.0
kernel   5.4.28-0.0.0-devel+git.9ef9ba76540e

Full error message:

$ sudo snap install hello-world
SIGILL: illegal instruction
PC=0x5a49c m=7 sigcode=1

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigFetchG called from 0xa643261b
stack: frame={sp:0xa3dfcb94, fp:0xa3dfcba4} stack=[0xa35fd0c4,0xa3dfccc4)
a3dfcb14:  a5a5d73d  a34005f8  a5afc1bc  000003fe
a3dfcb24:  a5a5f323  a5a5f305  00000000  a34005f8
a3dfcb34:  a5a5690b  a5afc1bc  a644801e  00000000
a3dfcb44:  00000001  00000000  a4f00780  a3dfcbb8
a3dfcb54:  0180c0a0  a4f00b7f  a3dfcbac  006f599c
a3dfcb64:  a5a56999  00000001  00000000  a3dfcbb8
a3dfcb74:  a34005f8  fbada488  a5a5d0f1  00000001
a3dfcb84:  00000400  00000001  00000400  0181c0a0
a3dfcb94: <a643261b  00000001  00000000  00000000
a3dfcba4: >a4f00780  00000000  00000000  0004c960 <runtime.schedule+712>
a3dfcbb4:  006d01fc  00010000  00000001  a34005f8
a3dfcbc4:  00000000  00000000  00000000  00000000
a3dfcbd4:  00000000  00000000  00000000  00000000
a3dfcbe4:  00000000  00000000  00000000  00000000
a3dfcbf4:  00000000  00000000  00000000  00000000
a3dfcc04:  00000000  00000000  00000000  00000000
a3dfcc14:  00000000  00000000  a3dfd7b0  a643255d
runtime.sigFetchG(0x0, 0x0)
        /usr/lib/go/src/runtime/signal_unix.go:397 +0xd0

goroutine 1 [syscall]:
runtime.cgocall(0x4cb6e0, 0x190f70c, 0x1810020)
        /usr/lib/go/src/runtime/cgocall.go:133 +0x5c fp=0x190f6f4 sp=0x190f6dc pc=0x12e7c
os/user._Cfunc_mygetpwnam_r(0x181c0a0, 0x180c0a0, 0xa4f00780, 0x400, 0x1810020, 0x0)
        _cgo_gotypes.go:157 +0x38 fp=0x190f708 sp=0x190f6f4 pc=0x1aef04
os/user.lookupUser.func1.1(0x181c0a0, 0x8, 0x8, 0x180c0a0, 0x1810028, 0x1810020, 0x1afb10)
        /usr/lib/go/src/os/user/cgo_lookup_unix.go:70 +0xbc fp=0x190f72c sp=0x190f708 pc=0x1b0488
os/user.lookupUser.func1(0x598af8)
        /usr/lib/go/src/os/user/cgo_lookup_unix.go:70 +0x44 fp=0x190f74c sp=0x190f72c pc=0x1b050c
os/user.retryWithBuffer(0x1810028, 0x190f7c0, 0x7, 0x181c0a0)
        /usr/lib/go/src/os/user/cgo_lookup_unix.go:247 +0x2c fp=0x190f774 sp=0x190f74c pc=0x1afc24
os/user.lookupUser(0x181a04a, 0x7, 0x0, 0x0, 0x0)
        /usr/lib/go/src/os/user/cgo_lookup_unix.go:61 +0x130 fp=0x190f7e4 sp=0x190f774 pc=0x1af3c0
os/user.Lookup(0x181a04a, 0x7, 0x181a04a, 0x7, 0x0)
        /usr/lib/go/src/os/user/lookup.go:36 +0x58 fp=0x190f808 sp=0x190f7e4 pc=0x1aebe0
github.com/snapcore/snapd/osutil.RealUser(0x5fe49a, 0x18, 0x0)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/osutil/user.go:162 +0x9c fp=0x190f824 sp=0x190f808 pc=0x1b8f50
github.com/snapcore/snapd/client.storeAuthDataFilename(0x0, 0x0, 0x1866000, 0x181a067)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/client/login.go:97 +0xac fp=0x190f854 sp=0x190f824 pc=0x3d2e70
github.com/snapcore/snapd/client.readAuthData(0x0, 0x0, 0x0)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/client/login.go:135 +0x4c fp=0x190f888 sp=0x190f854 pc=0x3d3144
github.com/snapcore/snapd/client.(*Client).setAuthorization(0x18991f0, 0x1866000, 0x190f964, 0x3)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/client/client.go:146 +0x14 fp=0x190f8ec sp=0x190f888 pc=0x3ce60c
github.com/snapcore/snapd/client.(*Client).raw(0x18991f0, 0x5f00a2, 0x3, 0x181c080, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/client/client.go:204 +0x39c fp=0x190f9d4 sp=0x190f8ec pc=0x3cee84
github.com/snapcore/snapd/client.(*Client).do(0x18991f0, 0x5f00a2, 0x3, 0x181c080, 0xe, 0x0, 0x0, 0x0, 0x0, 0x57c108, ...)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/client/client.go:262 +0xe4 fp=0x190fa68 sp=0x190f9d4 pc=0x3cf0f8
github.com/snapcore/snapd/client.(*Client).doSync(0x18991f0, 0x5f00a2, 0x3, 0x181c080, 0xe, 0x0, 0x0, 0x0, 0x0, 0x57c078, ...)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/client/client.go:306 +0x7c fp=0x190fab8 sp=0x190fa68 pc=0x3cf67c
github.com/snapcore/snapd/client.(*Client).Change(0x18991f0, 0x1947e58, 0x2, 0x18991f0, 0x19b0000, 0x1947e58)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/client/change.go:84 +0xac fp=0x190faf8 sp=0x190fab8 pc=0x3cd888
main.waitMixin.wait(0x18991f0, 0x5f0000, 0x1947e58, 0x2, 0x0, 0x0, 0x0)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/cmd/snap/wait.go:88 +0x28c fp=0x190fc8c sp=0x190faf8 pc=0x4be448
main.(*cmdInstall).installOne(0x18c9590, 0xbebc4ed5, 0xb, 0x0, 0x0, 0x1994720, 0x582850, 0x5d2090)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/cmd/snap/cmd_snap_op.go:445 +0x254 fp=0x190fcf8 sp=0x190fc8c pc=0x4a7160
main.(*cmdInstall).Execute(0x18c9590, 0x18a1640, 0x0, 0x2, 0x18c9590, 0x1)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/cmd/snap/cmd_snap_op.go:548 +0x394 fp=0x190fd44 sp=0x190fcf8 pc=0x4a7ff0
github.com/snapcore/snapd/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs(0x197a680, 0x1884108, 0x2, 0x3, 0xe8d18, 0x147, 0xa89758, 0xb6eee3ec, 0x0)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/vendor/github.com/jessevdk/go-flags/parser.go:333 +0x63c fp=0x190fe1c sp=0x190fd44 pc=0x114e28
github.com/snapcore/snapd/vendor/github.com/jessevdk/go-flags.(*Parser).Parse(...)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/vendor/github.com/jessevdk/go-flags/parser.go:190
main.run(0x1947040, 0xe)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/cmd/snap/main.go:481 +0x7c fp=0x190fed0 sp=0x190fe1c pc=0x4bce90
main.main()
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/cmd/snap/main.go:448 +0x2b0 fp=0x190ffb8 sp=0x190fed0 pc=0x4bc978
runtime.main()
        /usr/lib/go/src/runtime/proc.go:203 +0x208 fp=0x190ffe4 sp=0x190ffb8 pc=0x46950
runtime.goexit()
        /usr/lib/go/src/runtime/asm_arm.s:857 +0x4 fp=0x190ffe4 sp=0x190ffe4 pc=0x737f0

goroutine 24 [syscall]:
os/signal.signal_recv(0x0)
        /usr/lib/go/src/runtime/sigqueue.go:147 +0x130
os/signal.loop()
        /usr/lib/go/src/os/signal/signal_unix.go:23 +0x14
created by os/signal.Notify.func1
        /usr/lib/go/src/os/signal/signal.go:127 +0x34

goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0xa644df30, 0x72, 0xffffffff)
        /usr/lib/go/src/runtime/netpoll.go:203 +0x44
internal/poll.(*pollDesc).wait(0x19ba1f4, 0x72, 0x1000, 0x1000, 0xffffffff)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0x19ba1e0, 0x1973000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/go/src/internal/poll/fd_unix.go:169 +0x174
net.(*netFD).Read(0x19ba1e0, 0x1973000, 0x1000, 0x1000, 0x182a574, 0x70640, 0x18000e0)
        /usr/lib/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x199ac78, 0x1973000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/net.go:184 +0x64
net/http.(*persistConn).Read(0x1938140, 0x1973000, 0x1000, 0x1000, 0x18816c0, 0x393aec, 0x18816c0)
        /usr/lib/go/src/net/http/transport.go:1825 +0x168
bufio.(*Reader).fill(0x1994810)
        /usr/lib/go/src/bufio/bufio.go:100 +0x108
bufio.(*Reader).Peek(0x1994810, 0x1, 0x0, 0x0, 0x1, 0x1881600, 0x0)
        /usr/lib/go/src/bufio/bufio.go:138 +0x38
net/http.(*persistConn).readLoop(0x1938140)
        /usr/lib/go/src/net/http/transport.go:1978 +0x15c
created by net/http.(*Transport).dialConn
        /usr/lib/go/src/net/http/transport.go:1647 +0x9b8

goroutine 23 [select]:
net/http.(*persistConn).writeLoop(0x1938140)
        /usr/lib/go/src/net/http/transport.go:2277 +0xc4
created by net/http.(*Transport).dialConn
        /usr/lib/go/src/net/http/transport.go:1648 +0x9d4

goroutine 3 [runnable]:
main.waitMixin.wait.func1(0x19bc300, 0x18991f0, 0x19b0000, 0x1947e58, 0x2)
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/cmd/snap/wait.go:61
created by main.waitMixin.wait
        /workdir/torizon/build-torizon/tmp-torizon/work/armv7at2hf-neon-tdx-linux-gnueabi/snapd/2.38-r0/build/src/github.com/snapcore/snapd/cmd/snap/wait.go:61 +0xf0

trap    0x6
error   0x0
oldmask 0x0
r0      0x5aa50
r1      0x0
r2      0x24000000
r3      0x72
r4      0x1
r5      0x400
r6      0x181c0a0
r7      0xa3dfcbb8
r8      0x180c0a0
r9      0xa4f00b7f
r10     0x1800b60
fp      0xa4f00780
ip      0xa4f00780
sp      0xa3dfcb94
lr      0xa6432624
pc      0x5a49c
cpsr    0xa0070030
fault   0xe4

Greetings @wpupp,

There’s not a lot of direction I can give here as there could be a multitude of things that are wrong. First of all there’s quite a bit of difference between the 4.9 and 5.4 kernel that could affect snaps. Also snaps themselves make a lot of assumptions on the kernel they’re running on so there may be further issues even if this one gets resolved.

Also the meta-snappy layer you’re using only has support for the thud release of yocto, while Torizon is based on the zeus release. So there may be fixes that are needed for the zeus release.

If I may ask, is distributing all these various packages via snaps your only option? Could these not be modified for a container approach or just built into the image by default?

Best Regards,
Jeremias