It’s really recommended to build everything with Yocto. While you can build the kernel stand-alone for Torizon it’s very error prone to replace it properly in an already deployed image. Furthermore building the kernel stand-alone for Torizon is also not straightforward. Since the kernel config for Torizon is not in the kernel source, so you’ll need to extract it from the device to your build machine.
Again I really recommend to use Yocto if possible. That being said what kind of error did you encounter with Yocto? go_systemd is a generic package and is not something we control.