Hello,
We are starting to move many of our products to Toradex platforms. Since we use Gitlab for our development, I figured we could try using torizoncore builder to implement a build pipeline to keep our custom Torizoncore images in one place as GL job artifacts. I still don’t have a lot of experience writing CI jobs so I searched to see if anyone had the same problems, but no luck. Has anyone tried to do something similar?
Hi @spasoye !
This is the first time I see a question about TorizonCore Builder and CI.
Great question, by the way 
Could you give us more context about what you are trying and how would be the CI workflow for you?
I mean:
- Would you use GitLab CI with TorizonCore Builder to generate Torizon Platform packages and then push via OTA to the modules?
- Or the GitLab CI would use TorizonCore Builder to generate Toradex Easy Installer images that you would flash using Toradex Easy Installer?
In the meantime, I will ask internally about your question and share with you more information as soon as possible.
Best regards,
Henrique
I have a suggestion that may help somewhat.
Internally we use Gitlab CI pipelines as well for testing the various functions of TorizonCore Builder. This includes setting up TorizonCore Builder and performing various commands as one would in actual use.
The CI file we use is mirrored to our public Github repo here: torizoncore-builder/.gitlab-ci.yml at bullseye · toradex/torizoncore-builder · GitHub
In particular what’s most relevant to you would probably be the test-integration stage. This is where we do the test automation and actually use TorizonCore Builder in the pipeline.
It’s probably not a one to one with what you’re seeking to do. But perhaps it can be a good reference or inspiration for you to start off with.
Best Regards,
Jeremias
Hi @henrique.tx,
We are still only starting using Toradex in few of our designs and major thinks introducing something like Torizon Platform is overkill at this moment in our development. Currently I build Easy Installer images locally and uploads them on OneDrive cloud. This approach is making my code all over the place and difficult to navigate through it so I’m trying to keep my sources and build in the same place.
Hi @spasoye !
Thanks for the clarification.
You can take a look at @jeremias.tx’s answer to have an example to serve as an inspiration.
Also, there is a brief example about TorizonCore pipeline in this webinar: Beyond Development: Torizon
Also, @drew.tx has a kind of related script that he created for a demo, which might be helpful for you to have some ideas on how you can do your own setup: torizon-nfs-poc/tcb.sh at main · drewmoseley/torizon-nfs-poc · GitHub
Best regards,
Henrique
Hey @henrique.tx,
I was able to setup build job and create my first successful build few hours ago. This is .gitlab-ci.yaml file I used to create simple one job pipeline:
image: torizon/torizoncore-builder:3
variables:
# This should be set by docker image already, just to be sure...
DOCKER_HOST: tcp://docker:2375
# Use overlayfs driver for better performance
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
GIT_STRATEGY: clone
GIT_SUBMODULE_STRATEGY: none
IMAGE_NAME: $CI_COMMIT_TIMESTAMP
# Push application to the Torizon Platform Services
build-job:
tags:
- amd64
- docker
- gideon
- bigcpu
image:
name: torizon/torizoncore-builder:3
entrypoint: [ "" ]
# Scripts
before_script:
- apt update
- apt install -y openssh-client
# Prepare ssh directory for git
- mkdir -p $HOME/.ssh && touch $HOME/.ssh/config
- chmod 700 $HOME/.ssh && chmod 600 $HOME/.ssh/config
- printf '%s\n' 'Host *' 'StrictHostKeyChecking no' 'UserKnownHostsFile=/dev/null' >> $HOME/.ssh/config
# Start ssh-agent and load ssh keys for GitLab and Bitbucket from GitLab CI variables
- 'eval $(ssh-agent -s)'
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
# Test ssh connections
- ssh -T git@bitbucket.org
- ssh -T git@gitlab.com
# Login to GitLab container registry
#- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- git submodule update --jobs=8 --init --recursive
- torizoncore-builder --version
- torizoncore-builder -h
- ls -al
- mkdir /deploy
- torizoncore-builder build
--set IMAGE_VERSION="test_image" --file
cicd_tcbuild.yaml
- ls -al images
I still have to manage artifacts (in my case EasyInstaller image saved in image/test_image). Im not sure is it safe to pass docker folder path to artifacts: section. I was planning to upload it to Gitlabs package registry, but I’m opened to suggestions if there is smarter way to keep track of my builds.
Hi @spasoye,
Please forgive my ignorance here, I’m a noob when it comes to Gitlab CI. Can you clarify a bit what your concerns are? I don’t understand what it means to “pass a docker folder path to artifacts: section”.
Drew
@drew.tx , Gitlab CI pipeline consists of different jobs with their own environments.
In example above I used docker container in build job that is responsible for building torizoncore easy installer image that is stored in dockers volume, but now I don’t know how to pass that image to next job in the pipeline (deploy job for example). This is my workaround to do what I wanted to do in a first place:
image: torizon/torizoncore-builder:3
variables:
GIT_STRATEGY: clone
GIT_SUBMODULE_STRATEGY: none
IMAGE_NAME: $CI_COMMIT_TIMESTAMP
default:
tags:
- amd64
- docker
- gideon
- bigbw
stages:
- build
# include:
# - template: Security/Secret-Detection.gitlab-ci.yml
# - template: Security/SAST.gitlab-ci.yml
# Build and upload flashable image to package registry
build easy installer image:
variables:
BUILD_NUMBER: 0.0.1
stage: build
# TODO: check naming
rules:
- if: '$CI_COMMIT_TAG'
when: on_success
image:
name: torizon/torizoncore-builder:3
entrypoint: [""]
before_script:
- apt update
- apt install -y openssh-client zip
# Prepare ssh directory for git
- mkdir -p $HOME/.ssh && touch $HOME/.ssh/config
- chmod 700 $HOME/.ssh && chmod 600 $HOME/.ssh/config
- printf '%s\n' 'Host *' 'StrictHostKeyChecking no' 'UserKnownHostsFile=/dev/null' >> $HOME/.ssh/config
# Start ssh-agent and load ssh keys for GitLab and Bitbucket from GitLab CI variables
- 'eval $(ssh-agent -s)'
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
# Test ssh connections
- ssh -T git@bitbucket.org
- ssh -T git@gitlab.com
# Login to GitLab container registry
#- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- git submodule update --jobs=8 --init --recursive
- torizoncore-builder --version
- torizoncore-builder -h
- ls -al
- mkdir /deploy
# build easy installer image.
- torizoncore-builder build
--set IMAGE_VERSION="${CI_COMMIT_TAG}"
--set IMAGE_COMMIT_MESSAGE="${CI_COMMIT_MESSAGE}"
--file cicd_tcbuild.yaml
- zip -r ${CI_COMMIT_TAG}.zip ${CI_COMMIT_TAG}
# deploy image to package registry
# TODO: need to run build only on special tag.
- echo "link--->"${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${CI_PROJECT_TITLE}/${BUILD_NUMBER}/liftbot_HMI_test.zip?select=package_file""
- >
curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file "liftbot_HMI_test.zip" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${CI_PROJECT_TITLE}/${BUILD_NUMBER}/liftbot_HMI_test.zip?select=package_file"
This is tcbuild.yaml:
input:
easy-installer:
remote: https://artifacts.toradex.com:443/artifactory/torizoncore-oe-prod-frankfurt/dunfell-5.x.y/release/7/verdin-imx8mm/torizon-rt/torizon-core-docker/oedeploy/torizon-core-docker-rt-verdin-imx8mm-Tezi_5.3.0%2Bbuild.7.tar
customization:
# >> Splash screen:
splash-screen: custom/GIDEON_Logo_Core_rotated.png
# >> Directories overlayed to the base OSTree
filesystem:
- custom/
device-tree:
# >> Directories where to look for include files.
include-dirs:
- aunit-device-trees/include/
- aunit-device-trees/dts-arm64/
overlays:
remove:
- verdin-imx8mm_lt8912_overlay.dtbo
# >> Overlays to add to output image.
add:
# - aunit-device-trees/overlays/aunit_rtu_linux_overlay.dts
- aunit-device-trees/overlays/verdin-imx8mm_sn65dsi84_overlay.dts
- aunit-device-trees/overlays/verdin-imx8mm_sn65dsi84-lt170410_overlay.dts
- aunit-device-trees/overlays/ili2132_overlay.dts
- aunit-device-trees/overlays/max98357_overlay.dts
kernel:
# >> Custom kernel arguments.
# arguments:
# - key1=val1
# - key2=val2
# >> Modules to build and possibly load automatically.
modules:
# works for versions >= 5.2
- source-dir: ilitek_ts_i2c/
autoload: yes
- source-dir: max98357a/
autoload: yes
# >> The output section defines properties of the output image.
output:
# >> OSTree deployment configuration (relevant also for Easy Installer output).
ostree:
branch: liftbot_hmi
commit-subject: "${IMAGE_VERSION}"
commit-body: "${IMAGE_COMMIT_MESSAGE}"
# >> Parameters for deploying to an Easy Installer image.
easy-installer:
# >> Output directory of the customized image (REQUIRED):
local: ${IMAGE_VERSION}
# >> Information shown by Toradex Easy Installer:
name: "${IMAGE_VERSION}"
description: "${IMAGE_COMMIT_MESSAGE}"
I believe what you have is correct, if I understand properly. Running torizoncore-builder with the yaml file you have will generate the TEZI image in the local path with the name ${IMAGE_VERSION}. You should be able to simply use that path in the subsequent stages.
Drew
Hi @spasoye !
Do you have any update regarding this topic?
Did you manage to solve it?
Best regards,