Hello @Mowlwurf,
I could not install the bash debug extension that you use (I found it on Marketplace, but it does not show up after the installation).
It looks like a timing issue from what you are describing. It could be that the setup-environment is likely being sourced/executed before the repo sync has actually completed writing the full directory structure.
If you could verify that this is the problem, then I suggest either splitting the setup and build logic into two different bash scripts or adding a wait or file check logic before moving on to setting up the environment.