Here is some information about the findings to complete the documentation here in the community.
So the Problem here was that the driver did not stop the HDMI clock output when the HPD pin was pulled low but the HDMI was still plugged in. The question might arise why pull the HPD pin low to trigger a Hot Plug Detect event but leave the HDMI connector connected? Well, there are use cases where HDMI output into a DisplayPort (DP) converter and you trigger the HPD signal via the DP connection. So the HDMI lanes are permanently connected with the HDMI to DP converter.
FIX:
We have a patch that removes the clock from the line when the HPD pin is changed. in this case her kernel 5.15 was used. The change was merged into our downstream branch. Its on our public git for them to test: It is fixed in BSP 6.6
https://git.toradex.com/cgit/linux-toradex.git/commit/?h=toradex_5.15-2.2.x-imx&id=e63f274c8b6ecab8d94cfc53aa58744ffac97f8c
Best Regards,
Matthias