Thanks, I will thoroughly test if that helps.
Figured it out. The problem is a memory leak (not allowed to link for whatever reason) in the Linux kernel. Apparently should be fixed in 6.12. In the meantime, I have downgraded the kernel (still had 6.10.3) which solved the issue.