Multi-monitor setup with KVM & laptop dock changes screen order/position

Not sure where to ask so I am starting here in neon since I am using KDE neon, and I can hopefully be pointed in the right direction if this is the wrong place.

I got a 4-screen setup (2x2) connected to a KVM so I can switch between a Windows machine and a Linux laptop (KDE neon). The issue I am experiencing is that when switching between the machines my KDE neon machine seems to re-order the monitors. It keeps the 2x2 layout but the display sources get rearranged randomly and I have to keep rearrange my display configuration all the time.

This does not happen for the windows machine, but the laptop is using a dock and DisplayLink so it could be related to that. Anyone know if this can be fixed in anyway, or if I can report it as a bug anywhere?