On KDE Plasma 6.4 now there is an option in Legacy X11 App Support in System Settings:
Control of pointer and keyboard:
Allow without asking for permission
Turning this on would tame this annoying popup.
2 Likes