You can do it with Klassy window decoration. There used to be another way by adjusting the colorscheme, but I simply can’t remember how it was done.
Anywho…Klassy.
I remember now. A small hack I used back in the days was to add two lines to the ~/.config/kdeglobals file, section wm. That way you could color the frame. Dunno if it helps really and dunno what plasma 6 gives.
But, for example, two windows ( standard breeze) without the “hack”.