Just wanted to confirm that my solution was due to messing with display manager in one way or the other. Ultimately though I feel the most likely fixing of the solution comes from changing adaptive sync from always to off then automatic.
To add more detail I first changed my 2nd screens refresh rate to something lower which definitely seemed to decrease the occurrences but just as my hopes were high it happened and then happened a lot. After that I messed with adaptive sync which ultimately seemed to fix everything.
I have had a total of one screen freeze since I posted the issue and I believe that was whilst an application activated gsync and I was alt tabbing like sm1 taking LSD.
So if anyone has this problem just start by disabling gsync I guess and see if that solves it. If so you at least know what to look at other than having to go to an older driver if that is not an option for you.