Troubleshooting KRDP: A Guide to Connecting from Windows Native RDP Clients

General
1

Hello everyone,

As a recent convert to KDE (thanks to the Steam Deck!) and a CachyOS user, I’ve been transitioning my workflow toward a Wayland-native environment. One of the biggest hurdles I encountered was establishing a reliable RDP connection from a Windows 11 client to a CachyOS-based KRDP server.

After significant troubleshooting, I’ve managed to get the native Windows Remote Desktop Connection (mstsc.exe) client working perfectly. I wanted to share these findings to help others bypass the “protocol mismatch” errors and negotiation failures that currently lack deep documentation.

The Setup

  • Server: Beelink Mini PC (Ryzen 5 5800H) running CachyOS (KDE Plasma 6.5.5 / Wayland).

  • Clients: Windows 11 Pro (RX 7800XT), Android, and a CachyOS Laptop.

Summary of Client Compatibility

  • CachyOS (Linux): KRDC and Remmina work out of the box with no issues.

  • Android: The native Microsoft client currently fails due to H.264 decoding limitations. I recommend using aRDP as a stable alternative.

  • Windows (Native): Requires specific configuration changes to handle the H.264 stream correctly.

The Fix: Enabling Windows Native RDP Support

If you are receiving protocol errors or connection refusals from Windows, follow these three steps:

1. Disable Hardware Accelerated Decoding It appears that certain Windows configurations (in my case, using an RX 7800XT) struggle to negotiate the H.264 stream from KRDP via hardware.

  • Open gpedit.msc (Local Group Policy Editor).

  • Navigate to: Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Connection Client.

  • Enable the policy: “Turn off hardware-accelerated decoding”.

  • Run gpupdate /force in a Command Prompt to apply the change.

2. Adjust Experience Settings In the Windows Remote Desktop Connection client:

  • Go to the Experience tab.

  • Uncheck “Persistent bitmap caching”. This often resolves “black screen” or negotiation hangs.

3. Credential Handling KRDP can be sensitive to how credentials are passed.

  • Enter your Username in the client, but do not save the password initially.

  • Ensure “Allow me to save credentials” is checked.

  • Manually enter the password only when the Windows security pop-up appears during the connection attempt.

Additional notes

Third-party clients like Thincast worked immediately. Guacamole on docker only works with guacd package built with h.264 support included. I am using mwilson2764/guacd-h264 - Docker Image .

A note on Hardware Encoding: While my Beelink’s Ryzen 5800H supports H.264 hardware encoding (verified via Jellyfin/FFmpeg), KRDP currently seems to default to CPU encoding. If anyone has insights on forcing VA-API or hardware-accelerated encoding within the KRDP server environment, I would love to hear your suggestions.

A thank you to the Devs: The progress made on KRDP is fantastic. We are closer than ever to a seamless Wayland remote desktop experience. My only hope for the future is expanded documentation to help bridge the gap for users transitioning from X11 or Gnome.

I hope this helps anyone else struggling with “Protocol Mismatch” errors!

Disclaimer: Post rephrased using AI. Re-posted on Cachyos forum: Troubleshooting KRDP: A Guide to Connecting from Windows Native RDP Clients - General - CachyOS Forum

1 Like
3

@krunchynug8 Thanks for compiling this information, super useful. I was finally able to get connected up to my KRDP server using the Windows remote desktop client thanks to this.

I’ve been looking forward to more KRDP improvements and/or guides like this to bridge the gap. My solution up until this point was installing Windows Subsystems for Linux, installing Fedora, and running KRDC via wsl.exe to be able to connect up to my machine running KRDP. Not very elegant, but at least it worked.