Wayland Vs X Benchmarking results (updated july 30)

(Update, July 30 - Now that the FPS cap has been fixed, I am shocked to note a more than 60% drop in performance in Wayland vs X11 using DXVK. Native OpenGL itself has seen a 15% reduction in performance in Wayland vs X. My previous results are invalid and I will have to run all the suites again to get a clearer picture of what is going on).

Having managed to get Wayland working properly on both Kubuntu and Neon after a weekend experiencing every problem posted here since the release of P6, it is time to benchmark and see which one is faster. The results are interesting in that they are not interesting.

I benchmarked every form of rendering type: OpenGL, Vulkan, DXVK, VKD3D, Cuda, RT, and straight up CPU. The things in brackets () are the engine.

Due to some weirdness with Wayland imposing a 60 FPS caps on almost everything, I had to forgo most of my standard benchmarks as they all run well above 100fps and select only the heavy hitters that struggle to reach 60fps.

Results


DXVK - Fire Strike U 4k (3DM) - W:4666 vs X:4644
DXVK - Fire Strike E 2k (3DM) - W:8307 vs X:8471
DXVK - Fire Strike B 1k (3DM) - W10417 vs X 13330*

Vulkan - Spaceship (Unity) - W:51 vs X:56*

VKD3D - Cyberpunk 2077 (RedEngine) W:40.61 vs X:38.83

OGL - Superposition (Unigine) W:3708 vs X3546

Cuda - Indigo (Indigo) W:14.965 vs X:17.732

RT - EZBench (Unreal Engine 5.1) W:2734 vs X:2632

CPU - Cinebench (Cinema 4D) W:1310 vs X1298


(* these two are likely anomalies due to actually bumping into the FPS limit)

As you can see Wayland has a very slight advantage over X if you remove the outliers, but it is so slight it is within the margin of error. Quantitatively there is little to no difference between Wayland and X.

Qualitatively there is a lot. Wayland seems to surge when the FPS start to climb, where it seems that it is steady then speeds up, though it also can get rather jerky at times. X is the opposite… it is steady then occasionally lags or gets microstutters.

The worst culprit for this was Spaceship using the unity engine which had significant pauses and stutters within the benchmark in Wayland but runs smooth as silk in X.

The lack of hardware controls over the GPU means you can do no tuning, not even the fans. Due to this Wayland runs hot.

The lack of color controls in Wayland means it is also ugly. You are stuck with the default from your GPU, which may not work well with your monitor, and no way to adjust it from Plasma.

The lack of controls for 3D aspects like overrides for AA/AF/Vsync means games that need them are broken (some older games cannot use modern AA settings).

You also cannot build a video wall with Wayland using multiple GPUs and displays. It is not supported. We have been using video walls for 20 years, and after 15 years of development, Wayland still cannot even do basic things. Some of my benchmarks can span across multiple monitors.

In conclusion, you gain nothing with Wayland as a user experience, but you lose a lot. It is no faster, no more capable, and no easier to use than X (arguably less so), and you lose all control over your GPU and displays.

Wayland, at this time, has no advantage over X for desktop users who like to do ordinary things like play games or create media. It is more of a hindrance than help. It seems little more than reinventing the wheel, and they made it square. Wayland will only be useful when it matches the capabilities and features of Z11. As it stands, its not even close.

5 Likes

Hi shadow

Thanks for the super-duper tech approach.

Sounds like Wayland and AI have something in common.

I tried Wayland on my Porteux with KDE and it totally flopped.

If you want any info just ask but your testing is complete enough for me.

Thank you for the awesome report.

Vektor

3 Likes

Thanks. I have heard a lot of people complain that Wayland is slower in Steam and Games. Curious if this was true, I decided to check. On my rig… there is no difference.

1 Like

Now that I have gotten the frame rate limit sorted out, I will have to re-benchmark everything as a quick test looked to be about 20% short of X11 performance once it gets up above 60fps. But that was “seat of the pants”, not actual testing. One of the things I noticed to was the the initial synthetic benchmarks may not be telling the whole story as gameplay FPS seemed to be about 10% slower in Wayland than in X even though the benchmarks showed otherwise (not unexpected). I will need to control for that variable in my testing.

1 Like

Update: Things are not looking good for Wayland anymore. Early testing shows that with the frame cap eliminated the max FPS is way down compared to X11. I will run the full suite tomorrow, but as it stands, Wayland is looking like a bit of a pig.

update - Its getting late, but initial testing is showing a 14.5% reduction in performance with Wayland in OpenGL vs X11 now that the framerate cap is gone. I did not expect this based on earlier results. It is even worse with DXVK. I decided to run a quick benchmarks after loading up CyberPunk 2077. My jaw dropped upon seeing the same frame rate I had with my old GTX 970… which was shocking. FPS was literally reduced to under 30fps. This is a 62% drop in performance. 62%!!!

1 Like

You did not specify your CPU, GPU, RAM, distro, driver version or kernel version, to allow your results to be put in context, reproduced or compared to others.
Neither did you give graphs or numbers.
Please share this, se we can call it a proper benchmark.

Did your driver update have an impact on your results?

Are you using Neon for benchmarking?

Neon isn’t a great distro for performance especially on recent hardware, since it has oldish kernel and drivers by design (based on Ubuntu LTS).

I would recommend reading https://www.phoronix.com/ for linux benchmarking.

For instance https://www.phoronix.com/review/kde-plasma-6-amd-gaming with a AMD Ryzen 9 7950X system with AMD Radeon RX 7900 XTX graphics on Linux 6.5 + Mesa 23.2 with KDE Plasma 6.0.

The software used to run those benchmarks can be reused locally and are reproducible.

But they don’t as in detail in the games as you did and the variety of rendering pipeline, that’s a nice effort.

5 Likes

I use Phoronix. Remarkable suite. I decided to focus less on techy things and more on usery things that we like to do with our machines from an entertainment perspective such as games now that Linux/GNU is actually really good at it. I never had any interest in benchmarking Linux in the past for that reason.

Most benchmarking focuses on comparing the latest and greatest hardware or comparing it to windows, but that is not my focus at the moment. I am more interested in the relative differences are between Wayland and X on the same hardware. What that hardware is, is not particularly important (to a degree).

Sincerely, this has been a fantastic way to learn about Wayland, which to me seems to be a bit like learning Quantum Mechanics… the more I learn, the less I know. I have come to realize just how different a paradigm it is.

What I would like to do is set up a test bench for each of the main platforms: Intel/AMD/Nvidia to allow me to give relative results (not comparative) for the three brands of GPUs and to a lesser extent the CPUs.

I love tuning and benchmarking, so this stuff is a lot of fun. I do not care about the fastest PC ever (that is a hollow title), but making any PC the fastest it can be.

I am pondering ways to refine the testing to ensure it is will always be accurate and reproducible. I am not satisfied with how I did things with this testing (I made an LTT error of announcing results prematurely).

1 Like

It has everything to do with X and Wayland that are by definition tied to particular driver and their particular support of particular hardware.

The same recent driver can have very different behavior depending on the hardware generation for instance.

Benchmark numbers can never be extrapolated directly out of their own context.

Your numbers will ever be accurate to people with similar hardware and setup.

I love tuning and benchmarking, so this stuff is a lot of fun. I do not care about the fastest PC ever (that is a hollow title), but making any PC the fastest it can be.

That’s a nice goal, this does not mean your benchmark relevancy is not tied to its setup.

What I would like to do is set up a test bench for each of the main platforms: Intel/AMD/Nvidia to allow me to give relative results (not comparative) for the three brands of GPUs and to a lesser extent the CPUs.

To get relative results, you first need absolute ones to get the baseline…
Relative results are just a way to present data.

I am pondering ways to refine the testing to ensure it is will always be accurate and reproducible.

First fix the moving parts, here: software versions (distro/kernel) + hardware. And then you change only what you mean to measure (here would be Wayland session/ X11 session).

That’s the scientific method.

3 Likes

Benchmarking has always been, and remains, controversial. The problem when you use the term benchmark is most people assume hardware and hardware comparison and it usually turns into a **** size comparison which I want to avoid. The intention here is to generate a simple set of tests that others can run to compare relative performance on their own machines, and then add the deltas to a database.

One of the things with the scientific method is the recognition that conclusions often have to be extrapolated from limited datasets, as controlling for every singe variable is impossible. To do it the way you describe would require testing every single GPU, CPU, Motherboard, ram, and driver version combination. That would require 10s of thousands of computers.

But, over time, with individuals contributing to the database, accuracy can be refined. The information I am interested is: in general what is the relative difference between the two across an average of all use cases. Whether or not this is useful to you as a developer is not my main consideration; general user experience is.

I am also approaching this from the perspective of the average user, who is not running the latest and greatest hardware. Most are like me, somewhere in the middle (The Steam Database is awesome in that regard).

I intend to extend the testing to Gnome as well, to see what differences there are between mutter and Kwin. I do agree with you that inclusion of the OS and driver version is relevant, and while I did not “officially list them”, I did include Neon in the post, and mentioned the driver numbers used. Specific hardware used is not as relevant as generation. The major difference between two GPUs of the same generation and make (excluding budget rebrands of older hardware) is not the IPC accelerators, but clocks, compute core count, memory amount, and bus width.

Benchmarks are imperfect. There is no denying that. But, they do have their place, and do provide useful information.

2 Likes

Hello shadow

What is your plan to implement “individuals contributing to the database”?
Will you provide a software package for someone to use on their equipment?

That would be cool, because anyone could perform a regimented test and report back with information that would slide right in to a D-base with consistency. 1 click produces a file and sends it to a holding tank for inspection and inclusion in the D-base.

At the least your project would be consistent from one point of reference.

Vektor

2 Likes

Hey, thanks for the encouragement.

I am thinking a simplified suite of commonly available free tools that everybody can run. Phoronix is a powerful tool for this, but I find it cumbersome. The idea is to make it fun as well as informative, hence the use of visual benchmarks.

I completely agree with Meven about gathering as much useful data as possible. Granularity is important. I have been looking to see if this kind of data has been gathered empirically as it relates to gaming and other graphical workloads already, but so far I cannot glean anything useful. Everything is just anecdotal.

I also want to emulate the metrics and standards used by windows tech reviewers like GN and HWU so that Windows users can get a sense of the fact that there is now a legitimate alternative to Windows for gaming. I want to use as many tools that demographic would be familiar with (3DMark, Cinebench, Tomb Raider, Cyberpunk, etc) as possible.

First though, I need to build a relational database. While the hardware will be a sample size of one, I can start to track changes to drivers and Plasma/Gnome versions relative to performance and a few fuzzy metrics like stability or qualitative in game experience.

I should point out I am not doing this to crap in anybody’s cereal. Quite the contrary. Linux is not just a commandline wonder for technogeeks anymore, and it deserves to yell out “I am fun too!”. The fact that I can even do all this today blows my mind. Its amazing.

I am also a bit of an old dog having to learn new tricks. 35 years of using X means there is a lot of stale garbage in my brain. Wayland is a completely different paradigm from X11. Many old assumptions are not valid and I am still learning what those are, so I expect to make mistakes and will need to refine the process as time moves on.

1 Like

I’ve heard Wayland is all this and all that and everyone jumping on the Wayland bandwagon mainly because it’s new tech and suppossed to be better and everyone loves new tech just for the sake of it. But from your report it seems like it’s not worth it, at least not yet. Which is fine with me. I’ll stick to X11 and be a happy user. When Wayland is better than X11 than everyone will know. And the decision to switch will be a no brainer.

3 Likes

As far as i know, there no games 100% Wayland. DXVK, Wine, runs with XWayland, which is X11 in Wayland.

Maybe we could have this with Unvanquished or Xonotic compiled for supporting natively Wayland …

It’s the main problem with Wayland, compatibility and performances.

3 Likes

You make a good point in that I need to be more specific about what I mean when I say “Wayland”. As you say, when in a Wayland session I am using multiple tools that are not necessarily “Wayland” but the compatibility layer. I am currently looking too see what apps I might be able to employ to benchmark Wayland natively.

2 Likes

The mundane won’t notice that.

The professional will notice that.

Overall, when Wayland will do everything X can do in a decade, some other project to replace Wayland will have started.

The problem : Linux distros may stop supporting X in a couple of years.

Thank you for having shed some other type of light on this subject.

1 Like

Thanks, I have a lot to think about and plan for. I am inventorying spare parts to see if I can put together a second test bench PC. One AMD based, the other Intel, with a cross section of graphics cards from AMD/Nvidia. I will have to consider if I want to try to acquire an Intel Arc GPU. I might be able to score some donated parts to have two dedicated open benches and leave my personal PC alone. I am going to have to put my begging hat on.

Hi, sorry to bring up a dead post, but I’d like to add a few things, since this post ranks quite highly in google search results so I think people could be potentially misled.

There are more advantages to Wayland than just performance. Since it’s newer and will replace X in a few years (many distros are already using it by default), we will get constant improvements it is more secure by design.

  • Wayland is capable of mixed-DPI support and in my experience has worked a lot better on laptops when plugging in external monitors like TVs and projectors.
  • Also on laptops, I have found that touch gestures like swiping and scrolling have been much smoother and fluid. In X, it’s almost impossible for me to do things like 3-finger swipes unless I really focus on doing that gesture.
  • When using computers with Intel graphics, I have experienced screen tearing with X. This doesn’t happen on Wayland.
  • DE animations in GNOME, like going to the activities view are smoother. Also, dragging windows around has often felt glitchy for me in X. I don’t get this problem with wayland.
  • Wayland has native touchscreen support.

All in all, I have felt that I have had a smaller, more performant experience with wayland. I don’t have any data to back this up but after trying it on a few different machines, that is what I have noticed.

On powerful desktops, perhaps the difference between wayland and X is small, and you may as well just go with whatever is more performant for your system But on laptops where integrated graphics is usually required, and support for things like touchpad gestures, smooth scrolling, and touchscreen is important, wayland has been a game-changer for me.

On my main laptop, a Framework 13, I have also felt that my battery life lasts longer as well on wayland. And fans rev up less when doing light tasks like browsing the web. Again, this is all going off general observations and I don’t have any data to prove this. But overall, I have had a much nicer experience with wayland.

1 Like

Wayland has matured a lot in the most recent releases. While still a buggy disaster on Neon for me, in Kubuntu it is working very will for the most part. But, even then, there are still glitches that make Wayland a show stopper as a daily driver for me.

  1. Multi dedicated GPU with installed monitors support still fails in Wayland. I can plug in many screens to my primary GPU, but adding screens to the secondary or tertiary GPUs results in no desktop on those screens. In X, I can use all available outputs.

  2. Focus is a nightmare with apps often coming to the forground unbidden and then refusing to click through. Some full screen apps do not hide the task bar which stays visible but unusable. Frozen fullscreen apps are impossible to kill as I cannot get anything to get in front of them. In X, I can usually get to my taskbar or start menu and launch a process manager which will come to the foreground. In Wayland, they remain in the background.

  3. Mode switching. While Wayland would seem to handles resolution changes more smoothly, it often results in a corrupted desktop that leads to freezing and the requirement for a reboot.

  4. Lack of GPU and color controls - a total show stopper. I need calibrated displays, and there are times when a simple ICC profile does not get you where you need, especially as monitors age, or the monitors do not support them at all.

  5. Wine is … well… problematic. It seems Wayland does not like the way Wine does things, and many apps are glitchy in Wayland where they are stable in X11. Many outright fail.

  6. Moving to a virtual terminal (ctrl-alt-f#) results in a corrupted desktop when returning to the desktop.

The performance in Wayland has improved dramatically. In that regard, I no longer have any complaints. Stability, usability, and predictability however, are still issues. At least in KDE. I cannot speak about Gnome as I have not used it in 10 years.

As for efficiency, they appear to use about the same amount of energy on my systems. I am actually more than a little impressed by the growing efficiency of Linux as a whole. It has made remarkable strides in that area. I am sitting doing this writing on a full destop with multiple displays and my primary 300watt rated GPU is using 18watts. My secondary is idling at 135mhz using zero resources with the fans off and is only 25 degrees c.

Two years ago I thought Wayland would be “ready” in two years. Sadly, that is not the case, so I am looking at maybe another 2 years before Wayland even begins to approach X11 for usability and stability. In KDE.

I personally think the Wayland paradigm is flawed. It places the entire development emphasis on the DE makers which means that each version of Wayland is different from the others, resulting in a massive duplication of effort and the necessity for hardware vendors to support multiple iterations of Wayland instead of one standard iteration of X11.

That may help explain why Nvidia often comes across as hostile to Wayland. It costs them more, yet gives nothing back. It makes the experience less than pleasant, which reflects on Nvidia as a company. It should not, because its not their fault. It is the philosophy behind Wayland.

Wayland should, at the very least, be as capable of doing the basics (like color control) as well as X11. It is not and so it SHOULD NOT be the default option at this time. (In KDE).

If I were a rich man, I would have no problem throwing tons of money at X11 to keep it alive and viable. Wayland should be an alternative, not a replacement.

Running an 8700k, RTX 2070S and the 560.35.03 drivers installed via my package manager via the Launchpad PPA.

I’ve noticed the same drop in performance under Wayland vs X11 while playing certain games. CS2 simply refuses to go over ~80% GPU usage, while playing the same game under X11 results in 100% GPU usage no problem - As a result FPS is almost halved with no FPS limiters or vsync enabled under Wayland.

Furthermore, Scrolling in Firefox isn’t as smooth under Wayland, it’s notably smoother under X11. Even the slide animation while switching virtual desktops isn’t as smooth under Wayland vs X11. GSP firmware is disabled.

Then we have ongoing fractional scaling issues…