New ideas using Wayland Input Methods

Input Methods are a way of allowing third parties programs to manipulate text in text input boxes in applications.


This is a companion discussion topic for the original entry at http://blog.davidedmundson.co.uk/blog/new-ideas-using-wayland-input-methods/
15 Likes

Great work!

These days a lot of clients now have in-built support, including all GTK4 text areas, so it might not be needed. Maybe the ship for this has sailed, or there could be still value in centralising it and having a common pattern.

if we could remove the custom and duplicated code for this feature in NeoChat. Tokodon and KMail, this would make me very happy. One thing that NeoChat and Tokodon have are custom emojis support which might be a bit tricky to implement.

7 Likes

Wow, amazing stuff.

Each of these would be a blockbuster Plasma 6 feature so I sincerely hope at least a few of them manage to make it!

6 Likes

Amazing. I hope we will get a proper emoji picker on Plasma 6 based on this work. This is one of the thing I miss the most in windows. In windows you can just presss “win+.” and type and the selected emoji will automatically paste on the app.
Here is a baby wogue video demonstrating the problem. Ignoring the smugness, the points he pick up on are actually valid.

Direct Diacritic Display is also a fantastic thing as now I can easily and properly name people’s name.

Timely Translation Tasks, Simply Speak will greatly improve accessibility to many users.

All around an amazing job. Will be eagerly waiting for this.

3 Likes

In case you didn’t know, this works in Plasma as well :slight_smile: The UX isn’t the greatest, because it requires you to manually paste it into the application.

4 Likes

Yep. Knew that. But manually copy pasting emoji really sucks.

2 Likes

As another option to enter emoji, the ibus input method supports emoji, with ctrl+shift+e by default IIRC.

1 Like

I wish someone would make Kwin work with xeyes, or a similar alternative.

:eyes:

Handwriting support like in Windows 11 would be a nice - but not very important - additional input method. It probably would mean to include something like tesseract to work.

2 Likes

Do those features conflict with an existing input method? Like if I am already using fcitx5, can I still get those features?

2 Likes

That’s a really good question.
Right now all the examples in the blog post multiplex internally within a single application.

Kwin is set up so you have one input backend. Having two InputContexts alive at once both grabbing and inserting wouldn’t really work.
But being able to switch at runtime seems doable.

Code wise we need 3 steps:

  • seeing if we can port code in my playground to use KDE global shortcuts instead of sniffing keys to activate
  • add some “activate me please” signal to the InputMethod manager class
  • make kwin follow this, replaying as though we had just changed focus out to the old input context, and acting like a focus in to a new context on the new input method
  • then we can test against maliit and my demos
2 Likes

Especially the voice input is one of the features that would increase accessibility by a lot. Desktop Linux is lacking in the accessibility department and that would get us a bit closer. The other features are cool too but I think a system wide stt system would be the most must-have of them all.
In general the demos are sick and I hope at least some of that gets into plasma 6 in a polished state!

2 Likes

The first two ideas, “Convenient Clipboard Connections” and “Easy Emoji Entry”, have been provided in Chinese IMEs (e.g. sogoupinyin) for decades. If we are exploring this direction, then commercial CJK IMEs, especially modern smartphone IMEs, would be a good source of inspiration.

1 Like

Another example is ‘fig’, which uses the ibus IME framework to provide autocompletion for Linux shells.

The problem is that to do good autocompletion, you need not only text the user is currently typing, but also surrounding text. fig does it using shell hooks. But what if we do an IME that provides autocompletion for all text boxes? (probably using ChatGPT-like tech, e.g. like what Github Copilot does in vscode)
Then perhaps there should be someway that IME can get “surrounding text” from the app.

1 Like

There is an existing concept of surrounding text in IMEs.
How much surrounding text is provided is rather undefined. Qt gives you the current line.

1 Like

Yes, but wouldn’t it a killer feature to have custom emojis in all Plasma apps, which supports images or (hyperlink-)plugins?

Hi! I just recently found this discussion. The author of fcitx5 csslayer (the most advanced CJK input method framework under wayland at the moment) is an expert in developing input methods under Wayland. He has written several blog posts about Wayland input method issues:

  1. Chrome/Chromium 今旄 Wayland èŸ“ć…„æł•æ”ŻæŒçŽ°çŠ¶ | CS Slayer (Status of Wayland input method support in Chrome/Chromium today, written in Chinese and can be translated into English by Google Translate, etc.)

  2. A comprehensive guide about using Fcitx 5 on wayland | CS Slayer (A comprehensive guide about using Fcitx 5 on wayland)

He is also a fan of KDE.

I’m a fan off CS Slayer :slight_smile:

Hi all. I’m an ex dev who’s once promising career was stripped away by disability making keyboard and mouse usage difficult to impossible. I just discovered the amazing ibus-typing-booster (Any relationship between this thread’s Fabian and that app’s Mike Fabian?), and am doing my homework to use it correctly, and came across this thread.

Sorry I’m a little late to the party, but hopefully somebody here will see this reply, because there are a lot of important people I see here, and a very important subject was brought up.

Having two InputContexts alive at once both grabbing and inserting wouldn’t really work.

This is a MAJOR accessibility problem in Wayland. We have lots of tools which can be used to help with accessibility, for power users and disabled users alike, but the big problem is that because they all exclusively grab input devices, they cannot coexist, meaning that there’s no way to achieve the accessibility levels that are possible on X11 or Windows - we would need to have a single tool that can do literally everything, unless a new solution is developed.

I have been very lightly involved in a discussion with some application developers on github regarding this problem recently.

If any of you had any input to this discussion or any suggestions of directions to take to solve this problem, it would be deeply appreciated. The conversation on github is over here: https://github.com/kermitfrog/inputmangler/issues/2

Apologies for any interruption, but I hope some of you might find this a subject worthy of further consideration.

Thanks for your time
 and Happy New Year :slight_smile:

1 Like

The only feature I miss from proprietary OSes, is the visual input method, where you draw on the touch pad.

I don’t think anyone uses that for real, but it was great as a learning tool :slight_smile: