Occasionally freezes when using dbus to take screenshots

Development
,
1

I’m trying to write a Python script to recognize objects in images of specific windows/areas on a screen. My idea is to get the raw image data through the org.kde.KWin.ScreenShot2 interface (or, is there a better way to try?) and pass it to the machine learning part. However, the script occasionally gets stuck at the line f.read(expected_size) when looping through screenshots. I tried adding a breakpoint before that, and the dbus interface does indeed return a dbus.Dictionary (containing the format/height/scale/stride, etc., as described in the KWin source file).

I then checked the Qt documentation, and QImage::Format_ARGB32_Premultiplied is indeed 4 bytes/pixel, so it should read a size of 4 * width * height. I tried reading the data byte by byte, but the script runs very slowly (~80fps → ~30fps).

Am I missing some details about dbus? Or should modify the way fd object read in Python?

Thanks in advance :slight_smile:

class AreaMonitor:
    def __init__(self, interval_ms: int = 1000):
        self.bus = dbus.SessionBus()
        self.kwin_interface = dbus.Interface(
            self.bus.get_object(
                bus_name="org.kde.KWin.ScreenShot2",
                object_path="/org/kde/KWin/ScreenShot2",
            ),
            dbus_interface="org.kde.KWin.ScreenShot2",
        )
        self.interval = interval_ms / 1000.0
        self._capture_options = {
            "include-cursor": dbus.Boolean(False),
            "native-resolution": dbus.Boolean(True)
        }
    def capture_area(self, x: int, y: int, width: int, height: int):
        read_fd, write_fd = os.pipe()
        try:
            write_fd_dbus = dbus.types.UnixFd(write_fd)
            results = self.kwin_interface.CaptureArea(
                dbus.Int32(x), dbus.Int32(y),
                dbus.UInt32(width), dbus.UInt32(height),
                self._capture_options,
                write_fd_dbus
            )
            os.close(write_fd)
            image_width = results.get("width", 0)
            image_height = results.get("height", 0)
            expected_size = image_width * image_height * 4
            with os.fdopen(read_fd, 'rb') as f:
                image_data = f.read(expected_size)
            return image_data
        except Exception as e:
            os.close(write_fd)
            raise e
2

KWin has some restriction to which program can take screenshot:
https://invent.kde.org/plasma/kwin/-/blob/master/src/plugins/screenshot/screenshotdbusinterface2.cpp#L233

You can set the environment variable to get around it.
in /etc/env so that kwin_wayland process can see it.

export KWIN_SCREENSHOT_NO_PERMISSION_CHECKS=1

The other way is to have a desktop file for your app with a special key, like spectacle does:
https://invent.kde.org/plasma/spectacle/-/blob/master/desktop/org.kde.spectacle.desktop.cmake?ref_type=heads#L198

Those restrictions might be lifted they are a bit out of place, I introduced them a while back.

You could also capture video.

1 Like
3

Thanks for pointing that out. I created a corresponding desktop file similar to spectacle in ~/.local/share/applications (also tried put into /usr/share/applications) beforehand, and declared the permissionX-KDE-DBUS-Restricted-Interfaces. Otherwise, the script would directly raise an error like “This application is not authorized to take screenshot” when calling dbus.

But now it seems that dbus is returning results normally, so the permissions should be fine?

And there’s my .desktop file for python script:

[Desktop Entry]
Name=AreaMonitor
# Exec is Python binary to ensure proper permissions when using `python script.py`.
Exec=/home/myuser/.conda/envs/test/bin/python
Type=Application
StartupNotify=false
DBusActivatable=true
X-DBUS-StartupType=Unique
Categories=Utility;
X-KDE-DBUS-Restricted-Interfaces=org.kde.KWin.ScreenShot2
4

I think this answers itself.

5

My description may have been misleading (translated from non-English), apologize.
I created the .desktop file before the script got stuck,

I tried adding a breakpoint before that, and the dbus interface does indeed return a dbus.Dictionary

and I also received a dict object returned by dbus in Python, indicating that the permissions were correct. My problem was that while I set the theoretically correct byte length when executing the line f.read(), it suddenly got stuck at this line when calling the dbus interface frequently, so I guess that not enough data (expected to be 4 bytes per pixel) was written to the UnixFd correctly?
Additionally, I tried changing the area size and found that it never had a problem when the region was small, but this issue frequently occurred when it was set to 2560*1440 pixels, entire screen.
Anyway, thanks very much for your patient reply :slight_smile: