Test Failures Fatal by Default in KDE CI

Announcement
1

Hi all,

I’m excited to share an important update to our Continuous Integration system that will improve the stability and reliability of our software. Starting soon, test failures will be fatal by default in KDE CI.

Historically, making test failures fatal has been an opt-in feature. This allowed some projects to inadvertently overlook failing tests, which could lead to undetected breakages. To address this, we are transitioning to a new default behavior where test failures will block merging, ensuring that every commit keeps our code in a working state.

Key Details

New Opt-Out Option

We’ve introduced a new allow-failing-tests-on option for the .kde-ci.yml file for projects that need temporary flexibility. If your project isn’t ready for this change, you can opt out using this option.

Previously, the require-passing-tests-on option was used to make test failures fatal by adding it to the options in the .kde-ci.yml file like this:

require-passing-tests-on: [ 'Linux', 'Windows' ]

If you need to opt out of the new default behavior, you can add the allow-failing-tests-on option to your .kde-ci.yml file like this:

allow-failing-tests-on: [ 'Linux', 'Windows' ]

Full example of a .kde-ci.yml file with the new option:

# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

Dependencies:
- 'on': ['@all']
  'require':
    'frameworks/kcoreaddons': '@stable-kf6'

Options:
  allow-failing-tests-on: [ 'Windows' ]

The new opt-out option is available now, and the default behavior has not yet changed. We will have a transition period to allow projects to adjust before implementing the new default.

Transition Period

There will be a 6-week transition period to allow projects to adjust before making test failures fatal by default. During this time we will:

  • Monitor adoption and feedback.
  • Assist projects that haven’t been adjusted yet, especially those under community maintenance.

Final Steps

After the transition period, test failures will become fatal by default. Additionally:

  • The legacy require-passing-tests-on option will be removed.
  • We will communicate the change to the community via the mailing list and Discourse.

What You Need to Do

We encourage all maintainers to review and update their CI configurations accordingly. Your proactive adjustments and feedback will help ensure a smooth transition and maintain the high quality of KDE projects.

If your project needs to allow a platform to continue failing tests temporarily, add the allow-failing-tests-on option to your .kde-ci.yml file with the appropriate platforms listed.

If some tests are failing because the Windows CI can’t handle certain GUI tests, prefer conditionally skipping only those tests on Windows CI runs by adding the following to the problematic tests:

if (qEnvironmentVariableIntValue("KDECI_CANNOT_CREATE_WINDOWS")) {
    QSKIP("KDE CI can't create a window on this platform, skipping this test");
}

If you have any questions or need support during this process, reply here and we’ll do our best to assist.

TLDR

Test failures will be fatal by default in KDE CI. Use allow-failing-tests-on to opt out temporarily. Review and update your CI configurations to ensure a smooth transition. Reach out for support if needed.

Thank you for your attention and commitment to maintaining robust, high-quality code.

Cheers,
Kristen

8 Likes