Testing desktop notifications with NixOS

Desktop notifications are a nice way to be informed immediately about important changes on your system. In a terminal or interactive shell script this is easy enough: trap 'notify-send "💥"' ERR. But in a non-interactive or restricted context, such as a systemd service, we need more setup for desktop notifications to work. And with more setup comes more chances to mess up, so it would be good to test that this actually works on a real desktop. Enter NixOS tests!

• Nix makes it easy to write reproducible tests.
• The tests run in a virtual machine¹, so we don’t need to worry about how the host OS is configured.
• The test framework supports easily starting services and waiting for them to reach a desired state.
• Tests can run anywhere with a bit of RAM and disk space.

At minimum, we need the test to do the following:

1. Create a NixOS VM with some configuration.
2. Start the VM.
3. Run the commands to test the NixOS configuration.

The result is pretty simple (annotated and minimised):

{ pkgs }:
let
  machineName = "machine";
in
pkgs.testers.runNixOSTest {
  name = "ssh-server-audit";

  nodes.${machineName} = {
    # Import only the NixOS configuration relevant for this test
    imports = [ ../ssh-server.nix ];

    # Optional/irrelevant config omitted
  };

  testScript = ''
    ${machineName}.start()

    # Wait for SSH daemon service to start
    ${machineName}.wait_for_unit("sshd.service")

    # Should pass SSH server audit
    ${machineName}.succeed("${pkgs.ssh-audit}/bin/ssh-audit 127.0.0.1")
  '';
}

This test works great, so we can reuse the general format:

{ pkgs }:
let
  machineName = "machine";
in
pkgs.testers.runNixOSTest {
  name = "some-name";

  nodes.${machineName} = {
    # NixOS configuration
  };

  testScript = ''
    ${machineName}.start()

    ${machineName}.succeed("some command")
  '';
}

Desktop notification tests need several additional features, and some of these are non-trivial to implement or find examples of. The rest of the article explains how this was achieved. See the final result for the full code.

A functioning graphical desktop

This is available in the existing nixpkgs GNOME tests. I already use GNOME, so reusing this was an obvious choice. The relevant parts of the test are in nodes.${machineName}:

{
  services = {
    desktopManager.gnome.enable = true;
    xserver.enable = true;
  };
}

Auto-login to the desktop

We need to be logged in to the desktop so that we actually see the notifications. The relevant parts of the test, also based on the GNOME tests above, are in nodes.${machineName}:

let
  userName = "alice";
in
{
  services.displayManager.autoLogin = {
    enable = true;
    user = userName;
  };

  users.users.${userName} = {
    isNormalUser = true;
    uid = 1000; # Used later
  };
}

Wait for the desktop to be fully ready for interaction

This functionality is also found in the GNOME tests, but it’s a big chunk of code:

{ pkgs }:
let
  machineName = "machine";
  userName = "alice";
in
pkgs.testers.runNixOSTest {
  nodes.${machineName} = {
    # Allow Eval API access; see
    # https://github.com/NixOS/nixpkgs/blob/07700a890da3eaf57033bc408937e8955ca1c393/nixos/tests/gnome.nix#L25-L38
    systemd.user.services."org.gnome.Shell@".serviceConfig.ExecStart = [
      ""
      "${pkgs.gnome-shell}/bin/gnome-shell --unsafe-mode"
    ];
  };

  testScript =
    { nodes, ... }:
    let
      user = nodes.${machineName}.users.users.${userName};
      inherit (user) name;
      bus = "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/${toString user.uid}/bus";
    in
    ''
      start_all()

      # Wait for GNOME desktop. Correct output should be (true, 'false')
      ${machineName}.wait_until_succeeds(
          "su -c '${bus} gdbus call --session --dest=org.gnome.Shell --object-path=/org/gnome/Shell --method=org.gnome.Shell.Eval Main.layoutManager._startingUp' ${name} | grep --fixed-strings --line-regexp --quiet \"(true, 'false')\""
      )
    '';
}

Open a GUI where the notification shows permanently

Notifications often disappear after showing for a few seconds, and we want to guarantee that we don’t miss it! This can be done by sending the default key to open the notifications menu, [Super](https://en.wikibooks.org/wiki/QEMU/Monitor#sendkey_keys)-v. We also want to clear any existing notifications. Thus:

{ pkgs }:
let
  machineName = "machine";
in
pkgs.testers.runNixOSTest {
  testScript = ''
    # Open notifications menu
    ${machineName}.send_key("meta_l-v")

    # Go to "Clear" button
    ${machineName}.send_key("down")

    # Clear notifications
    ${machineName}.send_key("spc")
  '';
}

Wait for the service to fail rather than succeed

We want to see the error notification, after all. It was easy to extend the framework to do this:

{ pkgs }:
let
  machineName = "machine";
  userName = "alice";
in
pkgs.testers.runNixOSTest {
  testScript =
    { nodes, ... }:
    let
      user = nodes.${machineName}.users.users.${userName};
      inherit (user) name;
    in
    ''
      def wait_for_unit_state(node, unit: str, required_state: str, user: str | None = None, timeout: int = 900) -> None:
          def check_state(_last_try: bool) -> bool:
              return node.get_unit_property(unit, "ActiveState", user) == required_state

          with node.nested(
              f"waiting for unit {unit}"
              + (f" with user {user}" if user is not None else "")
              + f" to reach state {required_state}"
          ):
              retry(check_state, timeout)

      ${machineName}.systemctl("start android-backup.service", user="${name}")
      wait_for_unit_state(${machineName}, "android-backup", "failed", user="${name}")
    '';
}

OCR the screen repeatedly until the expected text shows up

OCR is supported out of the box with a single line of configuration, so this is actually super easy:

{ pkgs }:
let
  machineName = "machine";
in
pkgs.testers.runNixOSTest {
  enableOCR = true;

  testScript = ''
    # Wait for the relevant notification
    # (the notification text is trimmed to fit the GUI, so we can only check for the first few words)
    ${machineName}.wait_for_text(r"Could not pull OSMAnd files")
  '';
}

Run reliably with minimal resources in CI

This was harder than expected, because GitLab CI runners have limited resources, and a full NixOS GNOME desktop environment with default settings includes a lot of things. But we don’t need all of those things, so let’s get rid of what we can in nodes.${machineName}:

{
  environment = {
    defaultPackages = [ ];
    gnome.excludePackages = [
      pkgs.adwaita-icon-theme
      pkgs.baobab
      pkgs.decibels
      pkgs.epiphany
      pkgs.gnome-backgrounds
      pkgs.gnome-bluetooth
      pkgs.gnome-calculator
      pkgs.gnome-calendar
      pkgs.gnome-characters
      pkgs.gnome-clocks
      pkgs.gnome-color-manager
      pkgs.gnome-connections
      pkgs.gnome-console
      pkgs.gnome-contacts
      pkgs.gnome-control-center
      pkgs.gnome-font-viewer
      pkgs.gnome-logs
      pkgs.gnome-maps
      pkgs.gnome-menus
      pkgs.gnome-music
      pkgs.gnome-system-monitor
      pkgs.gnome-text-editor
      pkgs.gnome-tour # Avoid pop-up window at start
      pkgs.gnome-user-docs
      pkgs.gnome-weather
      pkgs.gtk3.out
      pkgs.loupe
      pkgs.nautilus
      pkgs.papers
      pkgs.showtime
      pkgs.simple-scan
      pkgs.snapshot
      pkgs.xdg-user-dirs
      pkgs.xdg-user-dirs-gtk
      pkgs.yelp
    ];
  };

  hardware.bluetooth.enable = false;

  networking = {
    dhcpcd.enable = false;
    firewall.enable = false;
    networkmanager.enable = false;
  };

  programs = {
    fuse.enable = false;
    nano.enable = false;
    geary.enable = false;
  };

  services = {
    avahi.enable = false;
    colord.enable = false;
    dleyna.enable = false;
    fstrim.enable = false;
    geoclue2.enable = false;
    gnome = {
      core-apps.enable = false;
      evolution-data-server.enable = mkForce false;
      gcr-ssh-agent.enable = false;
      gnome-browser-connector.enable = false;
      gnome-initial-setup.enable = false;
      gnome-keyring.enable = mkForce false;
      gnome-online-accounts.enable = false;
      gnome-remote-desktop.enable = false;
      gnome-user-share.enable = false;
      localsearch.enable = false;
      rygel.enable = false;
      sushi.enable = false;
      tinysparql.enable = false;
    };
    gvfs.enable = mkForce false;
    libinput.enable = false;
    lvm.enable = false;
    orca.enable = false;
    pipewire.enable = false;
    pipewire.wireplumber.enable = false;
    power-profiles-daemon.enable = false;
    speechd.enable = false;
    udisks2.enable = mkForce false;
    upower.enable = mkForce false;
  };

  systemd.user = {
    tmpfiles.enable = false;
  };

  xdg.portal.enable = mkForce false;
}

This was done with a few heuristics:

• Check for big dependencies with nix-tree "$(nix build --print-out-paths .#.checks.x86_64-linux.TEST_NAME.nodes.machine.system.build.toplevel)", and remove any which don’t seem to be necessary.
• Look at how environment.gnome.excludePackages and services.xserver.excludePackages are defined, to exclude everything in there.

• Disable programs and services in GNOME which are not necessary for the tests to run. To figure out which services are enabled in a test I ended up with the following horrible hack:

  builtins.attrNames (
  nixosConfigurations.ci-full-unstable.pkgs.lib.filterAttrs
    (name: svc: name != "frp" && svc.enable.isDefined or false && svc.enable.value)
    (checks.x86_64-linux.androidBackupService.extendNixOS {
      module =
        { options, ... }:
        {
          passthru = { inherit options; };
        };
    }).nodes.machine.passthru.options.services
  )
  

  • Don’t disable dbus.

Using ls --si --size "$(nix build --print-out-paths .#.checks.x86_64-linux.androidBackupService.nodes.machine.system.build.images.iso)/iso/nixos-test-x86_64-linux.iso" as a proxy for the test size², the size before any optimisations was 2.0 GB, and after it was 1.2 GB (40% smaller).

Heavily inspired by Jacek Galowicz’s excellent Unveiling the Power of the NixOS Integration Test Driver, and made possible by all the contributors to the NixOS test framework ♥️.

1. Many tests can run in containers, but that’s not supported for the full desktop environment we’ll be using in this article. It’s also easy to run tests in multiple containers/VMs, for example to test networking. ↩

2. Please let me know if you are aware of a better way to estimate how much space the VM will actually take when running the test. ↩