In the software industry, automated testing is a crucial part of the software development lifecycle. It allows for rapid issue detection and mitigation, encourages better compartmentalization of code, and provides confidence in overall application stability. While most Continuous Integration and Continuous Delivery (CI/CD) information on the web tends to focus on instrumenting browsers in order to test web applications, we need to do something a little different. We’ve got a Windows GUI application written in C++ and Qt, and we’re already using pywinauto for UIA instrumentation in order to run system tests against it. The challenge is to get these system tests to run on CI.
The starting point
We’re big fans of GitHub Actions. Our unit and integration tests run great on this platform, so it makes sense to extend the CI workflows to enable system test execution. This particular application uses a custom Windows driver, which cannot be installed on GitHub-hosted systems. Because of this, we need to use a self-hosted runner. With the new Windows VM-based runner successfully executing the other test types, the obvious first step is to simply try running the system tests and see what happens. Predictably, nothing useful happens, and the tests error out while trying to find objects in the app.
Delving into sessions
The tests appear to run correctly when executed manually while remoting into the VM. Clearly, there is an issue with the GitHub Actions-specific environment. The self-hosted runner executable from GitHub was installed as a Windows service, so that it would start up automatically. When running as a Windows service, there is no screen, physical or virtual, available for use. Traditionally, the answer to this has been enabling Windows auto-logon. That way, when the machine (virtual machine, in our case) boots up, it automatically logs in as a predetermined user, and then everything is executed in that user’s context.
Since this context would be running in the “console session”, i.e., what would be a real monitor on a physical computer, everything would be visible and accessible. Sadly, this isn’t a great security practice. Besides, enabling auto-logon necessarily stores the logon user’s password in plaintext on the machine. That’s an even worse security practice. So, how can we get this running?
Because we don’t want to use the console session, the only other reasonable choice is a remote desktop (RDP) session. How do we establish an RDP session as part of a CI workflow? After experimenting with Microsoft’s RDP ActiveX controls, we settled on a less finicky third-party solution. The FreeRDP project is an open source implementation of the Remote Desktop Protocol. Conveniently enough, the maintainers provide access to nightly builds of Windows binaries. wfreerdp.exe to the rescue! All right, now we can store the Windows VM’s user credentials as a GitHub secret and pass them to wfreerdp.exe to establish a session. But the GitHub Actions runner executable is still a service. How do we run the tests in our shiny new RDP session?
Finishing the puzzle
Enter PsExec. Some of the most powerful Windows utilities come from Sysinternals, which has been a part of Microsoft since 2006. PsExec allows us to execute commands as any user in any specified session. It can do much more, but that’s the functionality we’re focusing on. Running the system tests in the established RDP session via PsExec works beautifully! The last problem we need to solve is, how do we find the session number (or ID) of the RDP session?
Running query session in a command prompt will show current sessions. While we could try parsing that output, such a solution would be, shall we say, sub-optimal. In fact, it would be pretty janky. The admittedly complicated, but reliable, solution that we came up with is running the equivalent session query in PowerShell. Using C# interop and a number of [DllImport] statements bringing in wtsapi32.dll, which contains the Win32 APIs necessary to enumerate and inspect session information, we can get the list of active sessions and which users are connected to them. Finally, we filter the results and output the desired session number for use by PsExec.
The overall solution
To summarize, we have implemented the following steps for running system tests on a Windows GUI application inside a CI environment:
- Store user credentials as a GitHub secret to avoid having an always-logged-on machine
- Use FreeRDP at the beginning of system test execution to log in using the aforementioned user credentials
- Query and filter the CI machine’s session information using PowerShell and C# interop code to find the newly created, active RDP session
- Execute the system tests in the located session via PsExec
When specifying session numbers, PsExec may not output the stdout and stderr streams. To work around this, we pipe all the output to a temporary file. After execution is complete, we output the file to stdout, so that the results are viewable directly in the GitHub Actions logs.
This solution is more complicated than the equivalent in, say, Linux, where we have access to convenient projects like Xvfb for hosting a virtual X11 window, but given our operating system and security requirements, it works pretty well.
P.S. In some circumstances, it is possible to automate a Windows GUI application within a “session-less void”. Even if that had worked for us, we still needed to take screenshots of the entire desktop area in case of failure, because other applications are also potentially outputting info that could be useful for debugging. This scenario is what ultimately forced us down the whole RDP session path.
P.P.S. At first, we were establishing the RDP session and then terminating wfreerdp.exe before starting the system tests themselves. The disconnected session existed, but things still didn’t behave correctly with the system tests that required multiple applications to talk to each other. And the failure screenshots were blank. Instead, we let the RDP session exist during the entire run of the system tests. GitHub Actions’ cleanup phase then terminates wfreerdp.exe for us automatically, having detected it as an orphan process.
P.P.P.S. The featured image for this blog post was generated by the Stable Diffusion deep learning text-to-image model. The prompt was “Synthwave illustration of a robot sitting in front of a computer, artstation, 4k”.
