From 35792b1700b404b2eac7f0c60f6dfbf9a6bb3e31 Mon Sep 17 00:00:00 2001
From: Michal Nowak <mnowak@isc.org>
Date: Wed, 16 Aug 2023 16:19:30 +0200
Subject: [PATCH] Report hung system tests

At times, a problem might occur where a test is not responding,
especially in the CI, determining the specific test responsible can be
difficult. Fortunately, when running tests with the pytest runner,
pytest sets the PYTEST_CURRENT_TEST environment variable to the current
test nodeid and stage. Afterward, the variable can be examined to
identify the test that has stopped responding.

The monitoring script needs to be started in the background. Still, the
shell executor used for BSD and FIPS testing can't handle the background
process cleanly, and the script step will wait for the background
process for the entire duration of the background process (currently
3000 seconds). Therefore, run the monitoring script only when the Docker
executor is used where this is not a problem.
---
 .gitlab-ci.yml                   |  2 ++
 util/get-running-system-tests.py | 25 +++++++++++++++++++++++++
 2 files changed, 27 insertions(+)
 create mode 100755 util/get-running-system-tests.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index cfd11105b9..5665123a99 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -402,6 +402,8 @@ cross-version-config-tests:
     - *setup_interfaces
   script:
     - *find_pytest
+    - *find_python
+    - ( if [ "${CI_DISPOSABLE_ENVIRONMENT}" = "true" ]; then sleep 3000; "$PYTHON" "${CI_PROJECT_DIR}/util/get-running-system-tests.py"; fi ) &
     - cd bin/tests/system
     - >
       "$PYTEST" --junit-xml="$CI_PROJECT_DIR"/junit.xml -n "$TEST_PARALLEL_JOBS" | tee pytest.out.txt
diff --git a/util/get-running-system-tests.py b/util/get-running-system-tests.py
new file mode 100755
index 0000000000..1653e0a66d
--- /dev/null
+++ b/util/get-running-system-tests.py
@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+
+# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
+#
+# SPDX-License-Identifier: MPL-2.0
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0.  If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# See the COPYRIGHT file distributed with this work for additional
+# information regarding copyright ownership.
+
+import psutil
+
+for pid in psutil.pids():
+    try:
+        environ = psutil.Process(pid).environ()
+        if "PYTEST_CURRENT_TEST" in environ:
+            name = psutil.Process(pid).name()
+            print(
+                f'pytest process {name}/{pid} running: {environ["PYTEST_CURRENT_TEST"]}'
+            )
+    except (psutil.AccessDenied, psutil.NoSuchProcess):
+        pass