From 9588b06e0feb66e2e7ec6f80f1f5feef46ed91d6 Mon Sep 17 00:00:00 2001 From: Zygmunt Krynicki Date: Mon, 2 Dec 2024 14:38:28 +0100 Subject: [PATCH 1/5] Allow running exactly one test in utils/test The new check-one-test-% pattern rule allows running individual test scripts. This allows them to be tested in parallel across many Make worker threads or across many distinct machines with spread. Signed-off-by: Zygmunt Krynicki --- utils/test/Makefile | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/utils/test/Makefile b/utils/test/Makefile index 7f52c0d2c..5aa8d8ec6 100644 --- a/utils/test/Makefile +++ b/utils/test/Makefile @@ -84,6 +84,16 @@ clean: check: __libapparmor __parser export PYTHONPATH=$(PYTHONPATH) LD_LIBRARY_PATH=$(LD_LIBRARY_PATH) LC_ALL=C __AA_CONFDIR=$(CONFDIR) __AA_BASEDIR=$(BASEDIR) __AA_PARSER=$(PARSER) ; $(foreach test, $(wildcard test-*.py), echo ; echo === $(test) === ; $(call pyalldo, $(test))) +# FIXME: passing -v to test-$*.py below makes some tests fail. +# +# This is a static pattern rule that allows running individual tests via the +# check-one-test-NAME target, where NAME matches an exciting file test-NAME.py. +# The way it is coded allows make to complete it, making it easier +# to use interactively. +check-one-test-%: PYTHON ?= python3 +$(foreach t,$(wildcard test-*.py),check-one-test-$(patsubst test-%.py,%,$t)): check-one-test-%: __libapparmor __parser + PYTHONPATH=$(PYTHONPATH) LD_LIBRARY_PATH=$(LD_LIBRARY_PATH) LC_ALL=C __AA_CONFDIR=$(CONFDIR) __AA_BASEDIR=$(BASEDIR) __AA_PARSER=$(PARSER) $(PYTHON) test-$*.py + .coverage: $(wildcard ../aa-* ../apparmor/*.py test-*.py) __libapparmor __parser export PYTHONPATH=$(PYTHONPATH) LD_LIBRARY_PATH=$(LD_LIBRARY_PATH) LC_ALL=C __AA_CONFDIR=$(CONFDIR) __AA_BASEDIR=$(BASEDIR) __AA_PARSER=$(PARSER) ; $(COVERAGE_IGNORE_FAILURES_CMD) ; $(foreach test, $(wildcard test-*.py), echo ; echo === $(test) === ; $(PYTHON) -m coverage run --branch -p $(test); ) $(PYTHON) -m coverage combine From cc041815786c6a1f4c3bd95f3d70386767e94852 Mon Sep 17 00:00:00 2001 From: Zygmunt Krynicki Date: Mon, 25 Nov 2024 13:54:03 +0100 Subject: [PATCH 2/5] Allow running tests with spread Spread is a full-system, or integration test suite runner initially developed to test snapd. Over time it has spread to other projects where it provides a structured way to organize, run and debug complex full-system interactions. Spread is documented on https://github.com/canonical/spread and is used in production since late 2016. Spread has a notion of backends which are responsible for allocating and discarding test machines. For the purpose of running AppArmor regression tests, I've combined spread with my own tool, image garden. The tool provides off-the-shelf images, constructed on-the-fly from freely available images, and makes them easily available to spread. The reason for doing it this way is so that using non-free cloud systems is not required and anyone can repeat the test process locally, on their own computer. Vanilla spread is somewhat limited to x86-64 systems but the way I've used it here makes it equally possible to test x86_64 *and* aarch64 systems. I've done most of the development on an ARM single-board-computer running on my desk. Spread requires a top-level spread.yaml file and a collection of task.yaml files that describe individual tasks (for us, those are just tests). Tasks have no implied dependency except that to reach a given task, spread will run all the _prepare_ statements leading to that task, starting from the project, test suite and then task. With proper care one can then run a specific individual test with a one-line command, for example: ``` spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure ``` This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU architecture of the host), copy the project tree into the test machine, install all the build dependencies, build all the parts of apparmor and then run one specific variant of the regression test, namely the at_secure program. Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie), but also, if you have a Google cloud account, on Google Compute Engine or in one of the other backends either built into spread or available as a fork of spread or as a helper for ad-hoc backend. Spread can also create more than one worker per system and distribute the tests to all of the available instances. In no way are we locking ourselves out of the ability to run our test suite on our target of choice. Spread has other useful switches, such as: - `-reuse` for keeping machines around until discarded with -discard - `-resend` for re-sending updated copy of the project (useful for -reuse) - `-debug` for starting an interactive shell on any failure - `-shell` for starting an interactive shell instead of the `execute` phase This first patch contains just the spread elements, assuming that both spread and image-garden are externally installed. A GitLab continuous integration installing everything required and running a subset of tests will follow shortly. I've expanded the initial selection of systems to allow running all the tests on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check but also to showcase how practical spread is at covering real-world systems. A number of systems and tests are currently failing: - garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-12:tests/regression/apparmor:deleted - garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname - garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected - garden:debian-cloud-13:tests/regression/apparmor:deleted - garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server - garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted - garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname - garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server - garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname - garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected In addition, only on openSUSE, I've skipped the entire test suite of the utils directory, as it requires python3 ttk themes, which I cannot find in packaged form. Signed-off-by: Zygmunt Krynicki --- .gitignore | 7 + spread.yaml | 182 ++++++++++++++++++ tests/regression/apparmor/task.yaml | 86 +++++++++ tests/unit/binutils/task.yaml | 3 + tests/unit/libapparmor/task.yaml | 3 + tests/unit/parser-tst-caching/task.yaml | 3 + tests/unit/parser-tst-dirtest/task.yaml | 3 + tests/unit/parser-tst-equality/task.yaml | 7 + tests/unit/parser-tst-error-output/task.yaml | 3 + tests/unit/parser-tst-minimize/task.yaml | 3 + tests/unit/parser-tst-parser-sanity/task.yaml | 7 + tests/unit/parser/task.yaml | 16 ++ tests/unit/utils/task.yaml | 7 + 13 files changed, 330 insertions(+) create mode 100644 spread.yaml create mode 100644 tests/regression/apparmor/task.yaml create mode 100644 tests/unit/binutils/task.yaml create mode 100644 tests/unit/libapparmor/task.yaml create mode 100644 tests/unit/parser-tst-caching/task.yaml create mode 100644 tests/unit/parser-tst-dirtest/task.yaml create mode 100644 tests/unit/parser-tst-equality/task.yaml create mode 100644 tests/unit/parser-tst-error-output/task.yaml create mode 100644 tests/unit/parser-tst-minimize/task.yaml create mode 100644 tests/unit/parser-tst-parser-sanity/task.yaml create mode 100644 tests/unit/parser/task.yaml create mode 100644 tests/unit/utils/task.yaml diff --git a/.gitignore b/.gitignore index 3d0eb258b..725ac51e4 100644 --- a/.gitignore +++ b/.gitignore @@ -306,3 +306,10 @@ tests/regression/apparmor/xattrs_profile tests/regression/apparmor/coredump **/__pycache__/ *.orig +# Patterns related to spread integration tests +*.img +*.iso +*.log +*.qcow2 +*.run +.spread-reuse.yaml diff --git a/spread.yaml b/spread.yaml new file mode 100644 index 000000000..7e914e481 --- /dev/null +++ b/spread.yaml @@ -0,0 +1,182 @@ +project: apparmor +backends: + google: + key: '$(HOST: echo "$SPREAD_GOOGLE_KEY")' + halt-timeout: 1h + # Run only when explicitly named. This backend requires a Google Compute + # Engine (GCE) account and incurs cost on every use. It is most practical + # to scale-out tests once spread can express sufficient concurrency. + manual: true + # TODO: This needs to be adjusted to properly account for apparmor tests. + location: snapd-spread/europe-west2-b + systems: + - ubuntu-22.04-64: + workers: 4 + - ubuntu-24.04-64: + workers: 4 + - ubuntu-24.10-64: + workers: 4 + garden: + # The garden backend relies on https://gitlab.com/zygoon/image-garden + # TODO: Switch to a released version for better stability. + type: adhoc + # Use 2GB of RAM and four cores as otherwise we may not have enough memory + # to link the parser. It is better to have more workers than to have one + # big worker with lots of resources. + allocate: ADDRESS "$(QEMU_MEM_OPTION="-m 2048" QEMU_SMP_OPTION="-smp 4" image-garden allocate "$SPREAD_SYSTEM".$(uname -m))" + discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS" + systems: + # All systems except for the one Ubuntu system are marked as manual. + # This way we don't accidentally spin up everything when someone runs + # spread without knowing better. + - opensuse-cloud-15.6: + username: opensuse + password: opensuse + workers: 2 + manual: true # Run only when explicitly named. + environment: + # openSUSE 15 ships very old default python. + PYTHON: /usr/bin/python3.11 + PYTHON_CONFIG: /usr/bin/python3.11-config + - opensuse-cloud-tumbleweed: + username: opensuse + password: opensuse + workers: 2 + manual: true + - debian-cloud-12: + username: debian + password: debian + workers: 2 + manual: true + - debian-cloud-13: + username: debian + password: debian + workers: 2 + manual: true + - ubuntu-cloud-22.04: + username: ubuntu + password: ubuntu + workers: 2 + manual: true + - ubuntu-cloud-24.04: + username: ubuntu + password: ubuntu + workers: 2 + manual: true + - ubuntu-cloud-24.10: + username: ubuntu + password: ubuntu + workers: 2 + +exclude: + - .git + - "*.qcow2" + - "*.iso" + - "*.img" + - "*.log" + - "*.run" + +# Copy the project to this path on the test system. +# This is also available as $SPREAD_PATH. +path: /tmp/apparmor + +prepare: | + # Install build dependencies, depending on the type of system running. + case "$SPREAD_SYSTEM" in + debian-*|ubuntu-*) + apt-get update -qq + # TODO: extract this from README.md libapparmor section and unifiy with what is in .gitlab-ci.yml. + DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \ + attr \ + autoconf \ + autoconf-archive \ + automake \ + bison \ + build-essential \ + dejagnu \ + flake8 \ + flex \ + gettext \ + libdbus-1-dev \ + libtool \ + liburing-dev \ + pkg-config \ + python3-all-dev \ + python3-gi \ + python3-notify2 \ + python3-psutil \ + python3-setuptools \ + python3-tk \ + python3-ttkthemes \ + swig + ;; + opensuse-*) + # On openSUSE the default gcc and python are very old. We can use more + # recent version of Python quite easily but perl extension module system + # does not want us to modify the CC that's baked into perl and all my + # attempts at using gcc-14 have failed. + zypper install -y \ + attr \ + autoconf \ + autoconf-archive \ + automake \ + bison \ + dbus-1-devel \ + dejagnu \ + flex \ + gcc \ + gcc-c++ \ + gettext \ + gobject-introspection \ + libtool \ + liburing2-devel \ + make \ + pkg-config \ + python3-flake8 \ + python3-notify2 \ + python3-psutil \ + python3-setuptools \ + python3-setuptools \ + python3-tk \ + python311 \ + python311-devel \ + swig + ;; + *) + echo "Please add support for $SPREAD_SYSTEM to spread.yaml" + exit 1 + ;; + esac + + # TODO: add logic to skip this build phase and use prebuild binaries from + # GitLab pipeline. This should also reduce the number of dependencies we need + # to install above. + + # Configure libapparmor. We have to pass CC and CXX explicitly if provided in + # the environment. + ( + cd $SPREAD_PATH/libraries/libapparmor + sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python + ) + # Build libapparmor. + make -C $SPREAD_PATH/libraries/libapparmor -j"$(nproc)" + # Build apparmor_parser. + # The alternative builds sequentially to use less memory. + make -C $SPREAD_PATH/parser -j"$(nproc)" + # Build binary utilities (aa-exec and firends). + make -C $SPREAD_PATH/binutils -j"$(nproc)" + # Build python utilities. + make -C $SPREAD_PATH/utils -j"$(nproc)" + +# In case of failure, include the kernel version in the log. +debug-each: | + uname -a + +suites: + tests/unit/: + summary: Unit tests that do not exercise the kernel layer. + tests/regression/: + summary: Regression tests for parser-kernel interaction. + prepare: | + # FIXME: `make -C tests/regression` does not do anything. + make -C "$SPREAD_PATH/tests/regression/apparmor" -j"$(nproc)" diff --git a/tests/regression/apparmor/task.yaml b/tests/regression/apparmor/task.yaml new file mode 100644 index 000000000..02efe9b4e --- /dev/null +++ b/tests/regression/apparmor/task.yaml @@ -0,0 +1,86 @@ +summary: run all of the apparmor regression test suite +# TODO: This is somewhat tedious and it'd be nicer if we could somehow generate +# the variants without doing it "by hand" with the following one-liner: +# +# echo '$(foreach t,$(TESTS),$(info TEST/$t: 1))'| make -f Makefile -f /dev/stdin +environment: + TEST/aa_exec: 1 + TEST/access: 1 + TEST/attach_disconnected: 1 + TEST/at_secure: 1 + TEST/introspect: 1 + TEST/capabilities: 1 + TEST/changeprofile: 1 + TEST/onexec: 1 + TEST/changehat: 1 + TEST/changehat_fork: 1 + TEST/changehat_misc: 1 + TEST/chdir: 1 + TEST/clone: 1 + TEST/complain: 1 + TEST/coredump: 1 + TEST/deleted: 1 + TEST/e2e: 1 + TEST/environ: 1 + TEST/exec: 1 + TEST/exec_qual: 1 + TEST/fchdir: 1 + TEST/fd_inheritance: 1 + TEST/fork: 1 + TEST/i18n: 1 + TEST/link: 1 + TEST/link_subset: 1 + TEST/mkdir: 1 + TEST/mmap: 1 + TEST/mount: 1 + TEST/mult_mount: 1 + TEST/named_pipe: 1 + TEST/namespaces: 1 + TEST/net_raw: 1 + TEST/open: 1 + TEST/openat: 1 + TEST/pipe: 1 + TEST/pivot_root: 1 + TEST/posix_ipc: 1 + TEST/ptrace: 1 + TEST/pwrite: 1 + TEST/query_label: 1 + TEST/regex: 1 + TEST/rename: 1 + TEST/readdir: 1 + TEST/rw: 1 + TEST/socketpair: 1 + TEST/swap: 1 + TEST/sd_flags: 1 + TEST/setattr: 1 + TEST/symlink: 1 + TEST/syscall: 1 + TEST/sysv_ipc: 1 + TEST/tcp: 1 + TEST/unix_fd_server: 1 + TEST/unix_socket_pathname: 1 + TEST/unix_socket_abstract: 1 + TEST/unix_socket_unnamed: 1 + TEST/unix_socket_autobind: 1 + TEST/unlink: 1 + TEST/userns: 1 + TEST/xattrs: 1 + TEST/longpath: 1 + TEST/nfs: 1 + TEST/xattrs_profile: 1 + TEST/dbus_eavesdrop: 1 + TEST/dbus_message: 1 + TEST/dbus_service: 1 + TEST/dbus_unrequested_reply: 1 + TEST/io_uring: 1 + TEST/exec_stack: 1 + TEST/aa_policy_cache: 1 + TEST/nnp: 1 + TEST/stackonexec: 1 + TEST/stackprofile: 1 +execute: | + # Run the shell script that is named after the spread variant we are running + # now. The makefile runs them all sequentially via the "alltests" goal. Here + # we can parallelize it through spread and also have a way to run precisely + # the test we want, especially for debugging. + bash "$SPREAD_VARIANT".sh diff --git a/tests/unit/binutils/task.yaml b/tests/unit/binutils/task.yaml new file mode 100644 index 000000000..33bda2559 --- /dev/null +++ b/tests/unit/binutils/task.yaml @@ -0,0 +1,3 @@ +summary: Run unit tests of AppArmor binary utilities +execute: | + make -C "$SPREAD_PATH/binutils" check diff --git a/tests/unit/libapparmor/task.yaml b/tests/unit/libapparmor/task.yaml new file mode 100644 index 000000000..dc0dbaad6 --- /dev/null +++ b/tests/unit/libapparmor/task.yaml @@ -0,0 +1,3 @@ +summary: Run unit tests of libapparmor +execute: | + make -C "$SPREAD_PATH/libraries/libapparmor" check diff --git a/tests/unit/parser-tst-caching/task.yaml b/tests/unit/parser-tst-caching/task.yaml new file mode 100644 index 000000000..ed729ad9e --- /dev/null +++ b/tests/unit/parser-tst-caching/task.yaml @@ -0,0 +1,3 @@ +summary: Run apparmor_parser caching test from parser/tst +execute: | + make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" caching diff --git a/tests/unit/parser-tst-dirtest/task.yaml b/tests/unit/parser-tst-dirtest/task.yaml new file mode 100644 index 000000000..eb399a263 --- /dev/null +++ b/tests/unit/parser-tst-dirtest/task.yaml @@ -0,0 +1,3 @@ +summary: Run apparmor_parser dirtest test from parser/tst +execute: | + make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" dirtest diff --git a/tests/unit/parser-tst-equality/task.yaml b/tests/unit/parser-tst-equality/task.yaml new file mode 100644 index 000000000..5401e97ec --- /dev/null +++ b/tests/unit/parser-tst-equality/task.yaml @@ -0,0 +1,7 @@ +summary: Run apparmor_parser tests from parser/tst +# This test is particularly slow. Those values are aimed at running fast enough +# on a Raspberry Pi 5, representing a slower-ish computer. +warn-timeout: 20m +kill-timeout: 30m +execute: | + make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" equality diff --git a/tests/unit/parser-tst-error-output/task.yaml b/tests/unit/parser-tst-error-output/task.yaml new file mode 100644 index 000000000..f22f91413 --- /dev/null +++ b/tests/unit/parser-tst-error-output/task.yaml @@ -0,0 +1,3 @@ +summary: Run apparmor_parser error_output test from parser/tst +execute: | + make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" error_output diff --git a/tests/unit/parser-tst-minimize/task.yaml b/tests/unit/parser-tst-minimize/task.yaml new file mode 100644 index 000000000..6f1e43951 --- /dev/null +++ b/tests/unit/parser-tst-minimize/task.yaml @@ -0,0 +1,3 @@ +summary: Run apparmor_parser minimize test from parser/tst +execute: | + make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" minimize diff --git a/tests/unit/parser-tst-parser-sanity/task.yaml b/tests/unit/parser-tst-parser-sanity/task.yaml new file mode 100644 index 000000000..6c5fa34e8 --- /dev/null +++ b/tests/unit/parser-tst-parser-sanity/task.yaml @@ -0,0 +1,7 @@ +summary: Run apparmor_parser parser_sanity test from parser/tst +# This test is particularly slow. Those values are aimed at running fast enough +# on a Raspberry Pi 5, representing a slower-ish computer. +warn-timeout: 20m +kill-timeout: 30m +execute: | + make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" parser_sanity diff --git a/tests/unit/parser/task.yaml b/tests/unit/parser/task.yaml new file mode 100644 index 000000000..034368346 --- /dev/null +++ b/tests/unit/parser/task.yaml @@ -0,0 +1,16 @@ +summary: Run apparmor_parser unit tests from parser/ +details: | + The parser has a number of different tests. Those are all represented as + spread task variants so that they are directly visisble and runnable. +environment: + TEST/tst_regex: 1 + TEST/tst_misc: 1 + TEST/tst_symtab: 1 + TEST/tst_variable: 1 + TEST/tst_lib: 1 +prepare: | + # The test relies on make to build a binary. + make -C "$SPREAD_PATH/parser" -j"$(nproc)" "$SPREAD_VARIANT" +execute: | + cd "$SPREAD_PATH"/parser + ./"$SPREAD_VARIANT" diff --git a/tests/unit/utils/task.yaml b/tests/unit/utils/task.yaml new file mode 100644 index 000000000..37930898b --- /dev/null +++ b/tests/unit/utils/task.yaml @@ -0,0 +1,7 @@ +summary: Run unit tests of python utilities +# FIXME: Exclude openSUSE as the test depends on python3-ttkthemes +# which are not packaged in the distribution. +systems: + - -opensuse-* +execute: | + make -C "$SPREAD_PATH/utils" check From c95ac4d3500eec3f5713c1eebda28f72177d785e Mon Sep 17 00:00:00 2001 From: Zygmunt Krynicki Date: Tue, 26 Nov 2024 01:15:51 +0100 Subject: [PATCH 3/5] Second iteration of spread support Compared to v1 the following improvements have been made: - The cost of installing packages have been shifted from each startup to image preparation phase, thanks to the integration of custom cloud-init profiles into image-garden. This has dramatic impact on iteration time while also entirely removing requirement to be online to run once a prepared image is available. - Support for running on Google Compute Engine has been removed since it would not be able to use cloud-init the same way would currently only complicate setup. - The number of workers have been tuned for local iteration, aiming for comfortable work with 16GB of memory on the host. Once CI/CD pipeline support is introduced I will add a dedicated entry so that resources are utilized well both locally and when running in CI. - The set of regression tests listed in tests/regression/apparmor/task.yaml is now cross-checked so introduction of a new test to the makefile there is automatically flagged and causes spread to fail with a clear message. - The task tests/unit/utils has been improved to generate profiles. Thanks to Christian Boltz for explaining this relationship between tests. - A number of comments have been improved and cleaned up for readability, accuracy and sometimes better grammar. Signed-off-by: Zygmunt Krynicki --- .gitignore | 2 + .image-garden.mk | 78 +++++++++++++++ spread.yaml | 144 ++++++++-------------------- tests/regression/apparmor/task.yaml | 5 +- tests/unit/utils/task.yaml | 5 +- 5 files changed, 127 insertions(+), 107 deletions(-) create mode 100644 .image-garden.mk diff --git a/.gitignore b/.gitignore index 725ac51e4..9fe00e383 100644 --- a/.gitignore +++ b/.gitignore @@ -306,9 +306,11 @@ tests/regression/apparmor/xattrs_profile tests/regression/apparmor/coredump **/__pycache__/ *.orig + # Patterns related to spread integration tests *.img *.iso +*.lock *.log *.qcow2 *.run diff --git a/.image-garden.mk b/.image-garden.mk new file mode 100644 index 000000000..eda9f2cc6 --- /dev/null +++ b/.image-garden.mk @@ -0,0 +1,78 @@ +# This file is read by image-garden when spread is allocating test machines. +# All the package installation happens through cloud-init profiles defined +# below. + +# This is the cloud-init user-data profile for all Debian systems. Note that it +# is an extension of the default profile necessary for operation of +# image-garden. +define DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE +$(CLOUD_INIT_USER_DATA_TEMPLATE) +packages: +- attr +- autoconf +- autoconf-archive +- automake +- bison +- build-essential +- dejagnu +- flake8 +- flex +- gettext +- libdbus-1-dev +- libtool +- liburing-dev +- pkg-config +- python3-all-dev +- python3-gi +- python3-notify2 +- python3-psutil +- python3-setuptools +- python3-tk +- python3-ttkthemes +- swig +runcmd: +- apt clean +endef + +# Ubuntu shares cloud-init profile with Debian. +UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE=$(DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE) + +# On openSUSE Leap the default gcc and python are very old. We can use more +# recent version of Python quite easily but perl extension module system does +# not want us to modify the CC that's baked into perl and all my attempts at +# using gcc-14 have failed. +define OPENSUSE_CLOUD_INIT_USER_DATA_TEMPLATE +$(CLOUD_INIT_USER_DATA_TEMPLATE) +packages: +- attr +- autoconf +- autoconf-archive +- automake +- bison +- dbus-1-devel +- dejagnu +- flex +- gcc +- gcc-c++ +- gettext +- gobject-introspection +- libtool +- liburing2-devel +- make +- pkg-config +- python3-flake8 +- python3-notify2 +- python3-psutil +- python3-setuptools +- python3-setuptools +- python3-tk +- python311 +- python311-devel +- swig +endef + +# On openSUSE tumbleweed the set of packages may drift towards more recent +# versions more rapidly than on Leap but the moment we want to, for example, +# move to Python 3.13, we can define a separate entry with different package +# set or perhaps with $(patsubst)-computed package set. +OPENSUSE_tumbleweed_CLOUD_INIT_USER_DATA_TEMPLATE=$(OPENSUSE_CLOUD_INIT_USER_DATA_TEMPLATE) diff --git a/spread.yaml b/spread.yaml index 7e914e481..71106cb89 100644 --- a/spread.yaml +++ b/spread.yaml @@ -1,21 +1,5 @@ project: apparmor backends: - google: - key: '$(HOST: echo "$SPREAD_GOOGLE_KEY")' - halt-timeout: 1h - # Run only when explicitly named. This backend requires a Google Compute - # Engine (GCE) account and incurs cost on every use. It is most practical - # to scale-out tests once spread can express sufficient concurrency. - manual: true - # TODO: This needs to be adjusted to properly account for apparmor tests. - location: snapd-spread/europe-west2-b - systems: - - ubuntu-22.04-64: - workers: 4 - - ubuntu-24.04-64: - workers: 4 - - ubuntu-24.10-64: - workers: 4 garden: # The garden backend relies on https://gitlab.com/zygoon/image-garden # TODO: Switch to a released version for better stability. @@ -41,127 +25,61 @@ backends: - opensuse-cloud-tumbleweed: username: opensuse password: opensuse - workers: 2 + workers: 4 manual: true - debian-cloud-12: username: debian password: debian - workers: 2 + workers: 4 manual: true - debian-cloud-13: username: debian password: debian - workers: 2 + workers: 4 manual: true - ubuntu-cloud-22.04: username: ubuntu password: ubuntu - workers: 2 + workers: 4 manual: true - ubuntu-cloud-24.04: username: ubuntu password: ubuntu - workers: 2 + workers: 4 manual: true - ubuntu-cloud-24.10: username: ubuntu password: ubuntu - workers: 2 + workers: 4 exclude: - .git + - "*.o" + # Files related to spread and image-garden. - "*.qcow2" - "*.iso" - "*.img" - "*.log" - "*.run" + - "*.lock" # Copy the project to this path on the test system. # This is also available as $SPREAD_PATH. path: /tmp/apparmor prepare: | - # Install build dependencies, depending on the type of system running. - case "$SPREAD_SYSTEM" in - debian-*|ubuntu-*) - apt-get update -qq - # TODO: extract this from README.md libapparmor section and unifiy with what is in .gitlab-ci.yml. - DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \ - attr \ - autoconf \ - autoconf-archive \ - automake \ - bison \ - build-essential \ - dejagnu \ - flake8 \ - flex \ - gettext \ - libdbus-1-dev \ - libtool \ - liburing-dev \ - pkg-config \ - python3-all-dev \ - python3-gi \ - python3-notify2 \ - python3-psutil \ - python3-setuptools \ - python3-tk \ - python3-ttkthemes \ - swig - ;; - opensuse-*) - # On openSUSE the default gcc and python are very old. We can use more - # recent version of Python quite easily but perl extension module system - # does not want us to modify the CC that's baked into perl and all my - # attempts at using gcc-14 have failed. - zypper install -y \ - attr \ - autoconf \ - autoconf-archive \ - automake \ - bison \ - dbus-1-devel \ - dejagnu \ - flex \ - gcc \ - gcc-c++ \ - gettext \ - gobject-introspection \ - libtool \ - liburing2-devel \ - make \ - pkg-config \ - python3-flake8 \ - python3-notify2 \ - python3-psutil \ - python3-setuptools \ - python3-setuptools \ - python3-tk \ - python311 \ - python311-devel \ - swig - ;; - *) - echo "Please add support for $SPREAD_SYSTEM to spread.yaml" - exit 1 - ;; - esac - - # TODO: add logic to skip this build phase and use prebuild binaries from - # GitLab pipeline. This should also reduce the number of dependencies we need - # to install above. - - # Configure libapparmor. We have to pass CC and CXX explicitly if provided in - # the environment. - ( - cd $SPREAD_PATH/libraries/libapparmor - sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python - ) + # Configure libapparmor but only if a makefile is not already present. + # This makes repeated iteration with -reuse much faster, as the chain of + # invocations of make below are efficient if nothing needs to be done. + if [ ! -f "$SPREAD_PATH"/libraries/libapparmor/Makefile ]; then + ( + cd $SPREAD_PATH/libraries/libapparmor + sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python + ) + fi # Build libapparmor. make -C $SPREAD_PATH/libraries/libapparmor -j"$(nproc)" # Build apparmor_parser. - # The alternative builds sequentially to use less memory. make -C $SPREAD_PATH/parser -j"$(nproc)" # Build binary utilities (aa-exec and firends). make -C $SPREAD_PATH/binutils -j"$(nproc)" @@ -178,5 +96,27 @@ suites: tests/regression/: summary: Regression tests for parser-kernel interaction. prepare: | - # FIXME: `make -C tests/regression` does not do anything. - make -C "$SPREAD_PATH/tests/regression/apparmor" -j"$(nproc)" + # Spread does not support programmatically generated test variants. + # Ensure that the list baked into tests/regressin/apparmor/task.yaml + # contains all the tests defined in tests/regression/apparmor/Makefile. + echo '$(foreach t,$(TESTS),$(info TEST/$t))' | \ + make -n -f "$SPREAD_PATH"/tests/regression/apparmor/Makefile -f /dev/stdin | \ + grep -F TEST/ | \ + cut -d / -f 2 | \ + tee apparmor-regression-tests.txt + fail=0 + for V in $(cat apparmor-regression-tests.txt); do + if ! grep -xF ' TEST/'"$V"': 1' "$SPREAD_PATH"/tests/regression/apparmor/task.yaml; then + echo "tests/regression/task.yaml: missing test variant: TEST/$V" >&2 + fail=1 + fi + done + if [ "$fail" -ne 0 ]; then + echo "exiting due to missing variants listed above" >&2 + exit 1 + fi + + # Build all the apparmor regression test programs. + make -C "$SPREAD_PATH"/tests/regression/apparmor -j"$(nproc)" + restore: | + rm -f apparmor-regression-tests.txt diff --git a/tests/regression/apparmor/task.yaml b/tests/regression/apparmor/task.yaml index 02efe9b4e..4e6d795c8 100644 --- a/tests/regression/apparmor/task.yaml +++ b/tests/regression/apparmor/task.yaml @@ -1,8 +1,5 @@ summary: run all of the apparmor regression test suite -# TODO: This is somewhat tedious and it'd be nicer if we could somehow generate -# the variants without doing it "by hand" with the following one-liner: -# -# echo '$(foreach t,$(TESTS),$(info TEST/$t: 1))'| make -f Makefile -f /dev/stdin +# See spread.yaml for the code which verifies that nothing is missing from this list of variants environment: TEST/aa_exec: 1 TEST/access: 1 diff --git a/tests/unit/utils/task.yaml b/tests/unit/utils/task.yaml index 37930898b..cac37f28f 100644 --- a/tests/unit/utils/task.yaml +++ b/tests/unit/utils/task.yaml @@ -3,5 +3,8 @@ summary: Run unit tests of python utilities # which are not packaged in the distribution. systems: - -opensuse-* +prepare: | + # Generate the profiles that the test relies on. + make -C "$SPREAD_PATH"/parser/tst gen_xtrans gen_dbus execute: | - make -C "$SPREAD_PATH/utils" check + make -C "$SPREAD_PATH"/utils check From 1df91e2c8c936a887f34b5bd2d3afcb682b7c72c Mon Sep 17 00:00:00 2001 From: Zygmunt Krynicki Date: Tue, 26 Nov 2024 15:40:19 +0100 Subject: [PATCH 4/5] Third iteration of spread support - Tests defined in utils/test are now described by a task.yaml in the same directory and can run concurrently across many machines. - Tests for utils/ are now executed on openSUSE Tumbleweed since ttk themes is no longer a hard dependency in master. - Tests no longer run on openSUSE Leap 15.6 due to the age of default Python (3.6) and gcc/g++. The tight integration with SWIG which does not seem to support other Python versions very well. Perl hard-codes old GCC for extension modules. The upcoming openSUSE Leap 16 should be a viable target. In the meantime we can still test everything through rolling-release Tumbleweed. - Formatting of YAML files is now more uniform, at four spaces per tab. - The run-spread.sh script is now in the root of the tree. The script allows running all spread tests sequentially on one system, while collecting logs and artifacts for convenient analysis after the fact. - All systems are adjusted to run _four_ workers in parallel with _two_ virtual cores each and equipped with 1.5GB of virtual memory. This aims to best utilize the capacity of a typical CI worker with two to four cores and about 8GB of available memory. - Failing tests are marked as such, so that as a whole the entire spread suite can pass and be useful at catching regressions. Signed-off-by: Zygmunt Krynicki --- .gitignore | 3 + .image-garden.mk | 16 +- run-spread.sh | 29 +++ spread.yaml | 228 ++++++++++-------- tests/regression/apparmor/task.yaml | 204 ++++++++++------ tests/unit/binutils/task.yaml | 2 +- tests/unit/libapparmor/task.yaml | 2 +- tests/unit/parser-tst-caching/task.yaml | 2 +- tests/unit/parser-tst-dirtest/task.yaml | 2 +- tests/unit/parser-tst-equality/task.yaml | 6 +- tests/unit/parser-tst-error-output/task.yaml | 2 +- tests/unit/parser-tst-minimize/task.yaml | 2 +- tests/unit/parser-tst-parser-sanity/task.yaml | 6 +- tests/unit/parser/task.yaml | 22 +- tests/unit/utils/task.yaml | 10 - utils/test/task.yaml | 50 ++++ utils/vim/task.yaml | 3 + 17 files changed, 362 insertions(+), 227 deletions(-) create mode 100755 run-spread.sh delete mode 100644 tests/unit/utils/task.yaml create mode 100644 utils/test/task.yaml create mode 100644 utils/vim/task.yaml diff --git a/.gitignore b/.gitignore index 9fe00e383..13f1d2f89 100644 --- a/.gitignore +++ b/.gitignore @@ -315,3 +315,6 @@ tests/regression/apparmor/coredump *.qcow2 *.run .spread-reuse.yaml +.spread-reuse.*.yaml +spread-artifacts/ +spread-logs/ diff --git a/.image-garden.mk b/.image-garden.mk index eda9f2cc6..a93b65a94 100644 --- a/.image-garden.mk +++ b/.image-garden.mk @@ -30,18 +30,13 @@ packages: - python3-tk - python3-ttkthemes - swig -runcmd: -- apt clean endef # Ubuntu shares cloud-init profile with Debian. UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE=$(DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE) -# On openSUSE Leap the default gcc and python are very old. We can use more -# recent version of Python quite easily but perl extension module system does -# not want us to modify the CC that's baked into perl and all my attempts at -# using gcc-14 have failed. -define OPENSUSE_CLOUD_INIT_USER_DATA_TEMPLATE +# This is the cloud-init user-data profile for openSUSE Tumbleweed. +define OPENSUSE_tumbleweed_CLOUD_INIT_USER_DATA_TEMPLATE $(CLOUD_INIT_USER_DATA_TEMPLATE) packages: - attr @@ -67,12 +62,7 @@ packages: - python3-setuptools - python3-tk - python311 +- python3-devel - python311-devel - swig endef - -# On openSUSE tumbleweed the set of packages may drift towards more recent -# versions more rapidly than on Leap but the moment we want to, for example, -# move to Python 3.13, we can define a separate entry with different package -# set or perhaps with $(patsubst)-computed package set. -OPENSUSE_tumbleweed_CLOUD_INIT_USER_DATA_TEMPLATE=$(OPENSUSE_CLOUD_INIT_USER_DATA_TEMPLATE) diff --git a/run-spread.sh b/run-spread.sh new file mode 100755 index 000000000..da9ac89cf --- /dev/null +++ b/run-spread.sh @@ -0,0 +1,29 @@ +#!/bin/sh +# Run integration tests with spread sequentially on all the systems, using +# multiple workers per system. This mode is suitable to run on a single +# quad-core CPU with 8GB of RAM and no desktop session. +set -xeu + +if test -z "$(command -v spread)"; then + echo "You need to install spread from https://github.com/snapcore/spread with the Go compiler and the command: go install github.com/snapcore/spread/cmd/spread@latest" >&2 + exit 1 +fi + +if test -z "$(command -v image-garden)"; then + echo "You need to install image-garden from https://gitlab.com/zygoon/image-garden: make install prefix=/usr/local" >&2 + exit 1 +fi + +rm -rf spread-logs spread-artifacts +mkdir -p spread-logs +for system in \ + opensuse-cloud-tumbleweed \ + debian-cloud-12 \ + debian-cloud-13 \ + ubuntu-cloud-22.04 \ + ubuntu-cloud-24.04 \ + ubuntu-cloud-24.10; do + if ! spread -artifacts ./spread-artifacts -v "$system" | tee spread-logs/"$system".log; then + echo "Spread exited with code $?" >spread-logs/"$system".failed + fi +done diff --git a/spread.yaml b/spread.yaml index 71106cb89..d00cffba4 100644 --- a/spread.yaml +++ b/spread.yaml @@ -1,122 +1,138 @@ project: apparmor backends: - garden: - # The garden backend relies on https://gitlab.com/zygoon/image-garden - # TODO: Switch to a released version for better stability. - type: adhoc - # Use 2GB of RAM and four cores as otherwise we may not have enough memory - # to link the parser. It is better to have more workers than to have one - # big worker with lots of resources. - allocate: ADDRESS "$(QEMU_MEM_OPTION="-m 2048" QEMU_SMP_OPTION="-smp 4" image-garden allocate "$SPREAD_SYSTEM".$(uname -m))" - discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS" - systems: - # All systems except for the one Ubuntu system are marked as manual. - # This way we don't accidentally spin up everything when someone runs - # spread without knowing better. - - opensuse-cloud-15.6: - username: opensuse - password: opensuse - workers: 2 - manual: true # Run only when explicitly named. - environment: - # openSUSE 15 ships very old default python. - PYTHON: /usr/bin/python3.11 - PYTHON_CONFIG: /usr/bin/python3.11-config - - opensuse-cloud-tumbleweed: - username: opensuse - password: opensuse - workers: 4 - manual: true - - debian-cloud-12: - username: debian - password: debian - workers: 4 - manual: true - - debian-cloud-13: - username: debian - password: debian - workers: 4 - manual: true - - ubuntu-cloud-22.04: - username: ubuntu - password: ubuntu - workers: 4 - manual: true - - ubuntu-cloud-24.04: - username: ubuntu - password: ubuntu - workers: 4 - manual: true - - ubuntu-cloud-24.10: - username: ubuntu - password: ubuntu - workers: 4 + garden: + # The garden backend relies on https://gitlab.com/zygoon/image-garden + # TODO: Switch to a released version for better stability. + type: adhoc + # Use just enough RAM to link the parser on a virtual system with two + # cores. Using more cores may easily consume more memory, due to make + # -j$(nproc, used below than a small CI/CD system is typically granted. + # It is better to have more workers than to have one big worker with + # lots of resources. + allocate: ADDRESS "$(QEMU_MEM_OPTION="-m 1536" QEMU_SMP_OPTION="-smp 2" image-garden allocate "$SPREAD_SYSTEM".$(uname -m))" + discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS" + systems: + # All systems except for the one Ubuntu system are marked as + # manual. This way we don't accidentally spin up everything when + # someone runs spread without knowing better. + - opensuse-cloud-tumbleweed: + username: opensuse + password: opensuse + workers: 4 + manual: true + - debian-cloud-12: + username: debian + password: debian + workers: 4 + manual: true + - debian-cloud-13: + username: debian + password: debian + workers: 4 + manual: true + - ubuntu-cloud-22.04: + username: ubuntu + password: ubuntu + workers: 4 + manual: true + - ubuntu-cloud-24.04: + username: ubuntu + password: ubuntu + workers: 4 + manual: true + - ubuntu-cloud-24.10: + username: ubuntu + password: ubuntu + workers: 4 exclude: - - .git - - "*.o" - # Files related to spread and image-garden. - - "*.qcow2" - - "*.iso" - - "*.img" - - "*.log" - - "*.run" - - "*.lock" + - .git + - "*.o" + # Files related to spread and image-garden. + - "*.qcow2" + - "*.iso" + - "*.img" + - "*.log" + - "*.run" + - "*.lock" + - spread-logs + - spread-artifacts # Copy the project to this path on the test system. # This is also available as $SPREAD_PATH. path: /tmp/apparmor prepare: | - # Configure libapparmor but only if a makefile is not already present. - # This makes repeated iteration with -reuse much faster, as the chain of - # invocations of make below are efficient if nothing needs to be done. - if [ ! -f "$SPREAD_PATH"/libraries/libapparmor/Makefile ]; then - ( - cd $SPREAD_PATH/libraries/libapparmor - sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python - ) - fi - # Build libapparmor. - make -C $SPREAD_PATH/libraries/libapparmor -j"$(nproc)" - # Build apparmor_parser. - make -C $SPREAD_PATH/parser -j"$(nproc)" - # Build binary utilities (aa-exec and firends). - make -C $SPREAD_PATH/binutils -j"$(nproc)" - # Build python utilities. - make -C $SPREAD_PATH/utils -j"$(nproc)" + # Configure libapparmor but only if a makefile is not already present. + # This makes repeated iteration with -reuse much faster, as the chain of + # invocations of make below are efficient if nothing needs to be done. + if [ ! -f "$SPREAD_PATH"/libraries/libapparmor/Makefile ]; then + ( + cd $SPREAD_PATH/libraries/libapparmor + sh ./autogen.sh && sh ./configure --prefix=/usr --with-perl --with-python + ) + fi + # Build libapparmor. + make -C $SPREAD_PATH/libraries/libapparmor -j"$(nproc)" + # Build apparmor_parser. + make -C $SPREAD_PATH/parser -j"$(nproc)" + # Build binary utilities (aa-exec and firends). + make -C $SPREAD_PATH/binutils -j"$(nproc)" + # Build python utilities. + make -C $SPREAD_PATH/utils -j"$(nproc)" # In case of failure, include the kernel version in the log. debug-each: | - uname -a + uname -a suites: - tests/unit/: - summary: Unit tests that do not exercise the kernel layer. - tests/regression/: - summary: Regression tests for parser-kernel interaction. - prepare: | - # Spread does not support programmatically generated test variants. - # Ensure that the list baked into tests/regressin/apparmor/task.yaml - # contains all the tests defined in tests/regression/apparmor/Makefile. - echo '$(foreach t,$(TESTS),$(info TEST/$t))' | \ - make -n -f "$SPREAD_PATH"/tests/regression/apparmor/Makefile -f /dev/stdin | \ - grep -F TEST/ | \ - cut -d / -f 2 | \ - tee apparmor-regression-tests.txt - fail=0 - for V in $(cat apparmor-regression-tests.txt); do - if ! grep -xF ' TEST/'"$V"': 1' "$SPREAD_PATH"/tests/regression/apparmor/task.yaml; then - echo "tests/regression/task.yaml: missing test variant: TEST/$V" >&2 - fail=1 - fi - done - if [ "$fail" -ne 0 ]; then - echo "exiting due to missing variants listed above" >&2 - exit 1 - fi + utils/: + summary: Unit tests for the Python utilities. + prepare: | + # Generate apparmor profiles that the tests rely on. + make -C "$SPREAD_PATH"/parser/tst gen_xtrans gen_dbus + # Spread does not support programmatically generated test variants. + # Ensure that the list baked into utils/test/task.yaml contains all + # the files matching utils/test/test-*.py + fail=0 + for V in $SPREAD_PATH/utils/test/test-*.py; do + Vdash="$(basename "$V" | sed -e 's,^test-,,' -e 's,\.py$,,')" + Vunder="$(basename "$V" | sed -e 's,^test-,,' -e 's,\.py$,,' -e 's,-,_,g')" + if ! grep -xF ' TEST/'"$Vunder"': '"$Vdash" "$SPREAD_PATH"/utils/test/task.yaml; then + echo "utils/test/task.yaml: missing test variant: TEST/$Vunder: $Vdash" >&2 + fail=1 + fi + done + if [ "$fail" -ne 0 ]; then + echo "exiting due to missing variants listed above" >&2 + exit 1 + fi + tests/unit/: + summary: Unit tests that do not exercise the kernel layer. + tests/regression/: + summary: Regression tests for parser-kernel interaction. + prepare: | + # Spread does not support programmatically generated test variants. + # Ensure that the list baked into tests/regression/apparmor/task.yaml + # contains all the tests defined in tests/regression/apparmor/Makefile. + echo '$(foreach t,$(TESTS),$(info TEST/$t))' | \ + make -n -f "$SPREAD_PATH"/tests/regression/apparmor/Makefile -f /dev/stdin | \ + grep -F TEST/ | \ + cut -d / -f 2 | \ + tee apparmor-regression-tests.txt + fail=0 + while read -r V; do + if ! grep -xF ' TEST/'"$V"': 1' "$SPREAD_PATH"/tests/regression/apparmor/task.yaml; then + echo "tests/regression/task.yaml: missing test variant: TEST/$V" >&2 + fail=1 + fi + done &2 + exit 1 + fi - # Build all the apparmor regression test programs. - make -C "$SPREAD_PATH"/tests/regression/apparmor -j"$(nproc)" - restore: | - rm -f apparmor-regression-tests.txt + # Build all the apparmor regression test programs. + make -C "$SPREAD_PATH"/tests/regression/apparmor -j"$(nproc)" + restore: | + rm -f apparmor-regression-tests.txt diff --git a/tests/regression/apparmor/task.yaml b/tests/regression/apparmor/task.yaml index 4e6d795c8..e73cd8a83 100644 --- a/tests/regression/apparmor/task.yaml +++ b/tests/regression/apparmor/task.yaml @@ -1,83 +1,129 @@ summary: run all of the apparmor regression test suite # See spread.yaml for the code which verifies that nothing is missing from this list of variants environment: - TEST/aa_exec: 1 - TEST/access: 1 - TEST/attach_disconnected: 1 - TEST/at_secure: 1 - TEST/introspect: 1 - TEST/capabilities: 1 - TEST/changeprofile: 1 - TEST/onexec: 1 - TEST/changehat: 1 - TEST/changehat_fork: 1 - TEST/changehat_misc: 1 - TEST/chdir: 1 - TEST/clone: 1 - TEST/complain: 1 - TEST/coredump: 1 - TEST/deleted: 1 - TEST/e2e: 1 - TEST/environ: 1 - TEST/exec: 1 - TEST/exec_qual: 1 - TEST/fchdir: 1 - TEST/fd_inheritance: 1 - TEST/fork: 1 - TEST/i18n: 1 - TEST/link: 1 - TEST/link_subset: 1 - TEST/mkdir: 1 - TEST/mmap: 1 - TEST/mount: 1 - TEST/mult_mount: 1 - TEST/named_pipe: 1 - TEST/namespaces: 1 - TEST/net_raw: 1 - TEST/open: 1 - TEST/openat: 1 - TEST/pipe: 1 - TEST/pivot_root: 1 - TEST/posix_ipc: 1 - TEST/ptrace: 1 - TEST/pwrite: 1 - TEST/query_label: 1 - TEST/regex: 1 - TEST/rename: 1 - TEST/readdir: 1 - TEST/rw: 1 - TEST/socketpair: 1 - TEST/swap: 1 - TEST/sd_flags: 1 - TEST/setattr: 1 - TEST/symlink: 1 - TEST/syscall: 1 - TEST/sysv_ipc: 1 - TEST/tcp: 1 - TEST/unix_fd_server: 1 - TEST/unix_socket_pathname: 1 - TEST/unix_socket_abstract: 1 - TEST/unix_socket_unnamed: 1 - TEST/unix_socket_autobind: 1 - TEST/unlink: 1 - TEST/userns: 1 - TEST/xattrs: 1 - TEST/longpath: 1 - TEST/nfs: 1 - TEST/xattrs_profile: 1 - TEST/dbus_eavesdrop: 1 - TEST/dbus_message: 1 - TEST/dbus_service: 1 - TEST/dbus_unrequested_reply: 1 - TEST/io_uring: 1 - TEST/exec_stack: 1 - TEST/aa_policy_cache: 1 - TEST/nnp: 1 - TEST/stackonexec: 1 - TEST/stackprofile: 1 + TEST/aa_exec: 1 + TEST/aa_policy_cache: 1 + TEST/access: 1 + TEST/at_secure: 1 + TEST/attach_disconnected: 1 + TEST/capabilities: 1 + TEST/changehat: 1 + TEST/changehat_fork: 1 + TEST/changehat_misc: 1 + TEST/changeprofile: 1 + TEST/chdir: 1 + TEST/clone: 1 + TEST/complain: 1 + TEST/coredump: 1 + TEST/dbus_eavesdrop: 1 + TEST/dbus_message: 1 + TEST/dbus_service: 1 + TEST/dbus_unrequested_reply: 1 + TEST/deleted: 1 + TEST/e2e: 1 + TEST/environ: 1 + TEST/exec: 1 + TEST/exec_qual: 1 + TEST/exec_stack: 1 + TEST/fchdir: 1 + TEST/fd_inheritance: 1 + TEST/fork: 1 + TEST/i18n: 1 + TEST/introspect: 1 + TEST/io_uring: 1 + TEST/link: 1 + TEST/link_subset: 1 + TEST/longpath: 1 + TEST/mkdir: 1 + TEST/mmap: 1 + TEST/mount: 1 + TEST/mult_mount: 1 + TEST/named_pipe: 1 + TEST/namespaces: 1 + TEST/net_raw: 1 + TEST/nfs: 1 + TEST/nnp: 1 + TEST/onexec: 1 + TEST/open: 1 + TEST/openat: 1 + TEST/pipe: 1 + TEST/pivot_root: 1 + TEST/posix_ipc: 1 + TEST/ptrace: 1 + TEST/pwrite: 1 + TEST/query_label: 1 + TEST/readdir: 1 + TEST/regex: 1 + TEST/rename: 1 + TEST/rw: 1 + TEST/sd_flags: 1 + TEST/setattr: 1 + TEST/socketpair: 1 + TEST/stackonexec: 1 + TEST/stackprofile: 1 + TEST/swap: 1 + TEST/symlink: 1 + TEST/syscall: 1 + TEST/sysv_ipc: 1 + TEST/tcp: 1 + TEST/unix_fd_server: 1 + TEST/unix_socket_abstract: 1 + TEST/unix_socket_autobind: 1 + TEST/unix_socket_pathname: 1 + TEST/unix_socket_unnamed: 1 + TEST/unlink: 1 + TEST/userns: 1 + TEST/xattrs: 1 + TEST/xattrs_profile: 1 + # Some tests are currently failing. Those are listed below. For each variant + # listed above, the XFAIL variable contains a list of spread systems where this + # test is expected to fail. + # + # Error: unix_fd_server passed. Test 'ATTACH_DISCONNECTED (attach_disconnected.path rule at /)' was expected to 'fail' + XFAIL/attach_disconnected: opensuse-cloud-tumbleweed debian-cloud-12 debian-cloud-13 ubuntu-cloud-22.04 + # Error: unix_fd_server failed. Test 'fd passing; unconfined client' was expected to 'pass'. Reason for failure 'FAIL - bind failed: Permission denied' + # Error: unix_fd_server failed. Test 'fd passing; confined client w/ rw' was expected to 'pass'. Reason for failure 'FAIL - bind failed: Permission denied' + XFAIL/deleted: opensuse-cloud-tumbleweed debian-cloud-12 debian-cloud-13 + # Error: unix_fd_server failed. Test 'fd passing; confined -> unconfined' was expected to 'pass'. Reason for failure 'FAIL - bind failed: Permission denied' + # Error: unix_fd_server failed. Test 'fd passing; unconfined -> confined' was expected to 'pass'. Reason for failure 'FAIL CLIENT - connect Permission denied' + # Error: unix_fd_server failed. Test 'fd passing; unconfined -> confined (no perm)' was expected to 'pass'. Reason for failure 'FAIL CLIENT - connect Permission denied' + # Error: unix_fd_server failed. Test 'fd passing; confined -> confined' was expected to 'pass'. Reason for failure 'FAIL - bind failed: Permission denied' + XFAIL/unix_fd_server: opensuse-cloud-tumbleweed debian-cloud-12 debian-cloud-13 + # Error: unix_socket failed. Test 'AF_UNIX pathname socket (stream); confined server w/ access (rw)' was expected to 'pass'. Reason for failure 'FAIL - setsockopt (SO_RCVTIMEO): Permission denied' + # Error: unix_socket passed. Test 'AF_UNIX pathname socket (stream); confined server w/ a missing af_unix access (create)' was expected to 'fail' + # Error: unix_socket failed. Test 'AF_UNIX pathname socket (stream); confined client w/ access (rw)' was expected to 'pass'. Reason for failure 'FAIL - setsockopt (SO_RCVTIMEO): Permission denied' + # xpass: AF_UNIX pathname socket (dgram); confined server w/ access (rw) + # Error: unix_socket passed. Test 'AF_UNIX pathname socket (dgram); confined server w/ a missing af_unix access (create)' was expected to 'fail' + # xpass: AF_UNIX pathname socket (dgram); confined client w/ access (rw) + # Error: unix_socket failed. Test 'AF_UNIX pathname socket (seqpacket); confined server w/ access (rw)' was expected to 'pass'. Reason for failure 'FAIL - setsockopt (SO_RCVTIMEO): Permission denied' + # Error: unix_socket passed. Test 'AF_UNIX pathname socket (seqpacket); confined server w/ a missing af_unix access (create)' was expected to 'fail' + # Error: unix_socket failed. Test 'AF_UNIX pathname socket (seqpacket); confined client w/ access (rw)' was expected to 'pass'. Reason for failure 'FAIL - setsockopt (SO_RCVTIMEO): Permission denied' + XFAIL/unix_socket_pathname: opensuse-cloud-tumbleweed debian-cloud-12 debian-cloud-13 +artifacts: + - bash.log + - bash.err execute: | - # Run the shell script that is named after the spread variant we are running - # now. The makefile runs them all sequentially via the "alltests" goal. Here - # we can parallelize it through spread and also have a way to run precisely - # the test we want, especially for debugging. - bash "$SPREAD_VARIANT".sh + # Run the shell script that is named after the spread variant we are running + # now. The makefile runs them all sequentially via the "alltests" goal. Here + # we can parallelize it through spread and also have a way to run precisely + # the test we want, especially for debugging. + if ! bash -x "$SPREAD_VARIANT".sh >bash.log 2>bash.err; then + for xfail in ${XFAIL:-}; do + if [ "$SPREAD_SYSTEM" = "$xfail" ]; then + echo "Ignoring expected failure of $SPREAD_TASK on $SPREAD_SYSTEM" + exit 0 + fi + done + + echo "Test $SPREAD_VARIANT has unexpectedly failed" + echo "Test execution logs are in the files bash.{log.err} and are collected as artifacts" + exit 1 + else + for xfail in ${XFAIL:-}; do + if [ "$SPREAD_SYSTEM" = "$xfail" ]; then + echo "Test $SPREAD_VARIANT has unexpectedly passed" + echo "Test execution logs are in the files bash.{log.err} and are collected as artifacts" + exit 1 + fi + done + fi diff --git a/tests/unit/binutils/task.yaml b/tests/unit/binutils/task.yaml index 33bda2559..a9b0973a3 100644 --- a/tests/unit/binutils/task.yaml +++ b/tests/unit/binutils/task.yaml @@ -1,3 +1,3 @@ summary: Run unit tests of AppArmor binary utilities execute: | - make -C "$SPREAD_PATH/binutils" check + make -C "$SPREAD_PATH"/binutils check diff --git a/tests/unit/libapparmor/task.yaml b/tests/unit/libapparmor/task.yaml index dc0dbaad6..cd45b46ff 100644 --- a/tests/unit/libapparmor/task.yaml +++ b/tests/unit/libapparmor/task.yaml @@ -1,3 +1,3 @@ summary: Run unit tests of libapparmor execute: | - make -C "$SPREAD_PATH/libraries/libapparmor" check + make -C "$SPREAD_PATH"/libraries/libapparmor check diff --git a/tests/unit/parser-tst-caching/task.yaml b/tests/unit/parser-tst-caching/task.yaml index ed729ad9e..8ffa062fa 100644 --- a/tests/unit/parser-tst-caching/task.yaml +++ b/tests/unit/parser-tst-caching/task.yaml @@ -1,3 +1,3 @@ summary: Run apparmor_parser caching test from parser/tst execute: | - make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" caching + make -C "$SPREAD_PATH"/parser/tst -j"$(nproc)" caching diff --git a/tests/unit/parser-tst-dirtest/task.yaml b/tests/unit/parser-tst-dirtest/task.yaml index eb399a263..f68449c88 100644 --- a/tests/unit/parser-tst-dirtest/task.yaml +++ b/tests/unit/parser-tst-dirtest/task.yaml @@ -1,3 +1,3 @@ summary: Run apparmor_parser dirtest test from parser/tst execute: | - make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" dirtest + make -C "$SPREAD_PATH"/parser/tst -j"$(nproc)" dirtest diff --git a/tests/unit/parser-tst-equality/task.yaml b/tests/unit/parser-tst-equality/task.yaml index 5401e97ec..cb45b3562 100644 --- a/tests/unit/parser-tst-equality/task.yaml +++ b/tests/unit/parser-tst-equality/task.yaml @@ -1,7 +1,11 @@ summary: Run apparmor_parser tests from parser/tst # This test is particularly slow. Those values are aimed at running fast enough # on a Raspberry Pi 5, representing a slower-ish computer. +# +# TODO: figure out how to make this test more parallelizable. warn-timeout: 20m kill-timeout: 30m +# Start this task sooner rather than later. +priority: 100 execute: | - make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" equality + make -C "$SPREAD_PATH"/parser/tst -j"$(nproc)" equality diff --git a/tests/unit/parser-tst-error-output/task.yaml b/tests/unit/parser-tst-error-output/task.yaml index f22f91413..8c2ae0e13 100644 --- a/tests/unit/parser-tst-error-output/task.yaml +++ b/tests/unit/parser-tst-error-output/task.yaml @@ -1,3 +1,3 @@ summary: Run apparmor_parser error_output test from parser/tst execute: | - make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" error_output + make -C "$SPREAD_PATH"/parser/tst -j"$(nproc)" error_output diff --git a/tests/unit/parser-tst-minimize/task.yaml b/tests/unit/parser-tst-minimize/task.yaml index 6f1e43951..28e9fb401 100644 --- a/tests/unit/parser-tst-minimize/task.yaml +++ b/tests/unit/parser-tst-minimize/task.yaml @@ -1,3 +1,3 @@ summary: Run apparmor_parser minimize test from parser/tst execute: | - make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" minimize + make -C "$SPREAD_PATH"/parser/tst -j"$(nproc)" minimize diff --git a/tests/unit/parser-tst-parser-sanity/task.yaml b/tests/unit/parser-tst-parser-sanity/task.yaml index 6c5fa34e8..5b6e8196f 100644 --- a/tests/unit/parser-tst-parser-sanity/task.yaml +++ b/tests/unit/parser-tst-parser-sanity/task.yaml @@ -1,7 +1,11 @@ summary: Run apparmor_parser parser_sanity test from parser/tst # This test is particularly slow. Those values are aimed at running fast enough # on a Raspberry Pi 5, representing a slower-ish computer. +# +# TODO: figure out how to make this test more parallelizable. warn-timeout: 20m kill-timeout: 30m +# Start this task sooner rather than later. +priority: 100 execute: | - make -C "$SPREAD_PATH/parser/tst" -j"$(nproc)" parser_sanity + make -C "$SPREAD_PATH"/parser/tst -j"$(nproc)" parser_sanity diff --git a/tests/unit/parser/task.yaml b/tests/unit/parser/task.yaml index 034368346..0eb1a7170 100644 --- a/tests/unit/parser/task.yaml +++ b/tests/unit/parser/task.yaml @@ -1,16 +1,16 @@ summary: Run apparmor_parser unit tests from parser/ details: | - The parser has a number of different tests. Those are all represented as - spread task variants so that they are directly visisble and runnable. + The parser has a number of different tests. Those are all represented as + spread task variants so that they are directly visible and runnable. environment: - TEST/tst_regex: 1 - TEST/tst_misc: 1 - TEST/tst_symtab: 1 - TEST/tst_variable: 1 - TEST/tst_lib: 1 + TEST/tst_regex: 1 + TEST/tst_misc: 1 + TEST/tst_symtab: 1 + TEST/tst_variable: 1 + TEST/tst_lib: 1 prepare: | - # The test relies on make to build a binary. - make -C "$SPREAD_PATH/parser" -j"$(nproc)" "$SPREAD_VARIANT" + # The test relies on make to build a binary. + make -C "$SPREAD_PATH"/parser -j"$(nproc)" "$SPREAD_VARIANT" execute: | - cd "$SPREAD_PATH"/parser - ./"$SPREAD_VARIANT" + cd "$SPREAD_PATH"/parser + ./"$SPREAD_VARIANT" diff --git a/tests/unit/utils/task.yaml b/tests/unit/utils/task.yaml deleted file mode 100644 index cac37f28f..000000000 --- a/tests/unit/utils/task.yaml +++ /dev/null @@ -1,10 +0,0 @@ -summary: Run unit tests of python utilities -# FIXME: Exclude openSUSE as the test depends on python3-ttkthemes -# which are not packaged in the distribution. -systems: - - -opensuse-* -prepare: | - # Generate the profiles that the test relies on. - make -C "$SPREAD_PATH"/parser/tst gen_xtrans gen_dbus -execute: | - make -C "$SPREAD_PATH"/utils check diff --git a/utils/test/task.yaml b/utils/test/task.yaml new file mode 100644 index 000000000..1595f8e6b --- /dev/null +++ b/utils/test/task.yaml @@ -0,0 +1,50 @@ +summary: Run "make check" broken down by test name +systems: + - -opensuse-cloud-15.6 +environment: + # FIXME: Spread disallows dashes in variant names, most likely as a bug. + # As such we map the all_under_score variant name to actual test name. + TEST/aa_cli_bootstrap: aa-cli-bootstrap + TEST/aa_decode: aa-decode + TEST/aa_easyprof: aa-easyprof + TEST/aa_notify: aa-notify + TEST/aa: aa + TEST/aare: aare + TEST/abi: abi + TEST/alias: alias + TEST/all: all + TEST/baserule: baserule + TEST/boolean: boolean + TEST/capability: capability + TEST/change_profile: change_profile + TEST/common: common + TEST/config: config + TEST/dbus: dbus + TEST/example: example + TEST/file: file + TEST/include: include + TEST/io_uring: io_uring + TEST/libapparmor_test_multi: libapparmor-test_multi + TEST/logparser: logparser + TEST/logprof: logprof + TEST/minitools: minitools + TEST/mount: mount + TEST/mqueue: mqueue + TEST/network: network + TEST/notify: notify + TEST/parser_simple_tests: parser-simple-tests + TEST/pivot_root: pivot_root + TEST/profile_list: profile-list + TEST/profile_storage: profile-storage + TEST/profiles: profiles + TEST/ptrace: ptrace + TEST/regex_matches: regex_matches + TEST/rlimit: rlimit + TEST/severity: severity + TEST/signal: signal + TEST/translations: translations + TEST/unix: unix + TEST/userns: userns + TEST/variable: variable +execute: | + make --warn-undefined-variables check-one-test-"$TEST" diff --git a/utils/vim/task.yaml b/utils/vim/task.yaml new file mode 100644 index 000000000..dfe75c2d7 --- /dev/null +++ b/utils/vim/task.yaml @@ -0,0 +1,3 @@ +summary: Run "make check" +execute: | + make --warn-undefined-variables check From d27377a62f8847da176125d4ee45d7a63f31d920 Mon Sep 17 00:00:00 2001 From: Zygmunt Krynicki Date: Thu, 5 Dec 2024 13:14:03 +0100 Subject: [PATCH 5/5] Document spread tests in README.md Signed-off-by: Zygmunt Krynicki --- README.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/README.md b/README.md index a9310b6a6..d0be320d8 100644 --- a/README.md +++ b/README.md @@ -206,6 +206,30 @@ usage and how to update and add tests. Below is a quick overview of their location and how to run them. +Using spread with local virtual machines +---------------------------------------- + +It may be convenient to use the spread tool to provision and run the test suite +in an ephemeral virtual machine. This allows testing in isolation from the +host, as well as testing across different commonly used distributions and their +real kernels. + +```sh +sudo apt install git golang whois ovmf genisoimage qemu-utils qemu-system +go install github.com/snapcore/spread/cmd/spread@latest +git clone https://gitlab.com/zygoon/image-garden +make -C image-garden +sudo make -C image-garden install +image-garden make ubuntu-cloud-24.10.x86_64.run +cd $APPARMOR_PATH +git clean -xdf +~/go/bin/spread -artifacts ./spread-artifacts -v ubuntu-cloud-24.10 +# or ~/go/bin/spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure +``` + +Running the `run_spread.sh` script, with `spread` on `PATH` will run all the +tests across several supported systems (Debian, Ubuntu and openSUSE). + Regression tests ---------------- For details on structure and adding tests, see