- Tests defined in utils/test are now described by a task.yaml in the same
directory and can run concurrently across many machines.
- Tests for utils/ are now executed on openSUSE Tumbleweed since ttk themes is
no longer a hard dependency in master.
- Tests no longer run on openSUSE Leap 15.6 due to the age of default
Python (3.6) and gcc/g++. The tight integration with SWIG which does
not seem to support other Python versions very well. Perl hard-codes
old GCC for extension modules. The upcoming openSUSE Leap 16 should be
a viable target. In the meantime we can still test everything through
rolling-release Tumbleweed.
- Formatting of YAML files is now more uniform, at four spaces per tab.
- The run-spread.sh script is now in the root of the tree. The script allows
running all spread tests sequentially on one system, while collecting logs
and artifacts for convenient analysis after the fact.
- All systems are adjusted to run _four_ workers in parallel with _two_ virtual
cores each and equipped with 1.5GB of virtual memory. This aims to best
utilize the capacity of a typical CI worker with two to four cores and about
8GB of available memory.
- Failing tests are marked as such, so that as a whole the entire spread suite
can pass and be useful at catching regressions.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit 1df91e2c8c)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Compared to v1 the following improvements have been made:
- The cost of installing packages have been shifted from each startup to image
preparation phase, thanks to the integration of custom cloud-init profiles
into image-garden. This has dramatic impact on iteration time while also
entirely removing requirement to be online to run once a prepared image is
available.
- Support for running on Google Compute Engine has been removed since it would
not be able to use cloud-init the same way would currently only complicate
setup.
- The number of workers have been tuned for local iteration, aiming for
comfortable work with 16GB of memory on the host. Once CI/CD pipeline
support is introduced I will add a dedicated entry so that resources are
utilized well both locally and when running in CI.
- The set of regression tests listed in tests/regression/apparmor/task.yaml is
now cross-checked so introduction of a new test to the makefile there is
automatically flagged and causes spread to fail with a clear message.
- The task tests/unit/utils has been improved to generate profiles. Thanks to
Christian Boltz for explaining this relationship between tests.
- A number of comments have been improved and cleaned up for readability,
accuracy and sometimes better grammar.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit c95ac4d350)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Spread is a full-system, or integration test suite runner initially developed
to test snapd. Over time it has spread to other projects where it provides a
structured way to organize, run and debug complex full-system interactions.
Spread is documented on https://github.com/canonical/spread and is used in
production since late 2016.
Spread has a notion of backends which are responsible for allocating and
discarding test machines. For the purpose of running AppArmor regression tests,
I've combined spread with my own tool, image garden. The tool provides
off-the-shelf images, constructed on-the-fly from freely available images, and
makes them easily available to spread.
The reason for doing it this way is so that using non-free cloud systems is not
required and anyone can repeat the test process locally, on their own computer.
Vanilla spread is somewhat limited to x86-64 systems but the way I've used it
here makes it equally possible to test x86_64 *and* aarch64 systems. I've done
most of the development on an ARM single-board-computer running on my desk.
Spread requires a top-level spread.yaml file and a collection of task.yaml
files that describe individual tasks (for us, those are just tests). Tasks have
no implied dependency except that to reach a given task, spread will run all
the _prepare_ statements leading to that task, starting from the project, test
suite and then task. With proper care one can then run a specific individual
test with a one-line command, for example:
```
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
```
This will prepare a fresh ubuntu-cloud-24.04 system (matching the CPU
architecture of the host), copy the project tree into the test machine, install
all the build dependencies, build all the parts of apparmor and then run one
specific variant of the regression test, namely the at_secure program.
Importantly the same test can also run on, say debian-cloud-13 (Debian Trixie),
but also, if you have a Google cloud account, on Google Compute Engine or in
one of the other backends either built into spread or available as a fork of
spread or as a helper for ad-hoc backend. Spread can also create more than one
worker per system and distribute the tests to all of the available instances.
In no way are we locking ourselves out of the ability to run our test suite on
our target of choice.
Spread has other useful switches, such as:
- `-reuse` for keeping machines around until discarded with -discard
- `-resend` for re-sending updated copy of the project (useful for -reuse)
- `-debug` for starting an interactive shell on any failure
- `-shell` for starting an interactive shell instead of the `execute` phase
This first patch contains just the spread elements, assuming that both spread
and image-garden are externally installed. A GitLab continuous integration
installing everything required and running a subset of tests will follow
shortly.
I've expanded the initial selection of systems to allow running all the tests
on several versions of Ubuntu, Debian and openSUSE, mainly as a sanity check
but also to showcase how practical spread is at covering real-world systems.
A number of systems and tests are currently failing:
- garden:debian-cloud-12:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-12:tests/regression/apparmor:deleted
- garden:debian-cloud-12:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-12:tests/regression/apparmor:unix_socket_pathname
- garden:debian-cloud-13:tests/regression/apparmor:attach_disconnected
- garden:debian-cloud-13:tests/regression/apparmor:deleted
- garden:debian-cloud-13:tests/regression/apparmor:unix_fd_server
- garden:debian-cloud-13:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-15.6:tests/regression/apparmor:deleted
- garden:opensuse-cloud-15.6:tests/regression/apparmor:e2e
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-15.6:tests/regression/apparmor:unix_socket_pathname
- garden:opensuse-cloud-15.6:tests/regression/apparmor:xattrs_profile
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:attach_disconnected
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:deleted
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_fd_server
- garden:opensuse-cloud-tumbleweed:tests/regression/apparmor:unix_socket_pathname
- garden:ubuntu-cloud-22.04:tests/regression/apparmor:attach_disconnected
In addition, only on openSUSE, I've skipped the entire test suite of the utils
directory, as it requires python3 ttk themes, which I cannot find in packaged
form.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit cc04181578)
Signed-off-by: John Johansen <john.johansen@canonical.com>
The new check-one-test-% pattern rule allows running individual test scripts.
This allows them to be tested in parallel across many Make worker threads or
across many distinct machines with spread.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit 9588b06e0f)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Tests #466 but is marked as expected fail due to that bug not being resolved.
Depends on !1441 which adds the xfail infrastructure to the parser equality testing framework, and should be rebased on top of master once that MR is merged.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1443
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
(cherry picked from commit e1d8bf1888)
currently the equality tests require the tests to PASS as known equality
or inequality. Add the ability to add tests that are a known problem
and are expected to fail the equality, or inequality test.
This is done by using
verify_binary_xequality
verify_binary_xinequality
This allows new tests to be added to document a known issue, without
having to develop the fix for the issue. The use of this facility
is expected to be temporary, so any test marked as xequality or
xinequality will be noisy but not fail the other tests until they
are fixed, at which point they will cause the tests to fail to
force them to be updated to the correct equality or inequality
test.
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1441
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
(cherry picked from commit 53e322b755)
currently the equality tests require the tests to PASS as known equality
or inequality. Add the ability to add tests that are a known problem
and are expected to fail the equality, or inequality test.
This is done by using
verify_binary_xequality
verify_binary_xinequality
This allows new tests to be added to document a known issue, without
having to develop the fix for the issue. The use of this facility
is expected to be temporary, so any test marked as xequality or
xinequality will be noisy but not fail the other tests until they
are fixed, at which point they will cause the tests to fail to
force them to be updated to the correct equality or inequality
test.
Signed-off-by: John Johansen <john.johansen@canonical.com>
(cherry picked from commit b81ea65c1c)
Signed-off-by: John Johansen <john.johansen@canonical.com>
The regression test suite uses root with capabilities restricted in
several tests. This can cause the test suite to fail in weird and
confusing ways.
Add a test to check for DAC permissiosns from / to the testsuite
and abort running the tests with an error message if DAC permissions
are going to cause the test suite to fail.
Currently the test is pretty basic, but is better than nothing.
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1411
Approved-by: Georgia Garcia <georgia.garcia@canonical.com>
Merged-by: John Johansen <john@jjmx.net>
(cherry picked from commit c5b17d85ea)
The regression test suite uses root with capabilities restricted in
several tests. This can cause the test suite to fail in weird and
confusing ways.
Add a test to check for DAC permissiosns from / to the testsuite
and abort running the tests with an error message if DAC permissions
are going to cause the test suite to fail.
Currently the test is pretty basic, but is better than nothing.
Signed-off-by: John Johansen <john.johansen@canonical.com>
(cherry picked from commit 82e4b4ba00)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Using this version of make:
```
GNU Make 4.2.1
Built for x86_64-suse-linux-gnu
```
I'm not entirely sure why but the alternative syntax I've used works correctly.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit 4caf0aff81)
Signed-off-by: John Johansen <john.johansen@canonical.com>
This fixes the test to pass on openSUSE Tumbleweed, where the small size
prevented alloction of an inode for the `lost+found` directory:
```
garden:opensuse-cloud-tumbleweed .../tests/regression/apparmor# mkfs.ext2 -F -m 0 -N 10 /tmp/sdtest.32929-21402-6x826m/image.ext3
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 512 1k blocks and 8 inodes
Allocating group tables: done
Writing inode tables: done
ext2fs_mkdir: Could not allocate inode in ext2 filesystem while creating /lost+found
```
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit 32ee85cef8)
Signed-off-by: John Johansen <john.johansen@canonical.com>
This fixes an error with Python 3.11:
```
test/test-parser-simple-tests.py:420:21: E502 the backslash is redundant between brackets
```
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit 92fcdcab9e)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Which is technically not POSIX and command -v works everywhere. This fixes
building and running the test suite on openSUSE Tumbleweed.
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit 4b0adc63f5)
Signed-off-by: John Johansen <john.johansen@canonical.com>
On a test system without bison installed, make setup fails with:
/bin/sh: 1: bison: not found
/bin/sh: 1: test: -ge: unexpected operator
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
(cherry picked from commit f58fe9cd52)
Signed-off-by: John Johansen <john.johansen@canonical.com>
The file was quoted with the following space, making the test broken.
Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
(cherry picked from commit 8c16fb2700)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Match Flags convert output to hex but don't restore after outputting
the flag resulting in following numbers being hex encoded. This
results in dumps that can be confusing eg.
rule: \d2 -> \x2 priority=1001 (0x4/0)< 0x4>
rule: \d7 -> \a priority=3e9 (0x4/0)< 0x4>
rule: \d10 -> \n priority=3e9 (0x4/0)< 0x4>
rule: \d9 -> \t priority=3e9 (0x4/0)< 0x4>
rule: \d14 -> \xe priority=1001 (0x4/0)< 0x4>
where priority=3e9 is the hex encoded priority 1001.
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1419
Approved-by: Maxime Bélair <maxime.belair@canonical.com>
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
(cherry picked from commit 2e77129e15)
Match Flags convert output to hex but don't restore after outputting
the flag resulting in following numbers being hex encoded. This
results in dumps that can be confusing eg.
rule: \d2 -> \x2 priority=1001 (0x4/0)< 0x4>
rule: \d7 -> \a priority=3e9 (0x4/0)< 0x4>
rule: \d10 -> \n priority=3e9 (0x4/0)< 0x4>
rule: \d9 -> \t priority=3e9 (0x4/0)< 0x4>
rule: \d14 -> \xe priority=1001 (0x4/0)< 0x4>
where priority=3e9 is the hex encoded priority 1001.
Signed-off-by: John Johansen <john.johansen@canonical.com>
(cherry picked from commit a31343c5f7)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Most `tests/regression/apparmor/*.sh` scripts contain
. $bin/prologue.inc
This will explode if one of the parent directories contains a space.
Minimized reproducer:
```
# cat test.sh
pwd=`dirname $0`
pwd=`cd $pwd ; /bin/pwd`
bin=$pwd
echo "pwd: $bin"
. $bin/prologue.inc
# ./test.sh
pwd: /tmp/foo bar
./test.sh: line 9: /tmp/foo: No such file or directory
```
Notice that test.sh tries to source `/tmp/foo` instead of `/tmp/foo bar/prologue.inc`.
The fix is to quote the prologue.inc path:
. "$bin/prologue.inc"
While on it, also fix other uses of $bin - directly and indirectly - by quoting them.
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1418
Approved-by: Ryan Lee <rlee287@yahoo.com>
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
(cherry picked from commit a422d2ea17)
... to avoid issues with spaces in a parent directory's name.
"Indirect uses" means usage of $bin via another variable, for example
`foo=$bin/whatever`
(cherry picked from commit 55db4af979)
Signed-off-by: John Johansen <john.johansen@canonical.com>
... to avoid issues with spaces in a parent directory's name.
(cherry picked from commit 22cf88b7c7)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Most `tests/regression/apparmor/*.sh` scripts contain
. $bin/prologue.inc
This will explode if one of the parent directories contains a space.
Minimized reproducer:
```
pwd=`dirname $0`
pwd=`cd $pwd ; /bin/pwd`
bin=$pwd
echo "pwd: $bin"
. $bin/prologue.inc
pwd: /tmp/foo bar
./test.sh: line 9: /tmp/foo: No such file or directory
```
Notice that test.sh tries to source `/tmp/foo` instead of `/tmp/foo bar/prologue.inc`.
The fix - as done in this commit - is to quote the prologue.inc path:
. "$bin/prologue.inc"
(cherry picked from commit e1972eb22f)
Signed-off-by: John Johansen <john.johansen@canonical.com>
Instead of encoding permissions in the accept and accept2 tables
extended perms uses a permissions table and accept becomes an index
into the table.
Add the ability to dump the permissions table so that it can be
compared and debugged.
Signed-off-by: John Johansen <john.johansen@canonical.com>
(cherry picked from commit 45964d34e7)
Signed-off-by: John Johansen <john.johansen@canonical.com>
The chfa dump is missing information about the accept2 entry. The
accept2 information is necessary to help with debugging state machine
builds as accept2 is used to store quiet and audit information in the
old format or conditional information in the extended perms format.
Signed-off-by: John Johansen <john.johansen@canonical.com>
(cherry picked from commit 00dedf10ad)
Signed-off-by: John Johansen <john.johansen@canonical.com>
The Makefile is missing some of its .h depenedncies causing compiles
to either fail or worse result in bad builds when rebuilding in an
already built tree.
Move the header dependencies into a variable and use it for each
target. While some targets don't need every include as a dependency
and this will result in unnecessary rebuilds in some cases, it makes
the Makefile cleaner, easier to maintain and makes sure a dependency
isn't accidentally missed.
Signed-off-by: John Johansen <john.johansen@canonical.com>
(cherry picked from commit 7cc7f47424)
Signed-off-by: John Johansen <john.johansen@canonical.com>
The parser recently changed how/where deny information is applied.
commit 1fa45b7c1 ("parser: dfa minimization prepare for extended
permissions") removed the implicit filtering of explicit denies during
the minimization pass. The implicit clear allowed the explicit
information to be carried into the minimization pass and merged with
implicit denies. The end result being a minimized dfa with the explicit
deny information available to be applied post minimization, and
then dropped later at permission encoding in the accept entries.
Extended permission however enable carrying explicit deny information
into the kernel to fix certain bugs like complain mode not being
able to distinguish between implicit and explicit deny rules (ie.
deny rules get ignored in complain mode). However keeping explicit
deny information when unnecessary result in a larger state machine
than necessary and slower compiles.
commit 179c1c1ba ("parser: fix minimization check for filtering_deny")
Moved the explicit apply_and_clear_deny() pass to before minimization
to restore mnimization's ability to create a minimized dfa with
explicit and implicit deny information merged but this also cleared
the explicit deny information that used to be carried through
minimization. This meant that when the deny information was applied
post minimization it resulted in the audit and quiet information
being cleared.
This resulted in the query_label tests failing as they are checking
for the expected audit infomation in the permissions.
Fixes: 179c1c1ba ("parser: fix minimization check for filtering_deny")
Bug: https://gitlab.com/apparmor/apparmor/-/issues/461
Signed-off-by: John Johansen <john.johansen@canonical.com>
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1408
Approved-by: Ryan Lee <rlee287@yahoo.com>
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
(cherry picked from commit eb365b374d)
Signed-off-by: John Johansen <john.johansen@canonical.com>
If a user specifies a non-existing file to merge into the profiles
(`aa-mergeprof /file/not/found`), this results in a backtrace showing an
AppArmorBug because that file unsurprisingly doesn't end up in the
active_profiles filelist.
Handle this more gracefully by adding a read_error_fatal parameter to
read_profile() that, if set, forwards the exception. With that,
aa-mergeprof doesn't try to list the profiles in this non-existing file.
Note that all other callers of read_profile() continue to ignore read
errors, because aborting just because a single file in /etc/apparmor.d/
(for example a broken symlink) isn't readable would be a bad idea.
This bug was introduced in 4e09f315c3, therefore I propose this patch for 3.0..master
MR: https://gitlab.com/apparmor/apparmor/-/merge_requests/1403
Approved-by: John Johansen <john@jjmx.net>
Merged-by: John Johansen <john@jjmx.net>
(cherry picked from commit 5ebbe788ea)
Signed-off-by: John Johansen <john.johansen@canonical.com>
