tests: Add support for running system tests under retis.

Retis is very useful for debugging our system tests or debugging
kernel issues through our system tests.  This change adds a convenient
way to run any kernel system test with the retis capture on the
background.  E.g.:

  make check-kernel OVS_TEST_WITH_RETIS=yes TESTSUITEFLAGS='167 -d'

Retis 1.5 is required, since we're using ifdump profile, and it also
will mount debugfs for us in case of running in a different namespace.
It should be available in $PATH.

In addition to just capturing the retis.data, we're also running the
capture with --print to print all the events as they appear, and
producing the sorted output in the end.  This makes it easier to work
across systems with different versions of retis and saves time for
running the sort manually.  The raw data is still available for
advanced processing, if needed.

Not specifying any particular collector, capturing everything that's
enabled by default.  OVS tracking is turned on by default.

Since OVS tracking is used, it's required to start retis after the
kernel datapath is created, otherwise it will fail to obtain the map
of upcall PIDs.  That's why we need to start it after the bridge is
created.

Only adding support for kernel-related test suites for now.  For
userspace test suites it may also be useful at some point, but
currently that requires running without --ovs-track and isn't too
important.

Startup of the retis capture adds significant amount of time to each
test, so not running it by default.

Link: https://github.com/retis-org/retis
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>

Documentation/topics/testing.rst

@@ -455,6 +455,28 @@ datapath testsuite.
 
   https://git.kernel.org/pub/scm/network/iproute2/iproute2.git
 
+It is also possible to run `retis`_ capture along with the `check-kernel` and
+`check-offloads` tests by setting `OVS_TEST_WITH_RETIS` environment variable
+to 'yes'.  This can be useful for debugging the test cases.  For example, the
+following command can be used to run the test 167 under `retis`::
+
+    $ make check-kernel OVS_TEST_WITH_RETIS=yes TESTSUITEFLAGS='167 -d'
+
+After the test is completed, the following data will be available in the test
+directory:
+
+* `retis.err` - standard error stream of the `retis collect`.
+* `retis.log` - standard output of the `retis collect`, contains all captured
+  events in the order they appeared.
+* `retis.data` - raw events collected by retis, `retis sort` or other commands
+  can be used on this file for further analysis.
+* `retis.sorted` - text file containing the output of `retis sort` executed on
+  the `retis.data`, for convenience.
+
+Requires retis version 1.5 or newer.
+
+.. _retis: https://github.com/retis-org/retis
+
 .. _testing-static-analysis:
 
 Static Code Analysis

NEWS

@@ -29,6 +29,9 @@ Post-v3.5.0
    - ovs-tcpdump:
      * Update the --mirror-to option, adding support for specifying an
        existing port as a mirror interface.
+   - Added support for running tests from 'check-kernel' and 'check-offloads'
+     system test targets under retis by setting OVS_TEST_WITH_RETIS=yes.
+     See the 'Testing' section of the documentation for more details.
    - Updated documentation to remove most documentation of kernel module
      that was previously part of the OVS distribution. It was removed from
      the OVS distribution in the 3.0 release and is no longer present in

tests/ovs-macros.at

@@ -360,6 +360,16 @@ m4_ifndef([AT_FAIL_IF],
     [AT_CHECK([($1) \
     && exit 99 || exit 0], [0], [ignore], [ignore])])])
 
+dnl Start retis to track all the traffic passing through OVS.
+m4_define([RETIS_CHECK_AND_RUN],
+  [if test "$OVS_TEST_WITH_RETIS" = yes && retis --version > /dev/null; then
+       on_exit 'retis sort --utc retis.data > retis.sorted'
+       OVS_DAEMONIZE([retis -p ifdump collect --utc --allow-system-changes \
+                          --ovs-track --out --print 2>retis.err 1>retis.log],
+                     [retis.pid])
+       OVS_WAIT_UNTIL([grep -q 'loaded' retis.err])
+   fi])
+
 dnl Add a rule to always accept the traffic.
 dnl The first argument to this macro should be the command to run:
 dnl iptables or ip6tables

tests/system-kmod-macros.at

@@ -29,6 +29,9 @@ m4_define([OVS_TRAFFIC_VSWITCHD_START],
    _OVS_VSWITCHD_START([], [$3])
    dnl Add bridges, ports, etc.
    AT_CHECK([ovs-vsctl -- _ADD_BR([br0]) -- $1 m4_if([$2], [], [], [| uuidfilt])], [0], [$2])
+
+   dnl Start retis capture if requested.
+   RETIS_CHECK_AND_RUN()
 ])
 
 # OVS_TRAFFIC_VSWITCHD_STOP([ALLOWLIST], [extra_cmds])

tests/system-offloads-testsuite-macros.at

@@ -27,6 +27,9 @@ m4_define([OVS_TRAFFIC_VSWITCHD_START],
    _OVS_VSWITCHD_START([], [-- set Open_vSwitch . other_config:hw-offload=true $3])
    dnl Add bridges, ports, etc.
    AT_CHECK([ovs-vsctl -- _ADD_BR([br0]) -- $1 m4_if([$2], [], [], [| uuidfilt])], [0], [$2])
+
+   dnl Start retis capture if requested.
+   RETIS_CHECK_AND_RUN()
 ])
 
 # Macro to exclude tests that will fail with TC offload enabled.