Unlike the C IDL code, the Python version still monitors the
_Server DB with "monitor" instead of "monitor_cond". This results
in receiving an entire Database row every time the "index" value
is updated which includes the 'schema' column. Using "monitor_cond"
will result in "update2" notifications which just include the
changed "index" value.
Unlike the C IDL, the Python IDL requires a SchemaHelper object
to instantiate the IDL, leaving it to the user of the library to
call "get_schema" themselves. Since the Python IDL did not have
support for retrieving the schema automatically and did not have
a state for doing so, instead of transitioning on an error response
from retrieving the _Server schema to requesting the "data" schema,
this moves directly to monitoring the "data" DB.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
'pkg_resources' module is deprecated and no longer available in newer
versions of python, so pytest tests are skipped:
DeprecationWarning: pkg_resources is deprecated as an API.
See https://setuptools.pypa.io/en/latest/pkg_resources.html
Unfortunately, there is no direct replacement for it and functionality
is scattered between different packages.
Using a new standard library importlib.metadata to find installed
packages and their versions. Using packaging.requirements to parse
lines from the requirements file and compare versions. This covers
all we need.
The 'packaging' is a project used by pip and a dependency for many
other libraries, so should be available for any supported verison of
python. 'importlib' was introduced in python 3.8. Since we support
older versions of python and 'packaging' is not part of the standard
library, checking that import is possible and falling back to
'pkg_resources' if needed. We may remove the fallback when we stop
supporting python below 3.8.
Even though 'packaging' is a common dependency, added to the test
requirements so it will not be missed in CI.
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Before the patch, the size of the backlog depended on the type of socket
(UNIX vs INET) as well as on the language (C vs Python), specifically:
- python used backlog size = 10 for all sockets;
- C used 64 for UNIX sockets but 10 for INET sockets.
This consolidates the values across the board. It effectively bumps the
number of simultaneous connections to python unixctl servers to 64. Also
for INET C servers too.
The rationale to do it, on top of consistency, is as follows:
- fmt_pkt in ovn testsuite is limited by python server listen backlog,
and as was found out when adopting the tool, it is sometimes useful to
run lots of parallel calls to fmt_pkt unixctl server in some tests.
(See [1] for example.)
- there is a recent report [2] on discuss@ ML where the reporter noticed
significant listen queue overflows in some scenarios (large openstack
deployments; happens during leader transition when hundreds of neutron
nodes - with dozens of neutron api workers each - simultaneously
reconnect to the same northbound leader.) Note: While there is no
clear indication that this backlog size bump would resolve the
reported issues, it would probably help somewhat.
[1] 0baca3e519
[2] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-April/053049.html
Signed-off-by: Ihar Hrachyshka <ihrachys@redhat.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
After running ovs-tcpdump and inputs multiple CTRL+C, the program will
raise the following exception.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/bin/ovs-tcpdump", line 421, in cleanup_mirror
ovsdb = OVSDB(db_sock)
File "/usr/bin/ovs-tcpdump", line 168, in __init__
OVSDB.wait_for_db_change(self._idl_conn) # Initial Sync with DB
File "/usr/bin/ovs-tcpdump", line 155, in wait_for_db_change
while idl.change_seqno == seq and not idl.run():
The default handler of SIGINT is default_int_handler, so it was not
registered to the signal handler. When received CTRL+C again, the program
was broken, and calling hook could not be executed completely.
Signed-off-by: Daniel Ding <danieldin186@gmail.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
In general, most actions must be lists since the keys can be repeated.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
Avoid code duplication by moving the section testing code to its own
function.
Also, verify the length of the kv_list meets the expectations.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
Add the sample action to those that can be called in nested actions
(such as clone).
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
Parsing of info and matches was being tested as generic k-v parsing.
Also verify we don't find any unexpected field.
Also, verify the length of the kv_list meets the expectations.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
Currently python-ovs claims to be "db change aware" but does not
parse the "monitor_canceled" notification. Transactions can continue
being made, but the monitor updates will not be sent. This handles
monitor_cancel similarly to how ovsdb-cs currently does.
Fixes: c39751e44539 ("python: Monitor Database table to manage lifecycle of IDL client.")
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
Currently, if you use the python/test_requirements.txt file to
set up your test environment the MFEX tests will be skipped due
to the Scapy package not being included. This is fixed as part
of this patch.
The test case change will make sure the 'MFEX Configuration' test
will run without the need for Scapy and its auto-generated tests.
In addition, we exclude the traffic-related MFEX tests from running
on GitHub actions due to limited resources.
Signed-off-by: Eelco Chaudron <echaudro@redhat.com>
Acked-by: Simon Horman <horms@ovn.org>
This patch adds 'make check-kernel' to the GitHub actions ci.
However, to do this, some additional changes were needed. First,
some of the missing test and package dependencies had to be added.
Finally, we added an option to the GitHub run matrix that allows
the tests to be split up, to avoid lengthy single test runs.
Signed-off-by: Eelco Chaudron <echaudro@redhat.com>
Acked-by: Simon Horman <horms@ovn.org>
E721: "do not compare types, for exact checks use `is` / `is not`, for
instance checks use `isinstance()`"
This fixes `make flake8-check` target when running with
pycodestyle>=1.2.
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Ihar Hrachyshka <ihrachys@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
The unixctl implementation in Python has been split into three parts in
the past. During this process the UnixctlClient was duplicated, in
python/ovs/unixctl/client.py and python/ovs/unixctl/server.py. This
patch removes the duplicate from the latter.
Fixes: 53cf9963ccc6 ("python: Break unixctl implementation into registry, client, and server.")
Signed-off-by: Jakob Meng <code@jakobmeng.de>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
While sending a reply to the monitor_cond_since request, server
includes the last transaction ID. And it sends new IDs with each
subsequent update. Current implementation doesn't use the one
supplied with a monitor reply, and only takes into account IDs
provided with monitor updates. That may cause various issues:
1. Performance: During initialization, the last-id is set to zero.
If re-connection will happen after receiving a monitor reply,
but before any monitor update, the client will send a new
monitor request with an all-zero last-id and will re-download
the whole database again.
2. Data inconsistency: Assuming one of the clients sends a
transaction, but our python client disconnects before receiving
a monitor update for this transaction. The last-id will point
to a database state before this transaction. On re-connection,
this last-id will be sent and the monitor reply will contain
a diff with a new data from that transaction. But if another
disconnection happens right after that, on second re-connection
our python client will send another monitor_cond_since with
exactly the same last-id. That will cause receiving the same
set of updates again. And since it's an update2 message with
a diff of the data, the client will remove previously applied
result of the transaction. At this point it will have a
different database view with the server potentially leading
to all sorts of data inconsistency problems.
Fix that by always updating the last-id from the latest monitor
reply.
Fixes: 46d44cf3be0d ("python: idl: Add monitor_cond_since support.")
Acked-by: Simon Horman <horms@ovn.org>
Acked-by: Han Zhou <hzhou@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Quoting Paul Ganssle, setuptools maintainer:
* The setuptools project has stopped maintaining all direct
invocations of setup.py years ago, and distutils is deprecated.
There are undoubtedly many ways that your setup.py-based system is
broken today, even if it's not failing loudly or obviously.
* Direct invocations of setup.py cannot bootstrap their own
dependencies, and so some CLI is necessary for dependency
management.
* The setuptools project no longer wants to provide any public CLI,
and will be actively removing the existing interface (though the
time scale for this is long).
* PEP 517, 518 and other standards-based packaging are the future of
the Python ecosystem and a lot of progress has been made on making
this upgrade seamless.
As described in the recommendations in the end of the article: `python3
setup.py sdist` should be replaced by `python3 -m build --sdist`.
Link: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summary
Signed-off-by: Robin Jarry <rjarry@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
setup.py upload is now deprecated. When used, pypi.org returns an error:
Upload failed (400): Invalid value for blake2_256_digest. Error: Use
a valid, hex-encoded, BLAKE2 message digest.
Use twine which is the recommended replacement tool to upload on
pypi.org.
Link: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summary
Reported-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Robin Jarry <rjarry@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
The python/build folder contents are completely unrelated to the ovs
python bindings. These files are only used during the build for various
subsystems (docs, man pages, code generation, etc.).
Having that folder in that location prevents from running:
cd python && python3 -m build
Which is a way to generate PEP517 compatible source archives and binary
wheel packages.
Rename that folder to ovs_build_helpers which is more explicit. Update
all imports accordingly.
Link: https://peps.python.org/pep-0517/
Link: https://pypi.org/project/build/
Signed-off-by: Robin Jarry <rjarry@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This adds a Python version of the async DNS support added in:
771680d96 DNS: Add basic support for asynchronous DNS resolving
The above version uses the unbound C library, and this
implimentation uses the SWIG-wrapped Python version of that.
In the event that the Python unbound library is not available,
a warning will be logged and the resolve() method will just
return None. For the case where inet_parse_active() is passed
an IP address, it will not try to resolve it, so existing
behavior should be preserved in the case that the unbound
library is unavailable.
Intentional differences from the C version are as follows:
OVS_HOSTS_FILE environment variable can bet set to override
the system 'hosts' file. This is primarily to allow testing to
be done without requiring network connectivity.
Since resolution can still be done via hosts file lookup, DNS
lookups are not disabled when resolv.conf cannot be loaded.
The Python socket_util module has fallen behind its C equivalent.
The bare minimum change was done to inet_parse_active() to support
sync/async dns, as there is no equivalent to
parse_sockaddr_components(), inet_parse_passive(), etc. A TODO
was added to bring socket_util.py up to equivalency to the C
version.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
In some cases ovsdb server or relay gets restarted, ovsdb python clients
may keep the local socket open. Instead of reconnecting a lot of failures
will be logged.
This can be reproduced with ssl connections to the server/relay and
restarting it, so it has the same IP after restart.
This patch catches the Exceptions at do_handshake to recreate the
connection on the client side.
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Stefan Hoffmann <stefan.hoffmann@cloudandheat.com>
Signed-off-by: Luca Czesla <luca.czesla@mail.schwarz>
Signed-off-by: Max Lamprecht <max.lamprecht@mail.schwarz>
Co-authored-by: Luca Czesla <luca.czesla@mail.schwarz>
Co-authored-by: Max Lamprecht <max.lamprecht@mail.schwarz>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This patch adds ODP actions for SRv6 and its tests.
Signed-off-by: Nobuhiro MIKI <nmiki@yahoo-corp.jp>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Returning None in a constructor does not make sense and is just error
prone. Removing what was a leftover from an attempt to handle a common
error case of trying to parse what is commonly outputted by ovs-ofctl.
This should be done by the caller anyway.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
clone-like actions can also output to ports by specifying the port name.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
When CONTROLLER is used as free key, it means output=CONTROLLER which is
handled by decode_controller. However, it must output the KV in the
right format: "output": {"format": "CONTROLLER"}.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
OpenFlow actions names can be capitalized so in order to support this,
support case-insensitive KVDecoders and use it in Openflow actions.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Sometimes we don't want to return the result of a nested key-value
decoding as a dictionary but as a list of dictionaries. This happens
when we parse actions where keys can be repeated.
Refactor code that already takes that into account from ofp_act.py to
kv.py and use it for datapath action "clone".
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Currently, if a key is not found in the decoder information, we use the
default decoder which typically returns a string.
This not only means we can go out of sync with the C code without
noticing but it's also error prone as malformed flows could be parsed
without warning.
Make KeyValue parsing strict, raising an error if a decoder is not found
for a key.
This behaviour can be turned off globally by running 'KVDecoders.strict
= False' but it's generally not recommended. Also, if a KVDecoder does
need this default behavior, it can be explicitly configured specifying
it's default decoder.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
We were silently relying on some ofp actions to be decoded by the
default decoder which would yield decent string values.
In order to be more safe and robust, add an explicit decoder for all
missing actions.
This patch also reworks the learn action decoding to make it more
explicit and verify all the fields specified in the learn action are
actually valid fields.
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Fix the following erros in odp decoding:
- Missing push_mpls action
- Typos in collector_set_id, tp_src/tp_dst and csum
- Missing two fields in vxlan match
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
When initializing a monitor table the default monitor condition is
[True] which matches the behavior of the server (to send all rows of
that table). There's no need to include this default condition in the
initial monitor request so we can consider it implicitly acked by the
server.
This fixes the incorrect (one too large) expected condition sequence
number reported by ovsdb_idl_set_condition() when application is
trying to set a [True] condition for a new table.
Reported-by: Numan Siddique <numans@ovn.org>
Suggested-by: Ilya Maximets <i.maximets@ovn.org>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
ovsdb-server allows the OVSDB clients to specify the uuid for
the row inserts [1]. Both the C IDL client library and Python
IDL are missing this feature. This patch adds this support.
In C IDL, for each schema table, a new function is generated -
<schema_table>insert_persistent_uuid(txn, uuid) which can
be used the clients to persist the uuid.
ovs-vsctl and other derivatives of ctl now supports the same
in the generic 'create' command with the option "--id=<UUID>".
In Python IDL, the uuid to persist can be specified in
the Transaction.insert() function.
[1] - a529e3cd1f("ovsdb-server: Allow OVSDB clients to specify the UUID for inserted rows.:)
Acked-by: Adrian Moreno <amorenoz@redhat.com>
Acked-by: Han Zhou <hzhou@ovn.org>
Acked-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Numan Siddique <numans@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Fixes idl.Row's __str__ method to only print if the column exists on
the object. The Row object passed to the 'updates' argument of
Idl.notify only contains a subset of columns. Printing that argument
causes an AttributeError.
Fixes: 6a1c98461b46 ("Add a __str__ method to idl.Row")
Submitted-at: https://github.com/openvswitch/ovs/pull/392
Acked-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Christopher Aubut <christopher@aubut.me>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Python documentation suggests to do so "to avoid listing all the
PyTypeObject fields that you don't care about and also to avoid
caring about the fields' declaration order". And that does make
sense.
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
pyOpenSSL was recently switched for the Python standard library ssl
module in the cited commit. Python SSLsocket.send() does not allow
non-zero optional flag and it will explicitly raise an exception for
that. pyOpenSSL did nothing with this flag but kept it to be
compatible with socket API:
https://github.com/pyca/pyopenssl/blob/main/src/OpenSSL/SSL.py#L1844
Fixes: 68543dd523bd ("python: Replace pyOpenSSL with ssl.")
Reported-at: https://bugzilla.redhat.com/2115035
Acked-By: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Miro Tomaska <mtomaska@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
With just released flake8 5.0 we're getting a bunch of E275 errors:
utilities/bugtool/ovs-bugtool.in:959:23: E275 missing whitespace after keyword
tests/test-ovsdb.py:623:11: E275 missing whitespace after keyword
python/setup.py:105:8: E275 missing whitespace after keyword
python/setup.py:106:8: E275 missing whitespace after keyword
python/ovs/db/idl.py:145:15: E275 missing whitespace after keyword
python/ovs/db/idl.py:167:15: E275 missing whitespace after keyword
make[2]: *** [flake8-check] Error 1
This breaks CI on branches below 2.16. We don't see a problem right
now on newer branches because we're installing extra dependencies
that backtrack flake8 down to 4.1 or even 3.9.
Acked-by: Mike Pattrick <mkp@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
On Python 3.12, distutils will be removed and it's currently (3.10+)
deprecated (see PEP 632).
Since the suggested and simplest replacement is setuptools, this commit
replaces distutils to use setuptools instead.
setuptools < 59.0 doesn't have setuptools.errors and so, in this case,
distutils.errors is still used.
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
In order to correctly link with static libopenvswitch.a library,
users should also provide required cflags and all the libraries
libopenvswitch.a was actually built with and depends on. Otherwise,
it's not possible to link correctly.
Fixes: 671f93fe42d3 ("python: Allow building json C extension with static OVS library.")
Acked-by: Frode Nordahl <frode.nordahl@canonical.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
PyModule_AddObject() may fail and it doesn't steal references
in this case. The error condition should be handled to avoid
possible memory leaks.
And while it's not strictly specified if PyModule_Create may
fail, most of the examples in python documentation include
handling of a NULL case.
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Since Python 2 is not supported anymore, remove Python 2 support from C
extension too
Fixes: 1ca0323e7c29 ("Require Python 3 and remove support for Python 2.")
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Add unit test for OFFilter class.
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Add unit tests for OFPFlow class and ip-port range decoder
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Use pytest to run unit tests as part of the standard testsuite.
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
The json encoder can be used to convert Flows to json.
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Based on pyparsing, create a very simple filtering syntax.
It supports basic logic statements (and, &, or, ||, not, !), numerical
operations (<, >), equality (=, !=), and masking (~=). The latter is only
supported in certain fields (IntMask, EthMask, IPMask).
Masking operation is semantically equivalent to "includes",
therefore:
ip_src ~= 192.168.1.1
means that ip_src field is either a host IP address equal to 192.168.1.1
or an IPMask that includes it (e.g: 192.168.1.1/24).
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
