Before the patch, the size of the backlog depended on the type of socket
(UNIX vs INET) as well as on the language (C vs Python), specifically:
- python used backlog size = 10 for all sockets;
- C used 64 for UNIX sockets but 10 for INET sockets.
This consolidates the values across the board. It effectively bumps the
number of simultaneous connections to python unixctl servers to 64. Also
for INET C servers too.
The rationale to do it, on top of consistency, is as follows:
- fmt_pkt in ovn testsuite is limited by python server listen backlog,
and as was found out when adopting the tool, it is sometimes useful to
run lots of parallel calls to fmt_pkt unixctl server in some tests.
(See [1] for example.)
- there is a recent report [2] on discuss@ ML where the reporter noticed
significant listen queue overflows in some scenarios (large openstack
deployments; happens during leader transition when hundreds of neutron
nodes - with dozens of neutron api workers each - simultaneously
reconnect to the same northbound leader.) Note: While there is no
clear indication that this backlog size bump would resolve the
reported issues, it would probably help somewhat.
[1] 0baca3e519
[2] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-April/053049.html
Signed-off-by: Ihar Hrachyshka <ihrachys@redhat.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Simon Horman <horms@ovn.org>
This adds a Python version of the async DNS support added in:
771680d96 DNS: Add basic support for asynchronous DNS resolving
The above version uses the unbound C library, and this
implimentation uses the SWIG-wrapped Python version of that.
In the event that the Python unbound library is not available,
a warning will be logged and the resolve() method will just
return None. For the case where inet_parse_active() is passed
an IP address, it will not try to resolve it, so existing
behavior should be preserved in the case that the unbound
library is unavailable.
Intentional differences from the C version are as follows:
OVS_HOSTS_FILE environment variable can bet set to override
the system 'hosts' file. This is primarily to allow testing to
be done without requiring network connectivity.
Since resolution can still be done via hosts file lookup, DNS
lookups are not disabled when resolv.conf cannot be loaded.
The Python socket_util module has fallen behind its C equivalent.
The bare minimum change was done to inet_parse_active() to support
sync/async dns, as there is no equivalent to
parse_sockaddr_components(), inet_parse_passive(), etc. A TODO
was added to bring socket_util.py up to equivalency to the C
version.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
In some cases ovsdb server or relay gets restarted, ovsdb python clients
may keep the local socket open. Instead of reconnecting a lot of failures
will be logged.
This can be reproduced with ssl connections to the server/relay and
restarting it, so it has the same IP after restart.
This patch catches the Exceptions at do_handshake to recreate the
connection on the client side.
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Stefan Hoffmann <stefan.hoffmann@cloudandheat.com>
Signed-off-by: Luca Czesla <luca.czesla@mail.schwarz>
Signed-off-by: Max Lamprecht <max.lamprecht@mail.schwarz>
Co-authored-by: Luca Czesla <luca.czesla@mail.schwarz>
Co-authored-by: Max Lamprecht <max.lamprecht@mail.schwarz>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Currently, pyOpenSSL is half-deprecated upstream and so it's removed on
some distributions (for example on CentOS Stream 9,
https://issues.redhat.com/browse/CS-336), but since OVS only
supports Python 3 it's possible to replace pyOpenSSL with "import ssl"
included in base Python 3.
Stream recv and send had to be splitted as _recv and _send, since SSLError
is a subclass of socket.error and so it was not possible to except for
SSLWantReadError and SSLWantWriteError in recv and send of SSLStream.
TCPstream._open cannot be used in SSLStream, since Python ssl module
requires the SSL socket to be created before connecting it, so
SSLStream._open needs to create the socket, create SSL socket and then
connect the SSL socket.
Reported-by: Timothy Redaelli <tredaelli@redhat.com>
Reported-at: https://bugzilla.redhat.com/1988429
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Acked-by: Terry Wilson <twilson@redhat.com>
Tested-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
With other socket types, trying to connect and failing will return
an error code, but if an SSL Stream is used, then when
check_connection_completion(sock) is called, SSL will raise an
exception that doesn't derive from socket.error which is handled.
This adds handling for SSL.SysCallError which has the same
arguments as socket.error (errno, string). A future enhancement
could be to go through SSLStream class and implement error
checking for all of the possible exceptions similar to how
lib/stream-ssl.c's interpret_ssl_error() works across the various
methods that are implemented.
Fixes: d90ed7d65b ("python: Add SSL support to the python ovs client library")
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Thomas Neuman <thomas.neuman@nutanix.com>
Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Since Python 2 support was removed in 1ca0323e7c ("Require Python 3 and
remove support for Python 2."), python3-six is not needed anymore.
Moreover python3-six is not available on RHEL/CentOS7 without using EPEL
and so this patch is needed in order to release OVS 2.13 on RHEL7.
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Without shutting down the SSL connection, log messages like:
stream_ssl|WARN|SSL_read: unexpected SSL connection close
jsonrpc|WARN|ssl:127.0.0.1:47052: receive error: Protocol error
reconnect|WARN|ssl:127.0.0.1:47052: connection dropped (Protocol error)
would occur whenever the socket is closed. This just adds an
SSLStream.close() that calls shutdown() and ignores SSL errors, the
same way that lib/stream-ssl.c does in ssl_close().
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
On some systems in case where remote is not responding, socket could
remain in SYN_SENT state for a really long time without errors waiting
for connection. This leads to situations where open_blok() hangs for
a few minutes waiting for connection to the DOWN remote.
For example, our "multiple remotes" idl tests hangs waiting for
connection to the WRONG_PORT on FreeBSD in CirrusCI environment.
This leads to test failures because Alarm signal arrives much faster
than ETIMEDOUT from the socket.
This patch allowes to specify timeout value for 'open_block' function.
If the connection takes more time, socket will be closed with
ETIMEDOUT error code. Negative value or None in python could be
used to wait infinitely.
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
'sock.setsockopt' could throw exceptions. For example, if non-blocking
connection failed before the call:
Traceback (most recent call last):
File "../.././test-ovsdb.py", line 896, in <module>
main(sys.argv)
File "../.././test-ovsdb.py", line 891, in main
func(*args)
File "../.././test-ovsdb.py", line 604, in do_idl
ovs.stream.Stream.open(r))
File "/root/git_/ovs/python/ovs/stream.py", line 190, in open
error, sock = cls._open(suffix, dscp)
File "/root/git_/ovs/python/ovs/stream.py", line 744, in _open
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
File "/usr/local/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 54] Connection reset by peer
This fixes tests on FreeBSD.
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The python function ovs.socket_util.check_connection_completion() uses select()
(provided by python) to monitor the socket file descriptor. The select()
returns 1 when the file descriptor becomes ready. For error cases like -
111 (Connection refused) and 113 (No route to host) (POLLERR), ovs.poller._SelectSelect.poll()
expects the exceptfds list to be set by select(). But that is not the case.
As per the select() man page, writefds list will be set for POLLERR.
Please see "Correspondence between select() and poll() notifications" section of select(2)
man page.
Because of this behavior, ovs.socket_util.check_connection_completion() returns success
even if the remote is unreachable or not listening on the port.
This patch fixes this issue by using poll() to check the connection status similar to
the C implementation of check_connection_completion().
A new function 'get_system_poll() is added in ovs/poller.py which returns the
select.poll() object. If select.poll is monkey patched by eventlet/gevent, it
gets the original select.poll() and returns it.
The test cases added in this patch fails without the fix.
Suggested-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Mark Michelson <mmichels@redhat.com>
This patch is a simple implementation for the proposal discussed in
https://mail.openvswitch.org/pipermail/ovs-dev/2017-August/337038.html and
https://mail.openvswitch.org/pipermail/ovs-dev/2017-October/340013.html.
It enables ovs-vswitchd and other utilities to use DNS names when specifying
OpenFlow and OVSDB remotes.
Below are some of the features and limitations of this patch:
- Resolving is asynchornous in daemon context, avoiding blocking main loop;
- Resolving is synchronous in general utility context;
- Both IPv4 and IPv6 are supported;
- The resolving API is thread-safe;
- Depends on the unbound library;
- When multiple ip addresses are returned, only the first one is used;
- /etc/nsswitch.conf isn't respected as unbound library doesn't look at it;
- For async-resolving, caller need to retry later; there is no callback.
Signed-off-by: Yifeng Sun <pkusunyifeng@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Check if we are dealing with a Unicode string that needs
encoding for both Python 2 & 3.
Also, do the encoding the same way for Python 2 & 3 and avoid using
negation to make the code simpler.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Encoding from 'unicode' to 'str' that has been added to the Stream class
in commit 2254074e30 ("python: fix python3 encode/decode on Windows")
conflicts with SSLStream which already contains a quirk for pyopenssl
that does the same thing.
This results in a double encoding attempt when SSL is used and we crash
and burn due to:
Traceback (most recent call last):
File "../.././test-ovsdb.py", line 874, in <module>
main(sys.argv)
File "../.././test-ovsdb.py", line 869, in main
func(*args)
File "../.././test-ovsdb.py", line 655, in do_idl
idl_set(idl, command, step)
File "../.././test-ovsdb.py", line 526, in idl_set
status = txn.commit_block()
File "/home/jkbs/src/ovs/python/ovs/db/idl.py", line 1405, in commit_block
status = self.commit()
File "/home/jkbs/src/ovs/python/ovs/db/idl.py", line 1388, in commit
if not self.idl._session.send(msg):
File "/home/jkbs/src/ovs/python/ovs/jsonrpc.py", line 540, in send
return self.rpc.send(msg)
File "/home/jkbs/src/ovs/python/ovs/jsonrpc.py", line 244, in send
self.run()
File "/home/jkbs/src/ovs/python/ovs/jsonrpc.py", line 203, in run
retval = self.stream.send(self.output)
File "/home/jkbs/src/ovs/python/ovs/stream.py", line 808, in send
return super(SSLStream, self).send(buf)
File "/home/jkbs/src/ovs/python/ovs/stream.py", line 391, in send
buf = buf.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 83: ordinal not in range(128)
Remove the quirk from SSLStream as the base class now does encoding.
Reported-by: Marcin Mirecki <mmirecki@redhat.com>
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
At the moment we have WSAEventSelect in each if branch.
Since the call to the function is similar, we can move
it outside the if branch and create some local variables
which will be passed to WSAEventSelect.
This patch also remove the keyword argument passed when
the event for the connection overlapped structure is created.
The argument is not needed since it does not change the value
from the default one.
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
At the moment the sockets on Windows use the same events
that are being created for the pipes.
This is not correct because they should be different events.
This patch introduces a new event which should be used for sockets.
The new event needs to be set on automatic reset with its initial
state not signaled.
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Co-authored-by: Alin Gabriel Serdean <aserdean@ovn.org>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
Tested-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
Fix double encoding/decoding on data, caused by
'get_decoded_buffer' and 'get_encoded_buffer'.
The functions 'get_decoded_buffer' and 'get_encoded_buffer'
from winutils have been removed. They are no longer
necessary since the buffers received/returned are already
in the right form.
The necessary encoding has been moved before any sending
function (this also includes named pipes send on Windows).
Co-authored-by: Alin Serdean <aserdean@cloudbasesolutions.com>
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Signed-off-by: Alin Serdean <aserdean@cloudbasesolutions.com>
Acked-by: Lance Richardson <lrichard@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
Ensure that JSON is utf-8 encoded and that bytes sent/received on
the stream sockets are in utf-8 form. Add a test case to verify
that unicode data can be sent/received successfully using Python
IDL module.
Co-authored-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Lance Richardson <lrichard@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
Centos provides pyOpenSSL version pyOpenSSL-0.13.1-3.el7.x86_64.
There are 2 issues using this version, which this patch fixes
- The test case "simple idl verify notify - SSL" is skipped.
This is because "python -m OpenSSL.SSL" is used to detect the
presence of pyOpenSSL package. pyOpenSSL v0.13 has C python
modules because of which the above command returns 1.
So this patch fixes this by using 'python -c "import OpenSSL.SSL"'.
- The SSL.Context class does not have the function "set_session_cache_mode"
defined. Our usage here was only relevant for server-side connections,
(pssl), which is not yet supported by python-ovs, so just remove the
usage of this function. The default cache mode (server) will just
be ignored.
I have not tested with older versions (< 0.13) of pyOpenSSL.
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Acked-by: Lance Richardson <lrichard@redhat.com>
Tested-by: Marcin Mirecki <mmirecki@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
https://review.openstack.org/#/c/432906/
flake8-import-order adds 3 new flake8 warnings:
I100: Your import statements are in the wrong order.
I101: The names in your from import are in the wrong order.
I201: Missing newline between sections or imports.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Wait for clients to read from the pipe before disconnecting the server.
Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>
Signed-off-by: Gurucharan Shetty <guru@ovn.org>
When OVSDB server is aborted,
the SSL send function will throw SSL.SysCallError exception,
which we need to catch and return it's -errno.
While SSL.WantWriteError exception needs to return -EAGAIN
based on its parent class, not EAGAIN
Signed-off-by: Guoshuai Li <ligs@dtdream.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
stream_or_pstream_needs_probes always return 0. This causes TCP/SSL
connection not be probed, and no reconnect when the connection
is aborted
Signed-off-by: Guoshuai Li <ligs@dtdream.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Unix sockets (AF_UNIX) are not supported on Windows.
The replacement of Unix sockets on Windows is implemented
using named pipes, we are trying to mimic the behaviour
of unix sockets.
Instead of using Unix sockets to communicate
between components Named Pipes are used. This
makes the python sockets compatible with the
Named Pipe used in Windows applications.
Signed-off-by: Paul-Daniel Boca <pboca@cloudbasesolutions.com>
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Acked-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions>
Tested-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions>
Signed-off-by: Gurucharan Shetty <guru@ovn.org>
The do_handshake() function throws the exception OpenSSL.SSL.SysCallError
when the peer's SSL connection is closed, And the recv() function also
throws the exception OpenSSL.SSL.SysCallError when the peer's SSL
connection is abnormally closed, This commit catches the exception and
return error's errno.
Similarly, the recv() function also throws the exception
OpenSSL.SSL.ZeroReturnError when the peer's SSL connection is closed. This
exception refers to TCP connection normal closed, return (0, "")
Signed-off-by: Guoshuai Li <ligs@dtdream.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Numan Siddique <nusiddiq@redhat.com>
pycodestyle >= 2.1.0 reports E305 otherwise, and the flake8-check step
fails.
Signed-off-by: Daniele Di Proietto <diproiettod@vmware.com>
Acked-by: Ben Pfaff <blp@ovn.org>
SSL support is added to the ovs/stream.py. pyOpenSSL library is used
to support SSL. If this library is not present, then the SSL stream
is not registered with the Stream class.
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Instead of checking the raw version, use the six.PY2 and six.PY3 helpers
to determine if Python 2 or Python 3 are in use.
In one case, the check was to determine if the Python version was >=
2.6. We now only support >= 2.7, so this check would always be true.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Acked-by: Miguel Angel Ajo <majopela@redhat.com>
Python 3 has separate types for strings and bytes. Python 2 used the
same type for both. We need to convert strings to bytes before writing
them out to a socket. We also need to convert data read from the socket
to a string.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
In Python 2, dict.items(), dict.keys(), and dict.values() returned a
list. dict.iteritems(), dict.iterkeys(), and dict.itervalues() returned
an iterator.
As of Python 3, dict.iteritems(), dict.itervalues(), and dict.iterkeys()
are gone. items(), keys(), and values() now return an iterator.
In the case where we want an iterator, we now use the six.iter*()
helpers. If we want a list, we explicitly create a list from the
iterator.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
This patch fixes just the Python 3 problems found by running:
python3 setup.py install
There are still many other issues to be fixed, but this is a start.
Signed-off-by: Terry Wilson <twilson@redhat.com>
[russell@ovn.org resolved conflicts with current master]
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Fix the following pep8 errors:
E201 whitespace after '('
E203 whitespace before ','
E222 multiple spaces after operator
E225 missing whitespace around operator
E226 missing whitespace around arithmetic operator
E231 missing whitespace after ':'
E241 multiple spaces after ':'
E251 unexpected spaces around keyword / parameter equals
E261 at least two spaces before inline comment
E262 inline comment should start with '# '
E265 block comment should start with '# '
E271 multiple spaces after keyword
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Until now, "unix:" and "punix:" paths that are not absolute have
been considered relative to the current working directory. It
is more useful to consider them relative to the rundir, so this
commit makes that change to the C and Python implementations of
the stream code.
This commit also relaxes the whitelist check in the bridge code
so that any name that does not contain a "/" is considered OK.
Signed-off-by: Pavithra Ramesh <paramesh@vmware.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
If the loop condition in Stream.connect() was false, which is especially
likely for TCP connections, then Stream.connect() would return None,
which violates its documented behavior. This commit fixes the problem.
Reported-by: Isaku Yamahata <yamahata@valinux.co.jp>
Tested-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Python doesn't have select.POLL* constants on some architectures
(e.g. MacOSX). This code needs to define the constants for itself. It
uses select.POLL* constants only internally (doesn't pass them
outside). So there is no harm even if the definition would conflict
with Python's those.
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
c38f8724ae made stream.py not use class
decorator. So Stream.register need not to be decorator any more.
So simplify it.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Commit 8cc820 (python/ovs/stream: teach stream.py tcp socket) made a
change that used class decorators. Unfortunately, they were not
introduced until Python 2.6. XenServer uses Python 2.4, so the change
caused some Python-based daemons not to start. This commit uses an
alternate syntax suggested by Reid Price.
Bug #13580
Signed-off-by: Justin Pettit <jpettit@nicira.com>
Signed-off-by: Reid Price <reid@nicira.com>
Replaced all instances of Nicira Networks(, Inc) to Nicira, Inc.
Feature #10593
Signed-off-by: Raju Subramanian <rsubramanian@nicira.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
There isn't a lot of value in sending inactivity probes on unix
sockets. This patch changes the default to disable them.
Signed-off-by: Ethan Jackson <ethan@nicira.com>
There's no reason for a Unix domain client socket to bind a name. I don't
know why we've always done that. Stevens's "Unix Network Programming"
Unix domain socket client example doesn't do a bind.
Removes the 'unlink_path' parameter from new_fd_stream() since it is now
always passed as NULL.
Signed-off-by: Ben Pfaff <blp@nicira.com>
This patch does minor style cleanups to the code in the python and
tests directory. There's other code floating around that could use
similar treatment, but updating it is not convenient at the moment.
