The purpose of reconnect_deadline__() function is twofold:
1. Its result is used to tell if the state has to be changed right now
in reconnect_run().
2. Its result also used to determine when the process need to wake up
and call reconnect_run() for a next time, i.e. when the state may
need to be changed next time.
Since introduction of the 'receive-attempted' feature, the function
returns LLONG_MAX if the deadline is in the future. That works for
the first case, but doesn't for the second one, because we don't
really know when we need to call reconnect_run().
This is the problem for applications where jsonrpc connection is the
only source of wake ups, e.g. ovn-northd. When the network goes down
silently, e.g. server looses IP address due to DHCP failure, ovn-northd
will sleep in the poll loop indefinitely after being told that it
doesn't need to call reconnect_run() (deadline == LLONG_MAX).
Fixing that by actually returning the expected time if it is in the
future, so we will know when to wake up. In order to keep the
'receive-attempted' feature, returning 'now + 1' in case where the
time has already passed, but receive wasn't attempted. That will
trigger a fast wake up, so the application will be able to attempt the
receive even if there was no real events. In a correctly written
application we should not fall into this case more than once in a row.
'+ 1' ensures that we will not transition into a different state
prematurely, i.e. before the receive is actually attempted.
Fixes: 4241d652e465 ("jsonrpc: Avoid disconnecting prematurely due to long poll intervals.")
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
When reconnecting, if there are condition changes already sent to the
server but not yet acked, reset the db's 'last-id', esentially clearing
the local cache after reconnect.
This is needed because the client cannot easily differentiate between
the following cases:
a. either the server already processed the requested monitor
condition change but the FSM was restarted before the
client was notified. In this case the client should
clear its local cache because it's out of sync with the
monitor view on the server side.
b. OR the server hasn't processed the requested monitor
condition change yet.
Fixes: 46d44cf3be0d ("python: idl: Add monitor_cond_since support.")
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
When reconnecting forget about in-flight monitor condition changes
if the user requested a newer condition already.
This matches the C implementation, in ovsdb_cs_db_sync_condition().
Fixes: 46d44cf3be0d ("python: idl: Add monitor_cond_since support.")
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
Acked-By: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Add support for monitor_cond_since / update3 to python-ovs to
allow more efficient reconnections when connecting to clustered
OVSDB servers.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
ovsdb_atom_string and json_string are basically the same data structure
and ovsdb-server frequently needs to convert one to another. We can
avoid that by using json_string from the beginning for all ovsdb
strings. So, the conversion turns into simple json_clone(), i.e.
increment of a reference counter. This change gives a moderate
performance boost in some scenarios, improves the code clarity and
may be useful for future development.
Acked-by: Mike Pattrick <mkp@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Many python implementations pre-allocate space for multiple
objects in empty dicts and lists. Using a custom dict-like object
that only generates these objects when they are accessed can save
memory.
On a fairly pathological case where the DB has 1000 networks each
with 100 ports, with only 'name' fields set, this saves around
300MB of memory.
One could argue that if values are not going to change from their
defaults, then users should not be monitoring those columns, but
it's also probably good to not waste memory even if user code is
sub-optimal.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Acked-by: Flavio Fernandes <flavio@flaviof.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Python objects normally have a dictionary named __dict__ allocated
for handling dynamically assigned attributes. Depending on
architecture and Python version, that empty dict may be between
64 and 280 bytes.
Seeing as Atom and Datum objects do not need dynamic attribute
support and there can be millions of rows in a database, avoiding
this allocation with __slots__ can save 100s of MBs of memory per
Idl process.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Timothy Redaelli <tredaelli@redhat.com>
Tested-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Currently, pyOpenSSL is half-deprecated upstream and so it's removed on
some distributions (for example on CentOS Stream 9,
https://issues.redhat.com/browse/CS-336), but since OVS only
supports Python 3 it's possible to replace pyOpenSSL with "import ssl"
included in base Python 3.
Stream recv and send had to be splitted as _recv and _send, since SSLError
is a subclass of socket.error and so it was not possible to except for
SSLWantReadError and SSLWantWriteError in recv and send of SSLStream.
TCPstream._open cannot be used in SSLStream, since Python ssl module
requires the SSL socket to be created before connecting it, so
SSLStream._open needs to create the socket, create SSL socket and then
connect the SSL socket.
Reported-by: Timothy Redaelli <tredaelli@redhat.com>
Reported-at: https://bugzilla.redhat.com/1988429
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Acked-by: Terry Wilson <twilson@redhat.com>
Tested-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
In an upcoming patch, PyOpenSSL will be replaced with Python ssl module,
but in order to do an async connection with Python ssl module the ssl
socket must be created when the socket is created, but before the
socket is connected.
So, inet_open_active function is splitted in 3 parts:
- inet_create_socket_active: creates the socket and returns the family and
the socket, or (error, None) if some error needs to be returned.
- inet_connect_active: connect the socket and returns the errno (it
returns 0 if errno is EINPROGRESS or EWOULDBLOCK).
connect is replaced by connect_ex, since Python suggest to use it for
asynchronous connects and it's also cleaner since inet_connect_active
returns errno that connect_ex already returns, moreover due to a Python
limitation connect cannot not be used with ssl module.
inet_open_active function is changed in order to use the new functions
inet_create_socket_active and inet_connect_active.
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Acked-by: Terry Wilson <twilson@redhat.com>
Tested-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This ports the C IDL change f50714b to the Python IDL:
Until now the code here would happily try to send transactions to the
database server even if the database connection was not in the correct
state. In some cases this could lead to strange behavior, such as sending
a database transaction for a database that the IDL had just learned did not
exist on the server.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
ovsdb-server spends a lot of time cloning atoms for various reasons,
e.g. to create a diff of two rows or to clone a row to the transaction.
All atoms, except for strings, contains a simple value that could be
copied in efficient way, but duplicating strings every time has a
significant performance impact.
Introducing a new reference-counted structure 'ovsdb_atom_string'
that allows to not copy strings every time, but just increase a
reference counter.
This change allows to increase transaction throughput in benchmarks
up to 2x for standalone databases and 3x for clustered databases, i.e.
number of transactions that ovsdb-server can handle per second.
It also noticeably reduces memory consumption of ovsdb-server.
Next step will be to consolidate this structure with json strings,
so we will not need to duplicate strings while converting database
objects to json and back.
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
The ovsdb-cs layer triggers a forced reconnect in various cases:
- when an inconsistency is detected in the data received from the
remote server.
- when the remote server is running in clustered mode and transitioned
to "follower", if the client is configured in "leader-only" mode.
- when explicitly requested by upper layers (e.g., by the user
application, through the IDL layer).
In such cases it's desirable that reconnection should happen as fast as
possible, without the current exponential backoff maintained by the
underlying reconnect object. Furthermore, since 3c2d6274bcee ("raft:
Transfer leadership before creating snapshots."), leadership changes
inside the clustered database happen more often and, therefore,
"leader-only" clients need to reconnect more often too.
Forced reconnects call jsonrpc_session_force_reconnect() which will not
reset backoff. To make sure clients reconnect as fast as possible in
the aforementioned scenarios we first call the new API,
jsonrpc_session_reset_backoff(), in ovsdb-cs, for sessions that are in
state CS_S_MONITORING (i.e., the remote is likely still alive and
functioning fine).
jsonrpc_session_reset_backoff() resets the number of backoff-free
reconnect retries to the number of remotes configured for the session,
ensuring that all remotes are retried exactly once with backoff 0.
This commit also updates the Python IDL and jsonrpc implementations.
The Python IDL wasn't tracking the IDL_S_MONITORING state explicitly,
we now do that too. Tests were also added to make sure the IDL forced
reconnects happen without backoff.
Reported-at: https://bugzilla.redhat.com/1977264
Suggested-by: Ilya Maximets <i.maximets@ovn.org>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
When using eventlet monkey_patch()'d code, greenthreads can be
blocked on connection for several seconds while the database
contents are parsed. Eventlet recommends adding a sleep(0) call
to cooperatively yield in cpu-bound code. asyncio code has
asyncio.sleep(0). This patch adds an API method that defaults to
doing nothing, but can be overridden to yield as needed.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This is needed since the current bundled version doesn't work on Python
3.10+.
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Clients needs to re-connect from the relay that has no connection
with the database source. Also, relay acts similarly to the follower
from a clustered model from the consistency point of view, so it's not
suitable for leader-only connections.
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Fix an issue where Idl.run() returned False even if there was a change.
If Idl.run() reads multiple messages from the database server, some
may constitute changes and some may not. Changed the way change_seqno
is reset: if a message is not a change, reset change_seqno only to the
value before reading this message, not to the value before reading the
first message.
This will fix the return value in a scenario where some message was a
change and the last one wasn't. The new change_seqno will now be the
value after handling the message with the last change.
Fixes: c39751e44539 ("python: Monitor Database table to manage lifecycle of IDL client.")
Signed-off-by: Bodo Petermann <b.petermann@syseleven.de>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
As described in commit [1], it's possible that remote IP is backed by
a load-balancer and re-connection to this same IP will lead to
connection to a different server. This case is supported for C version
of IDL and should be supported in a same way for python implementation.
[1] ca367fa5f8bb ("ovsdb-idl.c: Allows retry even when using a single remote.")
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Dumitru Ceara <dceara@redhat.com>
The action headings were coming out all smashed together, like
"Theoutputaction". This fixes them so that they appear correctly, like
"The output action".
The previous code stripped starting and ending spaces on a per-node
basis, so that "The ", "<code>output</code>", and " action" each got
stripped down to "The", "output", "action" after processing. This
commit changes it so that stripping happens after concatenation, fixing
the problem.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-By: Timothy Redaelli <tredaelli@redhat.com>
Tested-By: Timothy Redaelli <tredaelli@redhat.com>
The Python IDL notification mechanism was sending a notification
for each processed update in a transaction as it was processed.
This causes issues with multi-row changes that contain references
to each other.
For example, if a Logical_Router_Port is created along with a
Gateway_Chassis, and the LRP.gateway_chassis set to that GC, then
when the notify() passes the CREATE event for the LRP, the GC will
not yet have been processed, so __getattr__ when _uuid_to_row fails
to find the GC, will return the default value for LRP.gateway_chassis
which is [].
This patch has the process_update methods return the notifications
that would be produced when a row changes, so they can be queued
and sent after all rows have been processed.
Fixes: d7d417fcddf9 ("Allow subclasses of Idl to define a notification hook")
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Brian Haley <haleyb.dev@gmail.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Tested-by: Flavio Fernandes <flavio@flaviof.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This follows up on commit 4241d652e465 ("jsonrpc: Avoid disconnecting
prematurely due to long poll intervals."), which implemented the same
thing in C.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Requested-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Ilya Maximets <i.maximets@ovn.org>
Update build system to ensure dirs.py is created when it is a
dependency for a build target. Also, update setup.py to
check for that dependency.
Fixes: 943c4a325045 ("python: set ovs.dirs variables with build system values")
Signed-off-by: Mark Gray <mark.d.gray@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
With other socket types, trying to connect and failing will return
an error code, but if an SSL Stream is used, then when
check_connection_completion(sock) is called, SSL will raise an
exception that doesn't derive from socket.error which is handled.
This adds handling for SSL.SysCallError which has the same
arguments as socket.error (errno, string). A future enhancement
could be to go through SSLStream class and implement error
checking for all of the possible exceptions similar to how
lib/stream-ssl.c's interpret_ssl_error() works across the various
methods that are implemented.
Fixes: d90ed7d65ba8 ("python: Add SSL support to the python ovs client library")
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Thomas Neuman <thomas.neuman@nutanix.com>
Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
ovs/dirs.py should be auto-generated using the template
ovs/dirs.py.template at build time. This will set the
ovs.dirs python variables with a value specified by the
environment or, if the environment variable is not set, from
the build system.
Signed-off-by: Mark Gray <mark.d.gray@redhat.com>
Acked-By: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Ian Stokes <ian.stokes@intel.com>
Builds on RHEL 8.2 systems are failing due to this issue.
See [1] as to why this is necessary.
I used the following command to identify files that need this fix:
find . -type f -executable | /usr/lib/rpm/redhat/brp-mangle-shebangs
I also updated the copyright notices as needed.
1. https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error
Fixes: 1ca0323e7c29 ("Require Python 3 and remove support for Python 2.")
Signed-off-by: Greg Rose <gvrose8192@gmail.com>
Acked-by: Aaron Conole <aconole@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Fixes two problems. First, the plural of chassis is also chassis.
Second, for linguistic analysis we need to consider plain words, not
words that have (e.g.) \fB and \fR pasted into them for nroff output.
This makes the OVN manpage for ovn-sb(5) talk about "set of Chassis"
not "set of Chassiss".
Acked-by: Numan Siddique <numans@ovn.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
For columns like QoS.queues where we have a map containing refTable
values, assigning w/ __setattr__ e.g. qos.queues={1: $queue_row}
works, but using using qos.setkey('queues', 1, $queue_row) results
in an Exception. The opdat argument can essentially just be the
JSON representation of the map column instead of trying to build
it.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Before the fix, headers split into multiple lines were producing bogus
quote characters in nroff output and failed to indent headers properly.
Specifically, it fixes a header and its indentation in
ovn-architecture(7).
Signed-off-by: Ihar Hrachyshka <ihrachys@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Since Python 2 support was removed in 1ca0323e7c29 ("Require Python 3 and
remove support for Python 2."), python3-six is not needed anymore.
Moreover python3-six is not available on RHEL/CentOS7 without using EPEL
and so this patch is needed in order to release OVS 2.13 on RHEL7.
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Python 2 reaches end-of-life on January 1, 2020, which is only
a few months away. This means that OVS needs to stop depending
on in the next release that should occur roughly that same time.
Therefore, this commit removes all support for Python 2. It
also makes Python 3 a mandatory build dependency.
Some of the interesting consequences:
- HAVE_PYTHON, HAVE_PYTHON2, and HAVE_PYTHON3 conditionals have
been removed, since we now know that Python3 is available.
- $PYTHON and $PYTHON2 are removed, and $PYTHON3 is always
available.
- Many tests for Python 2 support have been removed, and the ones
that depended on Python 3 now run unconditionally. This allowed
several macros in the testsuite to be removed, making the code
clearer. This does make some of the changes to the testsuite
files large due to indentation level changes.
- #! lines for Python now use /usr/bin/python3 instead of
/usr/bin/python.
- Packaging depends on Python 3 packages.
Acked-by: Numan Siddique <nusiddiq@redhat.com>
Tested-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
It's sometimes handy to log an entire Row object, so this just
adds a string representation of the object as:
Tablename(col1=val1, col2=val2, ..., coln=valn)
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Without shutting down the SSL connection, log messages like:
stream_ssl|WARN|SSL_read: unexpected SSL connection close
jsonrpc|WARN|ssl:127.0.0.1:47052: receive error: Protocol error
reconnect|WARN|ssl:127.0.0.1:47052: connection dropped (Protocol error)
would occur whenever the socket is closed. This just adds an
SSLStream.close() that calls shutdown() and ignores SSL errors, the
same way that lib/stream-ssl.c does in ssl_close().
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The Python IDL implementation supports ovsdb cluster connections.
This patch is a follow up to commit 31e434fc98, it adds the option of
connecting to the leader (the default) in the Raft-based cluster. It mimics
the exisiting C IDL support for clusters introduced in commit 1b1d2e6daa.
The _Server database schema is first requested, then a monitor of the
Database table in the _Server Database. Method __check_server_db verifies
the eligibility of the server. If the attempt to obtain a monitor of the
_Server database fails and a cluster id was not provided this implementation
proceeds to request the data monitor. If a cluster id was provided via the
set_cluster_id method then the connection is aborted and a connection to a
different node is instead attempted, until a valid cluster node is found.
Thus, when supplied, cluster id is interpreted as the intention to only
allow connections to a clustered database. If not supplied, connections to
standalone nodes, or nodes that do not have the _Server database are
allowed. change_seqno is not incremented in the case of Database table
updates.
Acked-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ted Elhourani <ted.elhourani@nutanix.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Commit 00fcc832d598 ("Update Python package requirements") added a
PEP 508 environment marker to install pywin32 on Windows systems.
This requires a new setuptools version (>= 20.5), but (at least)
RHEL/CentOS7 and Debian Jessie are using an older version of
setuptools and so python extension failed to build.
This commit adds "extras_require" instead of the PEP 508 environment
markers in order to have the conditional dependency of pywin32, but by
remaining compatible with the old setuptools versions.
CC: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>
CC: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Fixes: 00fcc832d598 ("Update Python package requirements")
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
'set_levels_from_string' doesn't check for exceptions that could
happen while opening syslog files or connecting to syslog sockets.
For example, if rsyslog stopped on a system:
$ test-unixctl.py -vFACILITY:daemon --detach
Traceback (most recent call last):
File "../../../../tests/test-unixctl.py", line 90, in <module>
main()
File "../../../../tests/test-unixctl.py", line 61, in main
ovs.vlog.handle_args(args)
File "python/ovs/vlog.py", line 463, in handle_args
msg = Vlog.set_levels_from_string(verbose)
File "python/ovs/vlog.py", line 345, in set_levels_from_string
Vlog.add_syslog_handler(words[1])
File "python/ovs/vlog.py", line 321, in add_syslog_handler
facility=syslog_facility)
File "/python2.7/logging/handlers.py", line 759, in __init__
self._connect_unixsocket(address)
File "/python2.7/logging/handlers.py", line 787, in _connect_unixsocket
self.socket.connect(address)
File "/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 111] Connection refused
In this case "/dev/log" file exists, so the check inside
'add_syslog_handler' doesn't help.
We need to catch the exceptions in 'set_levels_from_string' same way
as it done in 'init' function.
Also, we don't really need to check for '/dev/log' existence, because
exception will be catched on the upper layer and properly handled by
disabling the corresponding logger.
Fixes: d69d61c7c175 ("vlog: Ability to override the default log facility.")
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The Python ovs package relies on pywin32 for Windows support.
For this reason, pywin32 should be included in the requirements
list.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
This fix was reverted because it depended on a small bit of code
in a patch that was reverted that changed some python/ovs testing
and build. The fix is still necessary.
The OVS C-based JSON parser operates on bytes, so the parser_feed
function returns the number of bytes that are processed. The pure
Python JSON parser currently operates on unicode, so it expects
that Parser.feed() returns a number of characters. This difference
leads to parsing errors when unicode characters are passed to the
C JSON parser from Python.
Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Since python version 3.7 (and some 3.6+ versions) regexp engine
changed to treat the wrong escape sequences as errors. Previously,
if the replace string had something like '\u0000', '\u' was
qualified as a bad escape sequence and treated just as a sequence
of characters '\' and 'u'. But know this triggers an error:
Traceback (most recent call last):
File "/usr/lib/python3.7/sre_parse.py", line 1021, in parse_template
this = chr(ESCAPES[this][1])
KeyError: '\\u'
From the documentation [1]:
Unknown escapes consisting of '\' and an ASCII letter in replacement
templates for re.sub() were deprecated in Python 3.5, and will now
cause an error.
[1] https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals
We need to escape the backslash by another one to keep regexp engine
from errors. In case of '\\u000', '\\' is a valid escape sequence
and the 'u' is a simple character.
To be 100% safe we need to use 're.escape(replace)', but it escapes
too many characters making the logs hard to read.
This change fixes Python 3 tests on systems with python 3.7.
Should be backward compatible.
Reported-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
It appears that Python silently treats invalid escape sequences in
strings as literals, e.g. "\." is the same as "\\.". Newer versions of
checkpatch complain, and it does seem reasonable to me to fix these.
Acked-by: Numan Siddique <nusiddiq@redhat.com>
Tested-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Without this change, the fonts are wrong if a title contains formatting
like <code> or <var>.
Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The ordered lists that a .25in width produced looked OK in PostScript
or PDF output, but in text output every list item spanned two lines,
like this:
1.
First list item.
2.
Second list item.
With this change, they appear normally:
1. First list item.
2. Second list item.
Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
On some systems in case where remote is not responding, socket could
remain in SYN_SENT state for a really long time without errors waiting
for connection. This leads to situations where open_blok() hangs for
a few minutes waiting for connection to the DOWN remote.
For example, our "multiple remotes" idl tests hangs waiting for
connection to the WRONG_PORT on FreeBSD in CirrusCI environment.
This leads to test failures because Alarm signal arrives much faster
than ETIMEDOUT from the socket.
This patch allowes to specify timeout value for 'open_block' function.
If the connection takes more time, socket will be closed with
ETIMEDOUT error code. Negative value or None in python could be
used to wait infinitely.
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
If attempt to open non-blocking connection results with EINPROGRESS,
further polling will trigger DISCONNECT action in case of failures.
While handling this action, jsonrpc python library closes the
connection but does not change the current remote. This leads to
subsequent connection to the same remote. And the story starts from
the beginning producing infinite attempts to connect to a single
remote regardless of existense of others. Like this:
reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF
reconnect | INFO | tcp:127.0.0.1:45932: connecting...
reconnect | DBG | tcp:127.0.0.1:45932: entering CONNECTING
poller | DBG | 999-ms timeout
reconnect | INFO | tcp:127.0.0.1:45932: connection attempt timed out
reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF
poller | DBG | 0-ms timeout
reconnect | INFO | tcp:127.0.0.1:45932: connecting...
<...>
reconnect | DBG | tcp:127.0.0.1:45932: entering CONNECTING
poller | DBG | 1999-ms timeout
reconnect | INFO | tcp:127.0.0.1:45932: connection attempt timed out
reconnect | INFO | tcp:127.0.0.1:45932: waiting 4 seconds before reconnect
reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF
<...>
Fix that by always picking the new remote on disconnect.
This mimics the behaviour of jsonrpc C library.
Fixes "multiple remotes" tests on FreeBSD.
CC: Numan Siddique <nusiddiq@redhat.com>
Fixes: 31e434fc985c ("python jsonrpc: Allow jsonrpc_session to have more than one remote.")
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Otherwise failed non-blocking connection could be reported as
connected. This causes errors in all following operations with the
socket.
At least this is true on FreeBSD, where POLLHUP could be set without
POLLERR.
For example, stream_open_block() tests fails with the following error
reporting successful connection to the 'WRONG_PORT':
./ovsdb-idl.at:1817:
$PYTHON2 $srcdir/test-stream.py tcp:127.0.0.1:$WRONG_PORT
stdout:
./ovsdb-idl.at:1817: exit code was 0, expected 1
2399. ovsdb-idl.at:1817: FAILED (ovsdb-idl.at:1817)
Also added new tests to track this issue in C library:
'Check Stream open block - C - tcp'
'Check Stream open block - C - tcp6'
CC: Numan Siddique <nusiddiq@redhat.com>
Fixes: c1aa16d191d2 ("ovs python: ovs.stream.open_block() returns success even if the remote is unreachable")
Fixes: d6cedfd9d29d ("socket-util: Avoid using SO_ERROR.")
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
'sock.setsockopt' could throw exceptions. For example, if non-blocking
connection failed before the call:
Traceback (most recent call last):
File "../.././test-ovsdb.py", line 896, in <module>
main(sys.argv)
File "../.././test-ovsdb.py", line 891, in main
func(*args)
File "../.././test-ovsdb.py", line 604, in do_idl
ovs.stream.Stream.open(r))
File "/root/git_/ovs/python/ovs/stream.py", line 190, in open
error, sock = cls._open(suffix, dscp)
File "/root/git_/ovs/python/ovs/stream.py", line 744, in _open
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
File "/usr/local/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 54] Connection reset by peer
This fixes tests on FreeBSD.
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The memory returned by json_parser_finish needs to be freed by the caller.
Signed-off-by: Eric Lapointe <elapointe@corsa.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
This reverts commit a7be68a4d77791bbe02c37f7ad8ae60b02e5679e
and a subsequent commit 4617d1f6bd24c543f533f6485b42ebca6b0a8371.
There are too many issues with these patches. It's better to revert
them for now and make a separate fixed versions later if needed.
List of issues (maybe not full):
1. 'make clean' removes entire 'python' directory.
2. Fully broken Travis-CI testsuite build:
building 'ovs._json' extension
creating build/temp.linux-x86_64-2.7
error: could not create 'build/temp.linux-x86_64-2.7': \
Permission denied
https://travis-ci.org/openvswitch/ovs/jobs/440693765
3. Broken local testsuite build on Ubuntu 18.04:
running build_ext
building 'ovs._json' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/ovs
<...>
/usr/bin/ld: .libs/libopenvswitch.a(util.o): \
relocation R_X86_64_TPOFF32 against `var.7749' can not be \
used when making a shared object; recompile with -fPIC
<...>
collect2: error: ld returned 1 exit status
4. Fedora build failure because of 'setuptools' ('distutils')
hard dependency on 'redhat-rpm-config' package:
building 'ovs._json' extension
<...>
gcc: error: <...>/redhat-hardened-cc1: No such file or directory
5. Looks like 'setuptools' also could download and install
unwanted python modules during package build.
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The OVS C-based JSON parser operates on bytes, so the parser_feed
function returns the number of bytes that are processed. The pure
Python JSON parser currently operates on unicode, so it expects
that Parser.feed() returns a number of characters. This difference
leads to parsing errors when unicode characters are passed to the
C JSON parser from Python.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
