2
0
mirror of https://github.com/openvswitch/ovs synced 2025-08-26 11:57:38 +00:00

332 Commits

Author SHA1 Message Date
Terry Wilson
75ff71168a Un-revert Work around Python/C JSON unicode differences
This fix was reverted because it depended on a small bit of code
in a patch that was reverted that changed some python/ovs testing
and build. The fix is still necessary.

The OVS C-based JSON parser operates on bytes, so the parser_feed
function returns the number of bytes that are processed. The pure
Python JSON parser currently operates on unicode, so it expects
that Parser.feed() returns a number of characters. This difference
leads to parsing errors when unicode characters are passed to the
C JSON parser from Python.

Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2019-01-15 11:16:30 -08:00
Ilya Maximets
a5b6b6f3a9 python: Escape backslashes while formatting logs.
Since python version 3.7 (and some 3.6+ versions) regexp engine
changed to treat the wrong escape sequences as errors. Previously,
if the replace string had something like '\u0000', '\u' was
qualified as a bad escape sequence and treated just as a sequence
of characters '\' and 'u'. But know this triggers an error:

  Traceback (most recent call last):
    File "/usr/lib/python3.7/sre_parse.py", line 1021, in parse_template
      this = chr(ESCAPES[this][1])
  KeyError: '\\u'

From the documentation [1]:

  Unknown escapes consisting of '\' and an ASCII letter in replacement
  templates for re.sub() were deprecated in Python 3.5, and will now
  cause an error.

[1] https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals

We need to escape the backslash by another one to keep regexp engine
from errors. In case of '\\u000', '\\' is a valid escape sequence
and the 'u' is a simple character.

To be 100% safe we need to use 're.escape(replace)', but it escapes
too many characters making the logs hard to read.

This change fixes Python 3 tests on systems with python 3.7.
Should be backward compatible.

Reported-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2019-01-15 09:09:53 -08:00
Ben Pfaff
adb3f0b027 python: Avoid flake8 warning for unused variables.
Acked-by: Numan Siddique <nusiddiq@redhat.com>
Tested-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2019-01-11 08:45:10 -08:00
Ben Pfaff
145a7e88bb python: Fix invalid escape sequences.
It appears that Python silently treats invalid escape sequences in
strings as literals, e.g. "\." is the same as "\\.".  Newer versions of
checkpatch complain, and it does seem reasonable to me to fix these.

Acked-by: Numan Siddique <nusiddiq@redhat.com>
Tested-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2019-01-11 08:45:04 -08:00
Ben Pfaff
a31c38ad0b nroff: Fix fonts for h2, h3, h4.
Without this change, the fonts are wrong if a title contains formatting
like <code> or <var>.

Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2019-01-10 15:52:05 -08:00
Ben Pfaff
63187cff99 nroff: Increase width for .IP used for ordered lists.
The ordered lists that a .25in width produced looked OK in PostScript
or PDF output, but in text output every list item spanned two lines,
like this:

   1.
     First list item.
   2.
     Second list item.

With this change, they appear normally:

   1. First list item.
   2. Second list item.

Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2019-01-10 15:52:05 -08:00
Ilya Maximets
77f42ca535 stream: Allow timeout configuration for open_block.
On some systems in case where remote is not responding, socket could
remain in SYN_SENT state for a really long time without errors waiting
for connection. This leads to situations where open_blok() hangs for
a few minutes waiting for connection to the DOWN remote.

For example, our "multiple remotes" idl tests hangs waiting for
connection to the WRONG_PORT on FreeBSD in CirrusCI environment.
This leads to test failures because Alarm signal arrives much faster
than ETIMEDOUT from the socket.

This patch allowes to specify timeout value for 'open_block' function.
If the connection takes more time, socket will be closed with
ETIMEDOUT error code. Negative value or None in python could be
used to wait infinitely.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2019-01-10 15:39:48 -08:00
Ilya Maximets
257edb1ae0 python: jsonrpc: Pick new remote on disconnect.
If attempt to open non-blocking connection results with EINPROGRESS,
further polling will trigger DISCONNECT action in case of failures.
While handling this action, jsonrpc python library closes the
connection but does not change the current remote. This leads to
subsequent connection to the same remote. And the story starts from
the beginning producing infinite attempts to connect to a single
remote regardless of existense of others. Like this:

 reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF
 reconnect | INFO | tcp:127.0.0.1:45932: connecting...
 reconnect | DBG | tcp:127.0.0.1:45932: entering CONNECTING
 poller | DBG | 999-ms timeout
 reconnect | INFO | tcp:127.0.0.1:45932: connection attempt timed out
 reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF
 poller | DBG | 0-ms timeout
 reconnect | INFO | tcp:127.0.0.1:45932: connecting...
 <...>
 reconnect | DBG | tcp:127.0.0.1:45932: entering CONNECTING
 poller | DBG | 1999-ms timeout
 reconnect | INFO | tcp:127.0.0.1:45932: connection attempt timed out
 reconnect | INFO | tcp:127.0.0.1:45932: waiting 4 seconds before reconnect
 reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF
 <...>

Fix that by always picking the new remote on disconnect.
This mimics the behaviour of jsonrpc C library.

Fixes "multiple remotes" tests on FreeBSD.

CC: Numan Siddique <nusiddiq@redhat.com>
Fixes: 31e434fc985c ("python jsonrpc: Allow jsonrpc_session to have more than one remote.")
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-12-27 09:25:09 -08:00
Ilya Maximets
cfef5ae8f0 socket-util: Report POLLHUP as an error while connection completion checking.
Otherwise failed non-blocking connection could be reported as
connected. This causes errors in all following operations with the
socket.

At least this is true on FreeBSD, where POLLHUP could be set without
POLLERR.

For example, stream_open_block() tests fails with the following error
reporting successful connection to the 'WRONG_PORT':

  ./ovsdb-idl.at:1817:
             $PYTHON2 $srcdir/test-stream.py tcp:127.0.0.1:$WRONG_PORT
  stdout:
  ./ovsdb-idl.at:1817: exit code was 0, expected 1
  2399. ovsdb-idl.at:1817:  FAILED (ovsdb-idl.at:1817)

Also added new tests to track this issue in C library:
  'Check Stream open block - C - tcp'
  'Check Stream open block - C - tcp6'

CC: Numan Siddique <nusiddiq@redhat.com>
Fixes: c1aa16d191d2 ("ovs python: ovs.stream.open_block() returns success even if the remote is unreachable")
Fixes: d6cedfd9d29d ("socket-util: Avoid using SO_ERROR.")
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-12-24 11:50:43 -08:00
Ilya Maximets
bdc000b2aa python: Catch setsockopt exceptions for TCP stream.
'sock.setsockopt' could throw exceptions. For example, if non-blocking
connection failed before the call:

  Traceback (most recent call last):
    File "../.././test-ovsdb.py", line 896, in <module>
      main(sys.argv)
    File "../.././test-ovsdb.py", line 891, in main
      func(*args)
    File "../.././test-ovsdb.py", line 604, in do_idl
      ovs.stream.Stream.open(r))
    File "/root/git_/ovs/python/ovs/stream.py", line 190, in open
      error, sock = cls._open(suffix, dscp)
    File "/root/git_/ovs/python/ovs/stream.py", line 744, in _open
      sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
    File "/usr/local/lib/python2.7/socket.py", line 228, in meth
      return getattr(self._sock,name)(*args)
  socket.error: [Errno 54] Connection reset by peer

This fixes tests on FreeBSD.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-12-20 10:10:55 -08:00
Eric Lapointe
e120ff1f8e python-c-ext: Fix memory leak in Parser_finish
The memory returned by json_parser_finish needs to be freed by the caller.

Signed-off-by: Eric Lapointe <elapointe@corsa.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-10-31 10:34:17 -07:00
Ilya Maximets
e6703555e5 Revert "Test the Python C JSON extension"
This reverts commit a7be68a4d77791bbe02c37f7ad8ae60b02e5679e
and a subsequent commit 4617d1f6bd24c543f533f6485b42ebca6b0a8371.
There are too many issues with these patches. It's better to revert
them for now and make a separate fixed versions later if needed.

List of issues (maybe not full):

1. 'make clean' removes entire 'python' directory.

2. Fully broken Travis-CI testsuite build:
    building 'ovs._json' extension
    creating build/temp.linux-x86_64-2.7
    error: could not create 'build/temp.linux-x86_64-2.7': \
           Permission denied
    https://travis-ci.org/openvswitch/ovs/jobs/440693765

3. Broken local testsuite build on Ubuntu 18.04:
    running build_ext
    building 'ovs._json' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/ovs
    <...>
    /usr/bin/ld: .libs/libopenvswitch.a(util.o): \
        relocation R_X86_64_TPOFF32 against `var.7749' can not be \
        used when making a shared object; recompile with -fPIC
    <...>
    collect2: error: ld returned 1 exit status

4. Fedora build failure because of 'setuptools' ('distutils')
   hard dependency on 'redhat-rpm-config' package:
    building 'ovs._json' extension
    <...>
    gcc: error: <...>/redhat-hardened-cc1: No such file or directory

5. Looks like 'setuptools' also could download and install
   unwanted python modules during package build.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-10-15 11:13:48 -07:00
Terry Wilson
4617d1f6bd Work around Python/C JSON unicode differences
The OVS C-based JSON parser operates on bytes, so the parser_feed
function returns the number of bytes that are processed. The pure
Python JSON parser currently operates on unicode, so it expects
that Parser.feed() returns a number of characters. This difference
leads to parsing errors when unicode characters are passed to the
C JSON parser from Python.

Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
2018-10-11 15:00:59 -07:00
Terry Wilson
a7be68a4d7 Test the Python C JSON extension
The C JSON parser was added quite a while ago, but unless you
configure with --enable-shared and have the Python 2/3 development
libraries installed, and the resulting python-ovs module installed,
'make check' won't actually test it.

This patch changes Python-based tests to run from the
$builddir/python directory and makes the tests configurable to use
both JSON backends. There are some unicode failures in the C JSON
extension that I left unfixed in this patch to make it easy to
show run the new tests on broken code. The next patch in this set
works around the issue.

Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
2018-10-11 15:00:46 -07:00
Lucas Alvares Gomes
f192ba279e Python: Make Row's __getattr__ less error prone
Calling getattr() on a Row object after invoking delkey() with a value
that does not exist in the object will cause getattr() to fail with a
KeyError error. For example:

Oct 05 14:59:28 neutron-server[28435]:   File
"/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 122, in run
Oct 05 14:59:28 neutron-server[28435]:
txn.results.put(txn.do_commit())
Oct 05 14:59:28 neutron-server[28435]:   File
"/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py",
line 86, in do_commit
Oct 05 14:59:28 neutron-server[28435]:     command.run_idl(txn)
Oct 05 14:59:28 neutron-server[28435]:   File
"/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/command.py",
line 299, in run_idl
Oct 05 14:59:28 neutron-server[28435]:     if
isinstance(getattr(record, self.column), dict):
Oct 05 14:59:28 neutron-server[28435]:   File
"/usr/local/lib/python2.7/dist-packages/ovs/db/idl.py", line 843, in
__getattr__
Oct 05 14:59:28 neutron-server[28435]:     del dmap[key]
Oct 05 14:59:28 neutron-server[28435]: KeyError: 'bogusvalue'

This patch is replacing the "del dmap[key]" instruction with a
"dmap.pop(key, None)" instruction instead because a pop() (with a
default value) will not raise an exception in case the key does not
exist in the object in the first place, it will just ignore it.

Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-10-05 09:26:12 -07:00
Ilya Maximets
9551e80bef tests: Use environment variable for default timeout.
Introduce new 'OVS_CTL_TIMEOUT' environment variable
that, if set, will be used as a default timeout for
OVS control utilities. Setting it in 'atlocal.in' will
cover all the hangs inside the testsuite, even when
utils called in a subshell.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-08-15 11:36:18 -07:00
Numan Siddique
31e434fc98 python jsonrpc: Allow jsonrpc_session to have more than one remote.
Python IDL implementation doesn't have the support to connect to the
cluster dbs. This patch adds this support. We are still missing the
support in python idl class to connect to the cluster master. That
support will be added in an upcoming patch.

This patch is similar to the commit 8cf6bbb184 which added multiple remote
support in the C jsonrpc implementation.

Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-08-14 11:49:28 -07:00
Numan Siddique
c1aa16d191 ovs python: ovs.stream.open_block() returns success even if the remote is unreachable
The python function ovs.socket_util.check_connection_completion() uses select()
(provided by python) to monitor the socket file descriptor. The select()
returns 1 when the file descriptor becomes ready. For error cases like -
111 (Connection refused) and 113 (No route to host) (POLLERR), ovs.poller._SelectSelect.poll()
expects the exceptfds list to be set by select(). But that is not the case.
As per the select() man page, writefds list will be set for POLLERR.
Please see "Correspondence between select() and poll() notifications" section of select(2)
man page.

Because of this behavior, ovs.socket_util.check_connection_completion() returns success
even if the remote is unreachable or not listening on the port.

This patch fixes this issue by using poll() to check the connection status similar to
the C implementation of check_connection_completion().

A new function 'get_system_poll() is added in ovs/poller.py which returns the
select.poll() object. If select.poll is monkey patched by eventlet/gevent, it
gets the original select.poll() and returns it.

The test cases added in this patch fails without the fix.

Suggested-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Mark Michelson <mmichels@redhat.com>
2018-08-14 11:47:12 -07:00
Terry Wilson
2360464d62 Add ovs.compat module to python package
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Timothy Redaelli <tredaelli@redhat.com>
2018-08-10 11:35:37 -07:00
Ben Pfaff
e11f0c258d tests: Don't log to syslog during tests.
Until now, "make check" generated a huge amount of output to syslog.  This
commit suppresses it.

Acked-by: Ilya Maximets <i.maximets@samsung.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-08-09 15:40:09 -07:00
Flavio Leitner
b3d7debaab json: Use unnamed embedded union.
Otherwise the code does not build.

Fixes: fa37affad362 ("Embrace anonymous unions.")
Signed-off-by: Flavio Leitner <fbl@sysclose.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-08-06 16:46:37 -07:00
Yifeng Sun
771680d96f DNS: Add basic support for asynchronous DNS resolving
This patch is a simple implementation for the proposal discussed in
https://mail.openvswitch.org/pipermail/ovs-dev/2017-August/337038.html and
https://mail.openvswitch.org/pipermail/ovs-dev/2017-October/340013.html.

It enables ovs-vswitchd and other utilities to use DNS names when specifying
OpenFlow and OVSDB remotes.

Below are some of the features and limitations of this patch:
    - Resolving is asynchornous in daemon context, avoiding blocking main loop;
    - Resolving is synchronous in general utility context;
    - Both IPv4 and IPv6 are supported;
    - The resolving API is thread-safe;
    - Depends on the unbound library;
    - When multiple ip addresses are returned, only the first one is used;
    - /etc/nsswitch.conf isn't respected as unbound library doesn't look at it;
    - For async-resolving, caller need to retry later; there is no callback.

Signed-off-by: Yifeng Sun <pkusunyifeng@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-07-06 13:50:00 -07:00
Toms Atteka
7a68987a4d python: Update docstring in ovs.db.idl.Idl class.
Adjusted docstring and variable names according to previous code changes;
Fixed grammar "a attribute" > "an attribute".

Fixes: bf42f674 (idl: Convert python daemons to utilize SchemaHelper)
Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-06-05 11:43:59 -07:00
Ben Pfaff
fa37affad3 Embrace anonymous unions.
Several OVS structs contain embedded named unions, like this:

struct {
    ...
    union {
        ...
    } u;
};

C11 standardized a feature that many compilers already implemented
anyway, where an embedded union may be unnamed, like this:

struct {
    ...
    union {
        ...
    };
};

This is more convenient because it allows the programmer to omit "u."
in many places.  OVS already used this feature in several places.  This
commit embraces it in several others.

Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
Tested-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
2018-05-25 13:36:05 -07:00
Jakub Sitnicki
28058c298b python: Tighten the check if we need encoding
Check if we are dealing with a Unicode string that needs
encoding for both Python 2 & 3.

Also, do the encoding the same way for Python 2 & 3 and avoid using
negation to make the code simpler.

Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-04-19 09:27:32 -07:00
Jakub Sitnicki
c7ea0e6b79 python: Fix a double encoding attempt on an Unicode string
Encoding from 'unicode' to 'str' that has been added to the Stream class
in commit 2254074e3067 ("python: fix python3 encode/decode on Windows")
conflicts with SSLStream which already contains a quirk for pyopenssl
that does the same thing.

This results in a double encoding attempt when SSL is used and we crash
and burn due to:

Traceback (most recent call last):
  File "../.././test-ovsdb.py", line 874, in <module>
    main(sys.argv)
  File "../.././test-ovsdb.py", line 869, in main
    func(*args)
  File "../.././test-ovsdb.py", line 655, in do_idl
    idl_set(idl, command, step)
  File "../.././test-ovsdb.py", line 526, in idl_set
    status = txn.commit_block()
  File "/home/jkbs/src/ovs/python/ovs/db/idl.py", line 1405, in commit_block
    status = self.commit()
  File "/home/jkbs/src/ovs/python/ovs/db/idl.py", line 1388, in commit
    if not self.idl._session.send(msg):
  File "/home/jkbs/src/ovs/python/ovs/jsonrpc.py", line 540, in send
    return self.rpc.send(msg)
  File "/home/jkbs/src/ovs/python/ovs/jsonrpc.py", line 244, in send
    self.run()
  File "/home/jkbs/src/ovs/python/ovs/jsonrpc.py", line 203, in run
    retval = self.stream.send(self.output)
  File "/home/jkbs/src/ovs/python/ovs/stream.py", line 808, in send
    return super(SSLStream, self).send(buf)
  File "/home/jkbs/src/ovs/python/ovs/stream.py", line 391, in send
    buf = buf.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 83: ordinal not in range(128)

Remove the quirk from SSLStream as the base class now does encoding.

Reported-by: Marcin Mirecki <mmirecki@redhat.com>
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-04-18 11:33:35 -07:00
Terry Wilson
13973bc415 Add multi-column index support for the Python IDL
This adds multi-column index support for the Python IDL that is
similar to the feature in the C IDL. Since it adds sortedcontainers
as a dependency and some distros don't yet package it, the library
is copied in-tree and used if sortedcontainers is not installed.

Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-04-17 14:53:12 -07:00
Ben Pfaff
5ee527e223 reconnect: Add ability to do a number of retries without backoff.
This is aimed at an upcoming database clustering implementation, where it's
desirable to try all of the cluster members quickly before backing off to
retry them again in sequence.

Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
2018-03-24 12:04:51 -07:00
Ben Pfaff
62bba60935 ovsdb-idl: Use modern form of <monitor-requests>.
Long ago, a <monitor-requests> object in the OVSDB protocol mapped a table
name to a single <monitor-request>.  Since then, it has mapped a table name
to an *array of* <monitor-request> objects, but the OVSDB IDL has never
been updated to use the modern form.  This commit makes that change.

Reported-by: Anil Jangam <anilj.mailing@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
2018-03-14 11:27:19 -07:00
Timothy Redaelli
685e6983c6 python: KeyError shouldn't be raised from __getattr__
On Python 3 hasattr only intercepts AttributeError exception.
On Python2, instead, hasattr intercepts all the exceptions.

This means __getattr__ shouldn't return KeyError when the attribute
doesn't exists, but it should raise AttributeError instead.

Fixes: 2d54d8011e14 ("Python-IDL: getattr after mutate fix")
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
2018-03-14 11:34:05 -04:00
Guoshuai Li
980211b75d python: Fix decoding error when the received data is larger than 4096.
It can only receive 4096 bytes of data each time in jsonrpc,
when there are similar and Chinese characters occupy multiple bytes,
it may receive half a character, this time the decoding will be abnormal.
We need to receive the completed character to decode.

Signed-off-by: Guoshuai Li <ligs@dtdream.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-03-09 11:22:48 -08:00
Daniel Alvarez
eab138764e python: avoid useless JSON conversion to enhance performance
This patch removes a useless conversion to/from JSON in the
processing of any 'modify' operations inside the process_update2
method in Python IDL implementation.

Previous code will make resources creation take longer as the number
of elements in the row grows because of that JSON conversion. This
patch eliminates it and now the time remains consant regardless
of the database contents improving performance and scaling.

Reported-by: Daniel Alvarez <dalvarez@redhat.com>
Reported-at: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-February/046263.html
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>
Acked-by: Terry Wilson <twilson@redhat.com>
Tested-By: Terry Wilson <twilson@redhat.com>
Acked-by: Han Zhou <hzhou8@ebay.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-02-28 13:09:03 -08:00
Ben Pfaff
00386a6921 ovsdb-idlc: Add infrastructure for IDL schema extensions.
An IDL schema is an OVSDB schema with some extra stuff in it.  So far, all
of the extras have been at the top level.  This commit makes it possible
for IDL schemas to have extra information at the table and column levels as
long as it is in an "extensions" member.

No extensions are actually supported yet.

Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-02-16 15:08:08 -08:00
Ben Pfaff
0ea866bbc0 ovsdb-idlc: Add "cDecls" and "hDecls" IDL schema extensions.
An IDL schema is an OVSDB schema with some extra stuff in it: an idlPrefix
and an idlHeader at the top level to indicate what ovsdb-idlc needs to
generate the interface definitions.  This commit adds support for two more
optional IDL schema extensions that allow extra code to be written to the
.c and .h file that ovsdb-idlc generates.

Signed-off-by: Ben Pfaff <blp@ovn.org>
2018-02-16 15:08:08 -08:00
Ben Pfaff
12b84d50e0 ovsdb: Improve documentation.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
2017-12-14 11:21:42 -08:00
Ben Pfaff
74eaac06d6 tests: Convert soexpand build tool from Perl to Python.
Perl is unfashionable and Python is more widely available and understood,
so this commit converts one of the OVS uses of Perl into Python.

Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Aaron Conole <aconole@redhat.com>
2017-11-26 16:08:07 -08:00
Alin Balutoiu
4257675e00 windows,python: remove dead code in send_wait
Found while looking over the code.

Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Suggested-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Anand Kumar <kumaranand@vmware.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
2017-09-13 01:14:01 +03:00
Alin Balutoiu
0024e9e96f windows,python: remove unnecessary code
At the moment we have WSAEventSelect in each if branch.

Since the call to the function is similar, we can move
it outside the if branch and create some local variables
which will be passed to WSAEventSelect.

This patch also remove the keyword argument passed when
the event for the connection overlapped structure is created.
The argument is not needed since it does not change the value
from the default one.

Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
2017-09-13 00:38:37 +03:00
Alin Balutoiu
fef22d6871 windows, python: create a different event for sockets
At the moment the sockets on Windows use the same events
that are being created for the pipes.

This is not correct because they should be different events.

This patch introduces a new event which should be used for sockets.
The new event needs to be set on automatic reset with its initial
state not signaled.

Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Co-authored-by: Alin Gabriel Serdean <aserdean@ovn.org>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
Tested-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
2017-09-13 00:38:26 +03:00
Alin Balutoiu
29e34ce1c5 windows,python: Add restrictions to named pipes
Bump the security around named pipes to be more restrictive: disable network
access and allow only administrators and above to access the named pipes.

Acked-by: Anand Kumar <kumaranand@vmware.com>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
2017-09-13 00:04:38 +03:00
Alin Balutoiu
ba953e1e83 windows, python: Remove code duplication in send/recv functions
Move the return value at the end of the function
regardless of the pending/non-pending operation.

Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Acked-by: Russell Bryant <russell@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
2017-08-23 18:35:54 +03:00
Alin Balutoiu
e2e31d17f8 windows, python: Fix event type returned from poller
The function poll from poller should return a list of tuples
containing the events and their types.

On Windows the event type is not returned at the moment.
Instead of returning zero all the time, we check to see
the type of event and we set it accordingly before returning
the list.

This is used only for debugging purposes inside the function
"__log_wakeup" later on.

Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Acked-by: Russell Bryant <russell@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
2017-08-23 18:19:49 +03:00
Alin Balutoiu
2254074e30 python: fix python3 encode/decode on Windows
Fix double encoding/decoding on data, caused by
'get_decoded_buffer' and 'get_encoded_buffer'.

The functions 'get_decoded_buffer' and 'get_encoded_buffer'
from winutils have been removed. They are no longer
necessary since the buffers received/returned are already
in the right form.

The necessary encoding has been moved before any sending
function (this also includes named pipes send on Windows).

Co-authored-by: Alin Serdean <aserdean@cloudbasesolutions.com>
Signed-off-by: Alin Balutoiu <abalutoiu@cloudbasesolutions.com>
Signed-off-by: Alin Serdean <aserdean@cloudbasesolutions.com>
Acked-by: Lance Richardson <lrichard@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
2017-08-18 16:12:50 -04:00
Lance Richardson
e7164d96bc python: make python idl unicode-tolerant
Ensure that JSON is utf-8 encoded and that bytes sent/received on
the stream sockets are in utf-8 form. Add a test case to verify
that unicode data can be sent/received successfully using Python
IDL module.

Co-authored-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Lance Richardson <lrichard@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
2017-08-09 16:08:18 -04:00
Jason Wessel
fa145f1a53 Python3 compatibility: fix integer problems
In python3 maxint is not defined, but maxsize is defined in both
python2 and python3.

The put_text() will not automatically use a value which came in as
float due to a pior math function and python3 will throw an exception.
The simple answer is to convert it with int() and move on.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
2017-07-06 14:06:08 -07:00
Ben Pfaff
0b2c7e690a Replace most uses of and references to "ifconfig" by "ip".
It's becoming more common that OSes include "ip" but not "ifconfig", so
it's best to avoid using the latter.  This commit removes most references
to "ifconfig" and replaces them by "ip".  It also adds a build-time check
to make it harder to introduce new uses of "ifconfig".

There are important differences between "ifconfig" and "ip":

- An "ifconfig" command that sets an IP address also brings the interface
  up, but a similar "ip addr add" command does not, so it is often necessary
  (or at least precautionary) to add an "ip link set <dev> up" command.

- "ifconfig" can infer a netmask from an IP adddress, but "ip" always
  assumes /32 if none is given.

- "ifconfig" with address 0.0.0.0 removes any configured IP address, but
  "ip addr add" does not, so "ifconfig <dev> 0.0.0.0" must be replaced by
  "ip addr del" or "ip addr flush".

Signed-off-by: Ben Pfaff <blp@ovn.org>
2017-06-01 07:21:41 -07:00
Numan Siddique
ca9c2c56ff python ovs: Fix SSL exceptions with pyOpenSSL v0.13
Centos provides pyOpenSSL version pyOpenSSL-0.13.1-3.el7.x86_64.
There are 2 issues using this version, which this patch fixes

 - The test case "simple idl verify notify - SSL" is skipped.
   This is because "python -m OpenSSL.SSL" is used to detect the
   presence of pyOpenSSL package. pyOpenSSL v0.13 has C python
   modules because of which the above command returns 1.
   So this patch fixes this by using 'python -c "import OpenSSL.SSL"'.

 - The SSL.Context class does not have the function "set_session_cache_mode"
   defined.  Our usage here was only relevant for server-side connections,
   (pssl), which is not yet supported by python-ovs, so just remove the
   usage of this function.  The default cache mode (server) will just
   be ignored.

I have not tested with older versions (< 0.13) of pyOpenSSL.

Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Acked-by: Lance Richardson <lrichard@redhat.com>
Tested-by: Marcin Mirecki <mmirecki@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
2017-05-23 13:34:03 -04:00
Lucas Alvares Gomes
f73d562fc0 python: Allow tuning the session probe_interval from IDL
This patch is adding a new parameter called "probe_interval" to the
constructor of the Idl class. This new parameter will be used to tune
the database connection probing for that IDL session, some users might
want to tune it to be less agressive than the current 5s default in OVS
or even disable it.

Reported-at: https://bugs.launchpad.net/networking-ovn/+bug/1680146
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Acked-by: Daniel Alvarez <dalvarez@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
2017-04-13 12:08:01 -04:00
xurong00037997
6c7050b59c Adapt to flake8-import-order
https://review.openstack.org/#/c/432906/
flake8-import-order adds 3 new flake8 warnings:
I100: Your import statements are in the wrong order.
I101: The names in your from import are in the wrong order.
I201: Missing newline between sections or imports.

Signed-off-by: Ben Pfaff <blp@ovn.org>
2017-03-08 21:11:48 -08:00
Russell Bryant
dc42dbd0dc python: List 3.5 as a supported version.
All tests pass with Python 3.5 so list it as supported.

Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
2017-03-08 13:40:56 -05:00