As described in commit [1], it's possible that remote IP is backed by
a load-balancer and re-connection to this same IP will lead to
connection to a different server. This case is supported for C version
of IDL and should be supported in a same way for python implementation.
[1] ca367fa5f8 ("ovsdb-idl.c: Allows retry even when using a single remote.")
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Dumitru Ceara <dceara@redhat.com>
The Python IDL notification mechanism was sending a notification
for each processed update in a transaction as it was processed.
This causes issues with multi-row changes that contain references
to each other.
For example, if a Logical_Router_Port is created along with a
Gateway_Chassis, and the LRP.gateway_chassis set to that GC, then
when the notify() passes the CREATE event for the LRP, the GC will
not yet have been processed, so __getattr__ when _uuid_to_row fails
to find the GC, will return the default value for LRP.gateway_chassis
which is [].
This patch has the process_update methods return the notifications
that would be produced when a row changes, so they can be queued
and sent after all rows have been processed.
Fixes: d7d417fcdd ("Allow subclasses of Idl to define a notification hook")
Signed-off-by: Terry Wilson <twilson@redhat.com>
Acked-by: Brian Haley <haleyb.dev@gmail.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Tested-by: Flavio Fernandes <flavio@flaviof.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
For columns like QoS.queues where we have a map containing refTable
values, assigning w/ __setattr__ e.g. qos.queues={1: $queue_row}
works, but using using qos.setkey('queues', 1, $queue_row) results
in an Exception. The opdat argument can essentially just be the
JSON representation of the map column instead of trying to build
it.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Since Python 2 support was removed in 1ca0323e7c ("Require Python 3 and
remove support for Python 2."), python3-six is not needed anymore.
Moreover python3-six is not available on RHEL/CentOS7 without using EPEL
and so this patch is needed in order to release OVS 2.13 on RHEL7.
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
It's sometimes handy to log an entire Row object, so this just
adds a string representation of the object as:
Tablename(col1=val1, col2=val2, ..., coln=valn)
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The Python IDL implementation supports ovsdb cluster connections.
This patch is a follow up to commit 31e434fc98, it adds the option of
connecting to the leader (the default) in the Raft-based cluster. It mimics
the exisiting C IDL support for clusters introduced in commit 1b1d2e6daa.
The _Server database schema is first requested, then a monitor of the
Database table in the _Server Database. Method __check_server_db verifies
the eligibility of the server. If the attempt to obtain a monitor of the
_Server database fails and a cluster id was not provided this implementation
proceeds to request the data monitor. If a cluster id was provided via the
set_cluster_id method then the connection is aborted and a connection to a
different node is instead attempted, until a valid cluster node is found.
Thus, when supplied, cluster id is interpreted as the intention to only
allow connections to a clustered database. If not supplied, connections to
standalone nodes, or nodes that do not have the _Server database are
allowed. change_seqno is not incremented in the case of Database table
updates.
Acked-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ted Elhourani <ted.elhourani@nutanix.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Calling getattr() on a Row object after invoking delkey() with a value
that does not exist in the object will cause getattr() to fail with a
KeyError error. For example:
Oct 05 14:59:28 neutron-server[28435]: File
"/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 122, in run
Oct 05 14:59:28 neutron-server[28435]:
txn.results.put(txn.do_commit())
Oct 05 14:59:28 neutron-server[28435]: File
"/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py",
line 86, in do_commit
Oct 05 14:59:28 neutron-server[28435]: command.run_idl(txn)
Oct 05 14:59:28 neutron-server[28435]: File
"/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/command.py",
line 299, in run_idl
Oct 05 14:59:28 neutron-server[28435]: if
isinstance(getattr(record, self.column), dict):
Oct 05 14:59:28 neutron-server[28435]: File
"/usr/local/lib/python2.7/dist-packages/ovs/db/idl.py", line 843, in
__getattr__
Oct 05 14:59:28 neutron-server[28435]: del dmap[key]
Oct 05 14:59:28 neutron-server[28435]: KeyError: 'bogusvalue'
This patch is replacing the "del dmap[key]" instruction with a
"dmap.pop(key, None)" instruction instead because a pop() (with a
default value) will not raise an exception in case the key does not
exist in the object in the first place, it will just ignore it.
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Python IDL implementation doesn't have the support to connect to the
cluster dbs. This patch adds this support. We are still missing the
support in python idl class to connect to the cluster master. That
support will be added in an upcoming patch.
This patch is similar to the commit 8cf6bbb184 which added multiple remote
support in the C jsonrpc implementation.
Acked-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
This adds multi-column index support for the Python IDL that is
similar to the feature in the C IDL. Since it adds sortedcontainers
as a dependency and some distros don't yet package it, the library
is copied in-tree and used if sortedcontainers is not installed.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Long ago, a <monitor-requests> object in the OVSDB protocol mapped a table
name to a single <monitor-request>. Since then, it has mapped a table name
to an *array of* <monitor-request> objects, but the OVSDB IDL has never
been updated to use the modern form. This commit makes that change.
Reported-by: Anil Jangam <anilj.mailing@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
On Python 3 hasattr only intercepts AttributeError exception.
On Python2, instead, hasattr intercepts all the exceptions.
This means __getattr__ shouldn't return KeyError when the attribute
doesn't exists, but it should raise AttributeError instead.
Fixes: 2d54d8011e ("Python-IDL: getattr after mutate fix")
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
This patch removes a useless conversion to/from JSON in the
processing of any 'modify' operations inside the process_update2
method in Python IDL implementation.
Previous code will make resources creation take longer as the number
of elements in the row grows because of that JSON conversion. This
patch eliminates it and now the time remains consant regardless
of the database contents improving performance and scaling.
Reported-by: Daniel Alvarez <dalvarez@redhat.com>
Reported-at: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-February/046263.html
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>
Acked-by: Terry Wilson <twilson@redhat.com>
Tested-By: Terry Wilson <twilson@redhat.com>
Acked-by: Han Zhou <hzhou8@ebay.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
This patch is adding a new parameter called "probe_interval" to the
constructor of the Idl class. This new parameter will be used to tune
the database connection probing for that IDL session, some users might
want to tune it to be less agressive than the current 5s default in OVS
or even disable it.
Reported-at: https://bugs.launchpad.net/networking-ovn/+bug/1680146
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Acked-by: Daniel Alvarez <dalvarez@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
https://review.openstack.org/#/c/432906/
flake8-import-order adds 3 new flake8 warnings:
I100: Your import statements are in the wrong order.
I101: The names in your from import are in the wrong order.
I201: Missing newline between sections or imports.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Since __txn_process_reply always returns None, the existing code
will always hit the final else for replies and log a debug message
about receiving an unexpected reply. In the C version,
ovsdb_idl_txn_process_reply returns true any time the txn is found,
so that behavior is duplicated here.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
Most users of OVSDB react to whatever is currently in their view of the
database, as opposed to keeping track of changes and reacting to those
changes individually. The interface to conditional monitoring was
different, in that it expected the client to say what to add or remove from
monitoring instead of what to monitor. This seemed reasonable at the time,
but in practice it turns out that the usual approach actually works better,
because the condition is generally a function of the data visible in the
database. This commit changes the approach.
This commit also changes the meaning of an empty condition for a table.
Previously, an empty condition meant to replicate every row. Now, an empty
condition means to replicate no rows. This is more convenient for code
that gradually constructs conditions, because it does not need special
cases for replicating nothing.
This commit also changes the internal implementation of conditions from
linked lists to arrays. I just couldn't see an advantage to using linked
lists.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Liran Schour <lirans@il.ibm.com>
This patch fixes the scenario, where the mutate operation on a row
is sent in the same transaction as row insert operation. It was
obvserved that this mutate operation was not getting committed
to the OVSDB.
To get around the above problem the "where" condition in an
mutate operation is modified to use the named-uuid to identify
a row created in the current transaction.
Signed-off-by: Amitabha Biswas <abiswas@us.ibm.com>
Suggested-by: Richard Theis <rtheis@us.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
This patch fixes a couple of bugs in commit a59912a0
(python: add support for partial map and partial set updates)
and reverses a simplication added in commit 884d9bad
(Simplify partial map Py3 IDL test) to make the Python3 test
cases passes.
The following changes have been made:
1. Allow multiple map updates on the same column in a transaction.
2. Partial map Py3 IDL test can now support multiple elements.
3. SetAttr overrides pre-existing insert and remove updates.
4. addvalue/delvalue contains unique elements
Signed-off-by: Amitabha Biswas <abiswas@us.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Allow the python IDL to use mutate operations more freely
by mimicing the partial map and partial set operations now
available in the C IDL.
Unit tests for both of these types of operations are included.
They are not carbon copies of the C tests, because testing
idempotency is a bit difficult for the current python IDL
test harness.
Signed-off-by: Ryan Moats <rmoats@us.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
When python IDL calls the "notify" function after processing the "update2"
message from ovsdb-server, it is suppose to send the old values of the
updated columns as the last parameter. But the recent commit "897c8064"
sends the updated values. This breaks the behaviour.
This patch fixes this issue. It also updates the description of
the 'updates' param of the notify function to make it more clear.
Fixes: 897c8064 ("python: move Python idl to work with monitor_cond")
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Add to IDL API that allows the user to add and remove clauses on a table's condition
iteratively. IDL maintain tables condition and send monitor_cond_change to the server
upon condition change.
Add tests for conditional monitoring to IDL.
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Python idl works now with "monitor_cond" method. Add test
for backward compatibility with old "monitor" method.
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Python 3 no longer supports __cmp__. Instead, we have to implement the
"rich comparison" operators. We implement __eq__ and __lt__ and use
functools.total_ordering to implement the rest.
In one case, no __cmp__ method was provided and instead relied on the
default behavior provided in Python 2. We have to implement the
comparisons explicitly for Python 3.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
I've hit several bugs in this Python 3 work where the fix was some code
needed to be converted to use isinstance(). This has been primarily
around deadling with the changes to unicode handling. Go ahead and
convert the rest of the direct type comparisons to use isinstance(), as
it could avoid a bug I haven't hit yet and it's more Pythonic, anyway.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Python 2 has both long and int types. Python 3 only has int, which
behaves like long.
In the case of needing a set of integer types, we can use
six.integer_types which includes int and long for Python 2 and just int
for Python 3.
We can convert all cases of long(value) to int(value), because as of
Python 2.4, when the result of an operation would be too big for an int,
the type is automatically converted to a long.
There were several places in this patch doing type comparisons. The
preferred way to do this is using the isinstance() or issubclass()
built-in functions, so I converted the similar checks nearby while I was
at it.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
In Python 2, dict.items(), dict.keys(), and dict.values() returned a
list. dict.iteritems(), dict.iterkeys(), and dict.itervalues() returned
an iterator.
As of Python 3, dict.iteritems(), dict.itervalues(), and dict.iterkeys()
are gone. items(), keys(), and values() now return an iterator.
In the case where we want an iterator, we now use the six.iter*()
helpers. If we want a list, we explicitly create a list from the
iterator.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
This patch fixes just the Python 3 problems found by running:
python3 setup.py install
There are still many other issues to be fixed, but this is a start.
Signed-off-by: Terry Wilson <twilson@redhat.com>
[russell@ovn.org resolved conflicts with current master]
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
This patch resolves the following warnings from flake8:
E111 indentation is not a multiple of four
E112 expected an indented block
E113 unexpected indentation
It's critical to have correct indentation in Python code, so it seemed
worth enabling these warnings.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Fix the following pep8 errors:
E201 whitespace after '('
E203 whitespace before ','
E222 multiple spaces after operator
E225 missing whitespace around operator
E226 missing whitespace around arithmetic operator
E231 missing whitespace after ':'
E241 multiple spaces after ':'
E251 unexpected spaces around keyword / parameter equals
E261 at least two spaces before inline comment
E262 inline comment should start with '# '
E265 block comment should start with '# '
E271 multiple spaces after keyword
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Resolve pep8 errors:
E711 comparison to None should be 'if cond is None:'
The reason comparing against None with "is None" is preferred over
"== None" is because a class can define its own equality operator and
produce bizarre unexpected behavior. Using "is None" has a very
explicit meaning that can not be overridden.
E721 do not compare types, use 'isinstance()'
This one is actually a mistake by the tool in most cases.
'from ovs.db import types' looks just like types from the Python stdlib.
In those cases, use the full ovs.db.types name. Fix one case where it
actually was types from the stdlib.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
This code referred to "rows" where it meant to refer to "fetched_rows".
The patch resolves flake8 error:
F821 undefined name 'rows'
python/build/nroff.py used a function fatal() that was not defined,
which raised the same type of error.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
There is currently no mechanism in IDL to fetch specific column values
on-demand without having to register them for monitoring. In the case
where the column represent a frequently changing entity (e.g. counter),
and the reads are relatively infrequent (e.g. CLI client), there is a
significant overhead in replication.
This patch adds support in the Python IDL to register a subset of the
columns of a table as "readonly". Readonly columns are not replicated.
Users may "fetch" the readonly columns of a row on-demand. Once fetched,
the columns are not updated until the next fetch by the user. Writes by
the user to readonly columns does not change the value (both locally or
on the server).
The two main user visible changes in this patch are:
- The SchemaHelper.register_columns() method now takes an optionaly
argument to specify the subset of readonly column(s)
- A new Row.fetch(columns) method to fetch values of readonly columns(s)
Usage:
------
# Schema file includes all columns, including readonly
schema_helper = ovs.db.idl.SchemaHelper(schema_file)
# Register interest in columns with 'r' and 's' as readonly
schema_helper.register_columns("simple", [i, r, s], [r, s])
# Create Idl and jsonrpc, and wait for update, as usual
...
# Fetch value of column 'r' for a specific row
row.fetch('r')
txn.commit_block()
print row.r
print getattr(row, 'r')
# Writing to readonly column has no effect (locally or on server)
row.r = 3
print row.r # prints fetched value not 3
Signed-off-by: Shad Ansari <shad.ansari@hp.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
When 'alert' was turned off on a column, the code was erroring out when
value for that column was being set in a newly inserted row. This is
because the row._data was None at this time.
It seems that new rows are not initialized to defaults and that's why the
NULL error happens. IMO a newly inserted row should automatically get
intialized to default values. This new behavior can be implemented as a
separate improvement sometime in the future.
For now, I don't see an issue with adding the additional check. This new
check can continue as-is even after the new behavior is implemented.
Signed-off-by: Sumit Garg <sumit@extremenetworks.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
A bool (has_lock) was being accessed as a function call leading to a
runtime exception.
Signed-off-by: Sumit Garg <sumit@extremenetworks.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
It is useful to make the notification events that Idl processes
accessible to users of the library. This will make it possible to
keep external systems in sync, but does not impose any particular
notification pattern.
The Row.from_json() call is added to be able to convert the 'old'
JSON response on an update to a Row object to make it easy for
users of notify() to see what changed, though this usage of Row
is quite different than Idl's typical use.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Row.delete() handled the case of deleting a row that was added within the
current transaction, but not yet committed, but it did not correctly handle
the case of deleting a row that belonged to the database before the
transaction started. This fixes the problem.
Reported-by: Yeming Zhao <zhaoyeming@gmail.com>
Tested-by: Yeming Zhao <zhaoyeming@gmail.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
In some cases getattr(Row instance, attrname) doesn't raise AttributeError,
but TypeError
> File "python/ovs/db/idl.py", line 554, in __getattr__
> datum = self._data[column_name]
> TypeError: 'NoneType' object has no attribute '__getitem__'
So getattr(Row instance, attrname, default value) doesn't work.
This occurs when row._changes doesn't include attrname and row._data is None.
So teach Row.__getattr__ _data=None case.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Since Transaction._substitute doesn't substitute elements of list/tuple,
setting list references results in transaction error. Teach it such case.
Example:
{"op": "update",
"row":{"bridges":["set",[["uuid",
"1f42bc19-307f-42e7-a9c0-c12178bd8b51"],
["uuid",
"f97e0c76-7146-489d-9bed-29bc704f65fe"]]]},
"table": "Open_vSwitch",
"where":[["_uuid", "==", ["uuid",
"20c2a046-ae7e-4453-a576-11034db24985"]]]}
In the above case, uuid in "row" aren't replaced by "named-uuid" because
the function doesn't look into elements of lists.
When list/tuple is found, look into elements recursively.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
This is needed when using schema that was retrieved from ovsdb by get_schema
method.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Add register_table method to SchemaHelper as Python counterpart of
ovsdb_idl_add_table() in the C version of the IDL.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Replaced all instances of Nicira Networks(, Inc) to Nicira, Inc.
Feature #10593
Signed-off-by: Raju Subramanian <rsubramanian@nicira.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Originally the IDL transaction state machine had a return value
TXN_TRY_AGAIN to signal the client to wait for a change in the database and
then retry its transaction. However, this logic was incomplete, because
it was possible for the database to change before the reply to the
transaction RPC was received, in which case the client would wait for a
further change. Commit 4fdfe5ccf8 (ovsdb-idl: Prevent occasional hang
when multiple database clients race.) fixed the problem by breaking
TXN_TRY_AGAIN into two status codes, TXN_AGAIN_WAIT that meant to wait for
a further change and TXN_AGAIN_NOW that meant that a change had already
occurred so try again immediately.
This is correct enough, but it is more complicated than necessary. It is
simpler and just as correct to use a single "try again" status that
requires the client to wait for a change relative to the database contents
*before* the transaction was committed. This commit makes that change.
It also changes ovsdb_idl_run()'s return type from bool to void because
its return type is hardly useful anymore.
Signed-off-by: Ben Pfaff <blp@nicira.com>
The recently added SchemaHelper class significantly simplifies IDL
instantiation in Python. This commit converts all users of the old
method to the new method, and removes support for the old method.
Signed-off-by: Ethan Jackson <ethan@nicira.com>
