Long ago, a <monitor-requests> object in the OVSDB protocol mapped a table
name to a single <monitor-request>. Since then, it has mapped a table name
to an *array of* <monitor-request> objects, but the OVSDB IDL has never
been updated to use the modern form. This commit makes that change.
Reported-by: Anil Jangam <anilj.mailing@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
On Python 3 hasattr only intercepts AttributeError exception.
On Python2, instead, hasattr intercepts all the exceptions.
This means __getattr__ shouldn't return KeyError when the attribute
doesn't exists, but it should raise AttributeError instead.
Fixes: 2d54d8011e ("Python-IDL: getattr after mutate fix")
Signed-off-by: Timothy Redaelli <tredaelli@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
This patch removes a useless conversion to/from JSON in the
processing of any 'modify' operations inside the process_update2
method in Python IDL implementation.
Previous code will make resources creation take longer as the number
of elements in the row grows because of that JSON conversion. This
patch eliminates it and now the time remains consant regardless
of the database contents improving performance and scaling.
Reported-by: Daniel Alvarez <dalvarez@redhat.com>
Reported-at: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-February/046263.html
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>
Acked-by: Terry Wilson <twilson@redhat.com>
Tested-By: Terry Wilson <twilson@redhat.com>
Acked-by: Han Zhou <hzhou8@ebay.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
This patch is adding a new parameter called "probe_interval" to the
constructor of the Idl class. This new parameter will be used to tune
the database connection probing for that IDL session, some users might
want to tune it to be less agressive than the current 5s default in OVS
or even disable it.
Reported-at: https://bugs.launchpad.net/networking-ovn/+bug/1680146
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Acked-by: Daniel Alvarez <dalvarez@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
https://review.openstack.org/#/c/432906/
flake8-import-order adds 3 new flake8 warnings:
I100: Your import statements are in the wrong order.
I101: The names in your from import are in the wrong order.
I201: Missing newline between sections or imports.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Since __txn_process_reply always returns None, the existing code
will always hit the final else for replies and log a debug message
about receiving an unexpected reply. In the C version,
ovsdb_idl_txn_process_reply returns true any time the txn is found,
so that behavior is duplicated here.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Russell Bryant <russell@ovn.org>
Most users of OVSDB react to whatever is currently in their view of the
database, as opposed to keeping track of changes and reacting to those
changes individually. The interface to conditional monitoring was
different, in that it expected the client to say what to add or remove from
monitoring instead of what to monitor. This seemed reasonable at the time,
but in practice it turns out that the usual approach actually works better,
because the condition is generally a function of the data visible in the
database. This commit changes the approach.
This commit also changes the meaning of an empty condition for a table.
Previously, an empty condition meant to replicate every row. Now, an empty
condition means to replicate no rows. This is more convenient for code
that gradually constructs conditions, because it does not need special
cases for replicating nothing.
This commit also changes the internal implementation of conditions from
linked lists to arrays. I just couldn't see an advantage to using linked
lists.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Liran Schour <lirans@il.ibm.com>
This patch fixes the scenario, where the mutate operation on a row
is sent in the same transaction as row insert operation. It was
obvserved that this mutate operation was not getting committed
to the OVSDB.
To get around the above problem the "where" condition in an
mutate operation is modified to use the named-uuid to identify
a row created in the current transaction.
Signed-off-by: Amitabha Biswas <abiswas@us.ibm.com>
Suggested-by: Richard Theis <rtheis@us.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
This patch fixes a couple of bugs in commit a59912a0
(python: add support for partial map and partial set updates)
and reverses a simplication added in commit 884d9bad
(Simplify partial map Py3 IDL test) to make the Python3 test
cases passes.
The following changes have been made:
1. Allow multiple map updates on the same column in a transaction.
2. Partial map Py3 IDL test can now support multiple elements.
3. SetAttr overrides pre-existing insert and remove updates.
4. addvalue/delvalue contains unique elements
Signed-off-by: Amitabha Biswas <abiswas@us.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Allow the python IDL to use mutate operations more freely
by mimicing the partial map and partial set operations now
available in the C IDL.
Unit tests for both of these types of operations are included.
They are not carbon copies of the C tests, because testing
idempotency is a bit difficult for the current python IDL
test harness.
Signed-off-by: Ryan Moats <rmoats@us.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
When python IDL calls the "notify" function after processing the "update2"
message from ovsdb-server, it is suppose to send the old values of the
updated columns as the last parameter. But the recent commit "897c8064"
sends the updated values. This breaks the behaviour.
This patch fixes this issue. It also updates the description of
the 'updates' param of the notify function to make it more clear.
Fixes: 897c8064 ("python: move Python idl to work with monitor_cond")
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Add to IDL API that allows the user to add and remove clauses on a table's condition
iteratively. IDL maintain tables condition and send monitor_cond_change to the server
upon condition change.
Add tests for conditional monitoring to IDL.
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Python idl works now with "monitor_cond" method. Add test
for backward compatibility with old "monitor" method.
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Python 3 no longer supports __cmp__. Instead, we have to implement the
"rich comparison" operators. We implement __eq__ and __lt__ and use
functools.total_ordering to implement the rest.
In one case, no __cmp__ method was provided and instead relied on the
default behavior provided in Python 2. We have to implement the
comparisons explicitly for Python 3.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
I've hit several bugs in this Python 3 work where the fix was some code
needed to be converted to use isinstance(). This has been primarily
around deadling with the changes to unicode handling. Go ahead and
convert the rest of the direct type comparisons to use isinstance(), as
it could avoid a bug I haven't hit yet and it's more Pythonic, anyway.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Python 2 has both long and int types. Python 3 only has int, which
behaves like long.
In the case of needing a set of integer types, we can use
six.integer_types which includes int and long for Python 2 and just int
for Python 3.
We can convert all cases of long(value) to int(value), because as of
Python 2.4, when the result of an operation would be too big for an int,
the type is automatically converted to a long.
There were several places in this patch doing type comparisons. The
preferred way to do this is using the isinstance() or issubclass()
built-in functions, so I converted the similar checks nearby while I was
at it.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
In Python 2, dict.items(), dict.keys(), and dict.values() returned a
list. dict.iteritems(), dict.iterkeys(), and dict.itervalues() returned
an iterator.
As of Python 3, dict.iteritems(), dict.itervalues(), and dict.iterkeys()
are gone. items(), keys(), and values() now return an iterator.
In the case where we want an iterator, we now use the six.iter*()
helpers. If we want a list, we explicitly create a list from the
iterator.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
This patch fixes just the Python 3 problems found by running:
python3 setup.py install
There are still many other issues to be fixed, but this is a start.
Signed-off-by: Terry Wilson <twilson@redhat.com>
[russell@ovn.org resolved conflicts with current master]
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
This patch resolves the following warnings from flake8:
E111 indentation is not a multiple of four
E112 expected an indented block
E113 unexpected indentation
It's critical to have correct indentation in Python code, so it seemed
worth enabling these warnings.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Fix the following pep8 errors:
E201 whitespace after '('
E203 whitespace before ','
E222 multiple spaces after operator
E225 missing whitespace around operator
E226 missing whitespace around arithmetic operator
E231 missing whitespace after ':'
E241 multiple spaces after ':'
E251 unexpected spaces around keyword / parameter equals
E261 at least two spaces before inline comment
E262 inline comment should start with '# '
E265 block comment should start with '# '
E271 multiple spaces after keyword
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Resolve pep8 errors:
E711 comparison to None should be 'if cond is None:'
The reason comparing against None with "is None" is preferred over
"== None" is because a class can define its own equality operator and
produce bizarre unexpected behavior. Using "is None" has a very
explicit meaning that can not be overridden.
E721 do not compare types, use 'isinstance()'
This one is actually a mistake by the tool in most cases.
'from ovs.db import types' looks just like types from the Python stdlib.
In those cases, use the full ovs.db.types name. Fix one case where it
actually was types from the stdlib.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
This code referred to "rows" where it meant to refer to "fetched_rows".
The patch resolves flake8 error:
F821 undefined name 'rows'
python/build/nroff.py used a function fatal() that was not defined,
which raised the same type of error.
Signed-off-by: Russell Bryant <russell@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
There is currently no mechanism in IDL to fetch specific column values
on-demand without having to register them for monitoring. In the case
where the column represent a frequently changing entity (e.g. counter),
and the reads are relatively infrequent (e.g. CLI client), there is a
significant overhead in replication.
This patch adds support in the Python IDL to register a subset of the
columns of a table as "readonly". Readonly columns are not replicated.
Users may "fetch" the readonly columns of a row on-demand. Once fetched,
the columns are not updated until the next fetch by the user. Writes by
the user to readonly columns does not change the value (both locally or
on the server).
The two main user visible changes in this patch are:
- The SchemaHelper.register_columns() method now takes an optionaly
argument to specify the subset of readonly column(s)
- A new Row.fetch(columns) method to fetch values of readonly columns(s)
Usage:
------
# Schema file includes all columns, including readonly
schema_helper = ovs.db.idl.SchemaHelper(schema_file)
# Register interest in columns with 'r' and 's' as readonly
schema_helper.register_columns("simple", [i, r, s], [r, s])
# Create Idl and jsonrpc, and wait for update, as usual
...
# Fetch value of column 'r' for a specific row
row.fetch('r')
txn.commit_block()
print row.r
print getattr(row, 'r')
# Writing to readonly column has no effect (locally or on server)
row.r = 3
print row.r # prints fetched value not 3
Signed-off-by: Shad Ansari <shad.ansari@hp.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
When 'alert' was turned off on a column, the code was erroring out when
value for that column was being set in a newly inserted row. This is
because the row._data was None at this time.
It seems that new rows are not initialized to defaults and that's why the
NULL error happens. IMO a newly inserted row should automatically get
intialized to default values. This new behavior can be implemented as a
separate improvement sometime in the future.
For now, I don't see an issue with adding the additional check. This new
check can continue as-is even after the new behavior is implemented.
Signed-off-by: Sumit Garg <sumit@extremenetworks.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
A bool (has_lock) was being accessed as a function call leading to a
runtime exception.
Signed-off-by: Sumit Garg <sumit@extremenetworks.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
It is useful to make the notification events that Idl processes
accessible to users of the library. This will make it possible to
keep external systems in sync, but does not impose any particular
notification pattern.
The Row.from_json() call is added to be able to convert the 'old'
JSON response on an update to a Row object to make it easy for
users of notify() to see what changed, though this usage of Row
is quite different than Idl's typical use.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Row.delete() handled the case of deleting a row that was added within the
current transaction, but not yet committed, but it did not correctly handle
the case of deleting a row that belonged to the database before the
transaction started. This fixes the problem.
Reported-by: Yeming Zhao <zhaoyeming@gmail.com>
Tested-by: Yeming Zhao <zhaoyeming@gmail.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
In some cases getattr(Row instance, attrname) doesn't raise AttributeError,
but TypeError
> File "python/ovs/db/idl.py", line 554, in __getattr__
> datum = self._data[column_name]
> TypeError: 'NoneType' object has no attribute '__getitem__'
So getattr(Row instance, attrname, default value) doesn't work.
This occurs when row._changes doesn't include attrname and row._data is None.
So teach Row.__getattr__ _data=None case.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Since Transaction._substitute doesn't substitute elements of list/tuple,
setting list references results in transaction error. Teach it such case.
Example:
{"op": "update",
"row":{"bridges":["set",[["uuid",
"1f42bc19-307f-42e7-a9c0-c12178bd8b51"],
["uuid",
"f97e0c76-7146-489d-9bed-29bc704f65fe"]]]},
"table": "Open_vSwitch",
"where":[["_uuid", "==", ["uuid",
"20c2a046-ae7e-4453-a576-11034db24985"]]]}
In the above case, uuid in "row" aren't replaced by "named-uuid" because
the function doesn't look into elements of lists.
When list/tuple is found, look into elements recursively.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
This is needed when using schema that was retrieved from ovsdb by get_schema
method.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Add register_table method to SchemaHelper as Python counterpart of
ovsdb_idl_add_table() in the C version of the IDL.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Replaced all instances of Nicira Networks(, Inc) to Nicira, Inc.
Feature #10593
Signed-off-by: Raju Subramanian <rsubramanian@nicira.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Originally the IDL transaction state machine had a return value
TXN_TRY_AGAIN to signal the client to wait for a change in the database and
then retry its transaction. However, this logic was incomplete, because
it was possible for the database to change before the reply to the
transaction RPC was received, in which case the client would wait for a
further change. Commit 4fdfe5ccf8 (ovsdb-idl: Prevent occasional hang
when multiple database clients race.) fixed the problem by breaking
TXN_TRY_AGAIN into two status codes, TXN_AGAIN_WAIT that meant to wait for
a further change and TXN_AGAIN_NOW that meant that a change had already
occurred so try again immediately.
This is correct enough, but it is more complicated than necessary. It is
simpler and just as correct to use a single "try again" status that
requires the client to wait for a change relative to the database contents
*before* the transaction was committed. This commit makes that change.
It also changes ovsdb_idl_run()'s return type from bool to void because
its return type is hardly useful anymore.
Signed-off-by: Ben Pfaff <blp@nicira.com>
The recently added SchemaHelper class significantly simplifies IDL
instantiation in Python. This commit converts all users of the old
method to the new method, and removes support for the old method.
Signed-off-by: Ethan Jackson <ethan@nicira.com>
When a client of the IDL tries to commit a read-modify-write transaction
but the database has changed in the meantime, the IDL tells its client to
wait for the IDL to change and then try the transaction again by returning
TXN_TRY_AGAIN. The "wait for the IDL to change" part is important because
there's no point in retrying the transaction before the IDL has received
the database updates (the transaction would fail in the same way all over
again).
However, the logic was incomplete: the database update can be received
*before* the reply to the transaction RPC (I think that in the current
ovsdb-server implementation this will always happen, in fact). When this
happens, the right thing to do is to retry the transaction immediately;
if we wait, then we're waiting for an additional change to the database
that may never come, causing an indefinite hang.
This commit therefore breaks the "try again" IDL commit status code
into two, one that means "try again immediately" and another that means
"wait for a change then try again". When an update is processed after a
transaction is committed but before the reply is received, the "try again
now" tells the IDL client not to wait for another database change before
retrying its transaction.
Bug #5980.
Reported-by: Ram Jothikumar <rjothikumar@nicira.com>
Reproduced-by: Alex Yip <alex@nicira.com>
This patch does minor style cleanups to the code in the python and
tests directory. There's other code floating around that could use
similar treatment, but updating it is not convenient at the moment.
Until now, the Python bindings for OVSDB have not supported writing to the
database. Instead, writes had to be done with "ovs-vsctl" subprocesses.
This commit adds write support and brings the Python bindings in line with
the C bindings.
This commit deletes the Python-specific IDL tests in favor of using the
same tests as the C version of the IDL, which now pass with both
implementations.
This commit updates the two users of the Python IDL to use the new write
support. I tested this updates only by writing unit tests for them,
which appear in upcoming commits.
According to Reid, there may be some disadvantages to having this class be
anonymous, for example, cannot do instance/typechecking, might be
allocating a new class for every row as well, which isn't the most memory
efficient.
Suggested-by: Reid Price <reid@nicira.com>
Idl.__parse_row_update() assumed that every change that the database server
sent down actually modified the database. This is generally true, but
since Idl.__modify_row() already returns whether there was a change, we
might as well use it.
Reported-by: Reid Price <reid@nicira.com>
