Internal implementation of JSON array will be changed in the future
commits. Add access functions that users can rely on instead of
accessing the internals of 'struct json' directly and convert all the
users. Structure fields are intentionally renamed to make sure that
no code is using the old fields directly.
json_array() function is removed, as not needed anymore. Added new
functions: json_array_size(), json_array_at(), json_array_set()
and json_array_pop(). These are enough to cover all the use cases
within OVS.
The change is fairly large, however, IMO, it's a much overdue cleanup
that we need even without changing the underlying implementation.
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
We'll be changing the way strings are stored, so the direct access
will not be safe anymore. Change all the users to use the proper
API as they should have been doing anyway. This also means splitting
the handling of strings and serialized objects in most cases as
they will be treated differently.
The only code outside of json implementation for which direct access
is preserved is substitute_uuids() in test-ovsdb.c. It's an unusual
string manipulation that is only needed for the testing, so doesn't
seem worthy adding a new API function. We could introduce something
like json_string_replace() if this use case will appear somewhere
else in the future.
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Previously when an empty mutation was used to count the number of rows
in a table, OVSDB would iterate over all rows twice. First to perform an
RBAC check, and then to perform the no-operation.
This change adds a short circuit to mutate operations with no conditions
and an empty mutation set, returning immediately. One notable change in
functionality is not performing the RBAC check in this condition, as no
mutation actually takes place.
Reported-by: Terry Wilson <twilson@redhat.com>
Reported-at: https://issues.redhat.com/browse/FDP-359
Signed-off-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Database file contains the column diff, but it is discarded once
the 'new' state of a row is constructed. Keep it in the transaction
row, as it can be used later by other parts of the code.
Diffs do not live long, we keep them around only while transaction
is alive, so should not affect memory consumption.
Users for this data will be added in later commits.
Acked-by: Mike Pattrick <mkp@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Relay servers apply updates via ovsdb_table_execute_update(). XOR
updates contain datum diffs, and datum diffs can be larger than the
type constraints. Currently, relay will fail to parse such update
into ovsdb row triggering a syntax error and a re-connection.
Fix that by relaxing the size constraints for this kind of updates.
Fixes: 026c77c58ddb ("ovsdb: New ovsdb 'relay' service model.")
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Current version of ovsdb relay allows to scale out read-only
access to the primary database. However, many clients are not
read-only but read-mostly. For example, ovn-controller.
In order to scale out database access for this case ovsdb-server
need to process transactions that are not read-only. Relay is not
allowed to do that, i.e. not allowed to modify the database, but it
can act like a proxy and forward transactions that includes database
modifications to the primary server and forward replies back to a
client. At the same time it may serve read-only transactions and
monitor requests by itself greatly reducing the load on primary
server.
This configuration will slightly increase transaction latency, but
it's not very important for read-mostly use cases.
Implementation details:
With this change instead of creating a trigger to commit the
transaction, ovsdb-server will create a trigger for transaction
forwarding. Later, ovsdb_relay_run() will send all new transactions
to the relay source. Once transaction reply received from the
relay source, ovsdb-relay module will update the state of the
transaction forwarding with the reply. After that, trigger_run()
will complete the trigger and jsonrpc_server_run() will send the
reply back to the client. Since transaction reply from the relay
source will be received after all the updates, client will receive
all the updates before receiving the transaction reply as it is in
a normal scenario with other database models.
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
New database service model 'relay' that is needed to scale out
read-mostly database access, e.g. ovn-controller connections to
OVN_Southbound.
In this service model ovsdb-server connects to existing OVSDB
server and maintains in-memory copy of the database. It serves
read-only transactions and monitor requests by its own, but
forwards write transactions to the relay source.
Key differences from the active-backup replication:
- support for "write" transactions (next commit).
- no on-disk storage. (probably, faster operation)
- support for multiple remotes (connect to the clustered db).
- doesn't try to keep connection as long as possible, but
faster reconnects to other remotes to avoid missing updates.
- No need to know the complete database schema beforehand,
only the schema name.
- can be used along with other standalone and clustered databases
by the same ovsdb-server process. (doesn't turn the whole
jsonrpc server to read-only mode)
- supports modern version of monitors (monitor_cond_since),
because based on ovsdb-cs.
- could be chained, i.e. multiple relays could be connected
one to another in a row or in a tree-like form.
- doesn't increase availability.
- cannot be converted to other service models or become a main
active server.
Some performance test results can be found here:
https://mail.openvswitch.org/pipermail/ovs-dev/2021-July/385825.html
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This will be used to apply update3 type updates to ovsdb tables
while processing updates for future ovsdb 'relay' service model.
'ovsdb_datum_apply_diff' is allowed to fail, so adding support
to return this error.
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
According to RFC 7047, 'timeout' is an integer field:
5.2.6. Wait
The "wait" object contains the following members:
"op": "wait" required
"timeout": <integer> optional
...
For some reason initial implementation treated it as a real number.
This causes a build issue with clang that complains that LLONG_MAX
could not be represented as double:
ovsdb/execution.c:733:32: error: implicit conversion from 'long long'
to 'double' changes value from
9223372036854775807 to 9223372036854775808
timeout_msec = MIN(LLONG_MAX, json_real(timeout));
~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/sys/limits.h:69:19: note: expanded from macro 'LLONG_MAX'
#define LLONG_MAX __LLONG_MAX /* max for a long long */
^~~~~~~~~~~
/usr/include/x86/_limits.h:74:21: note: expanded from macro '__LLONG_MAX'
#define __LLONG_MAX 0x7fffffffffffffffLL /* max value for a long long */
^~~~~~~~~~~~~~~~~~~~
./lib/util.h:90:21: note: expanded from macro 'MIN'
#define MIN(X, Y) ((X) < (Y) ? (X) : (Y))
^ ~
Fix that by changing parser to treat 'timeout' as integer.
Fixes clang build on FreeBSD 12.1 in CirrusCI.
Fixes: f85f8ebbfac9 ("Initial implementation of OVSDB.")
Acked-by: Han Zhou <hzhou@ovn.org>
Acked-by: Numan Siddique <numans@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Several OVS structs contain embedded named unions, like this:
struct {
...
union {
...
} u;
};
C11 standardized a feature that many compilers already implemented
anyway, where an embedded union may be unnamed, like this:
struct {
...
union {
...
};
};
This is more convenient because it allows the programmer to omit "u."
in many places. OVS already used this feature in several places. This
commit embraces it in several others.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
Tested-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
This commit adds support for OVSDB clustering via Raft. Please read
ovsdb(7) for information on how to set up a clustered database. It is
simple and boils down to running "ovsdb-tool create-cluster" on one server
and "ovsdb-tool join-cluster" on each of the others and then starting
ovsdb-server in the usual way on all of them.
One you have a clustered database, you configure ovn-controller and
ovn-northd to use it by pointing them to all of the servers, e.g. where
previously you might have said "tcp:1.2.3.4" was the database server,
now you say that it is "tcp:1.2.3.4,tcp:5.6.7.8,tcp:9.10.11.12".
This also adds support for database clustering to ovs-sandbox.
Acked-by: Justin Pettit <jpettit@ovn.org>
Tested-by: aginwala <aginwala@asu.edu>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Names that begin with "_" are reserved, but ovsdb-server didn't previously
enforce this.
At the same time, make ovsdb-client ignore databases with reserved names
for the purpose of selecting a default database to work on.
This is in preparation for ovsdb-server starting to serve a new database,
full of meta-information, called _Server.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
This simplifies little bits of code here and there.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Tested-by: Yifeng Sun <pkusunyifeng@gmail.com>
Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com>
If there is an error after ovsdb_rbac_insert, 'row' is leaked.
So move the existing ovsdb_row_destroy to the function end.
Signed-off-by: Yifeng Sun <pkusunyifeng@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Add suport for ovsdb RBAC (role-based access control). This includes:
- Support for "RBAC_Role" table. A db schema containing a table
by this name will enable role-based access controls using
this table for RBAC role configuration.
The "RBAC_Role" table has one row per role, with each row having a
"name" column (role name) and a "permissions" column (map of
table name to UUID of row in separate permission table.) The
permission table has one row per access control configuration,
with the following columns:
"name" - name of table to which this row applies
"authorization" - set of column names and column:key pairs
to be compared against client ID to
determine authorization status
"insert_delete" - boolean, true if insertions and
authorized deletions are allowed.
"update" - Set of columns and column:key pairs for
which authorized updates are allowed.
- Support for a new "role" column in the remote configuration
table.
- Logic for applying the RBAC role and permission tables, in
combination with session role from the remote connection table
and client id, to determine whether operations modifying database
contents should be permitted.
- Support for specifying RBAC role string as a command-line option
to ovsdb-tool (Ben Pfaff).
Signed-off-by: Lance Richardson <lrichard@redhat.com>
Co-authored-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
When ovsdb-sever is running in the backup state, it would be nice to
make sure there is no un-intended changes to the backup database.
This patch makes the ovsdb server only accepts 'read' transactions as
a backup server. When the server role is changed into an active server,
all existing client connections will be reset. After reconnect, all
clinet transactions will then be accepted.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
When setting a where clause, if the timeout is set to a value of 0,
the clause is tested once and if it fails, a message of '"wait" timed
out' is returned. This can be misleading because there wasn't any
real time, so change the message to '"where" clause test failed'.
Signed-off-by: Ryan Moats <rmoats@us.ibm.com>
Reported-by: Ryan Moats <rmoats@us.ibm.com>
Reported-at: http://openvswitch.org/pipermail/dev/2016-August/077083.html
Fixes: f85f8ebb ("Initial implementation of OVSDB.")
Signed-off-by: Ben Pfaff <blp@ovn.org>
To easily allow both in- and out-of-tree building of the Python
wrapper for the OVS JSON parser (e.g. w/ pip), move json.h to
include/openvswitch. This also requires moving lib/{hmap,shash}.h.
Both hmap.h and shash.h were #include-ing "util.h" even though the
headers themselves did not use anything from there, but rather from
include/openvswitch/util.h. Fixing that required including util.h
in several C files mostly due to OVS_NOT_REACHED and things like
xmalloc.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Optimize ovsdb_condition_match_any_clause() to be in O(#columns in condition)
and not O(#clauses) in case condition's caluses function is boolean or "==".
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The following macros are renamed to avoid conflicts with other headers:
* WARN_UNUSED_RESULT to OVS_WARN_UNUSED_RESULT
* PRINTF_FORMAT to OVS_PRINTF_FORMAT
* NO_RETURN to OVS_NO_RETURN
Signed-off-by: Thomas Graf <tgraf@noironetworks.com>
Acked-by: Ben Pfaff <blp@nicira.com>
Although the check for negative timeout is present, the error string
is overwritten if an invalid "until" is found right after. This leaks
an error string and results in not reporting the negative timeout back
to the user even though it is encountered first.
Signed-off-by: Thomas Graf <tgraf@noironetworks.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
The MSVC C library printf() implementation does not support the 'z', 't',
'j', or 'hh' format specifiers. This commit changes the Open vSwitch code
to avoid those format specifiers, switching to standard macros from
<inttypes.h> where available and inventing new macros resembling them
where necessary. It also updates CodingStyle to specify the macros' use
and adds a Makefile rule to report violations.
Signed-off-by: Alin Serdean <aserdean@cloudbasesolutions.com>
Co-authored-by: Ben Pfaff <blp@nicira.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
This is a straight search-and-replace, except that I also removed #include
<assert.h> from each file where there were no assert calls left.
Signed-off-by: Ben Pfaff <blp@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
OVSDB has always had the ability to mark a column as "immutable", so that
its value cannot be changed in a given row after that row is initially
inserted. However, we discovered recently that ovsdb-server has never
enforced this constraint. This commit implements enforcement.
Reported-by: Paul Ingram <paul@nicira.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
Acked-by: Kyle Mestery <kmestery@cisco.com>
Replaced all instances of Nicira Networks(, Inc) to Nicira, Inc.
Feature #10593
Signed-off-by: Raju Subramanian <rsubramanian@nicira.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
The "lock" request requires the lock name to be an <id> but it is shown as
<string> in the "assert" operation. This corrects the "assert"
specification and fixes the suggested naming convention (since ":" is not
valid in an <id>).
This commit also updates the implementation to match the specification.
Reported-by: Jeremy Stribling <strib@nicira.com>
This provides clients a way to coordinate their access to the database.
This is a voluntary, not mandatory, locking protocols, that is, clients
are not prevented from modifying the database unless they cooperate with
the locking protocol. It is also not related to any of the ACID properties
of database transactions. It is strictly a way for clients to coordinate
among themselves.
The following commit will introduce one user.
This function took a struct ovsdb_table but only used the 'schema' member.
An upcoming patch needs to parse a column set when only the schema is
available, so to prepare for that this patch changes
ovsdb_column_set_from_json() to only take the schema that it really needs.
Errors from this function were being ignored, which meant that transactions
could use "mutate" to bypass number-of-elements constraints on sets and
maps. This fixes the problem and adds a test to prevent the problem from
recurring.
Bug #5781.
The name 'created' better reflects the actual meaning of this member: in
both ovsdb and ovs-vsctl, it is true if a row has been created with the
symbol's UUID and false otherwise.
ovsdb_execute_insert() tried to return a helpful error message when there
was a duplicate uuid-name, but ovsdb_execute() (its caller) makes any
parse error override a parse error. Since ovsdb_execute_insert() would
skip parsing the row when the uuid-name was a duplicate, this meant that
the error actually reported would be that "row" was not allowed here, which
wasn't at all helpful (since "row" is in fact mandatory).
This commit clears up the problem by always retrieving the "row" member,
which required a small amount of refactoring, and adds a test.
All of these changes avoid using the same name for two local variables
within a same function. None of them are actual bugs as far as I can tell,
but any of them could be confusing to the casual reader.
The one in lib/ovsdb-idl.c is particularly brilliant: inner and outer
loops both using (different) variables named 'i'.
Found with GCC -Wshadow.
The declaration of "error" that this commit removes shadowed an outer local
declaration of "error", which caused errors detected by this code not to be
propagated up to the outer level.
Found with GCC -Wshadow.
This also adds protocol compatibility to the database itself and to
ovsdb-client. It doesn't actually add multiple database support to
ovsdb-server, since we don't really need that yet.
The "declare" operation is specified to return a <uuid>, e.g.:
["uuid", "7b703002-f7b6-45c6-bfd6-7619b21a1a5f"]
It was actually just returning the second part, e.g.
"7b703002-f7b6-45c6-bfd6-7619b21a1a5f"
This commit makes the result match the specification.
Some of these are serious leaks, in that they could leak some amount of
memory for every transaction processed by the database server.
Found with valgrind.
The idea here is that transaction comments get copied to the ovsdb-server's
transaction log, which can then make it clear later why a particular change
was made to the database, to ease debugging.
This allows a transaction like this:
[{"op": "insert",
"table": "mytable",
"row": {"i": 0, "k": ["named-uuid", "self"]},
"uuid-name": "self"}]
It was already possible to do this by following up on the "insert" with an
"update", but since this was easy to implement I did it.
An upcoming commit will add support for replicating tables across JSON-RPC
connection. As a prerequisite ovsdb itself must support basic replication.
This commit adds that support and then reimplements the ovsdb file storage
in terms of that replication.
