Internal implementation of JSON array will be changed in the future
commits. Add access functions that users can rely on instead of
accessing the internals of 'struct json' directly and convert all the
users. Structure fields are intentionally renamed to make sure that
no code is using the old fields directly.
json_array() function is removed, as not needed anymore. Added new
functions: json_array_size(), json_array_at(), json_array_set()
and json_array_pop(). These are enough to cover all the use cases
within OVS.
The change is fairly large, however, IMO, it's a much overdue cleanup
that we need even without changing the underlying implementation.
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
We'll be changing the way strings are stored, so the direct access
will not be safe anymore. Change all the users to use the proper
API as they should have been doing anyway. This also means splitting
the handling of strings and serialized objects in most cases as
they will be treated differently.
The only code outside of json implementation for which direct access
is preserved is substitute_uuids() in test-ovsdb.c. It's an unusual
string manipulation that is only needed for the testing, so doesn't
seem worthy adding a new API function. We could introduce something
like json_string_replace() if this use case will appear somewhere
else in the future.
Acked-by: Mike Pattrick <mkp@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Set all the options for the source connection, not only the
inactivity probe interval.
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Add a new structure 'db_config' that holds the user-provided
configuration of the database. And attach this configuration
to each of the databases on the server.
Each database has a service model: standalone, clustered, relay
or active-backup. Relays and A-B databases have a source, each
source has its own set of JSON-RPC session options. A-B also
have an indicator of it being active or backup and an optional
list of tables to exclude from replication.
All of that should be stored per database in the temporary
configuration file that is used in order to restore the config
after the OVSDB crash. For that, the save/load functions are
also updates.
This change is written in generic way assuming all the databases
can have different configuration including service model.
The only user-visible change here is a slight modification of
the ovsdb-server/sync-status appctl, since it now needs to
skip databases that are not active-backup and also should report
active-backup databases that are currently active, i.e. not
added to the replication module.
If the service model is not defined in the configuration, it
is assumed to be standalone or clustered, and determined from
the storage type while opening the database. If the service
model is defined, but doesn't match the actual storage type
in the database file, ovsdb-server will fail to open the
database. This should never happen with internally generated
config file, but may happen in the future with user-provided
configuration files. In this case the service model is used
for verification purposes only, if administrator wants to
assert a particular model.
Since the database 'source' connections can't use 'role' or
'read-only' options, a new flag added to corresponding JSON
parsing functions to skip these fields.
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
When database is added to the replication, it should no longer
accept transactions that can modify it. When it's removed from
the replication, it should be writable again. Add this logic
to the replication module itself, so it can be removed from the
main ovsdb-server later.
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Refactoring of the replication code, so each database is handled
separately from each other. Supposed to work the same way as before
with the only difference that each backup database will have its own
connection to the source and will have its own state machine.
From the user's perspective, the only visible difference is that
ovsdb-server/sync-status appctl now shows the status of each
database separately.
If one of the connections is permanently broken, all the databases
will be switched to active. This is done in order to preserve the
old behavior where we had only one connection.
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Using SHORT version of the *_SAFE loops makes the code cleaner and less
error prone. So, use the SHORT version and remove the extra variable
when possible for hmap and all its derived types.
In order to be able to use both long and short versions without changing
the name of the macro for all the clients, overload the existing name
and select the appropriate version depending on the number of arguments.
Acked-by: Dumitru Ceara <dceara@redhat.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This will be used to apply update3 type updates to ovsdb tables
while processing updates for future ovsdb 'relay' service model.
'ovsdb_datum_apply_diff' is allowed to fail, so adding support
to return this error.
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
These functions will be used later for ovsdb 'relay' service model, so
moving them to a common code.
Warnings translated to ovsdb errors, caller in replication.c only
printed inconsistency warnings, but mostly ignored them. Implementing
the same logic by checking the error tag.
Also ovsdb_execute_insert() previously printed incorrect warning about
duplicate row while it was a syntax error in json. Fixing that by
actually checking for the duplicate and reporting correct ovsdb error.
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
There is one remaining use under datapath. That change should happen
upstream in Linux first according to our usual policy.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
When ovsdb-server is in backup mode and connects to the active
ovsdb-server for replication, and if takes more than 5 seconds to
get the dump of the whole database, it will drop the connection
soon after as the default probe interval is 5 seconds. This
results in a snowball effect of reconnections to the active
ovsdb-server.
This patch handles or mitigates this issue by setting the
default probe interval value to 60 seconds and provide the option to
configure this value from the unixctl command.
Other option could be increase the value of 'RECONNECT_DEFAULT_PROBE_INTERVAL'
to a higher value.
Acked-by: Mark Michelson <mmichels@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Numan Siddique <numans@ovn.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Presently, replication is not allowed if there is a schema version mismatch between
the schema returned by the active ovsdb-server and the local db schema. This is
causing failures in OVN DB HA deployments during uprades.
In the case of OpenStack tripleo deployment with OVN, OVN DB ovsdb-servers are
deployed on a multi node controller cluster in active/standby mode. During
minor updates or major upgrades, the cluster is updated one at a time. If
a node A is running active OVN DB ovsdb-servers and when it is updated, another
node B becomes active. After the update when OVN DB ovsdb-servers in A are started,
these ovsdb-servers fail to replicate from the active if there is a schema
version mismatch.
This patch addresses this issue by allowing replication even if there is a
schema version mismatch only if all the active db schema tables and its colums are
present in the local db schema.
This should not result in any data loss.
Signed-off-by: Numan Siddique <numans@ovn.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The present code resets the database when it is in the state -
'RPL_S_SCHEMA_REQUESTED' and repopulates the database when it
receives the monitor reply when it is in the state -
'RPL_S_MONITOR_REQUESTED'. If however, it goes to active mode
before it processes the monitor reply, the whole data is lost.
This patch alleviates the problem by resetting the database when it
receives the monitor reply (before processing it). So that
reset and repopulation of the db happens in the same state.
This approach still has a window for data loss if the function
process_notification() when processing the monitor reply fails for
some reason or ovsdb-server crashes for some reason during
process_notification().
Reported-by: Han Zhou <zhouhan@gmail.com>
Reported-at: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047161.html
Tested-by: aginwala <aginwala@ebay.com>
Acked-by: Han Zhou <zhouhan@gmail.com>
Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Several OVS structs contain embedded named unions, like this:
struct {
...
union {
...
} u;
};
C11 standardized a feature that many compilers already implemented
anyway, where an embedded union may be unnamed, like this:
struct {
...
union {
...
};
};
This is more convenient because it allows the programmer to omit "u."
in many places. OVS already used this feature in several places. This
commit embraces it in several others.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
Tested-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
This commit adds support for OVSDB clustering via Raft. Please read
ovsdb(7) for information on how to set up a clustered database. It is
simple and boils down to running "ovsdb-tool create-cluster" on one server
and "ovsdb-tool join-cluster" on each of the others and then starting
ovsdb-server in the usual way on all of them.
One you have a clustered database, you configure ovn-controller and
ovn-northd to use it by pointing them to all of the servers, e.g. where
previously you might have said "tcp:1.2.3.4" was the database server,
now you say that it is "tcp:1.2.3.4,tcp:5.6.7.8,tcp:9.10.11.12".
This also adds support for database clustering to ovs-sandbox.
Acked-by: Justin Pettit <jpettit@ovn.org>
Tested-by: aginwala <aginwala@asu.edu>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The ovs_assert() macro always evaluates its argument, even when NDEBUG is
defined so that failure is ignored. This behavior wasn't documented, and
thus a lot of code didn't rely on it. This commit documents the behavior
and simplifies bits of code that heretofore didn't rely on it.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com>
The break statement would never be executed as OVS_NOT_REACHED()
internally invokes abort() and causes process termination.
Signed-off-by: Bhanuprakash Bodireddy <bhanuprakash.bodireddy@intel.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Code that calls ovsdb_txn_row_delete() should avoid referencing the
deleted row again, because it might be freed. In practice this shouldn't
really happen in this case because of the particular circumstances, but it
costs little to be careful.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
Shadowing is when a variable with a given name in an inner scope hides a
different variable with the same name in a surrounding scope. This is
generally undesirable because it can confuse programmers. This commit
eliminates most of it.
Found with -Wshadow=local in GCC 7. The repo is not really ready to enable
this option by default because of a few cases that are harder to fix, and
harmless, such as nested use of CMAP_FOR_EACH.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Andy Zhou <azhou@ovn.org>
Replication OVSDB server from itself is usually caused by configuration
errors. Such configuration errors can lead to OVSDB server data loss.
See "reported-at" for more details.
This patch adds logics that prevent OVSDB server from replicating
itself.
Reported-by: Guishuai Li <ligs@dtdream.com>
Reported-at: https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/326963.html
Suggested-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Found by inspection.
The 'replication_dbs' structure was not freed after use.
Fix by adding a new function replication_dbs_destroy().
Also remove unnecessary global pointer variables initializer.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
The code here wasn't careful about parsing JSON received from the remote
OVSDB server. It assumed, for example, that a row that the remote server
implied was new was actually new, without looking to see whether there was
already a row with that UUID. This commit improves this validation. It
also rewrites code that translated updates locally into calls into the
query engine, via JSON, into simple lookups by UUID.
For me, this fixes a test failure in test 1866
(ovsdb-server/active-backup-role-switching), which caused the following
valgrind report:
==18725== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==18725== Access not within mapped region at address 0x0
==18725== at 0x43937E: ovsdb_datum_compare_3way (ovsdb-data.c:1626)
==18725== by 0x439344: ovsdb_datum_equals (ovsdb-data.c:1616)
==18725== by 0x4166CC: update_monitor_row_data (monitor.c:310)
==18725== by 0x414A90: ovsdb_monitor_changes_update (monitor.c:1255)
==18725== by 0x417009: ovsdb_monitor_change_cb (monitor.c:1339)
==18725== by 0x41DB52: ovsdb_txn_for_each_change (transaction.c:906)
==18725== by 0x416CC9: ovsdb_monitor_commit (monitor.c:1553)
==18725== by 0x41D993: ovsdb_txn_commit_ (transaction.c:868)
==18725== by 0x41D6F5: ovsdb_txn_commit (transaction.c:893)
==18725== by 0x418185: process_notification (replication.c:576)
==18725== by 0x417705: replication_run (replication.c:185)
==18725== by 0x408240: main_loop (ovsdb-server.c:198)
==18725== by 0x406432: main (ovsdb-server.c:429)
I don't know the exact cause of the problem, but this new implementation
leaves me more confident due to its simplicity.
Reported-by: Joe Stringer <joe@ovn.org>
Reported-at: http://openvswitch.org/pipermail/dev/2016-September/079315.html
Fixes: 60e0cd041958 ("ovsdb: Replication usability improvements")
Signed-off-by: Ben Pfaff <blp@ovn.org>
Tested-by: Joe Stringer <joe@ovn.org>
Acked-by: Andy Zhou <azhou@ovn.org>
Valgrind reports:
==18725== 32 bytes in 1 blocks are definitely lost in loss record 339 of 497
==18725== at 0x4C29BBE: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18725== by 0x450F1F: xmalloc (util.c:112)
==18725== by 0x41748E: replication_add_local_db (replication.c:137)
==18725== by 0x40803B: ovsdb_replication_init (ovsdb-server.c:146)
==18725== by 0x407C9E: ovsdb_server_connect_active_ovsdb_server
(ovsdb-server.c:1165)
==18725== by 0x450AB3: process_command (unixctl.c:313)
==18725== by 0x4500DC: run_connection (unixctl.c:347)
==18725== by 0x44FFB6: unixctl_server_run (unixctl.c:400)
==18725== by 0x4081AC: main_loop (ovsdb-server.c:182)
==18725== by 0x406432: main (ovsdb-server.c:429)
Fixes: 60e0cd041958 ("ovsdb: Replication usability improvements")
Signed-off-by: Joe Stringer <joe@ovn.org>
Acked-by: Flavio Fernandes <flavio@flaviof.com>
Acked-by: Ben Pfaff <blp@ovn.org>
Based on feedbacks from initial HA manager integration, added the
'--active' command line option and appctl command
"ovsdb-server/sync-status. See man page updates for details.
Added the RPL_S_INIT state in the state machine. This state is
not strictly necessary for the replication state machine, but is
introduced to make sure the state is update immediately when
the state machine is reset, via replication_init(). Without it
ovsdb/sync-status may display "replicating" or crash, if the command
is issued between after replication_init() is called, but before
the state variable is updated from replication_run().
Added a test to simulate the integration of HA manager with OVSDB
server using replication.
Other documentation and API improvements.
Tested-by: Numan Siddique <nusiddiq@redhat.com>
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Current replication uses blocking transactions, which are error prone
in practice, especially in handling RPC connection flapping to the
active server.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
It seems odd that the currently replication implementation moves the
struct db from ovsdb-server.c (file private) to replication.h (global).
This patch moves the 'struct db' defintion back into ovsdb-server.c,
Signed-off-by: Andy Zhou <azhou@ovn.org>
When starting, the replication logic may issue multiple requests at
a time, for example, one monitor request for each databases. The
request_ids keeps track of all outsanding request IDs that are used
for matching reply message with. It also provides the 'db' context
for the reply.
Future patches will make use of this facility.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Currently, 'sync-exclude-tables' command line options are simply stored
in a string. Change the implementation to store it in an shash instead
to improve modularity.
One additional benefit of this change is that errors can be detected
and reported to user earlier. Adde a 'dryrun' option to
set_blacklist_tables() API to make this feature available to the
command line option parsing and unixctl command parsing.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Fix a memory leak in case of error. The error object was not properly
disposed. Since the error to reset DB is not expected, log it and
exit.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Poll_block() requires the run() function to be paired with a wait()
function. Add one for replication.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Current replication code refers the other ovsdb-sever instance as
a 'remote'. which is overloaded in ovsdb.
Switching to use active/backup instead to make it less confusing.
Active is the server that should be servicing the client, backup
server is the server that boots with the --sync-from option.
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Valgrind testcase 1804 ovsdb-server.at:1023 insert rows, update rows by value
reports the following leak.
json_from_string (json.c:1025)
execute_update (replication.c:614), similarily at execute_delete()
process_table_update (replication.c:502)
process_notification.part.5 (replication.c:445)
process_notification (replication.c:402)
check_for_notifications (replication.c:418)
replication_run (replication.c:110)
Signed-off-by: William Tu <u9012063@gmail.com>
Acked-by: Andy Zhou <azhou@ovn.org>
Signed-off-by: Andy Zhou <azhou@ovn.org>
Currently disconnecting from the replicator server means closing the
jsonrpc connection and destroying the monitored table names and
blacklisted table names.
This patch makes a distinction between disconnecting from the
remote server, applicable when the replication incurs in an error,
and destroying the remote server info, applicable when ovsdb-server
exits gracefully.
Signed-off-by: Mario Cabrera <mario.cabrera@hpe.com>
To easily allow both in- and out-of-tree building of the Python
wrapper for the OVS JSON parser (e.g. w/ pip), move json.h to
include/openvswitch. This also requires moving lib/{hmap,shash}.h.
Both hmap.h and shash.h were #include-ing "util.h" even though the
headers themselves did not use anything from there, but rather from
include/openvswitch/util.h. Fixing that required including util.h
in several C files mostly due to OVS_NOT_REACHED and things like
xmalloc.
Signed-off-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Set and get the server to replicate from:
ovsdb-server/set-remote-ovsdb-server {server}
ovsdb-server/get-remote-ovsdb-server
Set and get the replicated table blacklist:
ovsdb-server/set-sync-excluded-tables {DB:table,...}
ovsdb-server/get-sync-excluded-tables
Connect to the configured server and start replication:
ovsdb-server/connect-remote-ovsdb-server
Disconnect from the remote server and stop replication, without dropping
the replicated data:
ovsdb-server/disconnect-remote-ovsdb-server
Signed-off-by: Mario Cabrera <mario.cabrera@hpe.com>
Signed-off-by: Andy Zhou <azhou@ovn.org>
Acked-by: Andy Zhou <azhou@ovn.org>
Optimize ovsdb_condition_match_any_clause() to be in O(#columns in condition)
and not O(#clauses) in case condition's caluses function is boolean or "==".
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Clang reports variable 'error' never been used. Fix by
returning error when table_update->type != JSON_OBJECT.
Signed-off-by: William Tu <u9012063@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
With this patch ovsdb-server no longer blocks waiting for the remote server
connection when doing replication.
Signed-off-by: Mario Cabrera <mario.cabrera@hpe.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
A blacklist of tables that will be excluded from replication can be
specified by the following option:
--sync-exclude-tables=db:table[,db:table]…
Where 'table' corresponds to a table name, and 'db' corresponds to the
database name where the table resides.
Signed-off-by: Mario Cabrera <mario.cabrera@hpe.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Replication is enabled by using the following option when starting the
database server:
--sync-from=server
Where 'server' can take any form described in the ovsdb-client(1)
manpage as an active connection. If this option is specified, the
replication process is immediately started.
Signed-off-by: Mario Cabrera <mario.cabrera@hpe.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
