If a new database server added to the cluster, or if one of the
database servers changed its IP address or port, then you need to
update the list of remotes for the client. For example, if a new
OVN_Southbound database server is added, you need to update the
ovn-remote for the ovn-controller.
However, in the current implementation, the ovsdb-cs module always
closes the current connection and creates a new one. This can lead
to a storm of re-connections if all ovn-controllers will be updated
simultaneously. They can also start re-dowloading the database
content, creating even more load on the database servers.
Correct this by saving an existing connection if it is still in the
list of remotes after the update.
'reconnect' module will report connection state updates, but that
is OK since no real re-connection happened and we only updated the
state of a new 'reconnect' instance.
If required, re-connection can be forced after the update of remotes
with ovsdb_cs_force_reconnect().
Acked-by: Dumitru Ceara <dceara@redhat.com>
Acked-by: Han Zhou <hzhou@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
RAFT messages could be fairly big. If something abnormal happens to
one of the servers in a cluster it may not be able to process all the
incoming messages in a timely manner. This results in jsonrpc backlog
growth on the sender's side. For example if follower gets many new
clients at once that it needs to serve, or it decides to take a
snapshot in a period of high number of database changes.
If backlog grows large enough it becomes harder and harder for follower
to process incoming raft messages, it sends outdated replies and
starts receiving snapshots and the whole raft log from the leader.
Sometimes backlog grows too high (60GB in this example):
jsonrpc|INFO|excessive sending backlog, jsonrpc: ssl:<ip>,
num of msgs: 15370, backlog: 61731060773.
In this case OS might actually decide to kill the sender to free some
memory. Anyway, It could take a lot of time for such a server to catch
up with the rest of the cluster if it has so much data to receive and
process.
Introducing backlog thresholds for jsonrpc connections.
If sending backlog will exceed particular values (500 messages or
4GB in size), connection will be dropped and re-created. This will
allow to drop all the current backlog and start over increasing
chances of cluster recovery.
Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1888829
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This commit adds support for OVSDB clustering via Raft. Please read
ovsdb(7) for information on how to set up a clustered database. It is
simple and boils down to running "ovsdb-tool create-cluster" on one server
and "ovsdb-tool join-cluster" on each of the others and then starting
ovsdb-server in the usual way on all of them.
One you have a clustered database, you configure ovn-controller and
ovn-northd to use it by pointing them to all of the servers, e.g. where
previously you might have said "tcp:1.2.3.4" was the database server,
now you say that it is "tcp:1.2.3.4,tcp:5.6.7.8,tcp:9.10.11.12".
This also adds support for database clustering to ovs-sandbox.
Acked-by: Justin Pettit <jpettit@ovn.org>
Tested-by: aginwala <aginwala@asu.edu>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The implementation cycles through the remotes in random order. This allows
clients to perform some load balancing across alternative implementations
of a service.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
Add suport for ovsdb RBAC (role-based access control). This includes:
- Support for "RBAC_Role" table. A db schema containing a table
by this name will enable role-based access controls using
this table for RBAC role configuration.
The "RBAC_Role" table has one row per role, with each row having a
"name" column (role name) and a "permissions" column (map of
table name to UUID of row in separate permission table.) The
permission table has one row per access control configuration,
with the following columns:
"name" - name of table to which this row applies
"authorization" - set of column names and column:key pairs
to be compared against client ID to
determine authorization status
"insert_delete" - boolean, true if insertions and
authorized deletions are allowed.
"update" - Set of columns and column:key pairs for
which authorized updates are allowed.
- Support for a new "role" column in the remote configuration
table.
- Logic for applying the RBAC role and permission tables, in
combination with session role from the remote connection table
and client id, to determine whether operations modifying database
contents should be permitted.
- Support for specifying RBAC role string as a command-line option
to ovsdb-tool (Ben Pfaff).
Signed-off-by: Lance Richardson <lrichard@redhat.com>
Co-authored-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
We've been warning about the change since 2.1, which was released a year
ago.
Signed-off-by: Justin Pettit <jpettit@nicira.com>
Acked-by: Ben Pfaff <blp@nicira.com>
If ovs-vsctl has to wait for ovs-vswitchd to reconfigure itself
according to the new database, then sometimes ovs-vsctl could
end up stuck in the event loop if OVSDB connection was dropped
while ovs-vsctl was still running.
This patch fixes this problem by letting ovs-vsctl to reconnect
to the OVSDB, if it has to wait cur_cfg field to be updated.
Issue: 1191997
Reported-by: Spiro Kourtessis <spiro@nicira.com>
Signed-Off-By: Ansis Atteka <aatteka@nicira.com>
The OVS code has always made a distinction between the unencrypted (TCP)
and SSL port numbers for the OpenFlow and OVSDB protocols. The default
port numbers for both protocols has changed, and there continues to be
no distinction between the unencrypted and SSL versions. This
commit removes the distinction in port numbers. A future patch will
recognize the change in default port number.
Signed-off-by: Justin Pettit <jpettit@nicira.com>
Acked-by: Ben Pfaff <blp@nicira.com>
Until now, ovs-vsctl has kept trying to the database server until it
succeeded or the timeout expired (if one was specified with --timeout).
This meant that if ovsdb-server wasn't running, then ovs-vsctl would hang.
The result was that almost every ovs-vsctl invocation in scripts specified
a timeout on the off-chance that the database server might not be running.
But it's difficult to choose a good timeout. A timeout that is too short
can cause spurious failures. A timeout that is too long causes long delays
if the server really isn't running.
This commit should alleviate this problem. It changes ovs-vsctl's behavior
so that, if it fails to connect to the server, it exits unsuccessfully.
This makes --timeout obsolete for the purpose of avoiding a hang if the
database server isn't running. (--timeout is still useful to avoid a hang
if ovsdb-server is running but ovs-vswitchd is not, for ovs-vsctl commands
that modify the database. --no-wait also avoids that issue.)
Bug #2393.
Bug #15594.
Reported-by: Jeff Merrick <jmerrick@vmware.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
2012-09-14T05:38:26Z|00001|jsonrpc|WARN|tcp:127.0.0.1:6634: receive error: Con
ovsdb-client: transaction failed (Connection reset by peer)
NOTE: This occurs intermittently depending on how ovsdb-server runs.
Running ovsdb-client on a remote machine increases the possibility.
This is because ovsdb-server closes newly accepted tcp connection.
The following changesets caused it. struct jsonrpc_session::dscp isn't set
based on listening socket's dscp value.
- ovsdb-server creates passive connection and listens on it.
- ovsdb-server accepts connection by ovsdb_jsonrpc_server_run().
The accepted socket inherits from the listening sockets.
ovsdb_jsonrpc_server_run() creates json session, but leaves dscp of
struct jsonrpc_session zero.
- On calling reconfigure_from_db(), it resets dscp value to
all jsonrpc sessions. Eventually jsonrpc_session_set_dscp() is called.
Then jsonrpc_session_force_reconnect() closes the connection.
With this patch,
- struct jsonrpc_session::dscp is correctly set based on
listening sockets dscp value.
- dscp of listening socket is changed dynamically by setsockopt.
This leaves a window where accepted socket may have old dscp.
But it is ignored for now because it would complicates codes
too much.
The related change sets:
- 0442efd9b1a88d923b56eab6b72b6be8231a49f7
Reapplying the dscp changes: No need to restart DB/OVS on changing
dscp value.
- 59efa47adf3234ec51541405726d033173851285
Revert DSCP update changes.
- b2e18db292cd4962af3248f11e9f17e6eaf9c033
No need to restart DB / OVS on changing dscp value.
- f125905cdd3dc0339ad968c0a70128807884b400
Allow configuring DSCP on controller and manager connections.
Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>
Signed-off-by: Mehak Mahajan <mmahajan@nicira.com>
Until now, the jsonrpc code has only counted receiving a full JSON-RPC
messages as activity. This could theoretically time out, then, while a
very long message is in transit or if a slow link is involved. This commit
changes this code to count receiving any part of a message as activity.
This isn't a problem for OpenFlow connections because OpenFlow messages are
at most 64 kB in size.
This problem hasn't actually been observed in practice.
Bug #12789.
Signed-off-by: Ben Pfaff <blp@nicira.com>
Replaced all instances of Nicira Networks(, Inc) to Nicira, Inc.
Feature #10593
Signed-off-by: Raju Subramanian <rsubramanian@nicira.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
The changes allow the user to specify a separate dscp value for the
controller connection and the manager connection. The value will take
effect on resetting the connections. If no value is specified a default
value of 192 is chosen for each of the connections.
Feature #10074
Requested-by: Rajiv Ramanathan <rramanathan@nicira.com>
Signed-off-by: Mehak Mahajan <mmahajan@nicira.com>
This function is an implementation detail. The JSONRPC unit test used it,
but not for any good reason, so this commit changes the test to avoid
using it.
Signed-off-by: Ben Pfaff <blp@nicira.com>
This commit makes the status of manager connections visible via the Manager
table in the database. Two new columns have been created for this purpose:
'is_connected' and 'status'. The former is a boolean flag, and the latter is a
string-string map which may contain the keys "last_error", "state", and
"time_in_state".
Requested-by: Keith Amidon <keith@nicira.com>
Reviewed by: Ben Pfaff.
Feature #3692.
I'm retaining the "managers" column in the Open_vSwitch table for now, but
I hope that applications transition to using "manager_options" eventually
so that we could drop it.
CC: Andrew Lambeth <wal@nicira.com>
CC: Jeremy Stribling <strib@nicira.com>
