docs: Add documentation for ovsdb relay mode.

Main documentation for the service model and tutorial with the use case
and configuration examples.

Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>

Documentation/automake.mk

@@ -52,6 +52,7 @@ DOC_SOURCE = \
 	Documentation/topics/networking-namespaces.rst \
 	Documentation/topics/openflow.rst \
 	Documentation/topics/ovs-extensions.rst \
+	Documentation/topics/ovsdb-relay.rst \
 	Documentation/topics/ovsdb-replication.rst \
 	Documentation/topics/porting.rst \
 	Documentation/topics/record-replay.rst \

Documentation/ref/ovsdb.7.rst

@@ -121,13 +121,14 @@ schema checksum from a schema or database file, respectively.
 Service Models
 ==============
 
-OVSDB supports three service models for databases: **standalone**,
+OVSDB supports four service models for databases: **standalone**,
-**active-backup**, and **clustered**.  The service models provide different
+**active-backup**, **relay** and **clustered**.  The service models provide
-compromises among consistency, availability, and partition tolerance.  They
+different compromises among consistency, availability, and partition tolerance.
-also differ in the number of servers required and in terms of performance.  The
+They also differ in the number of servers required and in terms of performance.
-standalone and active-backup database service models share one on-disk format,
+The standalone and active-backup database service models share one on-disk
-and clustered databases use a different format, but the OVSDB programs work
+format, and clustered databases use a different format, but the OVSDB programs
-with both formats.  ``ovsdb(5)`` documents these file formats.
+work with both formats.  ``ovsdb(5)`` documents these file formats.  Relay
+databases have no on-disk storage.
 
 RFC 7047, which specifies the OVSDB protocol, does not mandate or specify
 any particular service model.
@@ -406,6 +407,50 @@ following consequences:
   that the client previously read.  The OVSDB client library in Open vSwitch
   uses this feature to avoid servers with stale data.
 
+Relay Service Model
+-------------------
+
+A **relay** database is a way to scale out read-mostly access to the
+existing database working in any service model including relay.
+
+Relay database creates and maintains an OVSDB connection with another OVSDB
+server.  It uses this connection to maintain an in-memory copy of the remote
+database (a.k.a. the ``relay source``) keeping the copy up-to-date as the
+database content changes on the relay source in the real time.
+
+The purpose of relay server is to scale out the number of database clients.
+Read-only transactions and monitor requests are fully handled by the relay
+server itself.  For the transactions that request database modifications,
+relay works as a proxy between the client and the relay source, i.e. it
+forwards transactions and replies between them.
+
+Compared to the clustered and active-backup models, relay service model
+provides read and write access to the database similarly to a clustered
+database (and even more scalable), but with generally insignificant performance
+overhead of an active-backup model.  At the same time it doesn't increase
+availability that needs to be covered by the service model of the relay source.
+
+Relay database has no on-disk storage and therefore cannot be converted to
+any other service model.
+
+If there is already a database started in any service model, to start a relay
+database server use ``ovsdb-server relay:<DB_NAME>:<relay source>``, where
+``<DB_NAME>`` is the database name as specified in the schema of the database
+that existing server runs, and ``<relay source>`` is an OVSDB connection method
+(see `Connection Methods`_ below) that connects to the existing database
+server.  ``<relay source>`` could contain a comma-separated list of connection
+methods, e.g. to connect to any server of the clustered database.
+Multiple relay servers could be started for the same relay source.
+
+Since the way relays handle read and write transactions is very similar
+to the clustered model where "cluster" means "set of relay servers connected
+to the same relay source", "follower" means "relay server" and the "leader"
+means "relay source", same consistency consequences as for the clustered
+model applies to relay as well (See `Understanding Cluster Consistency`_
+above).
+
+Open vSwitch 2.16 introduced support for relay service model.
+
 Database Replication
 ====================
 
@@ -414,7 +459,8 @@ Replication, in this context, means to make, and keep up-to-date, a read-only
 copy of the contents of a database (the ``replica``).  One use of replication
 is to keep an up-to-date backup of a database.  A replica used solely for
 backup would not need to support clients of its own.  A set of replicas that do
-serve clients could be used to scale out read access to the primary database.
+serve clients could be used to scale out read access to the primary database,
+however `Relay Service Model`_ is more suitable for that purpose.
 
 A database replica is set up in the same way as a backup server in an
 active-backup pair, with the difference that the replica is never promoted to

Documentation/topics/index.rst

@@ -44,6 +44,7 @@ OVS
    openflow
    bonding
    networking-namespaces
+   ovsdb-relay
    ovsdb-replication
    dpdk/index
    windows

Documentation/topics/ovsdb-relay.rst

@@ -0,0 +1,124 @@
+..
+      Copyright 2021, Red Hat, Inc.
+
+      Licensed under the Apache License, Version 2.0 (the "License"); you may
+      not use this file except in compliance with the License. You may obtain
+      a copy of the License at
+
+          http://www.apache.org/licenses/LICENSE-2.0
+
+      Unless required by applicable law or agreed to in writing, software
+      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+      License for the specific language governing permissions and limitations
+      under the License.
+
+      Convention for heading levels in Open vSwitch documentation:
+
+      =======  Heading 0 (reserved for the title in a document)
+      -------  Heading 1
+      ~~~~~~~  Heading 2
+      +++++++  Heading 3
+      '''''''  Heading 4
+
+      Avoid deeper levels because they do not render well.
+
+===============================
+Scaling OVSDB Access With Relay
+===============================
+
+Open vSwitch 2.16 introduced support for OVSDB Relay mode with the goal to
+increase database scalability for a big deployments.  Mainly, OVN (Open Virtual
+Network) Southbound Database deployments.  This document describes the main
+concept and provides the configuration examples.
+
+What is OVSDB Relay?
+--------------------
+
+Relay is a database service model in which one ``ovsdb-server`` (``relay``)
+connects to another standalone or clustered database server
+(``relay source``) and maintains in-memory copy of its data, receiving
+all the updates via this OVSDB connection.  Relay server handles all the
+read-only requests (monitors and transactions) on its own and forwards all the
+transactions that requires database modifications to the relay source.
+
+Why is this needed?
+-------------------
+
+Some OVN deployment could have hundreds or even thousands of nodes.  On each of
+these nodes there is an ovn-controller, which is connected to the
+OVN_Southbound database that is served by a standalone or clustered OVSDB.
+Standalone database is handled by a single ovsdb-server process and clustered
+could consist of 3 to 5 ovsdb-server processes.  For the clustered database,
+higher number of servers may significantly increase transaction latency due
+to necessity for these servers to reach consensus.  So, in the end limited
+number of ovsdb-server processes serves ever growing number of clients and this
+leads to performance issues.
+
+Read-only access could be scaled up with OVSDB replication on top of
+active-backup service model, but ovn-controller is a read-mostly client, not
+a read-only, i.e. it needs to execute write transactions from time to time.
+Here relay service model comes into play.
+
+2-Tier Deployment
+-----------------
+
+Solution for the scaling issue could look like a 2-tier deployment, where
+a set of relay servers is connected to the main database cluster
+(OVN_Southbound) and clients (ovn-conrtoller) connected to these relay
+servers::
+
+                                    172.16.0.1
+   +--------------------+   +----+ ovsdb-relay-1 +--+---+ client-1
+   |                    |   |                       |
+   |    Clustered       |   |                       +---+ client-2
+   |     Database       |   |                        ...
+   |                    |   |                       +---+ client-N
+   |    10.0.0.2        |   |
+   |  ovsdb-server-2    |   |       172.16.0.2
+   |   +        +       |   +----+ ovsdb-relay-2 +--+---+ client-N+1
+   |   |        |       |   |                       |
+   |   |        +       +---+                       +---+ client-N+2
+   |   |   10.0.0.1     |   |                        ...
+   |   | ovsdb-server-1 |   |                       +---+ client-2N
+   |   |        +       |   |
+   |   |        |       |   |
+   |   +        +       |   +      ... ... ... ... ...
+   |  ovsdb-server-3    |   |
+   |    10.0.0.3        |   |                       +---+ client-KN-1
+   |                    |   |       172.16.0.K      |
+   +--------------------+   +----+ ovsdb-relay-K +--+---+ client-KN
+
+In practice, the picture might look a bit more complex, because all relay
+servers might connect to any member of a main cluster and clients might
+connect to any relay server of their choice.
+
+Assuming that servers of a main cluster started like this::
+
+  $ ovsdb-server --remote=ptcp:6642:10.0.0.1 ovn-sb-1.db
+
+The same for other two servers.  In this case relay servers could be
+started like this::
+
+  $ REMOTES=tcp:10.0.0.1:6642,tcp:10.0.0.2:6642,tcp:10.0.0.3:6642
+  $ ovsdb-server --remote=ptcp:6642:172.16.0.1 relay:OVN_Southbound:$REMOTES
+  $ ...
+  $ ovsdb-server --remote=ptcp:6642:172.16.0.K relay:OVN_Southbound:$REMOTES
+
+Every relay server could connect to any of the cluster members of their choice,
+fairness of load distribution is achieved by shuffling remotes.
+
+For the actual clients, they could be configured to connect to any of the
+relay servers.  For ovn-controllers the configuration could look like this::
+
+  $ REMOTES=tcp:172.16.0.1:6642,...,tcp:172.16.0.K:6642
+  $ ovs-vsctl set Open_vSwitch . external-ids:ovn-remote=$REMOTES
+
+Setup like this allows the system to serve ``K * N`` clients while having only
+``K`` actual connections on the main clustered database keeping it in a
+stable state.
+
+It's also possible to create multi-tier deployments by connecting one set
+of relay servers to another (smaller) set of relay servers, or even create
+tree-like structures with the cost of increased latency for write transactions,
+because they will be forwarded multiple times.

NEWS

@@ -7,6 +7,9 @@ Post-v2.15.0
        limits in addition to the previously configurable byte rate settings.
        This is not supported in the userspace datapath yet.
    - OVSDB:
+     * Introduced new database service model - "relay".  Targeted to scale out
+       read-mostly access (ovn-controller) to existing databases.
+       For more information: ovsdb(7) and Documentation/topics/ovsdb-relay.rst
      * New command line options --record/--replay for ovsdb-server and
        ovsdb-client to record and replay all the incoming transactions,
        monitors, etc.  More datails in Documentation/topics/record-replay.rst.

ovsdb/ovsdb-server.1.in

@@ -10,6 +10,7 @@ ovsdb\-server \- Open vSwitch database server
 .SH SYNOPSIS
 \fBovsdb\-server\fR
 [\fIdatabase\fR]\&...
+[\fIrelay:schema_name:remote\fR]\&...
 [\fB\-\-remote=\fIremote\fR]\&...
 [\fB\-\-run=\fIcommand\fR]
 .so lib/daemon-syn.man
@@ -35,12 +36,15 @@ For an introduction to OVSDB and its implementation in Open vSwitch,
 see \fBovsdb\fR(7).
 .PP
 Each OVSDB file may be specified on the command line as \fIdatabase\fR.
-If none is specified, the default is \fB@DBDIR@/conf.db\fR.  The database
+Relay databases may be specified on the command line as
-files must already have been created and initialized using, for
+\fIrelay:schema_name:remote\fR.  For a detailed description of relay database
-example, \fBovsdb\-tool\fR's \fBcreate\fR, \fBcreate\-cluster\fR, or
+argument, see \fBovsdb\fR(7).
-\fBjoin\-cluster\fR command.
+If none of database files or relay databases is specified, the default is
+\fB@DBDIR@/conf.db\fR.  The database files must already have been created and
+initialized using, for example, \fBovsdb\-tool\fR's \fBcreate\fR,
+\fBcreate\-cluster\fR, or \fBjoin\-cluster\fR command.
 .PP
-This OVSDB implementation supports standalone, active-backup, and
+This OVSDB implementation supports standalone, active-backup, relay and
 clustered database service models, as well as database replication.
 See the Service Models section of \fBovsdb\fR(7) for more information.
 .PP
@@ -50,7 +54,9 @@ successfully join a cluster (if the database file is freshly created
 with \fBovsdb\-tool join\-cluster\fR) or connect to a cluster that it
 has already joined.  Use \fBovsdb\-client wait\fR (see
 \fBovsdb\-client\fR(1)) to wait until the server has successfully
-joined and connected to a cluster.
+joined and connected to a cluster.  The same is true for relay databases.
+Same commands could be used to wait for a relay database to connect to
+the relay source (remote).
 .PP
 In addition to user-specified databases, \fBovsdb\-server\fR version
 2.9 and later also always hosts a built-in database named
@@ -243,10 +249,11 @@ not list remotes added indirectly because they were read from the
 database by configuring a
 \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR remote.
 .
-.IP "\fBovsdb\-server/add\-db \fIdatabase\fR"
+.IP "\fBovsdb\-server/add\-db \fIdatabase\fR
-Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR.  The database
+Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR.  \fIdatabase\fR
-file must already have been created and initialized using, for example,
+could be a database file or a relay description in the following format:
-\fBovsdb\-tool create\fR.
+\fIrelay:schema_name:remote\fR.  The database file must already have been
+created and initialized using, for example, \fBovsdb\-tool create\fR.
 .
 .IP "\fBovsdb\-server/remove\-db \fIdatabase\fR"
 Removes \fIdatabase\fR from the running \fBovsdb\-server\fR.  \fIdatabase\fR