According to profiling data, converting UUIDs to strings is a frequent
operation in some workloads. This typically results in a call to
xasprintf(), which internally calls vsnprintf() twice, first to
calculate the required buffer size, and then to format the string.
This patch introduces specialized functions for printing UUIDs, which
both reduces code duplication and improves performance.
For example, on my laptop, 10,000,000 calls to the new uuid_to_string()
function takes 1296 ms, while the same number of xasprintf() calls using
UUID_FMT take 2498 ms.
Signed-off-by: Dmitry Porokh <dporokh@nvidia.com>
Signed-off-by: Eelco Chaudron <echaudro@redhat.com>
Statically allocated datum objects should be properly initialized with
a special function instead of doing that manually.
WARNING: MemorySanitizer: use-of-uninitialized-value
0 0x58789c in ovsdb_datum_compare_3way lib/ovsdb-data.c:1846:19
1 0x52bbab in evaluate_relop lib/db-ctl-base.c:731:16
2 0x52b042 in check_condition lib/db-ctl-base.c:844:22
3 0x522fea in cmd_wait_until lib/db-ctl-base.c:1935:22
4 0x4c704b in do_vsctl utilities/ovs-vsctl.c:3001:13
5 0x4c4429 in main utilities/ovs-vsctl.c:204:17
6 0x7f5ad5 in __libc_start_call_main
7 0x7f5ad5 in __libc_start_main@GLIBC_2.2.5
8 0x432b04 in _start (utilities/ovs-vsctl+0x432b04)
In this case the reference counter ended up not initialized.
While at it, also updating the list_record() function to use a proper
initialization, even if it doesn't cause any issues for now.
Fixes: 485ac63d10f8 ("ovsdb: Add lazy-copy support for ovsdb_datum objects.")
Acked-by: Dumitru Ceara <dceara@redhat.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Coverity was flagged that the uninitialized output variable was used
in the ctl_context_init_command() function. This patch initializes
the variable.
In addition it also destroys the ds string in ctl_context_done()
in case it's not cleared properly.
Fixes: 07ff77ccb82a ("db-ctl-base: Make common database command code into library.")
Acked-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Paolo Valerio <pvalerio@redhat.com>
Signed-off-by: Eelco Chaudron <echaudro@redhat.com>
Variable "want_key" in function check_condition and variable "key"
in function set_column were not destroyed in exception branch.
This patch calls ovsdb_atom_destroy to release resources to avoid
memory leak.
Fixes: 79c1a00fb5a5 ("db-ctl-base: Don't die in set_column() on error.")
Fixes: e09b3af3e249 ("db-ctl-base: Don't die in is_condition_satisfied() on error")
Acked-by: Simon Horman <horms@ovn.org>
Signed-off-by: Zengyuan Wang <wangzengyuan@huawei.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
The commit b8bf410a5 [0] broke the `ovs-vsctl add` command
which now overwrites the value if it existed already.
This patch reverts the code around the `cmd_add` function
to restore the previous behavior. It also adds testing coverage
for this functionality.
[0] b8bf410a5c
Fixes: b8bf410a5c94 ("db-ctl-base: Use partial map/set updates for last add/set commands.")
Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=2182767
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Daniel Alvarez Sanchez <dalvarez@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Currently, command to add one item into a large set generates the
transaction with the full new content of that set plus 'wait'
operation for the full old content of that set. So, if we're adding
one new load-balancer into a load-balancer group in OVN using
ovn-nbctl, transaction will include all the existing load-balancers
from that groups twice.
IDL supports partial updates for sets and maps. The problem with that
is changes are not visible to the IDL user until the transaction
is committed. That will cause problems for chained ctl commands.
However, we still can optimize the very last command in the list.
It makes sense to do, since it's a common case for manual invocations.
Updating the 'add' command as well as 'set' for a case where we're
actually adding one new element to the map.
One downside is that we can't check the set size without examining
it and checking for duplicates, so allowing the transaction to be
sent and constraints to be checked on the server side in that case.
Not touching 'remove' operation for now, since removals may have
different type, e.g. if elements from the map are removed by the key.
The function will likely need to be fully re-written to accommodate
all the corner cases.
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
ovsdb-server allows the OVSDB clients to specify the uuid for
the row inserts [1]. Both the C IDL client library and Python
IDL are missing this feature. This patch adds this support.
In C IDL, for each schema table, a new function is generated -
<schema_table>insert_persistent_uuid(txn, uuid) which can
be used the clients to persist the uuid.
ovs-vsctl and other derivatives of ctl now supports the same
in the generic 'create' command with the option "--id=<UUID>".
In Python IDL, the uuid to persist can be specified in
the Transaction.insert() function.
[1] - a529e3cd1f("ovsdb-server: Allow OVSDB clients to specify the UUID for inserted rows.:)
Acked-by: Adrian Moreno <amorenoz@redhat.com>
Acked-by: Han Zhou <hzhou@ovn.org>
Acked-by: Terry Wilson <twilson@redhat.com>
Signed-off-by: Numan Siddique <numans@ovn.org>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Currently ovsdb-server is using shallow copies of some JSON objects
by keeping a reference counter. JSON string objects are also used
directly as ovsdb atoms in database rows to avoid extra copies.
Taking this approach one step further ovsdb_datum objects can also
be mostly deduplicated by postponing the copy until it actually
needed. datum object itself contains a type and 2 pointers to
data arrays. Adding a one more pointer to a reference counter
we may create a shallow copy of the datum by simply copying type
and pointers and increasing the reference counter.
Before modifying the datum, special function needs to be called
to perform an actual copy of the object, a.k.a. unshare it.
Most of the datum modifications are performed inside the special
functions in ovsdb-data.c, so that is not very hard to track.
A few places like ovsdb-server.c and column mutations are accessing
and changing the data directly, so a few extra unshare() calls
has to be added there.
This change doesn't affect the maximum memory consumption too much,
because most of the copies are short-living. However, not actually
performing these copies saves up to 40% of CPU time on operations
with large sets.
Reported-at: https://bugzilla.redhat.com/2069089
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
ovsdb-server spends a lot of time cloning atoms for various reasons,
e.g. to create a diff of two rows or to clone a row to the transaction.
All atoms, except for strings, contains a simple value that could be
copied in efficient way, but duplicating strings every time has a
significant performance impact.
Introducing a new reference-counted structure 'ovsdb_atom_string'
that allows to not copy strings every time, but just increase a
reference counter.
This change allows to increase transaction throughput in benchmarks
up to 2x for standalone databases and 3x for clustered databases, i.e.
number of transactions that ovsdb-server can handle per second.
It also noticeably reduces memory consumption of ovsdb-server.
Next step will be to consolidate this structure with json strings,
so we will not need to duplicate strings while converting database
objects to json and back.
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Current algorithm of ovsdb_datum_union looks like this:
for-each atom in b:
if not bin_search(a, atom):
push(a, clone(atom))
quicksort(a)
So, the complexity looks like this:
Nb * log2(Na) + Nb + (Na + Nb) * log2(Na + Nb)
Comparisons clones Comparisons for quicksort
for search
ovsdb_datum_union() is heavily used in database transactions while
new element is added to a set. For example, if new logical switch
port is added to a logical switch in OVN. This is a very common
use case where CMS adds one new port to an existing switch that
already has, let's say, 100 ports. For this case ovsdb-server will
have to perform:
1 * log2(100) + 1 clone + 101 * log2(101)
Comparisons Comparisons for
for search quicksort.
~7 1 ~707
Roughly 714 comparisons of atoms and 1 clone.
Since binary search can give us position, where new atom should go
(it's the 'low' index after the search completion) for free, the
logic can be re-worked like this:
copied = 0
for-each atom in b:
desired_position = bin_search(a, atom)
push(result, a[ copied : desired_position - 1 ])
copied = desired_position
push(result, clone(atom))
push(result, a[ copied : Na ])
swap(a, result)
Complexity of this schema:
Nb * log2(Na) + Nb + Na
Comparisons clones memory copy on push
for search
'swap' is just a swap of a few pointers. 'push' is not a 'clone',
but a simple memory copy of 'union ovsdb_atom'.
In general, this schema substitutes complexity of a quicksort
with complexity of a memory copy of Na atom structures, where we're
not even copying strings that these atoms are pointing to.
Complexity in the example above goes down from 714 comparisons
to 7 comparisons and memcpy of 100 * sizeof (union ovsdb_atom) bytes.
General complexity of a memory copy should always be lower than
complexity of a quicksort, especially because these copies usually
performed in bulk, so this new schema should work faster for any input.
All in all, this change allows to execute several times more
transactions per second for transactions that adds new entries to sets.
Alternatively, union can be implemented as a linear merge of two
sorted arrays, but this will result in O(Na) comparisons, which
is more than Nb * log2(Na) in common case, since Na is usually
far bigger than Nb. Linear merge will also mean per-atom memory
copies instead of copying in bulk.
'replace' functionality of ovsdb_datum_union() had no users, so it
just removed. But it can easily be added back if needed in the future.
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Han Zhou <hzhou@ovn.org>
Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
I would have found these useful for the OVN tests. The {in} operator
is the same as {<=}, but it's still useful to have the alternate syntax
because most of the time we think of set inclusion separately from
set subsets. The {not-in} operator is different from any existing
operator though.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Ilya Maximets <i.maximets@ovn.org>
Valgrind reported:
2491: database commands -- negative checks
==19245== 36 (32 direct, 4 indirect) bytes in 1 blocks are definitely lost in loss record 36 of 53
==19245== at 0x4C2FD5F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19245== by 0x431AB4: xrealloc (util.c:149)
==19245== by 0x41656D: ovsdb_datum_reallocate (ovsdb-data.c:1883)
==19245== by 0x41656D: ovsdb_datum_union (ovsdb-data.c:1961)
==19245== by 0x4107B2: cmd_add (db-ctl-base.c:1494)
==19245== by 0x406E2E: do_vsctl (ovs-vsctl.c:2626)
==19245== by 0x406E2E: main (ovs-vsctl.c:183)
==19252== 16 bytes in 1 blocks are definitely lost in loss record 9 of 52
==19252== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19252== by 0x430F74: xmalloc (util.c:138)
==19252== by 0x414D07: clone_atoms (ovsdb-data.c:990)
==19252== by 0x4153F6: ovsdb_datum_clone (ovsdb-data.c:1012)
==19252== by 0x4104D3: cmd_remove (db-ctl-base.c:1564)
==19252== by 0x406E2E: do_vsctl (ovs-vsctl.c:2626)
==19252== by 0x406E2E: main (ovs-vsctl.c:183)
This patch fixes them.
Acked-by: William Tu <u9012063@gmail.com>
Signed-off-by: Yifeng Sun <pkusunyifeng@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Tables and columns may be abbreviated to unique prefixes, but until
now the error messages have just said there's more than one match.
This commit makes the error messages list the possibilities.
Reviewed-by: Yifeng Sun <pkusunyifeng@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Memory leak occured in case specified key was not found in table
record.
Signed-off-by: Damijan Skvarc <damjan.skvarc@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Let the caller decide how to handle the error. Prepare for using the
parser in ovn-nbctl daemon mode.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Let the caller handle the error. Needed for ovn-nbctl daemon mode.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error via the context instead of calling ctl_fatal() so that
the caller can decide how to handle it.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Traditionally, for boolean variables we use boolean values.
Lets keep to that tradition.
Hopefully, using false with a bool works with gcc 6.3.1;
I use both recent versions of gcc (7.3) and older
versions (4.x), but did not see the issue found in
165c1f0649af commit.
Cc: Ian Stokes<ian.stokes@intel.com>
Fixes: 165c1f0649af ("db-ctl-base: Fix compilation warnings.")
Signed-off-by: Darrell Ball <dlu998@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
This commit fixes uninitialized variable warnings in functions
cmd_create() and cmd_get() when compiling with gcc 6.3.1 and -Werror
by initializing variables 'symbol' and 'new' to NULL.
Cc: Alex Wang <alexw@nicira.com>
Fixes: 07ff77ccb82a ("db-ctl-base: Make common database command code into library.")
Signed-off-by: Ian Stokes <ian.stokes@intel.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error via the context instead of calling ctl_fatal() so that
the caller can decide how to handle it.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error via the context instead of calling ctl_fatal() so that
the caller can decide how to handle it.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error via the context instead of calling ctl_fatal() so that
the caller can decide how to handle it.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error via the context instead of calling ctl_fatal() so that
the caller can decide how to handle it.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Release resources now that we are returning to the caller on error.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Release resources now that we are returning to the caller on error.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Release resources now that we are returning to the caller on error.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Propagate the error via the context for the caller to handle it.
Result of applying the following semantic patch (Coccinelle):
@@
expression s;
@@
- die_if_error(s);
+ ctx->error = s;
+ if (ctx->error) {
+ return;
+ }
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Prepare for the command handlers (pre_cmd_*() cmd_*() functions) to
report errors by storing them in the context.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Also, we no longer return the column as it was not used by any of
existing callers.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Signal that multiple rows match the record identifier via a new output
parameter instead of reporting the problem and dying, so that the caller
can handle the error without terminating the process if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Also, rename the function as it is no longer a typical predicate, so
that the users don't assume that the result is passed in return value.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Return the error message to the caller instead of reporting it and dying
so that the caller can handle the error without terminating the process
if needed.
Signed-off-by: Jakub Sitnicki <jkbs@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
These utilities logged the command very early, before parsing the options
or the command. This meant that logging options (like --log-file or
-vsyslog:off) weren't considered for the purpose of logging the command.
This fixes the problem.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
Several OVS structs contain embedded named unions, like this:
struct {
...
union {
...
} u;
};
C11 standardized a feature that many compilers already implemented
anyway, where an embedded union may be unnamed, like this:
struct {
...
union {
...
};
};
This is more convenient because it allows the programmer to omit "u."
in many places. OVS already used this feature in several places. This
commit embraces it in several others.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Justin Pettit <jpettit@ovn.org>
Tested-by: Alin Gabriel Serdean <aserdean@ovn.org>
Acked-by: Alin Gabriel Serdean <aserdean@ovn.org>
The ovn-nbctl, ovn-sbctl, and ovs-vsctl manpages are inconsistent in
their "Database Commands" section when it comes to referring to what
database tables exist. This commit amends this by making each *ctl
manpage reference the corresponding database manpage instead.
To aid in having a more handy list, the --help text of ovn-nbctl,
ovn-sbctl, and ovs-vsctl have been modified to list the available
tables. This is also referenced in the manpages for those applications.
Signed-off-by: Mark Michelson <mmichels@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
In C++, 'class' is a keyword. If this is used as the name for a field,
then C++ compilers can get confused about the context and fail to
compile references to such fields. Rename the field to 'class_' to
avoid this issue.
Signed-off-by: Joe Stringer <joe@ovn.org>
Acked-by: Ben Pfaff <blp@ovn.org>
Shadowing is when a variable with a given name in an inner scope hides a
different variable with the same name in a surrounding scope. This is
generally undesirable because it can confuse programmers. This commit
eliminates most of it.
Found with -Wshadow=local in GCC 7. The repo is not really ready to enable
this option by default because of a few cases that are harder to fix, and
harmless, such as nested use of CMAP_FOR_EACH.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Andy Zhou <azhou@ovn.org>
