Until now, when the poll_loop module's log level was turned up to "debug",
it would log a backtrace of the call stack for the event that caused poll()
to wake up in poll_block(). This was pretty useful from time to time to
find out why ovs-vswitchd was using more CPU than expected, because we
could find out what was causing it to wake up.
But there were some issues. One is simply that the backtrace was printed
as a series of hexadecimal numbers, so GDB or another debugger was needed
to translate it into human-readable format. Compiler optimizations meant
that even the human-readable backtrace wasn't, in my experience, as helpful
as it could have been. And, of course, one needed to have the binary to
interpret the backtrace. When the backtrace couldn't be interpreted or
wasn't meaningful, there was essentially nothing to fall back on.
This commit changes the way that "debug" logging for poll_block() wakeups
works. Instead of logging a backtrace, it logs the source code file name
and line number of the call to a poll_loop function, using __FILE__ and
__LINE__. This is by itself much more meaningful than a sequence of
hexadecimal numbers, since no additional interpretation is necessary. It
can be useful even if the Open vSwitch version is only approximately known.
In addition to the file and line, this commit adds, for wakeups caused by
file descriptors, information about the file descriptor itself: what kind
of file it is (regular file, directory, socket, etc.), the name of the file
(on Linux only), and the local and remote endpoints for socket file
descriptors.
Here are a few examples of the new output format:
932-ms timeout at ../ofproto/in-band.c:507
[POLLIN] on fd 20 (192.168.0.20:35388<->192.168.0.3:6633) at ../lib/stream-fd.c:149
[POLLIN] on fd 7 (FIFO pipe:[48049]) at ../lib/fatal-signal.c:168
This commit makes the status of manager connections visible via the Manager
table in the database. Two new columns have been created for this purpose:
'is_connected' and 'status'. The former is a boolean flag, and the latter is a
string-string map which may contain the keys "last_error", "state", and
"time_in_state".
Requested-by: Keith Amidon <keith@nicira.com>
Reviewed by: Ben Pfaff.
Feature #3692.
Many OVS functions return 0, EOF, or errno. There are several places in the
codebase where a return value is converted to a string. All must decide whether
the return value is set, and if it is, whether it is an errno value, EOF, or
otherwise invalid. This commit consolidates that code.
Reviewed by Ben Pfaff.
ovs_queue doesn't seem very useful; it's just a singly-linked list. It's
more generally useful to use a general-purpose "struct list" for lists of
packets, so this commit adds such a member to "struct ofpbuf" and shifts
the existing users to use it.
I'm retaining the "managers" column in the Open_vSwitch table for now, but
I hope that applications transition to using "manager_options" eventually
so that we could drop it.
CC: Andrew Lambeth <wal@nicira.com>
CC: Jeremy Stribling <strib@nicira.com>
Adding a macro to define the vlog module in use adds a level of
indirection, which makes it easier to change how the vlog module must be
defined. A followup commit needs to do that, so getting these widespread
changes out of the way first should make that commit easier to review.
In the ovsdb-server log there are fairly continuous messages like these:
Apr 26 11:27:55|15254|reconnect|INFO|unix:/tmp/stream-unix.31734.0: connected
Apr 26 11:27:55|15255|reconnect|INFO|unix:/tmp/stream-unix.31734.0: connection dropped
Apr 26 11:28:00|15256|reconnect|INFO|unix:/tmp/stream-unix.31810.0: connecting...
Apr 26 11:28:00|15257|reconnect|INFO|unix:/tmp/stream-unix.31810.0: connected
Apr 26 11:28:00|15258|reconnect|INFO|unix:/tmp/stream-unix.31810.0: connection dropped
These just indicate that ovs-vsctl is connecting to ovsdb-server from,
for example, the "vif" script. But there's no need to log all that detail;
it's simply not useful. This commit suppresses it.
Bug #2715.
Always passing 0 to reconnect_disconnected() means that it uses a generic
log message ("connection dropped"). By passing the error code, as done by
this commit, reconnect_disconnected() can log a more specific message.
Sometimes, when a user asks me to help debug a problem, it turns out that
an SSL connection was being made on a TCP port, or vice versa, or that an
OpenFlow connection was being made on a JSON-RPC port, or vice versa, and
so on. This commit adds log messages that diagnose this kind of problem,
e.g. "tcp:127.0.0.1:6633: received JSON-RPC data on OpenFlow channel".
The fatal-signal library notices and records fatal signals (e.g. SIGTERM)
and terminates the process on the next trip through poll_block(). But
some special utilities do not always invoke poll_block() promptly, e.g.
"ovs-ofctl monitor" does not call poll_block() as long as OpenFlow messages
are available. But these special cases seem like they are all likely to
call into functions that themselves block (those with "_block" in their
names). So make a new rule that such functions should always call
fatal_signal_run(), either directly or through poll_block(). This commit
implements and documents that rule.
Bug #2625.
This is unlikely to occur very often in practice, because s->stream
usually gets stuffed into s->rpc before long, but it is still a good idea
to fix it.
Some upcoming code wants to serialize JSON into a "struct ds" dynamic
string buffer, so expose an interface to do this.
This commit doesn't change much, but it renames some functions internal
to json.c to make the naming more consistent.
Also, make jsonrpc_log_msg() use this new function, since it is a more
straightforward way to do what it wants.
jsonrpc_session_connect() indirectly called reconnect_disconnected(), which
told the reconnect object that the connection had failed, before it told it
that the connection attempt had started. When the connection didn't
complete immediately, this caused the connection to time out immediately,
without any backoff.
Reported by Jeremy Stribling.
SSL, which will be added in an upcoming commit, requires some background
processing, which is best done in a "run" function in our architecture.
This commit adds stream_run() and stream_run_wait() and calls to them from
the places where they will be required.
