Use the backtrace functions that is provided by libc, this allows us
to get backtrace that is independent of the current memory map of the
process. Which in turn can be used for debugging/tracing purpose.
The backtrace is not 100% accurate due to various optimizations, most
notably "-fomit-frame-pointer" and LTO. This might result that the
line in source file doesn't correspond to the real line. However, it
should be able to pinpoint at least the function where the backtrace
was called.
The implementation is determined during compilation based on available
libraries. Libunwind has higher priority if both methods are available
to keep the compatibility with current behavior.
The backtrace is not marked as signal safe however the backtrace manual
page gives more detailed explanation why it might be the case [0].
Load the "libgcc" or equivalent in advance within the "fatal_signal_init"
which should ensure that subsequent calls to backtrace* do not call
malloc and are signal safe.
The typical backtrace will look similar to the one below:
/lib64/libopenvswitch-3.1.so.0(backtrace_capture+0x1e) [0x7fc5db298dfe]
/lib64/libopenvswitch-3.1.so.0(log_backtrace_at+0x57) [0x7fc5db2999e7]
/lib64/libovsdb-3.1.so.0(ovsdb_txn_complete+0x7b) [0x7fc5db56247b]
/lib64/libovsdb-3.1.so.0(ovsdb_txn_propose_commit_block+0x8d) [0x7fc5db563a8d]
ovsdb-server(+0xa661) [0x562cfce2e661]
ovsdb-server(+0x7e39) [0x562cfce2be39]
/lib64/libc.so.6(+0x27b4a) [0x7fc5db048b4a]
/lib64/libc.so.6(__libc_start_main+0x8b) [0x7fc5db048c0b]
ovsdb-server(+0x8c35) [0x562cfce2cc35]
backtrace.h elaborates on how to effectively get the line information
associated with the addressed presented in the backtrace.
[0]
backtrace() and backtrace_symbols_fd() don't call malloc() explicitly,
but they are part of libgcc, which gets loaded dynamically when first
used. Dynamic loading usually triggers a call to malloc(3). If you
need certain calls to these two functions to not allocate memory (in
signal handlers, for example), you need to make sure libgcc is loaded
beforehand
Reported-at: https://bugzilla.redhat.com/2177760
Signed-off-by: Ales Musil <amusil@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Until now, the monitor process held its log file open while it waited for
the child to exit, and then it reopened it after the child exited. Holding
the log file open this way prevented log rotation from freeing disk space.
This commit avoids the problem by closing the log file before waiting, then
reopening it afterward.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Reported-by: Antonin Bas <abas@vmware.com>
VMware-BZ: #2743409
Signed-off-by: William Tu <u9012063@gmail.com>
Due to not acquiring lock, clang reports:
lib/vlog.c:618:12: error: reading variable 'log_fd' requires holding mutex
'log_file_mutex' [-Werror,-Wthread-safety-analysis]
return log_fd;
The patch fixes it by creating a function in vlog.c to write
directly to log file unsafely.
Tested-at: https://travis-ci.org/github/williamtu/ovs-travis/builds/666165883
Fixes: ecd4a8fcdf ("fatal-signal: Log backtrace when no monitor daemon.")
Suggested-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Ilya Maximets <i.maximets@ovn.org>
Signed-off-by: William Tu <u9012063@gmail.com>
Currently the backtrace logging is only available when monitor
daemon is running. This patch enables backtrace logging when
no monitor daemon exists. At signal handling context, it detects
whether monitor daemon exists. If not, write directly the backtrace
to the vlog fd. Note that using VLOG_* macro doesn't work due to
it's buffer I/O, so this patch directly issue write() syscall to
the file descriptor.
For some system we stop using monitor daemon and use systemd to
monitor ovs-vswitchd, thus need this patch. Example of
ovs-vswitchd.log (note that there is no timestamp printed):
2020-03-23T14:42:12.949Z|00049|memory|INFO|175332 kB peak resident
2020-03-23T14:42:12.949Z|00050|memory|INFO|handlers:2 ports:3 reva
SIGSEGV detected, backtrace:
0x0000000000486969 <fatal_signal_handler+0x49>
0x00007f7f5e57f4b0 <killpg+0x40>
0x000000000047daa8 <pmd_thread_main+0x238>
0x0000000000504edd <ovsthread_wrapper+0x7d>
0x00007f7f5f0476ba <start_thread+0xca>
0x00007f7f5e65141d <clone+0x6d>
0x0000000000000000 <+0x0>
Acked-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: William Tu <u9012063@gmail.com>
It seems like if the user wanted a specific logfile but that request
cannot be fulfilled, OVS/OVN shouldn't just continue as if nothing
really happened (besides logging a warning).
Signed-off-by: Dan Williams <dcbw@redhat.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
The asynchronous IO library in glibc starts threads that show up as memory
leaks in valgrind. This commit attempts to avoid the warnings by flushing
all the asynchronous I/O to the log file before exiting. This only does
part of the job for glibc since it keeps the threads around for some
undefined idle time before killing them, so in addition this commit adds a
valgrind suppression to stop displaying these warnings in any case.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: William Tu <u9012063@gmai.com>
When we start ovs-appctl with no argument option "syslog-method",
the process crashed.
backtrace:
0 0x00000000004180c4 in vlog_set_syslog_method (method=0x0) at lib/vlog.c:587
1 0x0000000000408b27 in parse_command_line (argv=0x7fff097e83b8, argc=2) at utilities/ovs-appctl.c:175
2 main (argc=2, argv=0x7fff097e83b8) at utilities/ovs-appctl.c:53
Signed-off-by: Binbin Xu <xu.binbin1@zte.com.cn>
Signed-off-by: Ben Pfaff <blp@ovn.org>
A recent change changed this constructor to "init_this_module". This
breaks the build on windows according to AppVeyor. Include the module
name in the constructor name to work around the issue.
Fixes: 922fed065e ("vlog: Make the most common module reference more direct.")
Signed-off-by: Russell Bryant <russell@ovn.org>
Signed-off-by: Ben Pfaff <blp@ovn.org>
Most vlog calls are for the log module owned by the translation unit being
compiled, but this module was referenced indirectly through a pointer
variable. That seems silly, so this commit changes the code so that the
local vlog module is referred to directly, as &this_module.
We could get rid of the global variables for vlog modules entirely, but
I like getting linker errors when there's a duplicate module name.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
Until now, vlog had a macro VLOG_DEFINE_THIS_MODULE, which expanded using
VLOG_DEFINE_MODULE, which expanded using VLOG_DEFINE_MODULE__, and the
latter macros didn't have any other users. This commit combines them for
clarity.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
I think we once used this variable from an inline function in vlog.h, so
that we had to make it "extern", but these days it's only used from vlog.c,
so it can be static now.
Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Russell Bryant <russell@ovn.org>
vlog log file can be created when parsing --log-file option, before
switching user, in case the --user option is also specified. While this
does not directly cause errors for the running daemons, it can
leave the log files on the disk as created under the "root" user.
This patch fix the log file ownership to the user specified with --user.
Signed-off-by: Andy Zhou <azhou@nicira.com>
Acked-by: Ansis Atteka <aatteka@nicira.com>
This patch allows to query logging format at the runtime for each destination
with "vlog/list-pattern" command.
Signed-off-by: Ansis Atteka <aatteka@nicira.com>
Acked-by: Ben Pfaff <blp@nicira.com>
This patch helps to address two issues that are present on Ubuntu
15.04 (and most likely other Linux distributions) where rsyslog daemon
is configured to relay log messages from OVS to a remote log collector
and syslog format being used is something other than the one defined in
RFC 3164. These two issues are:
1. libc syslog() function always adds RFC 3164 prefix to syslog
messages before sending them over /dev/log Unix domain socket.
This does not allow us to use libc syslog() function to log in
RFC 5424 format; and
2. rsyslogd daemon that comes with Ubuntu 15.04 is too old and
uses hardcoded syslog message parser when it received messages
over /dev/log UNIX domain socket.
Solution to those two issues would be to use the newly introduced
--syslog-method=udp:127.0.0.1:514 command line argument when starting
OVS.
Signed-off-by: Ansis Atteka <aatteka@nicira.com>
Acked-by: Ben Pfaff <blp@nicira.com>
Without this commit, starting a daemon with just '--syslog-target'
causes a segmentation fault.
Signed-off-by: Gurucharan Shetty <gshetty@nicira.com>
Acked-by: Ben Pfaff <blp@nicira.com>
In OVS, we currently use the term 'facility' to mean the place
where we log (syslog, console or file). In Linux's syslog() and
rfc5424, the term 'facility' is used to specify what type of program
is logging the message (e.g: LOG_DAEMON). This causes confusion
while reading vlog's code. This commit changes the term 'facility'
to 'destination'.
Signed-off-by: Gurucharan Shetty <gshetty@nicira.com>
Acked-by: Ben Pfaff <blp@nicira.com>
A new function vlog_insert_module() is introduced to avoid using
list_insert() from the vlog.h header.
Signed-off-by: Thomas Graf <tgraf@noironetworks.com>
Acked-by: Ben Pfaff <blp@nicira.com>
