Currently the backtrace logging is only available when monitor
daemon is running. This patch enables backtrace logging when
no monitor daemon exists. At signal handling context, it detects
whether monitor daemon exists. If not, write directly the backtrace
to the vlog fd. Note that using VLOG_* macro doesn't work due to
it's buffer I/O, so this patch directly issue write() syscall to
the file descriptor.
For some system we stop using monitor daemon and use systemd to
monitor ovs-vswitchd, thus need this patch. Example of
ovs-vswitchd.log (note that there is no timestamp printed):
2020-03-23T14:42:12.949Z|00049|memory|INFO|175332 kB peak resident
2020-03-23T14:42:12.949Z|00050|memory|INFO|handlers:2 ports:3 reva
SIGSEGV detected, backtrace:
0x0000000000486969 <fatal_signal_handler+0x49>
0x00007f7f5e57f4b0 <killpg+0x40>
0x000000000047daa8 <pmd_thread_main+0x238>
0x0000000000504edd <ovsthread_wrapper+0x7d>
0x00007f7f5f0476ba <start_thread+0xca>
0x00007f7f5e65141d <clone+0x6d>
0x0000000000000000 <+0x0>
Acked-by: Ben Pfaff <blp@ovn.org>
Signed-off-by: William Tu <u9012063@gmail.com>
The patch catches the SIGSEGV signal and prints the backtrace
using libunwind at the monitor daemon. This makes debugging easier
when there is no debug symbol package or gdb installed on production
systems.
The patch works when the ovs-vswitchd compiles even without debug symbol
(no -g option), because the object files still have function symbols.
For example:
|daemon_unix(monitor)|WARN|SIGSEGV detected, backtrace:
|daemon_unix(monitor)|WARN|0x0000000000482752 <fatal_signal_handler+0x52>
|daemon_unix(monitor)|WARN|0x00007fb4900734b0 <killpg+0x40>
|daemon_unix(monitor)|WARN|0x00007fb49013974d <__poll+0x2d>
|daemon_unix(monitor)|WARN|0x000000000052b348 <time_poll+0x108>
|daemon_unix(monitor)|WARN|0x00000000005153ec <poll_block+0x8c>
|daemon_unix(monitor)|WARN|0x000000000058630a <clean_thread_main+0x1aa>
|daemon_unix(monitor)|WARN|0x00000000004ffd1d <ovsthread_wrapper+0x7d>
|daemon_unix(monitor)|WARN|0x00007fb490b3b6ba <start_thread+0xca>
|daemon_unix(monitor)|WARN|0x00007fb49014541d <clone+0x6d>
|daemon_unix(monitor)|ERR|1 crashes: pid 122849 died, killed \
(Segmentation fault), core dumped, restarting
However, if the object files' symbols are stripped, then we can only
get init function plus offset value. This is still useful when trying
to see if two bugs have the same root cause, Example:
|daemon_unix(monitor)|WARN|SIGSEGV detected, backtrace:
|daemon_unix(monitor)|WARN|0x0000000000482752 <_init+0x7d68a>
|daemon_unix(monitor)|WARN|0x00007f5f7c8cf4b0 <killpg+0x40>
|daemon_unix(monitor)|WARN|0x00007f5f7c99574d <__poll+0x2d>
|daemon_unix(monitor)|WARN|0x000000000052b348 <_init+0x126280>
|daemon_unix(monitor)|WARN|0x00000000005153ec <_init+0x110324>
|daemon_unix(monitor)|WARN|0x0000000000407439 <_init+0x2371>
|daemon_unix(monitor)|WARN|0x00007f5f7c8ba830 <__libc_start_main+0xf0>
|daemon_unix(monitor)|WARN|0x0000000000408329 <_init+0x3261>
|daemon_unix(monitor)|ERR|1 crashes: pid 106155 died, killed \
(Segmentation fault), core dumped, restarting
Most C library functions are not async-signal-safe, meaning that
it is not safe to call them from a signal handler, for example
printf() or fflush(). To be async-signal-safe, the handler only
collects the stack info using libunwind, which is signal-safe, and
issues 'write' to the pipe, where the monitor thread reads and
prints to ovs-vswitchd.log.
Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/590503433
Signed-off-by: William Tu <u9012063@gmail.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>
We have some common code between daemon-unix.c and
daemon-windows.c. Move them to daemon.c
Signed-off-by: Gurucharan Shetty <gshetty@nicira.com>
Acked-by: Ben Pfaff <blp@nicira.com>
