kopano-coredump(5)

Description

Coredump generation is largely controlled by the surrounding operating system. Kopano daemons only have little control over it (see end of this document).

System configuration

  Permission to generate

Most kopano daemons are configured to switch user IDs during operation (cf. "run_as_user" and "run_as_group" configuration file directives), and so the fs.suid_dumpable sysctl needs to be changed to non-zero. The recommended value for when you want to enable Kopano coredumps is 2. See the procfs(5) manual page for other options.

When setting fs.suid_dumpable=2, kernel.core_pattern must not be a relative path.

  Location for dumps

Ensure that the kernel.core_pattern sysctl specifies a template which is writable for the (unprivileged) daemon process. If the template is left at the default value ("core") and the working directory of the daemon, controlled with the "running_path" directive in the config file, happens to be an unwritable directory such as the root directory, a dump cannot be generated.

If in doubt, set kernel.core_pattern to something like /tmp/core.%E.%p, which produces standard uncompressed ELF cores. Ensure enough disk space is available. You should compress it before sending it to Kopano.

sysctls may not necessarily be settable from within a Linux container, in which case this has to be done in the top namespace.

  Applicable limits

When starting the service via systemd/systemctl: The coredump size limit is by default infinite. It is possible to define a global coredump limit in /etc/systemd/system.conf, so ensure that no custom value (cf. "DefaultLimitCORE" directive) has been set.

To check for a particular service's current applied policy, you can also use: `systemctl show kopano-server` and look at the LimitCORE= directive.

Alternatively, when starting the program directly from a shell prompt without the use of a service manager: The user shell limits are defined in /etc/security/limits.conf and often need to be adjusted, such as by adding a line like:
	* hard core unlimited

Kopano service settings

Once the system is set up for core generation, there is one more option on the kopano side.

•   kopano-dagent, kopano-gateway and kopano-spooler offer a "coredump_enabled" directive in their configuration file, and its default value is "systemdefault".

•   kopano-server has a "coredump_enabled" directive as well, and it defaults to "yes".

All other daemons which recognize no such directive in their config file behave as if coredump_enabled=systemdefault was in effect.

•   When coredump_enabled=no, the daemon sets the maximum core size to zero, which effectively deactivates dump generation even if otherwise permitted by the system.

•   When coredump_enabled=systemdefault, the system configuration is used as is.

•   When coredump_enabled=yes, the daemon attempts to increase the inherited system default value for itself to infinity. This is only really necessary if the system defaults (LimitNCORE, see above) specify a low limit.

Dump generation via gdb

There is an alternate way of generating dumps, using gdb, which is helpful if the automatic generation as explained above does not work for whatever reason (such as the inability to edit kernel.core_pattern).

Using gdb, one can attach to a particular process or start a new instance, and then inspect its state.

  Generate dumpfile

Thread 1 "kopano-dagent" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) gcore
warning: target file /proc/25097/cmdline contained unexpected null characters
Saved corefile core.25097

  Backtrace

A backtrace produced by gdb is usually a higher quality than the one produced by the process itself. For this, the debuginfo must be available in the system. (When running a version built yourself, the info is inside the executable. (When running a version from prebuilt binaries, look for package names containing "debug", "dbg" or similar.)

(gdb) thread apply all bt

  Signal interruptions

Some signals are received under normal conditions, such as SIGPIPE. To have gdb not stop and require user input all the time, SIGPIPE should be ignored:

(gdb) handle SIGPIPE nostop noprint
If you expect to observe a process for a really long time, it may make sense to do the same for SIGHUP too.

Note that openssl will, on certain architectures (ARM, PPC, s390x, sparcv9), use "smoke testing" to determine CPU capabilities, where it issues CPU instructions that not all CPUs understand. openssl sets up a signal handler to deal with it, but said signals will still stop gdb and you need to issue "continue" a few times.