Core dump file is not generated

Linux Problem Overview

Every time, my application crash a core dump file is not generated. I remember that few days ago, on another server it was generated. I'm running the app using screen in bash like this:

#!/bin/bash
ulimit -c unlimited
while true; do ./server; done

As you can see I'm using ulimit -c unlimited which is important if I want to generate a core dump, but it still doesn't generate it, when I got an segmentation fault. How can I make it work?

Linux Solutions

Solution 1 - Linux

This link contains a good checklist why core dumps are not generated:

The core would have been larger than the current limit.
You don't have the necessary permissions to dump core (directory and file). Notice that core dumps are placed in the dumping process' current directory which could be different from the parent process.
Verify that the file system is writeable and have sufficient free space.
If a sub directory named core exist in the working directory no core will be dumped.
If a file named core already exist but has multiple hard links the kernel will not dump core.
Verify the permissions on the executable, if the executable has the suid or sgid bit enabled core dumps will by default be disabled. The same will be the case if you have execute permissions but no read permissions on the file.
Verify that the process has not changed working directory, core size limit, or dumpable flag.
Some kernel versions cannot dump processes with shared address space (AKA threads). Newer kernel versions can dump such processes but will append the pid to the file name.
The executable could be in a non-standard format not supporting core dumps. Each executable format must implement a core dump routine.
The segmentation fault could actually be a kernel Oops, check the system logs for any Oops messages.
The application called exit() instead of using the core dump handler.

Solution 2 - Linux

Make sure your current directory (at the time of crash -- server may change directories) is writable. If the server calls setuid, the directory has to be writable by that user.

Also check /proc/sys/kernel/core_pattern. That may redirect core dumps to another directory, and that directory must be writable. More info here.

Solution 3 - Linux

Check:

$ sysctl kernel.core_pattern

to see how your dumps are created (%e will be the process name, and %t will be the system time).

For Ubuntu, dumps are created by apport in /var/crash, but in different format (see inside file).

You can test it by:

sleep 10 &
killall -SIGSEGV sleep

If core dumping is successful, you will see “(core dumped)” after the segmentation fault indication.

Solution 4 - Linux

For systemd systems¹, install the package systemd-coredump. Coredumps can be found via:

ls /var/lib/systemd/coredump

Furthermore, these coredumps are compressed in the lz4 format. To decompress, you can use the package liblz4-tool like this: lz4 -d FILE. To be able to debug the decompressed coredump using gdb, I also had to rename the utterly long filename into something shorter...

^{¹ Debian 9 Stretch}

Solution 5 - Linux

Remember if you are starting the server from a service, it will start a different bash session so the ulimit won't be effective there. Try to put this in your script itself:

ulimit -c unlimited

Solution 6 - Linux

Also, check to make sure you have enough disk space on /var/core or wherever your core dumps get written. If the partition is almos full or at 100% disk usage then that would be the problem. My core dumps average a few gigs so you should be sure to have at least 5-10 gig available on the partition.

Solution 7 - Linux

Note: If you have written any crash handler yourself, then the core might not get generated. So search for code with something on the line:

signal(SIGSEGV, <handler> );

so the SIGSEGV will be handled by handler and you will not get the core dump.

Solution 8 - Linux

The answers given here cover pretty well most scenarios for which core dump is not created. However, in my instance, none of these applied. I'm posting this answer as an addition to the other answers.

If your core file is not being created for whatever reason, I recommend looking at the /var/log/messages. There might be a hint in there to why the core file is not created. In my case there was a line stating the root cause:

Executable '/path/to/executable' doesn't belong to any package

To work around this issue edit /etc/abrt/abrt-action-save-package-data.conf and change ProcessUnpackaged from 'no' to 'yes'.

ProcessUnpackaged = yes

This setting specifies whether to create core for binaries not installed with package manager.

Solution 9 - Linux

If you call daemon() and then daemonize a process, by default the current working directory will change to /. So if your program is a daemon then you should be looking for a core in / directory and not in the directory of the binary.

Solution 10 - Linux

If one is on a Linux distro (e.g. CentOS, Debian) then perhaps the most accessible way to find out about core files and related conditions is in the man page. Just run the following command from a terminal:

man 5 core

Solution 11 - Linux

Although this isn't going to be a problem for the person who asked the question, because they ran the program that was to produce the core file in a script with the ulimit command, I'd like to document that the ulimit command is specific to the shell in which you run it (like environment variables). I spent way too much time running ulimit and sysctl and stuff in one shell, and the command that I wanted to dump core in the other shell, and wondering why the core file was not produced.

I will be adding it to my bashrc. The sysctl works for all processes once it is issued, but the ulimit only works for the shell in which it is issued (maybe also the descendents too) - but not for other shells that happen to be running.

Solution 12 - Linux

Just in case someone else stumbles on this. I was running someone else's code - make sure they are not handling the signal, so they can gracefully exit. I commented out the handling, and got the core dump.

Solution 13 - Linux

In centos,if you are not root account to generate core file: you must be set the account has a root privilege or login root account:

> vim /etc/security/limits.conf

> account soft core unlimited
account hard core unlimited

then if you in login shell with securecrt or other:

logout and then relogin

Solution 14 - Linux

Allow Dump from Daemons To allow all daemons witch are started by systemd to core dump.

Edit: /etc/systemd/system.conf add following

DefaultLimitCORE=infinity Edit: /etc/sysctl.d/core.conf add following

kernel.core_pattern = /var/lib/coredumps/core-%e-sig%s-user%u-group%g-pid%p-time%t kernel.core_uses_pid = 1 fs.suid_dumpable = 2

more detail: https://pve.proxmox.com/wiki/Enable_Core_Dump_systemd

Solution 15 - Linux

Our application stopped producing core dumps when a capability was set to it.

setcap 'cap_sys_nice=eip' /usr/bin/${our_app}

Removing it allowed the re-generation of coredumps.

setcap '-r' /usr/bin/${our_app}

Content Type	Original Author	Original Content on Stackoverflow
Question	Cyclone	View Question on Stackoverflow
Solution 1 - Linux	Philipp Claßen	View Answer on Stackoverflow
Solution 2 - Linux	Employed Russian	View Answer on Stackoverflow
Solution 3 - Linux	kenorb	View Answer on Stackoverflow
Solution 4 - Linux	Seth	View Answer on Stackoverflow
Solution 5 - Linux	user18853	View Answer on Stackoverflow
Solution 6 - Linux	chown	View Answer on Stackoverflow
Solution 7 - Linux	user18853	View Answer on Stackoverflow
Solution 8 - Linux	Srki Rakic	View Answer on Stackoverflow
Solution 9 - Linux	zapstar	View Answer on Stackoverflow
Solution 10 - Linux	tejus	View Answer on Stackoverflow
Solution 11 - Linux	Brenda J. Butler	View Answer on Stackoverflow
Solution 12 - Linux	Geoff Lentsch	View Answer on Stackoverflow
Solution 13 - Linux	LuciferJack	View Answer on Stackoverflow
Solution 14 - Linux	user2170180	View Answer on Stackoverflow
Solution 15 - Linux	Philippe Brais	View Answer on Stackoverflow