How do I analyze a program's core dump file with GDB when it has command-line parameters?

My program operates like this:

exe -p param1 -i param2 -o param3

It crashed and generated a core dump file, core.pid.

I want to analyze the core dump file by

gdb ./exe -p param1 -i param2 -o param3 core.pid

But GDB recognizes the parameters of the EXE file as GDB's input.

How do I analyze a core dump file in this situation?

You can use the core with GDB in many ways, but passing parameters which is to be passed to the executable to GDB is not the way to use the core file. This could also be the reason you got that error. You can use the core file in the following ways:

gdb <executable> <core-file> or gdb <executable> -c <core-file> or

gdb <executable>
...
(gdb) core <core-file>

When using the core file you don't have to pass arguments. The crash scenario is shown in GDB (checked with GDB version 7.1 on Ubuntu).

For example:

$ ./crash -p param1 -o param2
Segmentation fault (core dumped)
$ gdb ./crash core
GNU gdb (GDB) 7.1-ubuntu
...
Core was generated by `./crash -p param1 -o param2'. <<<<< See this line shows crash scenario
Program terminated with signal 11, Segmentation fault.
#0  __strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99    ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
    in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)

If you want to pass parameters to the executable to be debugged in GDB, use --args.

For example:

$ gdb --args ./crash -p param1 -o param2
GNU gdb (GDB) 7.1-ubuntu
...
(gdb) r
Starting program: /home/@@@@/crash -p param1 -o param2

Program received signal SIGSEGV, Segmentation fault.
__strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99    ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
    in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)

Man pages will be helpful to see other GDB options.

Most useful commands are:

• bt (backtrace)
• info locals (show values of local variables)
• info registers (show values of CPU registers)
• frame X (change to stack frame X)
• up and down (navigate in the stack frame (call chain))

Simple usage of GDB, to debug coredump files:

gdb <executable_path> <coredump_file_path>

A coredump file for a "process" gets created as a "core.pid" file.

After you get inside the GDB prompt (on execution of the above command), type:

...
(gdb) where

This will get you with the information, of the stack, where you can analayze the cause of the crash/fault. Other command, for the same purposes is:

...
(gdb) bt full

This is the same as above. By convention, it lists the whole stack information (which ultimately leads to the crash location).

objdump + gdb minimal runnable example

TL;DR:

• GDB can be used to find failing line, previously mentioned at: https://stackoverflow.com/questions/8305866/how-to-analyze-a-programs-core-dump-file-with-gdb/8306805#8306805
• the core file contains the CLI arguments, no need to pass them again
• objdump -s core can be used to dump memory in bulk

Now for the full educational test setup:

main.c

#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int myfunc(int i) {
    *(int*)(NULL) = i; /* line 7 */
    return i - 1;
}

int main(int argc, char **argv) {
    /* Setup some memory. */
    char data_ptr[] = "string in data segment";
    char *mmap_ptr;
    char *text_ptr = "string in text segment";
    (void)argv;
    mmap_ptr = (char *)malloc(sizeof(data_ptr) + 1);
    strcpy(mmap_ptr, data_ptr);
    mmap_ptr[10] = 'm';
    mmap_ptr[11] = 'm';
    mmap_ptr[12] = 'a';
    mmap_ptr[13] = 'p';
    printf("text addr: %p\n", text_ptr);
    printf("data addr: %p\n", data_ptr);
    printf("mmap addr: %p\n", mmap_ptr);

    /* Call a function to prepare a stack trace. */
    return myfunc(argc);
}

Compile, and run to generate core:

gcc -ggdb3 -std=c99 -Wall -Wextra -pedantic -o main.out main.c
ulimit -c unlimited
rm -f core
./main.out

Output:

text addr: 0x4007d4
data addr: 0x7ffec6739220
mmap addr: 0x1612010
Segmentation fault (core dumped)

GDB points us to the exact line where the segmentation fault happened, which is what most users want while debugging:

gdb -q -nh main.out core

then:

Reading symbols from main.out...done.
[New LWP 27479]
Core was generated by `./main.out'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000400635 in myfunc (i=1) at main.c:7
7           *(int*)(NULL) = i;
(gdb) bt
#0  0x0000000000400635 in myfunc (i=1) at main.c:7
#1  0x000000000040072b in main (argc=1, argv=0x7ffec6739328) at main.c:28

which points us directly to the buggy line 7.

CLI arguments are stored in the core file and don't need to be passed again

To answer the specific CLI argument questions, we see that if we change the cli arguments e.g. with:

rm -f core
./main.out 1 2

then this does get reflected in the previous bactrace without any changes in our commands:

Reading symbols from main.out...done.
[New LWP 21838]
Core was generated by `./main.out 1 2'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000564583cf2759 in myfunc (i=3) at main.c:7
7           *(int*)(NULL) = i; /* line 7 */
(gdb) bt
#0  0x0000564583cf2759 in myfunc (i=3) at main.c:7
#1  0x0000564583cf2858 in main (argc=3, argv=0x7ffcca4effa8) at main.c:2

So note how now argc=3. Therefore this must mean that the core file stores that information. I'm guessing it just stores it as the arguments of main, just like it stores the arguments of any other functions.

This makes sense if you consider that the core dump must be storing the entire memory and register state of the program, and so it has all the information needed to determine the value of function arguments on the current stack.

Less obvious is how to inspect the environment variables: https://stackoverflow.com/questions/44686478/how-to-get-environment-variable-from-a-core-dump Environment variables are also present in memory so the objdump does contain that information, but I'm not sure how to list all of them in one go conveniently, one by one as follows did work on my tests though:

p __environ[0]

Binutils analysis

By using binutils tools like readelf and objdump, we can bulk dump information contained in the core file such as the memory state.

Most/all of it must also be visible through GDB, but those binutils tools offer a more bulk approach which is convenient for certain use cases, while GDB is more convenient for a more interactive exploration.

First:

file core

tells us that the core file is actually an ELF file:

core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './main.out'

which is why we are able to inspect it more directly with usual binutils tools.

A quick look at the ELF standard shows that there is actually an ELF type dedicated to it:

Elf32_Ehd.e_type == ET_CORE

Further format information can be found at:

man 5 core

Then:

readelf -Wa core

gives some hints about the file structure. Memory appears to be contained in regular program headers:

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  NOTE           0x000468 0x0000000000000000 0x0000000000000000 0x000b9c 0x000000     0
  LOAD           0x002000 0x0000000000400000 0x0000000000000000 0x001000 0x001000 R E 0x1000
  LOAD           0x003000 0x0000000000600000 0x0000000000000000 0x001000 0x001000 R   0x1000
  LOAD           0x004000 0x0000000000601000 0x0000000000000000 0x001000 0x001000 RW  0x1000

and there is some more metadata present in a notes area, notably prstatus contains the PC:

Displaying notes found at file offset 0x00000468 with length 0x00000b9c:
  Owner                 Data size       Description
  CORE                 0x00000150       NT_PRSTATUS (prstatus structure)
  CORE                 0x00000088       NT_PRPSINFO (prpsinfo structure)
  CORE                 0x00000080       NT_SIGINFO (siginfo_t data)
  CORE                 0x00000130       NT_AUXV (auxiliary vector)
  CORE                 0x00000246       NT_FILE (mapped files)
    Page size: 4096
                 Start                 End         Page Offset
    0x0000000000400000  0x0000000000401000  0x0000000000000000
        /home/ciro/test/main.out
    0x0000000000600000  0x0000000000601000  0x0000000000000000
        /home/ciro/test/main.out
    0x0000000000601000  0x0000000000602000  0x0000000000000001
        /home/ciro/test/main.out
    0x00007f8d939ee000  0x00007f8d93bae000  0x0000000000000000
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93bae000  0x00007f8d93dae000  0x00000000000001c0
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93dae000  0x00007f8d93db2000  0x00000000000001c0
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93db2000  0x00007f8d93db4000  0x00000000000001c4
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93db8000  0x00007f8d93dde000  0x0000000000000000
        /lib/x86_64-linux-gnu/ld-2.23.so
    0x00007f8d93fdd000  0x00007f8d93fde000  0x0000000000000025
        /lib/x86_64-linux-gnu/ld-2.23.so
    0x00007f8d93fde000  0x00007f8d93fdf000  0x0000000000000026
        /lib/x86_64-linux-gnu/ld-2.23.so
  CORE                 0x00000200       NT_FPREGSET (floating point registers)
  LINUX                0x00000340       NT_X86_XSTATE (x86 XSAVE extended state)

objdump can easily dump all memory with:

objdump -s core

which contains:

Contents of section load1:

 4007d0 01000200 73747269 6e672069 6e207465  ....string in te
 4007e0 78742073 65676d65 6e740074 65787420  xt segment.text 

Contents of section load15:

 7ffec6739220 73747269 6e672069 6e206461 74612073  string in data s
 7ffec6739230 65676d65 6e740000 00a8677b 9c6778cd  egment....g{.gx.

Contents of section load4:

 1612010 73747269 6e672069 6e206d6d 61702073  string in mmap s
 1612020 65676d65 6e740000 11040000 00000000  egment..........

which matches exactly with the stdout value in our run.

This was tested on Ubuntu 16.04 amd64, GCC 6.4.0, and binutils 2.26.1.

Just skip the parameters. GDB doesn't need them:

gdb ./exe core.pid

From RMS's GDB debugger tutorial:

prompt > myprogram
Segmentation fault (core dumped)
prompt > gdb myprogram
...
(gdb) core core.pid
...

Make sure your file really is a core image -- check it using file.

A slightly different approach will allow you to skip GDB entirely. If all you want is a backtrace, the Linux-specific utility 'catchsegv' will catch SIGSEGV and display a backtrace.

It doesn't matter if the executable has arguments or not. To run GDB on any binary with a generated core file, the syntax is below.

Syntax:
gdb <binary name> <generated core file>
Eg:
gdb l3_entity 6290-corefile

Let me take the below example for more understanding.

bash-4.1$ **gdb l3_entity 6290-corefile**

**Core was generated** by `/dir1/dir2/dir3/l3_entity **Program terminated with signal SIGABRT, Aborted.**
#0
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10
(gdb)

From the above output, you can guess something about core, whether it is a NULL access, SIGABORT, etc..

These numbers #0 to #10 are the stack frames of GDB. These stack frames are not of your binary. In the above 0 - 10 frames if you suspect anything wrong select that frame

(gdb) frame 8

Now to see more details about it:

(gdb) list +

To investigate the issue further, you can print the suspected variable values here at this point in time.

(gdb) print thread_name

Simply type the command:

$ gdb <Binary> <codeDump>

Or

$ gdb <binary>

$ gdb) core <coreDump>

There isn't any need to provide any command line argument. The code dump generated due to an earlier exercise.

I just use coredumpctl debug (on Fedora 32) and it gives me a GDB console to debug my most recent core dump.