Skip to main content

Core Dumps

Core dump = snapshot of crashed program's memory. Essential for debugging crashes in production where debugger can't be attached.

Crash Snapshot

Core dump preserves: stack, heap, registers, loaded libraries. Analyze crash after it happened.

Enabling Core Dumps​

# Check current limit
ulimit -c

# Enable unlimited core dumps
ulimit -c unlimited

# Make permanent (add to ~/.bashrc)
echo "ulimit -c unlimited" >> ~/.bashrc

# System-wide (Linux)
echo "kernel.core_pattern = /tmp/core.%e.%p" | sudo tee /etc/sysctl.d/core.conf
sudo sysctl -p /etc/sysctl.d/core.conf

Core pattern variables:

  • %e = executable name
  • %p = PID
  • %t = timestamp
  • %s = signal number

Generating Core Dump​

// crash.cpp
#include <iostream>

void crash() {
    int* ptr = nullptr;
    *ptr = 42;  // Crash here
}

int main() {
    std::cout << "About to crash\n";
    crash();
    return 0;
}
# Compile with debug symbols
g++ -g crash.cpp -o crash

# Run and crash
./crash
Segmentation fault (core dumped)

# Core file created
ls -lh core*
-rw------- 1 user user 352K Mar 5 10:00 core.crash.12345

Analyzing with GDB​

# Load core dump
gdb ./program core.12345

# Or specify both
gdb -c core.12345 ./program

Example Session​

$ gdb ./crash core.crash.12345
Core was generated by `./crash'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000400567 in crash () at crash.cpp:6
6           *ptr = 42;

(gdb) bt
#0  0x0000000000400567 in crash () at crash.cpp:6
#1  0x0000000000400589 in main () at crash.cpp:11

(gdb) frame 0
#0  crash () at crash.cpp:6

(gdb) list
1    #include <iostream>
2
3    void crash() {
4        int* ptr = nullptr;
5        *ptr = 42;
6    }

(gdb) print ptr
$1 = (int *) 0x0

(gdb) info locals
ptr = 0x0

Information from Core Dump​

# Where it crashed
(gdb) where
(gdb) bt

# Registers at crash
(gdb) info registers

# Local variables
(gdb) info locals

# Arguments
(gdb) info args

# Thread information
(gdb) info threads
(gdb) thread apply all bt

# Memory at crash location
(gdb) x/10x $rsp    # Stack

Multi-Threaded Crash​

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7f... main () at main.cpp:20
  2    Thread 0x7e... worker () at worker.cpp:45
  3    Thread 0x7d... helper () at helper.cpp:10

# See all backtraces
(gdb) thread apply all bt

# Switch to thread that crashed
(gdb) thread 2
(gdb) bt

Common Crash Signals​

SignalMeaningCommon Cause
SIGSEGVSegmentation faultNull/invalid pointer
SIGABRTAbortassert() or abort()
SIGFPEFloating pointDivision by zero
SIGILLIllegal instructionCorrupted code pointer
SIGBUSBus errorMisaligned access

Generating Core Manually​

#include <csignal>
#include <unistd.h>

// Trigger core dump manually
void save_core_dump() {
    raise(SIGABRT);  // Abort and dump core
}

// Or
void save_core_dump_alt() {
    kill(getpid(), SIGSEGV);  // Simulate segfault
}

Core Dump Without Crash​

# Generate core of running process
gcore <PID>

# Create core.12345 without stopping process

Preventing Core Dumps​

# Disable core dumps
ulimit -c 0

# Prevent specific program from dumping
prctl(PR_SET_DUMPABLE, 0);  // In C/C++ code

Core Dump Location​

# Check where cores are saved
cat /proc/sys/kernel/core_pattern

# Common patterns:
# core                  → Current directory
# /tmp/core.%e.%p      → /tmp with name.PID
# |/usr/lib/systemd... → Systemd journal

# Change location
echo "/tmp/core.%e.%p.%t" | sudo tee /proc/sys/kernel/core_pattern

Examining Without GDB​

# Basic info
file core.12345
# core.12345: ELF 64-bit LSB core file, x86-64...

# Strings in core dump
strings core.12345 | less

# Binary diff (compare cores)
cmp core.old core.new

Production Debugging​

# 1. User reports crash
# 2. They send you core dump
# 3. You need matching binary and debug symbols

# Load core with correct binary
gdb /exact/same/binary core.dump

# If symbols separate
symbol-file /path/to/symbols

Automated Core Analysis​

# Script to analyze core dumps
#!/bin/bash
CORE=$1
BINARY=$2

gdb -batch -ex "bt" -ex "info locals" -ex "quit" $BINARY $CORE
./analyze_core.sh core.12345 ./program

Best Practices​

DO
  • Enable cores in development (ulimit -c unlimited)
  • Keep matching binaries for production cores
  • Strip binaries, but keep debug symbols separately
  • Set sensible core_pattern (name + PID)
  • Analyze cores from production crashes
DON'T
  • Enable cores in production (large files, sensitive data)
  • Delete cores before analysis
  • Mix binaries and cores from different versions
  • Assume core always generated (signals can be caught)

Summary​

info

Core dump = crashed program's memory snapshot.

  • Enable: ulimit -c unlimited
  • Analyze: gdb ./program core.12345
  • Check:
    • bt (backtrace)
    • info locals (variables)
    • info threads (threads)
  • Signals:
    • SIGSEGV (segfault)
    • SIGABRT (abort)
    • SIGFPE (div-by-zero)
  • Manual dump: gcore <PID>
  • Production: keep matching binary + symbols
  • Location: Check /proc/sys/kernel/core_pattern
# Workflow:
# 1. ulimit -c unlimited
# 2. Program crashes → core dump created
# 3. gdb ./program core.12345
# 4. bt (see call stack)
# 5. info locals (see variables)
# 6. Fix bug, recompile, test