interview-prep

strace, lsof, and tcpdump — what they do and when to use them

strace — system call tracer

strace intercepts and records every system call a process makes. A syscall is a request from user space to the kernel (open a file, read data, connect to a socket, allocate memory, etc.).

strace ls /tmp                       # Trace ls with all syscalls
strace -p <pid>                      # Attach to running process
strace -p <pid> -c                   # Count and summarise syscalls (low overhead)
strace -p <pid> -e trace=file        # Only file-related syscalls
strace -p <pid> -e trace=network     # Only network syscalls
strace -p <pid> -e trace=open,read,write  # Specific syscalls
strace -p <pid> -f                   # Follow forks (trace child processes too)
strace -p <pid> -T                   # Show time spent in each syscall
strace -o /tmp/trace.log -p <pid>    # Write to file (for long traces)

When to use strace:

  • Process is doing nothing but consuming CPU → are syscalls tight-looping?
  • Process hangs → what syscall is it stuck on? (typically futex, poll, epoll_wait)
  • "File not found" errors → which open() call is failing and with what path?
  • Debugging dynamic library issues → what .so files is it trying to load?
  • Process exits unexpectedly → which syscall returned what error?

Warning: strace adds significant overhead (2-10x slowdown) to the traced process. Use -c for statistics in production; use live tracing only in dev/staging or during an incident where the process is already broken.

# Example: find why a process can't open a config file
strace -p 1234 -e trace=open,openat 2>&1 | grep -E "(config|ENOENT|EACCES)"
# openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)

lsof — list open files

lsof shows every file, socket, pipe, and device that processes have open. In Linux, "everything is a file" — so lsof covers more than you'd think.

lsof -p <pid>               # Everything open by a specific PID
lsof -u joyson              # Everything open by a user
lsof /var/log/app.log       # What processes have this file open
lsof +D /var/log            # All files open in a directory (recursive)
lsof -i                     # All network connections
lsof -i :443                # What's listening/connected on port 443
lsof -i :443 -i :80         # Multiple ports
lsof -i tcp                 # All TCP connections
lsof -i @10.0.0.1           # All connections to a specific host
lsof -nP -i tcp -i udp      # Network, numeric host/port (faster, no DNS)

When to use lsof:

  • "Cannot delete a file" error: lsof /path/to/file — what process has it open? (File is "deleted" from the directory but data stays until all FDs are closed — disk space not freed)
  • Port conflict: lsof -i :8080 — what's already on port 8080?
  • Too many open files (EMFILE error): lsof -p <pid> | wc -l — is the process leaking file descriptors?
  • Investigating a process's connections: lsof -p <pid> -i — all network connections open by the process
# Example: find why /var/log/app.log won't delete
lsof /var/log/app.log
# COMMAND   PID   USER  FD  TYPE  DEVICE  SIZE  NODE  NAME
# app      1234  app   3w  REG   8,1     1.2G  4567  /var/log/app.log (deleted)

# The file is "deleted" from the directory but app (PID 1234) still has FD 3 open.
# Disk space is not freed until app closes or is restarted.
kill -HUP 1234              # Send SIGHUP to reopen log files (if app handles it)

tcpdump — packet capture

tcpdump captures raw network packets on an interface. It's the CLI alternative to Wireshark. Requires CAP_NET_RAW capability (usually root).

tcpdump -i any                       # Capture on all interfaces
tcpdump -i eth0                      # Specific interface
tcpdump -i any port 443              # Filter by port
tcpdump -i any host 10.0.0.1         # Filter by host
tcpdump -i any host 10.0.0.1 and port 443  # Combined filter
tcpdump -i any src 10.0.0.1          # Source IP only
tcpdump -i any dst 10.0.0.1          # Destination IP only
tcpdump -i any -w /tmp/capture.pcap  # Write to file for Wireshark
tcpdump -r /tmp/capture.pcap         # Read pcap file
tcpdump -i any -c 100                # Capture 100 packets then stop
tcpdump -i any -n                    # No DNS resolution (faster)
tcpdump -i any -X port 8080          # Show packet content in hex + ASCII
tcpdump -i any 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0'  # SYN/FIN packets only

When to use tcpdump:

  • Is the traffic even reaching the server? Capture on the server side to confirm packets arrive before checking application logs
  • TLS handshake failing? Capture and inspect with Wireshark
  • Service mesh or proxy debugging: Is traffic going through the proxy?
  • Confirming firewall rules: Did the packet get dropped or did it arrive?
  • DNS issues: tcpdump -i any port 53 to see what queries are being made
  • Latency analysis: Timestamps on packets to measure RTT
# Example: is DNS traffic leaving the server?
tcpdump -i any -n port 53
# 10:00:01.123456 IP 10.0.0.5.45678 > 10.0.0.1.53: A? example.com.
# 10:00:01.124321 IP 10.0.0.1.53 > 10.0.0.5.45678: example.com. A 93.184.216.34

# Example: capture HTTPS to file, analyse in Wireshark
tcpdump -i eth0 port 443 -w /tmp/tls.pcap

What to say in the interview:

"Three very different tools. strace intercepts syscalls — I use it to diagnose what a process is actually doing when it's slow or failing, like finding which file it can't open. lsof shows open files and sockets — most useful for 'what's holding this file open' and 'what's already on this port'. tcpdump captures raw packets — I reach for it when I need to confirm traffic is actually reaching the host, or for diagnosing network-level issues that don't show up in application logs. All three slow things down, so I prefer strace -c and tcpdump -c 100 for live production use."

My notes