strace in Production: When Your Process Won't Start and Logs Say Nothing

The "silent startup failure" is a common operational hurdle where a service fails to initialize under systemd despite running correctly when executed manually. This discrepancy typically occurs because the process terminates during the early boot sequence, before the application’s logging framework has initialized or successfully connected to its sink. When this happens, systemd often swallows the process's stderr, leaving administrators with no visibility into the crash.

When application-level logs fail to surface a root cause, you must move down the stack to the kernel interface using strace. This utility intercepts and records system calls made by a process, providing direct insight into how the application interacts with the operating system—such as failing to resolve a filesystem path, permission denied errors, or unexpected getcwd() failures.

To diagnose the failure, you can attach strace to the hanging process:

• Follow child processes: Use -f to track workers spawned by the daemon.
• Filter syscalls: Utilize -e trace=open,openat,read,write to isolate the specific interactions causing the hang.
• Analyze timing: Use -r for relative timestamps or -t for absolute timestamps to identify latency gaps before the timeout.
• Output capture: Redirect results to a file with -o /tmp/trace.log for post-mortem analysis.

A critical configuration detail often overlooked is that systemd may default to suppressing stderr. Ensure your unit file includes StandardOutput=journal or StandardError=journal to verify that the daemon's initial error stream is captured by the journal. Be aware that strace introduces significant performance overhead by pausing the process at every syscall, making it unsuitable for high-throughput production environments. Use it strictly during a maintenance window or within an isolated staging instance that replicates the production environment configuration.

When a service is running but unresponsive, and its logs contain nothing useful, application-level debugging may be insufficient. Attaching strace to the live process provides a direct view of the system calls the kernel is executing on the process's behalf. This is a diagnostic technique, not a permanent monitoring solution.

Start by locating the process ID (PID) of the daemon:

systemctl status your-daemon

The status output lists the main PID and child processes. With that PID, attach strace:

strace -p 1234 -f -e trace=write

• -p attaches to an already running process.
• -f follows forked child processes. Many daemons spawn workers, so this is often essential.
• -e trace=write filters the trace to the write syscall. This reveals output attempts, including messages written to stdout or stderr.

When application logging is silent, a filtered write trace can expose failures that occur before logging is initialized. In one observed case, the process repeatedly attempted to open a working directory under /var/run/ that did not exist. strace captured both the failed openat and a subsequent write(2, "failed to chdir", 15) to stderr. The error had been swallowed by systemd because StandardOutput was not set to journal.

Useful variations:

• strace -p -f — tail all syscalls as they happen.
• strace -p -e trace=open,openat,read,write — filter to specific syscall families.
• strace -p -f -t — add wall-clock timestamps.
• strace -p -f -r — print relative timestamps from the attach moment.
• strace -p -f -o /tmp/trace.log — save the trace for later analysis.
• strace -p -c -f — run for a period, then press Ctrl+C to print syscall counts.

The common output format is SYSCALL(arg1, arg2) = RESULT. A negative result value such as -1 ENOENT indicates an error. strace adds measurable overhead, so avoid leaving it attached to high-throughput production processes. Use a maintenance window, a test instance, or counter-based summary mode for profiling.

When a service starts and produces no log output, application-level logging has already failed you. At that point, strace shows the raw conversation between the process and the kernel. Its output format is consistent: SYSCALL(arg1, arg2) = RESULT. A failed syscall returns -1, and the kernel records the reason in errno, which strace prints as a symbolic suffix.

Consider this trace from a daemon that starts, sits silently, and then times out under systemd:

openat(AT_FDCWD, "/var/run/daemon/workdir", O_RDONLY) = -1 ENOENT
write(2, "failed to chdir", 15) = 15

The first line attempts to open /var/run/daemon/workdir read-only. AT_FDCWD tells the kernel to resolve that path relative to the process's current working directory; O_RDONLY requests read-only access. The result -1 ENOENT means the directory does not exist. The daemon expected the init script to create this workdir before startup; with the path absent, it cannot resolve its intended working directory and takes a failure path.

The second line is the application reporting that failure. File descriptor 2 is stderr. The process writes 15 bytes of the literal string failed to chdir, and the return value 15 confirms every byte reached the kernel. The error message was written, but under systemd it can be swallowed unless the service unit routes stderr to the journal, for example with StandardOutput=journal (or StandardError=journal). That is why the service looks silent while actually emitting an error.

Practical points for debugging silent startup failures:

• Attach to the PID with strace -p -f -e trace=open,openat,read,write; -f follows forked worker processes.
• Filter by syscall type to cut noise, and add -t for timestamps or -o /tmp/trace.log to capture output for analysis.
• Use strace in a maintenance window or against a test instance, because it adds overhead to every syscall.

In this case the fix was a one-line correction in the init script to create the missing workdir, but the cause was only discoverable by reading the kernel's answer.

When a daemon fails to start cleanly under systemd but runs when launched by hand, application-level logging is usually the first casualty. The process exits before its logging subsystem initializes, so the journal stays silent. In the observed case, attaching strace -p <pid> -f -e trace=write showed the failure in two syscalls:

openat(AT_FDCWD, "/var/run/daemon/workdir", O_RDONLY) = -1 ENOENT
write(2, "failed to chdir", 15) = 15

The openat call failed because the expected working directory did not exist. The subsequent write(2) went to stderr, which systemd discards unless StandardOutput=journal is configured. The application internally called getcwd() and died silently when the path could not be resolved—no log line, no config or permission error to inspect.

The fix was one line in the init script to create /var/run/daemon/workdir before daemon startup. Nothing in the application code was wrong, and nothing in the configuration indicated the directory was missing.

This is discoverable only with strace because strace records the literal syscall interface between process and kernel. Application logging happens after that interface; when a process dies before logging initializes, the kernel's ENOENT is the only verifiable signal. strace does not interpret—it exposes.

Enterprise debugging guidance:

• Use -f to follow forked children; daemons often fail in worker processes spawned after the parent exits.
• Filter with -e trace=open,openat,read,write to keep output readable; the write filter captures stderr that the init system swallows.
• Add -t or -r timestamps to correlate syscall timing with service timeout behavior.
• Never leave strace attached to high-throughput production processes; use a maintenance window, a test instance, or the -c summary flag after detach.

The takeaway: strace is not magic, but it is direct access to what the kernel tells the process. When logs have nothing to say, go one layer lower.

strace intercepts and records system calls made by a process, exposing the kernel’s interaction with that process when application-level logging is silent. A common production scenario is a daemon that exits before writing any log output; attaching strace reveals the exact syscall failure, such as openat(AT_FDCWD, "/var/run/daemon/workdir", O_RDONLY) = -1 ENOENT followed by a write(2) to stderr that systemd suppresses. The following flags form a distilled reference for diagnosing such cases.

Attach to an already running process with -p PID. Add -f to trace forked child processes—essential for daemons that spawn workers. Without -f, strace detaches from child processes immediately, missing the failing syscall.

strace -p -f

Filter by syscall type with -e trace=. This reduces noise and focuses attention on relevant operations. Common filters include:

• -e trace=open,openat,read,write — trace file and I/O operations
• -e trace=write — isolate output attempts, useful when looking for stderr messages
• -e trace=network — trace socket-related syscalls

strace -p -e trace=open,openat,read,write

Timestamp lines with -t to see wall-clock time per syscall. This helps correlate strace output with other event logs. Use -tt for microseconds when higher resolution is needed. Print relative timestamps with -r to measure elapsed time since the previous syscall, which is useful for identifying slow calls or retry loops.

strace -p -f -t
strace -p -f -r

Save output for later analysis with -o /tmp/trace.log. This avoids terminal scrollback limits and lets you inspect a long trace without obstructing the attaching terminal. It also preserves evidence for post-mortem review.

strace -p -f -o /tmp/trace.log

The output format is SYSCALL(arg1, arg2) = RESULT. Understanding this format is necessary before interpreting traces: the result is the return value, with -1 indicating an error and errno shown symbolically (e.g., ENOENT). In the example below, the process attempts to open a working directory, fails, and writes an error to file descriptor 2 (stderr):

openat(AT_FDCWD, "/var/run/daemon/workdir", O_RDONLY) = -1 ENOENT
write(2, "failed to chdir", 15) = 15

Note that strace adds measurable overhead. Do not run it continuously on high-throughput production processes. Instead, use it in a maintenance window or against a test instance. The -c flag prints a summary of syscall counts after detach (Ctrl+C), useful for profiling without per-line noise:

strace -p -c -f

Use strace when logs fail you: it provides direct visibility into the syscall layer, often revealing root causes that application logging never reaches.

strace instruments a process through ptrace, interrupting the target on every system call. That makes tracing inherently expensive: each syscall incurs a context-switch and dispatch cost inside the tracer. On a high-throughput production process, leaving strace attached permanently can therefore distort latency, inflate CPU usage, and change the very behavior you are observing. It is not a monitoring agent; it is a diagnostic tool.

For this reason, plan to run strace only during a maintenance window, on a canary, or against a dedicated test instance that represents the production workload. When you do attach, keep the session bounded. This is where the -c flag earns its place: after you detach, strace prints a summary table of syscall counts, instead of dumping every individual call to the terminal.

# Attach to PID 1234, follow child processes, and collect counts
strace -p 1234 -f -c

# Let it run for the desired interval, then press Ctrl+C
# strace detaches and prints a summary:
#   % time     seconds  usecs/call     calls    errors syscall
#   ------ ----------- ----------- --------- --------- ----------------
#   45.20    0.123456       12.34     10000           read
#   ...

The output is generated after detach, which means you get a concise profile of syscall distribution and frequency without the noise of per-call tracing. Appropriate flags to pair with -c:

• -p PID — attach to an already-running process.
• -f — follow forked child processes, essential for daemons that spawn workers.
• -e trace=open,openat,read,write — limit tracing to specific syscall groups when you already know the culprit.
• -o /tmp/trace.log — write the full trace to a file for post-mortem analysis; with -c, the summary is still printed on detach.

The same mechanism that exposes silent startup failures—such as a process repeatedly failing to open a path under /var/run before logging is available—can be used for profiling. The key is to keep the trace short and deliberate, then let -c convert that kernel-level visibility into an actionable summary.