Health checks passed. PM2 was online. SSL was valid. Yet every request returned 500. How a crashing Nginx worker hid behind a perfectly healthy application.

When a Healthy Service Goes Dark: Debugging an Nginx Segmentation Fault in Production

There is a particular kind of production incident that breaks the usual debugging instinct.

It is not the kind where the application crashes with a stack trace and a clear error message. It is not the kind where the logs point you straight to the problem. It is the kind where every signal you trust says everything is fine — and yet users cannot reach the service at all.

That is exactly what we ran into.

A Node.js service, running behind Nginx on a production server, stopped serving traffic without warning. No deployment had occurred. No configuration had changed. The application itself was healthy, PM2 reported normal operation, SSL certificates were valid, and health checks returned 200s. From the inside, the system looked perfectly alive.

From the outside, every request returned a 500 Internal Server Error.

This post documents the investigation: the steps taken, the misleading signals, the actual root cause, and what the incident taught us about how we monitor and trust production infrastructure.

The Stack

The setup was straightforward.

Internet
    │
    ▼
 Nginx  (reverse proxy, TLS termination)
    │
    ▼
Node.js via PM2  (application)

Nginx handled TLS termination and proxied traffic to the Node.js process running on localhost port 3000. PM2 managed the Node.js process lifecycle. This is a common, battle-tested configuration.

Nothing about the stack was exotic or experimental. It had been running in production for months without incident.

The Symptoms

Users started reporting failures. The domain was returning:

500 Internal Server Error

At first glance, the assumption was straightforward: the Node.js application was failing. That is almost always the first suspect.

The immediate response was to look at the application.

Step 1: Verify the Application

The quickest way to isolate whether the failure was in the application or in Nginx was to bypass the proxy entirely.

curl http://127.0.0.1:3000/api/health

Response:

{
  "api_health": "OK",
  "upstream_health": "OK",
  "health": "OK"
}

The application was healthy. PM2 confirmed it:

┌─────┬──────────────┬─────────┬─────────┬──────────┬────────┬──────┬──────┐
│ id  │ name         │ version │ mode    │ pid      │ uptime │ ↺    │ status │
├─────┼──────────────┼─────────┼─────────┼──────────┼────────┼──────┼──────┤
│ 0   │ app          │ 1.0.0   │ cluster │ 12847    │ 47m    │ 0    │ online │
└─────┴──────────────┴─────────┴─────────┴──────────┴────────┴──────┴──────┘

Zero restarts. Stable uptime. The application was not the problem.

The failure was somewhere between Nginx and the application.

Step 2: Verify SSL and Networking

The next category to eliminate was TLS and networking. A broken certificate, a DNS misconfiguration, or a network routing issue could cause the proxy layer to fail in ways that surface as 500 responses.

Testing the HTTPS connection locally:

curl -vk https://localhost \
  -H "Host: services.example.com"

TLS negotiation completed successfully. The certificate was valid and not expired. DNS was resolving to the correct IP address. The server was reachable on port 443.

This eliminated TLS and networking as the cause.

Step 3: Read the Nginx Error Log

Configuration and networking looked clean. The next step was to look at what Nginx itself was reporting.

sudo tail -n 50 /var/log/nginx/error.log

This is the line that changed the investigation entirely:

2026/06/10 03:12:47 [alert] 18423#18423: worker process 18501 exited on signal 11 (core dumped)
2026/06/10 03:12:47 [alert] 18423#18423: worker process 18502 exited on signal 11 (core dumped)

Signal 11. Segmentation fault.

This is not a configuration error. This is not an upstream connection failure. This is Nginx crashing.

A segmentation fault means a process attempted to access a memory address it was not permitted to access. The kernel killed the worker process immediately. When Nginx's worker processes crash, the master process may attempt to spawn replacements — but under certain conditions, those replacements also crash. The result is an Nginx that appears to be running but cannot actually process requests.

That explains every symptom precisely:

The application was healthy because the application was never involved
Nginx accepted connections at the TCP level — the master process was still alive
SSL worked because TLS termination happened before the crashing workers took over
Every request returned 500 because the workers responsible for proxying were crashing before completing the response

The 500 errors were not application errors. They were reverse proxy failures.

Why This Took Time to Find

The failure pattern mapped cleanly onto application failure. That is the first assumption any engineer makes when users report 500 errors.

The diagnostic flow that most engineers follow looks like this:

500 Error reported
    │
    ▼
Check application logs
    │
    ▼
Check application health
    │
    ▼
Check application process
    │
    ▼
Restart application

Every step in that flow returned reassuring results. The application was fine. Which makes it easy to keep looking inside the application for a problem that does not exist there.

The actual failure was one layer up, in the infrastructure sitting in front of the application. Without checking the Nginx error log specifically, this incident could have consumed hours of application-level debugging before the real cause surfaced.

The log had the answer the entire time.

The Fix

Once the root cause was clear, the resolution was direct.

The Nginx binary itself was the suspect. Segmentation faults in a long-running, stable Nginx process are most commonly caused by a corrupted executable, an inconsistent package state from an interrupted or partial system update, or a runtime library mismatch — OpenSSL, PCRE, or zlib becoming incompatible with the installed Nginx build.

The fix was to reinstall from clean packages and restart:

sudo apt update
sudo apt install --reinstall nginx nginx-common nginx-core
sudo systemctl restart nginx

After restart, verifying service recovery:

curl https://services.example.com/api/health

{
  "api_health": "OK",
  "upstream_health": "OK",
  "health": "OK"
}

Traffic recovered immediately. The service was fully operational.

What Probably Caused It

We could not prove the root cause with certainty, but the candidates were clear.

Corrupted binary from an interrupted package update. Unattended upgrades or a system update that was interrupted mid-write can leave the Nginx binary in an inconsistent state. The process appears to start correctly but crashes under actual load when it hits a damaged code path.

Runtime library version mismatch. Nginx links at runtime against OpenSSL, PCRE, and zlib. If any of those libraries were updated while Nginx was running — or updated to a version incompatible with the installed Nginx build — worker processes can segfault when they invoke affected code paths. The mismatch is invisible until a request exercises the broken dependency.

Third-party module fault. If a compiled module (gzip, headers, custom) was installed separately from the main package, its binary state might not match the core after an OS-level update. Module-level segfaults can occur even when all configuration syntax is valid.

Any of these would produce exactly what we observed: a stable master process, crashing workers, and clean configuration that gave no indication of the problem.

Lessons Learned

Application health checks only prove the application is healthy

This is a critical distinction that is easy to forget under pressure.

When a health endpoint returns 200, it confirms the application process is alive and can handle requests on its bound port. It does not confirm that requests are actually reaching it from the public internet. It does not confirm the proxy layer is functioning. It does not confirm that the full request path is intact.

Health checks are necessary. They are not sufficient for diagnosing end-to-end failures.

Monitoring needs to verify the complete path — from an external vantage point through to the application — not just individual component health in isolation.

Check logs before touching configuration

The immediate instinct during a 500 incident is to look at Nginx configuration. Maybe a proxy_pass is wrong. Maybe an upstream block is misconfigured. Maybe a recent change broke something.

In this incident, the configuration was never the problem. The binary was.

Editing configuration to fix a binary problem would have changed nothing — and would have introduced the risk of misconfiguration on top of an already broken system.

One minute in the error log would have skipped every other step.

Separate each layer before drawing conclusions

The investigation only moved forward because each layer was tested independently.

Client → Nginx         tested via:  curl -vk https://localhost
Nginx  → Application   tested via:  curl http://127.0.0.1:3000/api/health
Application → Upstream checked via: health endpoint, PM2 status

When each layer is tested in isolation, the fault boundary becomes specific. In this case, the curl to port 3000 returning healthy was the signal that moved the investigation from "application is broken" to "something between the internet and the application is broken." That redirection to the Nginx error log followed directly from that isolation step.

Skipping that step would have meant debugging the wrong component.

Infrastructure can be the culprit

Production engineering instinct tends to treat infrastructure as stable and applications as the source of failures. Applications change constantly. Infrastructure rarely changes.

But infrastructure is software. It runs on packages that get updated, on binaries that can be corrupted, on libraries that can drift out of compatibility. Infrastructure fails quietly — not with loud stack traces, but with crash signals in logs that no one is watching.

The application being innocent is not proof that it is the wrong place to look. It is proof that the right place to look is elsewhere.

Final Thoughts

This incident was not dramatic. There was no cascading failure, no data loss, no extended outage. The fix took minutes once the root cause was known.

But the time between the first 500 error and finding signal 11 in the error log was longer than it should have been. The misleading signals — healthy health checks, valid SSL, running PM2 processes — were convincing enough to keep attention on the application side for longer than warranted.

The log had the answer immediately. We just did not look there first.

In a layered system, a failure at any layer can look like a failure at any other layer from the outside. The only reliable way to find the real fault is to isolate each component, test it directly, and read what the system is actually telling you.

Trust evidence over assumptions. The logs usually know before we do.