A staging PostgreSQL database stopped accepting connections.
The first thing that stood out was the disk.
99.98% full.
That made sense as an explanation for why Postgres was unresponsive.
What did not make sense was this:
The actual database was only around 3 GB.
The disk was 32 GiB.
So what was using the rest?
Turns out, almost 30 GB was PostgreSQL WAL.
And that sent us down a Debezium rabbit hole.
The slot that wasn't moving
We use Debezium for CDC.
Debezium reads changes from PostgreSQL through a logical replication slot. PostgreSQL keeps the WAL needed by that slot until the consumer confirms that it has moved forward.
Normally that is exactly what you want.
But there is another side to that guarantee.
If the consumer stops advancing, PostgreSQL cannot throw the old WAL away.
It just keeps it.
And keeps it.
A logical replication slot had stopped making meaningful progress.
The storage history made this pretty obvious once we knew what to look for.
WAL was growing by roughly 5 GB a day.
There were a couple of moments where it dropped back down, presumably when the consumer briefly caught up.
Then it started climbing again.
5 GB.
10 GB.
15 GB.
20 GB.
25 GB.
30 GB.
Eventually there was nowhere left to go.
Postgres stopped accepting connections.
Why did the quieter environment fail?
This was the part that bothered me most.
Other environments used the same CDC setup.
Why weren't they filling their disks too?
We compared the environments.
They were still processing captured changes regularly.
Every time those environments processed changes, their replication positions moved forward and PostgreSQL got a chance to clear WAL that was no longer needed.
The quieter environment had gone without captured changes long enough for its slot to fall behind.
Its position had stopped moving.
That exposed another issue.
The setup lacked a heartbeat that could advance the slot when captured tables were quiet.
The busier environments were getting away with it because they were active.
The affected environment was quieter.
The environment with less happening in it was the one that failed.
I found that interesting.
It is easy to look at a healthy environment and assume its configuration is healthy too.
Sometimes it is just getting enough traffic to hide the problem.
The failure was bigger than the slot
Comparing environments also exposed configuration and alerting gaps. They were separate from the immediate cause of the full disk, but they made the incident harder to detect and recover from.
An alert is only useful when it reaches someone who can act. The review pushed us to check the full path from a signal to a person, not merely whether an alert rule existed.
Adding disk space fixed nothing
At this point Postgres was so full that we could not even connect to it to remove the replication slot.
So we increased the disk from 32 GiB to 64 GiB.
Postgres came back.
Problem solved?
Not really.
The 30 GB of WAL was still sitting there.
We had only created enough space to get back inside the database.
Once we confirmed the affected consumer was stopped, we dropped the inactive replication slot and checkpointed PostgreSQL.
WAL dropped from roughly:
30 GB → 0.5 GB
Disk usage fell dramatically.
That was the actual fix.
Increasing the disk was just a recovery step.
At the rate WAL had been growing, doubling storage without fixing the slot would have bought us about another week.
Then we would have been back in exactly the same place.
What I took away from it
The obvious lesson is:
Don't leave a PostgreSQL replication slot sitting around without a consumer that advances it.
But the more interesting lesson for me was about comparing environments.
My first question was:
What is wrong with the quiet environment?
A better question turned out to be:
What are the busier environments doing that allows them to avoid the same problem?
The answer was activity.
They were healthy partly because they were busy enough to keep moving the replication position.
The quiet environment exposed the assumption we had been relying on.
Healthy systems do not always mean healthy design.
Sometimes they just haven't met the condition that breaks it yet.