On September 7, our DF PostgreSQL server stopped accepting connections.
pgAdmin timed out. The API could no longer reach the database. Azure reported storage at 99.98%, is_db_alive = 0, and even a server restart failed.
At first, that did not make much sense.
The database itself contained roughly 3 GB of application data.
The disk was 32 GiB.
So what was using the other 29 GB?
It wasn't application data
The answer was PostgreSQL WAL.
About 30 GB of it.
Our DF environment used Debezium for change data capture. PostgreSQL maintains a logical replication slot for Debezium so that changes remain available until the consumer confirms it has processed them.
That guarantee has an important consequence.
If the replication position stops moving, PostgreSQL cannot simply discard the old WAL. It has to keep it because, as far as PostgreSQL knows, the consumer may still need it.
That was exactly what was happening.
The debezium_slot was no longer advancing, so PostgreSQL kept retaining WAL until almost the entire disk was occupied. The actual relations were only around 3 GB.
The storage graph made the pattern obvious in hindsight.
WAL had been climbing at roughly 5 GB per day, with two brief periods where the consumer caught up and storage dropped back down. Then it started climbing again until the server eventually ran out of space.
The interesting question became something else.
Why DF?
We had Debezium in dev and production too.
Neither of those environments had gone down.
What was different about DF?
Comparing the environments exposed the real difference.
Production was actively processing CDC messages, so its replication position continued moving forward. PostgreSQL could recycle old WAL.
Development was also active enough to keep advancing, and it had an hourly CDC probe that reported replication-slot lag.
DF was different.
By the time we compared the environments, DF had not successfully sent CDC messages since around August 31. Its replication position had effectively stopped moving.
The environment was quieter.
And because none of the environments had a Debezium heartbeat configured, dev and prod had effectively been getting away with the same weakness because they generated enough activity naturally.
DF did not.
It was a useful inversion of what I initially expected.
The busy environments were healthy partly because they were busy.
The quieter environment was the one accumulating state.
There were other differences too. DF had no equivalent hourly replication-slot watchdog, and its application settings were not automatically reasserted on every main deployment.
The investigation also uncovered separate configuration gaps: multiple DF Debezium instances had been competing for the same replication slot, and the contract_obligations and debezium_signal Event Hubs were missing.
Those were important problems to fix, but they were distinct from the immediate outage mechanism.
The direct failure chain was much simpler:
CDC position stops advancing
|
v
Replication slot retains old WAL
|
v
WAL keeps accumulating
|
v
32 GiB disk fills
|
v
PostgreSQL can no longer operate normally
|
v
Connections failThe first fix wasn't actually the fix
At 100% disk usage, we had another problem.
We could not connect to PostgreSQL to clean anything up.
An active replication slot cannot simply be removed while its consumer is using it, and deleting files directly from pg_wal would risk corrupting the database. The safe recovery required getting PostgreSQL writable again first.
We increased the server storage from 32 GiB to 64 GiB.
A few minutes later, PostgreSQL came back.
But the disk increase had not solved the underlying problem.
The roughly 30 GB of retained WAL was still there. We had only purchased enough breathing room to access the database again.
After confirming the DF Debezium consumer was stopped, we inspected the slot, dropped the inactive debezium_slot, and forced a checkpoint.
The result was immediate.
WAL dropped from roughly 30 GB to 0.5 GB.
Disk usage fell from effectively 100% of 32 GiB to roughly 9% of 64 GiB.
PostgreSQL started accepting connections normally again.
That distinction mattered.
At the rate WAL had been accumulating, simply doubling the disk without fixing the slot would have bought us only about six more days.
Capacity restored service.
It did not remove the failure mechanism.
We had also been warned
One detail from the investigation was uncomfortable.
The monitoring system had already detected the problem.
An Azure alert fired when PostgreSQL storage crossed 85%, roughly a day before the outage.
But the action group was not connected to email, Teams, SMS, or our normal incident channels.
Azure knew.
Nobody responsible for the system did.
That turned what could have been an early intervention into an outage.
We already monitored disk usage, but disk usage was also a late symptom.
The more useful signal was replication-slot lag.
If CDC remains enabled in DF, the environment needs the same kind of watchdog already present in dev: monitor how much WAL the slot is retaining and alert before storage becomes the limiting factor.
We also need a heartbeat strategy so a quiet environment can continue advancing its replication position instead of relying on application traffic to do it accidentally.
The lesson
The obvious lesson is that an inactive replication slot can fill a disk.
The more useful lesson is that healthy environments can hide broken assumptions.
Dev and prod were active enough that their replication positions kept moving. That made the absence of a heartbeat look harmless.
DF exposed the flaw because it was quiet.
And our first recovery action, adding storage, made PostgreSQL healthy again without making the system correct.
This incident changed the question I ask when comparing environments.
Not just:
What's broken in the failing environment?
But also:
What activity or circumstance is allowing the healthy environments to get away with the same design?
Sometimes the unusual environment is not the one behaving incorrectly.
It is the one finally exposing the assumption.