We had accumulated a large number of scheduled Azure DevOps jobs each day.
Not to build the application.
Not to deploy it.
Mostly to monitor it.
I checked the pipeline schedules and job matrices to understand how the total had grown. One availability check alone was launching multiple agent jobs on a short interval.
The checks were useful.
Where we were running them was the problem.
How we got there
Using a pipeline for monitoring is easy.
Write a script.
Add a schedule.
Run it on an Azure DevOps agent.
Done.
Need another check?
Add another job.
Another service?
Add another pipeline.
Each decision makes sense on its own.
Then one day you realize your monitoring system is launching more than a thousand CI jobs every day.
And those jobs are competing with builds and releases for agents.
That was where we were.
What actually needs an agent?
This became the useful question:
Does this check genuinely need an Azure DevOps agent?
Take something simple:
Is this endpoint returning the expected status?
We do not need a CI agent for that.
Same for TLS expiry.
Same for basic frontend availability.
Same for a lot of the HTTP checks we had accumulated.
We already ran Prometheus and Blackbox Exporter as part of our observability stack.
So those checks moved there.
Instead of:
Schedule
|
v
Azure DevOps agent
|
v
Start job
|
v
Run HTTP request
|
v
Exitwe now have:
Prometheus
|
v
Blackbox
|
v
EndpointNo hosted agent involved.
Not every monitor is a ping
Some checks actually do need more context.
We still needed to know things like:
- are background jobs making progress?
- is a message consumer moving?
- does an authenticated user path still work?
Those aren't all equivalent to GET /health.
So we consolidated the checks that needed application context into a focused watchdog.
We kept a separate end-to-end check for a user path that a healthy HTTP endpoint could not prove.
And we moved the cheap availability checks to Blackbox.
Different checks now use different mechanisms based on what they are trying to prove.
That sounds obvious when written down.
It wasn't how the system had evolved.
Who monitors the monitoring system?
Moving more responsibility into the observability stack created another problem.
What happens if the observability stack itself disappears?
If Prometheus is down, Prometheus cannot exactly alert us that Prometheus is down.
So we added an external sentinel whose job is to check the monitoring infrastructure from outside it.
Small thing.
But I like the principle behind it.
A system should not be the only authority on whether it is alive.
The numbers
Across the checks we changed, consolidation reduced scheduled agent jobs by roughly 70%. Across all scheduled monitoring, including the checks we kept, agent usage fell by more than half.
And all the checks moved to Blackbox now use zero Azure DevOps agents.
The interesting part wasn't optimizing YAML
I initially thought about this as a pipeline optimization.
I don't anymore.
The bigger problem was that two separate concerns had slowly collapsed into the same tool.
CI/CD answers questions like:
Can we safely build, test and ship this change?
Monitoring answers:
Is the system behaving correctly right now?
Azure DevOps can technically do both.
That does not mean it should.
Once I started looking at the system that way, the redesign became fairly straightforward.
Simple availability checks belong with observability.
Checks that need application context belong in a focused watchdog.
Real user paths sometimes deserve proper canaries.
CI agents should be left for work that actually needs CI agents.
The biggest optimization wasn't making the existing pipelines faster.
It was realizing that a lot of those jobs should never have been pipelines in the first place.