CI Is Not a Monitoring System: How We Removed 984 Azure DevOps Jobs a Day

At some point, our monitoring system had quietly become one of the largest consumers of our CI infrastructure.

Four scheduled Azure DevOps pipelines were launching 1,416 agent jobs every day.

The API monitoring pipeline alone started nine parallel jobs every 15 minutes.

That worked out to:

9 jobs x 4 runs/hour x 24 hours = 864 jobs every day.

Just to monitor APIs.

The checks themselves were useful.

The architecture was not.

Monitoring was competing with delivery

Azure DevOps hosted agents are useful because they give us isolated compute for builds, tests, migrations and releases.

We were also using them to answer questions like:

• Is this HTTP endpoint responding?
• Is the TLS certificate valid?
• Is the frontend reachable?
• Is this Function alive?

Every one of those checks required a scheduled pipeline run, agent allocation, checkout and job startup.

That meant monitoring workloads were competing with actual deployments for the same agent capacity.

The problem became especially visible when scheduled watchdog jobs overlapped with releases.

The question was not whether to remove monitoring.

We needed to keep coverage across the API, Azure Functions, frontend, TLS, AI inference, embeddings, outbox processing, CDC and the monitoring stack itself.

The question was:

Which of these checks genuinely requires a CI agent?

For most of them, the answer was none.

Move simple probes closer to the monitoring system

We already had an observability VM running our monitoring stack.

So ordinary HTTP and TLS checks moved to Prometheus Blackbox Exporter.

Instead of this:

Azure DevOps schedule
        |
        v
Hosted agent
        |
        v
Start job
        |
        v
Run HTTP probe
        |
        v
Report result

we could do this:

Prometheus
    |
    v
Blackbox Exporter
    |
    v
Endpoint

The same endpoint still gets checked.

But Azure DevOps is no longer involved.

Those probes now consume zero hosted agents.

This also places the responsibility where it belongs. Endpoint availability is an observability concern, not a CI/CD concern.

Consolidate the checks that actually need application knowledge

Some monitors could not be replaced with simple HTTP probes.

Embedding behavior, outbox processing and passive CDC health require more context about the platform.

Instead of maintaining several independent scheduled pipelines, we consolidated those checks into a Platform Watchdog.

AI inference remained separate.

A successful HTTP response from an AI endpoint does not necessarily prove the inference path works, so we retained a focused authenticated canary that exercises the real inference flow.

That gave us three different monitoring mechanisms for three different kinds of questions:

Blackbox
    HTTP / TLS / basic reachability

Platform Watchdog
    embedding / outbox / passive CDC

AI canary
    authenticated inference path

Each check now uses the cheapest mechanism capable of proving what we actually care about.

Then there was the monitoring stack itself

Moving more checks into the observability stack introduces another question.

What happens if the observability stack goes down?

A monitoring system cannot reliably report its own disappearance.

So we added an external observability sentinel.

The idea is simple: something outside the monitoring stack periodically verifies that the stack itself remains reachable.

It is the same problem that appears repeatedly in infrastructure design.

You cannot make the component being observed your only source of truth about whether that component exists.

The result

I went back through the actual YAML schedules and job matrices rather than estimating from pipeline count.

The four retired pipelines were responsible for:

Their replacements use:

That removed 984 Azure DevOps jobs per day from the workload we redesigned.

A reduction of roughly 70%.

Looking at the entire scheduled monitoring estate, including monitors we deliberately retained such as the CDC canary, migration watchdog and Debezium log poller:

Before: 1,770 jobs/day
After:    786 jobs/day
Saved:    984 jobs/day

That is a 56% reduction in total scheduled monitoring agent usage.

And the HTTP and TLS probes that moved to Blackbox now require no Azure DevOps agents at all.

The interesting part wasn't the YAML

It would be easy to describe this as pipeline optimization.

That is not really what changed.

The important change was recognizing that we had allowed two different concerns to collapse into the same execution model.

CI asks:

Has this change been built, tested and safely delivered?

Monitoring asks:

Is the system behaving correctly right now?

Both can technically be implemented with scheduled pipelines.

That does not mean they should be.

Once we separated those responsibilities, the architecture became simpler.

Cheap continuous probes moved to the monitoring system.

Checks requiring platform context were consolidated.

Checks requiring real authenticated behavior remained as focused canaries.

And Azure DevOps agents went back to doing the work for which they are valuable.

The lesson

Infrastructure waste does not always look like an oversized VM or an expensive database.

Sometimes it is thousands of tiny executions that individually look harmless.

Nine jobs every 15 minutes did not look alarming inside one YAML file.

Across a day, it was 864 jobs.

Across the monitoring estate, we were running 1,770 scheduled jobs every day.

The biggest improvement came from asking a basic architectural question:

Why does this workload need to exist here?

In this case, much of it didn't.

CI can run a monitor.

That does not make CI a monitoring system.