Zero functions, two different causes

The production Function App was Running.

It had no functions.

That distinction mattered because a Service Bus queue had stopped moving.

Compliance rules that should have been sent to an embedding service were piling up. Active rules were missing stored vectors, so semantic search was falling back to text-only results.

We had an alert for embedding coverage.

We also had a green status beside the Function App.

Only one of those signals described whether the work was actually happening.

Running is a resource state, not a workload state

We followed the stalled messages from the queue to the Functions host, then from the host to its container image.

The app was configured to run an image that no longer existed in the registry.

A cleanup step in our deployment pipeline retained a small number of recent images. What it did not check was whether an older image was still deployed to an environment.

That gave us a strange combination:

Azure resource       Running
Configured image     Missing
Discovered functions 0
Queue                 Growing

The Azure resource existed and was started.

The workload inside it did not.

This time, zero functions really did mean zero functions.

The cleanup job had more authority than context

Keeping only recent container images sounds sensible.

But recency and relevance are not the same thing.

An image can be old and still be the image production is running. Deleting it turns a storage policy into a deployment failure, even though the cleanup job has no idea it is changing production.

We changed retention so an image is protected when it is:

• referenced by a deployed environment
• the current deployment candidate
• pointed to by the latest tag
• inside the retention window

We also moved cleanup until after production deployment verification.

The order matters. A pipeline should prove that the replacement is healthy before removing anything that may still be serving traffic.

Recovery had to produce runtime evidence

Deploying a replacement image was only the beginning of the recovery.

We wanted evidence from the layer doing the work.

Startup logs showed the Python worker initializing. The host indexed the function. The Service Bus trigger started. The host acquired its lock.

Then the queue began to drain.

Those signals formed a useful chain:

Image deployed
      |
      v
Worker initialized
      |
      v
Function indexed
      |
      v
Trigger started
      |
      v
Messages consumed

The consumer was back.

The deployment pipeline still said the app had zero functions.

The same answer was now wrong

At first, the pipeline's zero matched reality.

After recovery, the same value came from Azure's management-plane function listing, which had not caught up with the running host.

Nothing about the number itself revealed that its meaning had changed.

Before recovery              After recovery

Management plane: 0          Management plane: 0
Runtime host:      0          Runtime host:      1
Queue:             growing    Queue:             draining

If we had treated the management response as conclusive, we would have marked a healthy recovery as a failed deployment.

So we asked the Functions host directly.

Its runtime view showed the loaded function while the control plane still showed none.

The important question was no longer:

How many functions does Azure say exist?

It was:

Which layer answered, and is that layer authoritative for what we are trying to prove?

Verification should follow the work

We updated deployment verification to use the runtime host's view of loaded functions, with management-plane discovery retained as a fallback.

But function discovery alone still does not prove the whole path.

The checks now cover:

• the image selected by the deployed app
• the queue and trigger configuration
• the state of the Function App
• the functions loaded by the runtime host
• the health of the consumer

We also added tests for stale listings, genuinely missing functions, successful deployment, and image drift.

No single signal carries the entire conclusion anymore.

The queue reached zero, but the incident did not

The active queue fell from 477 messages to zero.

Embedding coverage recovered and its alert resolved.

A separate alert caught new dead-letter activity during backlog recovery and resolved when that activity stopped.

But 144 historical dead-lettered messages remain to be investigated, including 14 added during the recovery.

A drained active queue proves that the consumer caught up.

It does not prove that every message completed successfully.

That is another version of the same lesson: a reassuring number can answer a much narrower question than the one you think you asked.

What I took away from it

Dashboards and APIs do not report truth in the abstract.

They report what one layer can see, at one moment, through one model of the system.

Before trusting the number, find out which system produced it and what question it can actually answer.