I wrote an Azure Durable function which makes a phone call to out of hours support engineers when an alert is raised within their production Azure environment, and I wanted to talk about how I did it and what I used.

When an alert is raised with the customers Azure environment I send an HTTP Post to my Azure durable function endpoint from the reporting tool we use, which is PRTG, you can do the same from Azure just as easily, we use PRTG to monitor Azure resources for things like High CPU and the amount of free disk space remaining, etc.

Durable functions was chosen so that I can make use of what’s called an orchestration durable function – you can read more about durable functions: – https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp

If you read the above articles you’ll get a good grasp of what an orchestrator in durable functions can do, to convey why I used them I have the following workflow requirements:-

  1. Receive details of the alert.
  2. Retrieve the support people’s phone numbers.
  3. If an alert is raised call the first number 3 times in 5 minutes, if answered by a human, read out the alert message and some extra content and ask the user to acknowledge the issue by pressing 1 on the keypad.
  4. If the support engineer doesn’t answer after 3 attempts then move onto the next number.
  5. If the support engineer answers and presses 1 stop the orchestration.

1 – Receive details of the alert
This is really easy to do, here I have a template set up in PRTG which forwards the details of the alert to my durable function like so:-

2 – Retrieve the support people’s phone numbers
I am storing the support people’s phone numbers in a CSV file which is uploaded to a simple Azure storage account, this allows the customer to edit the support numbers easily.

3 – Making the call
Here I make use of Twilio Rest API and I create a CallResource object and then call the Create method, Twilio has a thing called Twiml which you can create a message of your own out of and it will read this message out to the person who picks up the phone call. All of the details about who you call, what the call says, and the action they need to take are stored in config so it can be very easily changed for different customers.

The code to make a call is actually really simple.

var call = CallResource.Create(twiml: 
new Twilio.Types.Twiml($"<Response><Gather action='{callbackhandlerURL}'>{messageToReadToUser}"),
to: to,
from: from);

4 / 5 – Answering the call
This was the tricky part, figuring out if they had picked up the call was my initial challenge and I tried numerous things from the Twilio docs which were misleading, didn’t seem to work as I expected. The samples for this part of the documentation are sadly lacking.
Now when the call comes in the support engineer is asked to press 1 to acknowledge they have received the call and the orchestration can end, part of this involves having a callback URL so that Twilio can send you details back to a URL of your choice so that you can get the details of the call, things like call length, etc and if they pressed 1 during the call.

Orchestration
The orchestration part was pretty tricky for me to get right, huge thanks to @marcduiker he was an enormous help to me on this, figuring out how to do some of the steps proved tricky but very interesting!

Marc is putting together an Azure Functions University series where you can go and learn all about Azure Functions – please go check that out.

The orchestration logic was something like the following:-

MainOrchestrator – this function’s job is to be the orchestrator, within this function we call sub orchestrators, and also activity functions, think of an activity function as a separate function that does something, I had a GetNumbersFromStorage activity function and a SendNotification activity function. so the idea behind durable functions is to be able to call multiple azure functions using patterns, one of which is the orchestrator pattern.

RetryOrchestrator – this function’s job is to work out what to do when the call wasn’t answered the first time, do we need to make another call, how many times have we called this number, and have we ensured that the calls are spread out of 5 minutes so we don’t make multiple calls at the same time.

Twilio
To make this all work I created a Twilio account and purchased a number, this means you can use this number to make the calls. It costs 2 pence per call and 7 pence per call if you want to detect if someone answered the call using answering machine detection, so there are options available.

Summary
Durable functions have a lot of great use cases, definetly check them out and build something yourself to get a handle on how they work. The Azure durable function docs are really good.