Making a Azure poison queue Slack notifier

I'm currently working at a place were we are using queue triggered Webjobs to handle the sending of messages like email and SMS (using Send Grid and Twilio). Using a queue based system for this is great because it allows us to replay any queue messages, should one of the 3rd party's (or our code) fail to send the message.

Since we are connecting into 3rd party's you can almost guarantee there's going to be some form of failure. So its always good practice to leverage on this type of architecture to handle the unknown. We have the following setup:

Website > Storage Queue > Web Job > Send Grid
Website > Storage Queue > Web Job > Twillio

When a failure occurs, queue messages are automatically moved from the
message queue into a poison queue, these queues are always suffixed with "poison" (MS really wanted to highlight how toxic your problems are) like so:

email - For normal operation
email-poison - Messages moved here when a failure occurs
sms
sms-poison

Gaining visibility of what's in a poison queue is really important in knowing
the health of your system. So I embarked upon a task in seeking out an
alert setting buried deep somewhere in the Azure portal to help surface
any messages going into the poison queue. I knew this would be a metric
alert of some kind either in the 'Storage Account', 'Alerts' or perhaps
even 'Application Insights' blade.

After having spent a while searching for it as well as posting this Stack Overflow question (it wasn't a popular one..), I started doubting whether it even existed!

I even tried the search box at the top of the azure dashboard as a last ditch effort, hoping it will provide answers. You think this would
exists somewhere (if it does and my eyes have deceived me please do get
in touch) or at the very least be visible and easily findable? Alas this was not the case..

So I decided to do something about it,

why not have an Azure function that takes a storage account and looks through all the queues to check if any poison queue messages exist.

Whilst were at it we could also check if messages are stacking up in the non-poison queues (just in-case a Webjob has been turned off or cant process a certain message), and even provide the content of a problematic queue message. Since our team uses slack for communication I decided to send the notification to Slack. Below are the steps I took:

#Step 1 - Setting up slack

Setting up slack is quick and easy, just create a 'poison-queue' channel, and create a new integration in the custom integrations section (Note your gonna have to get admin access to do this (I have provided a link at the bottom of this article as its nested deep in their UI). An integration is essentially a web hook endpoint for us to post JSON data to (I have added a link for Slacks JSON format below too, as well as a message builder to help customise the look and feel).

The picture below show where you can get your web hook URL from.

Azure Queue Notifier - Slack Integration

#Step 2 - Create your Azure Function

Since this is not a tutorial on Azure Functions, I'm going to skip going into detail here. Microsoft however have provided some great documentation on this (with pictures!) to help you out. Links are at the end until MS break them. By the way your gonna need a cron expression to define the timeframe for this function to work in, if you hate cron as much as I do worry not! Use my cron expression for a daily sobering alert at 9:00 - 0 0 9 * * *

#Step 3 - Create your slack message structure

Next we can create the basic structure needed for our Slack message, expressed as a C# class. My class is actually quite simple and missing quite a few properties, to get a sense of all the customisations Slack offers have a look at the links below.

1#r "Newtonsoft.Json"
2
3#load "Attachments.csx"
4
5using Newtonsoft.Json;
6using System.Collections.Generic;
7
8public sealed class SlackMessage
9{
10    public SlackMessage()
11    {
12        Attachments = new List<Attachments>();
13    }
14
15    [JsonProperty("channel")]
16    public string Channel { get; set; }
17
18    [JsonProperty("username")]
19    public string UserName { get; set; }
20
21    [JsonProperty("text")]
22    public string Text { get; set; }
23
24    [JsonProperty("attachments")]
25    public List<Attachments> Attachments { get; set; }
26
27    [JsonProperty("icon_emoji")]
28    public string Icon
29    {
30        get { return ":computer:"; }
31    }
32}
33
34#r "Newtonsoft.Json"
35
36using Newtonsoft.Json;
37
38public class Attachments
39{
40    [JsonProperty("color")]
41    public string Colour { get; set; }
42
43    [JsonProperty("title")]
44    public string Title { get; set; }
45
46    [JsonProperty("text")]
47    public string Text { get; set; }
48}

Remember to create these classes as .csx files for the Azure function to understand them.

#Step 4 - Create a slack client to post the message

Now that we have our message structure we can create a class to serialize and post the JSON to Slack using the Webhook created in Step 1, below is the code to do this,

1#r "Newtonsoft.Json"
2#r "System.Web.Extensions"
3#r "System.Web"
4
5#load "SlackMessage.csx"
6#load "Attachments.csx"
7
8using System.Net;
9using Newtonsoft.Json;
10using System.Collections.Specialized;
11
12public class SlackClient
13{
14    public static readonly string WebHook = @"https://hooks.slack.com/services/XXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXX";
15
16    public void SendMessage(SlackMessage message)
17    {
18        string payloadJson = JsonConvert.SerializeObject(message);
19        
20        using (WebClient client = new WebClient())
21        {
22            NameValueCollection data = new NameValueCollection();
23            data["payload"] = payloadJson;
24            client.UploadValues(WebHook, "POST", data);
25        }
26    }
27}

Its good practice to move the Webhook URL into the settings file, for simplicity I have included it into this class.

#Step 5 - Queue Checker

Next we need to add code to loop through any connections string we pass it, check all the queues and send messages if we think there's something wrong.

1#r "Microsoft.WindowsAzure.Storage"
2
3#load "SlackClient.csx"
4#load "SlackMessage.csx"
5#load "Attachments.csx"
6
7using System.Collections.Generic;
8using System.Linq;
9using Microsoft.WindowsAzure.Storage;
10using Microsoft.WindowsAzure.Storage.Auth;
11
12public class PoisonQueueChecker
13{
14    public void CheckPoisonQueues(Dictionary<string, string> storageConnectionStrings)
15    {
16        var slackClient = new SlackClient();
17        var slackMessage = new SlackMessage { Text = "Poison Queue Alerts", Channel = "poison-queue" };
18
19        foreach (var storageConnectionString in storageConnectionStrings)
20        {
21            var storageCredentials = new StorageCredentials(storageConnectionString.Key, storageConnectionString.Value);
22            var storageAccount = new CloudStorageAccount(storageCredentials, true);
23            var queueClient = storageAccount.CreateCloudQueueClient();
24
25            var queues = queueClient.ListQueues();
26            foreach (var queue in queues)
27            {
28                queue.FetchAttributes();
29                //Gets the total messages in the queue
30                var queueCount = queue.ApproximateMessageCount;
31
32                if (queueCount > 0)
33                {
34                    var isPoisonQueue = queue.Name.EndsWith("poison");
35                    var attachment = new Attachments();
36                    attachment.Title = $"Queue: {queue.Name}, Message Count: {queueCount}";
37                    attachment.Colour = isPoisonQueue ? "danger" : "warning";
38
39                    //Note the peek function will not dequeue the message
40                    var message = queue.PeekMessage();
41                    attachment.Text = $@"Insertion Time: {message.InsertionTime}, Sample Contents:\n" +
42                                        $" {message.AsString}";                        
43
44                    slackMessage.Attachments.Add(attachment);
45                }
46            }
47
48            //Add a message showing all is well
49            if (!slackMessage.Attachments.Any())
50            {
51                slackMessage.Attachments.Add(new Attachments { Title = "All queues are operational and empty", Colour = "good" });
52            }
53        }
54
55        slackClient.SendMessage(slackMessage);
56    }
57}

#Step 6 - Being it all together

Final step is to hook up the functions run method like so:

1#load "PoisonQueueChecker.csx"
2
3using System;
4using System.Collections.Generic;
5
6public static void Run(TimerInfo myTimer, TraceWriter log)
7{
8    log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
9
10    var storageConnectionStrings = new Dictionary();
11    storageConnectionStrings.Add("storagename", "storagekey");
12
13    var poisonQueueChecker = new PoisonQueueChecker();
14    poisonQueueChecker.CheckPoisonQueues(storageConnectionStrings);
15}

And that's it, 9 O'clock tomorrow you can finally start gaining visibility of those poison queues and start worrying about those dodgy lines of code causing your messages to be poisoned.

#Helpful Links

Custom Integrations

https://<>.slack.com/apps/manage/custom-integrations

Customising your slack message

https://api.slack.com/docs/messages/builder

How to send a slack message to your web hook:

https://api.slack.com/custom-integrations/incoming-webhooks

How to create a azure function:

https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-azure-function

How to code up a azure function:

https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-csharp

Share this article