How to re-run the poison queue in Azure Webjobs

We just had a case where one of our queue based webjobs stopped working. Azure webjobs try to process a message a few times before it gives up. When it gives up it moves the message to the poison queue. Sadly there is not super easy way to rerun these messages after the issue is fixed. In theory it might be possible to point the webjob to the poison queue but we find it easier to just copy the failed messages back to the original queue. We do this from time to time so we built a small C# application and now we had the chance to update it to the latest Azure SDKs.

Code snippet

While we use this specifically for moving messages in the poison queue to the normal queue this can be used to transfer messages between any queues.

using Azure.Storage.Queues;
using System;
using System.Threading;
using System.Threading.Tasks;

namespace AzureQueueTransfer
{
    internal class Program
    {
        // Need Read, Update & Process (full url, can create in storage explorer)
        private const string sourceQueueSAS = ""; 

        // Need Add (full url, can create in storage explorer)
        private const string targetQueueSAS = "";
        private static async Task Main(string[] args)
        {
            var sourceQueue = new QueueClient(new Uri(sourceQueueSAS));
            var targetQueue = new QueueClient(new Uri(targetQueueSAS));

            var queuedAny = true;
            while (queuedAny)
            {
                Thread.Sleep(30000); // Sleep to make sure we dont build too much backlog so we can process new messages on higher prio than old ones
                queuedAny = false;
                foreach (var message in sourceQueue.ReceiveMessages(maxMessages: 32).Value)
                {
                    queuedAny = true;
                    var res = await targetQueue.SendMessageAsync(message.Body);

                    Console.WriteLine($"Transfered: {message.MessageId}");
                    await sourceQueue.DeleteMessageAsync(message.MessageId, message.PopReceipt);
                }

                Console.WriteLine($"Finished batch");
            } 
        }
    }
}

If you are using PackageReference you can install the dependency with <PackageReference Include="Azure.Storage.Queues" Version="12.6.0" />. If there is one thing I would improve for the future it would be to make sure that instead of doing that Thread.Sleep for a fixed time I would check that target queue is almost empty. Because in our cases it's more important that new messages get in quickly that the ones that our users have already been waiting on for a while.

How to get the SAS-tokens

SAS-tokens are great for this kind of scripts. We create short lived ones with just the permissions we need, on the resource we need. That lowers the risk that any bug cause any major issues. Specially since scripts in general is a bit less tested than application code.

To get these we use storage explorer. Right-click the queue you want to generate the token for and pick "Get Shared Access Signature". 

Example of Azure SAS-token creation

This is an example of the "target queue" in the code snippet. By only having Add permissions we can't accidentally overwrite or delete any other data on that queue. Having a very short time limit on that queue is also good in case we accidentally commit the code with the token or something similar.