Adding alerts to Azure Storage metrics

For every system you use, monitoring and alerting is critical. This holds true for Azure storage as well.

In this quick post I’ll explain how you can setup alert rules for hitting TPS limits.

Setting up the alert

To set up the alerts, navigate to your storage account, look for Alerts, and hit the “New alert rule” button.

Next, you need to define the rules you want to follow for this alert. The first question you get asked if which resource you want to monitor. This could be a single storage account, multiple accounts in a resource group, or all accounts in a subscription. In my case, I’ll create an alert for this specific storage account.

Next, you’ll need to select a condition to match. You can select diffent Signal types, which in our case will be Metrics. Then you can select transactions.

Next, you will see an overview of the current transactions, and you’ll get the opportunity to specify specific dimensions to split the metric.

Then, you’ll have to define which threshold to hit. Azure supports two thresholds here, either they are static (i.e. a metric you provide, e.g. close to the max of the storage account) or dynamic (showing you anomalies in your usage). In my case, I’ll configure a static alert at 1000 transactions. Note as well that you can define the aggregation granularity (over how much transactions will be counted) and how frequent this will be checked.

After that, you need to create an action group. Actions can be multiple things, ranging from sending an e-mail, to calling a webhook to triggering an automation runbook to run. In my case, I’ll send an e-mail to myself. If you don’t have action groups precreated, you can create them for here by hitting the Create button in stead of the Add button. You can even configure multiple actions as part of a single action group.

Finally, we’ll give our rule a name and enable it.

It can take up to 10 minutes for the alert to become active, and then it’ll take about 5 minutes for the aggregation window to catch up. Let’s wait a little and then trigger our alert.

Triggering the alert

To trigger my alert based on total amount of transactions, I’ll start a azcopy bench job. Azcopy bench is a benchmarking tool to verify bandwidth to azure storage. To do this, I’ll start up a Linux VM, and download azcopy on there with the following command:

wget https://aka.ms/downloadazcopy-v10-linux
tar -xvzf downloadazcopy-v10-linux
cd azcopy_linux_amd64_10.3.4

While in that folder, we can setup our benchmark tool. First, we’ll need a SAS token to connect to our storage account. Head over to the storage blade and look for SAS. Create a new SAS token that at least can write to blob. In my case, I gave full permissions to blob.

Once the token is generated, copy it somewhere safe. Next, we’ll create a new container for our benchmark.

Once that is done, we can format our bench command:

azcopy bench "https://<account>.blob.core.windows.net/<container>?<SAS>" --file-count 100 --size-per-file 2G

With that done, let’s run the benchmark and see if the alert rule is triggered.

And for sure – after waiting a couple minutes – I got an alert in my inbox, explaining we hit the 1000 transactions limit:

Automating this setup

Doing this setup using the portal is easy for a single storage account, but requires some automation to set this up across all your storage accounts. The following Powershell snipper will create a rule that does the same we just created in the portal.

$action = Get-AzActionGroup -Name "email_nills" -ResourceGroupName "Default-ActivityLogAlerts"
$act = New-AzActionGroup -ActionGroupId $action.Id
$crit = New-AzMetricAlertRuleV2Criteria -MetricName transactions -TimeAggregation Total -Operator GreaterThan -Threshold 1000

$staccs = Get-AzStorageAccount
foreach ($stacc in $staccs){
    $alertrulename = $stacc.StorageAccountName + " exceeded transactions" 
    Add-AzMetricAlertRuleV2 -ResourceGroupName $stacc.ResourceGroupName -Name $alertrulename -WindowSize 00:05:00 -ActionGroup $act `
    -TargetResourceId $stacc.Id -Frequency 00:05:00 -Condition $crit -severity 2
}

Summary

In this post we quickly evaluated how we can setup alerts based on the amount of transactions in a storage account. This will trigger automatically and do the required actions.