you can find the script on Github<\/a>. I’ll walk you through the steps here.<\/p>\n\n\n\nFirst up, we’ll setup a number of variables. We’ll use those throughout the setup.<\/p>\n\n\n\n
# Setup couple of variables\n$staccname = \"nfadfkvread\"\n$staccname2 = \"nfadfkvwrite\"\n$rgname = \"kv-adf\"\n$location = \"westus2\"\n$kvname = \"kv-nf-adf-sas\"\n$keyVaultSpAppId = \"cfa8b339-82a2-471a-a3c9-0fc0be7a4093\"\n$storageAccountKey = \"key1\"\n$SASDefinitionName = \"readFromAccount1\"\n$SASDefinitionName2 = \"writeToAccount2\"<\/code><\/pre>\n\n\n\nNext up, we’ll login to Azure. Then we’ll create the actual resources. Meaning resource group, storage accounts and key vault:<\/p>\n\n\n\n
# Login\nConnect-AzAccount \n\n# Create all resources\nWrite-Output \"Create all resources\"\n\nNew-AzResourceGroup -Name $rgname -Location $location\n$stacc = New-AzStorageAccount -ResourceGroupName $rgname -Location $location -Name $staccname -SkuName Standard_LRS\n$stacc2 = New-AzStorageAccount -ResourceGroupName $rgname -Location $location -Name $staccname2 -SkuName Standard_LRS\n$kv = New-AzKeyVault -VaultName $kvname -ResourceGroupName $rgname -Location $location<\/code><\/pre>\n\n\n\nThen, we’ll do some role assignments. First, we’ll give key vault permission to rotate the keys in the storage account. Then we’ll give my user account permissions in the key vault itself (FYI: Even if you are owner of a Key Vault that doesn’t give you access to the objects in the vault. The control (Azure API) and data plane (Key Vault itself) are configured independently).<\/em><\/p>\n\n\n\n# Give KV permissions on Storage to rotate keys\nWrite-Output \"Give KV permissions on Storage to rotate keys\"\n\nNew-AzRoleAssignment -ApplicationId $keyVaultSpAppId -RoleDefinitionName 'Storage Account Key Operator Service Role' -Scope $stacc.Id\nNew-AzRoleAssignment -ApplicationId $keyVaultSpAppId -RoleDefinitionName 'Storage Account Key Operator Service Role' -Scope $stacc2.Id\n\n# Give my user access to KV storage permissions\nWrite-Output \"Give my user access to KV storage permissions\"\n\n$userId = (Get-AzContext).Account.Id\nSet-AzKeyVaultAccessPolicy -VaultName $kvname -UserPrincipalName $userId -PermissionsToStorage get, list, delete, set, update, regeneratekey, getsas, listsas, deletesas, setsas, recover, backup, restore, purge<\/code><\/pre>\n\n\n\nWhen that is done, we’ll need to wait a couple of seconds for the role assignments to propagate fully in Azure. The role assignment that is the most critical here is the permission of key vault over the storage accounts. I have a 30 second sleep in the script itself. After that sleep, we can add the storage accounts to key vault.<\/p>\n\n\n\n
# Add storage accounts to key vault\n$regenPeriod = [System.Timespan]::FromDays(2)\nWrite-Output \"Sleeping 30 seconds to have role assignments propagate and catch up\"\nStart-Sleep -Seconds 30\nWrite-Output \"Done sleeping. Add storage accounts to key vault\"\n\nAdd-AzKeyVaultManagedStorageAccount -VaultName $kvname -AccountName $staccname -AccountResourceId $stacc.Id -ActiveKeyName $storageAccountKey -RegenerationPeriod $regenPeriod\nAdd-AzKeyVaultManagedStorageAccount -VaultName $kvname -AccountName $staccname2 -AccountResourceId $stacc2.Id -ActiveKeyName $storageAccountKey -RegenerationPeriod $regenPeriod<\/code><\/pre>\n\n\n\nWith that done, we can onboard the first storage account. What we need to do here is configure a SAS definition in Key Vault. For this first storage account, we’ll configure very fine permissions: only read and list allowed for the blob service. This will protect the account in case the SAS token would potentially leak.<\/p>\n\n\n\n
# Onboard first account with list\/read permissions only\nWrite-Output \"Onboard first account with list\/read permissions only\"\n\n$storageContext = New-AzStorageContext -StorageAccountName $staccname -Protocol Https -StorageAccountKey Key1 \n$start = [System.DateTime]::Now.AddDays(-1)\n$end = [System.DateTime]::Now.AddMonths(1)\n\n$sasToken = New-AzStorageAccountSasToken -Service blob -ResourceType Container,Object -Permission \"rl\" -Protocol HttpsOnly -StartTime $start -ExpiryTime $end -Context $storageContext\n\nSet-AzKeyVaultManagedStorageSasDefinition -AccountName $staccname -VaultName $kvname `\n-Name $SASDefinitionName -TemplateUri $sasToken -SasType 'account' -ValidityPeriod ([System.Timespan]::FromDays(1))\n<\/code><\/pre>\n\n\n\nThen we do the same for the second storage account. For the second one, we will configure a SAS policy that will only allow write and list operations. (I’m going to say it here already, having no read access on the destination account will not allow ADF to do integrity validation of the data that is written. This can be fine if you don’t want integrity validation, but if you want this, you’ll also want to add read permissions<\/em>. Later on in the demo you’ll see my first pipeline run fail because I turn on integrity validation, but don’t have read permissions.<\/em>)<\/p>\n\n\n\n# Onboard second account with write\/list permissions only\nWrite-Output \"Onboard second account with list\/read permissions only\"\n\n$storageContext = New-AzStorageContext -StorageAccountName $staccname2 -Protocol Https -StorageAccountKey Key1 \n$start = [System.DateTime]::Now.AddDays(-1)\n$end = [System.DateTime]::Now.AddMonths(1)\n\n$sasToken = New-AzStorageAccountSasToken -Service blob -ResourceType Container,Object -Permission \"wl\" -Protocol HttpsOnly -StartTime $start -ExpiryTime $end -Context $storageContext\n\nSet-AzKeyVaultManagedStorageSasDefinition -AccountName $staccname2 -VaultName $kvname `\n-Name $SASDefinitionName2 -TemplateUri $sasToken -SasType 'account' -ValidityPeriod ([System.Timespan]::FromDays(1))<\/code><\/pre>\n\n\n\nFinally, we can check that the secrets work correctly by getting a SAS token for each account. The name of the secrets is important here. The secrets don’t show up in the Azure portal. The secrets have the naming pattern storageAccountName-SASDefinitionName<\/code>.<\/p>\n\n\n\n# Getting secrets to verify everything works\nWrite-Host \"Getting secrets to verify things work.\"\n\n$secret = Get-AzKeyVaultSecret -VaultName $kvname -Name \"$staccname-$SASDefinitionName\"\n$secret.SecretValueText\n$secret = Get-AzKeyVaultSecret -VaultName $kvname -Name \"$staccname2-$SASDefinitionName2\"\n$secret.SecretValueText<\/code><\/pre>\n\n\n\nAnd this concludes setting up the storage accounts and key vaults. We can now use both in Azure Data Factory.<\/p>\n\n\n\n
Setting up the Azure Data Factory<\/h2>\n\n\n\n Since the ADF piece is the interesting piece I wanted to dive into, I’ll do this work via the portal. To start, we’ll create the actual data factory. Look for Azure Data Factory in either the Azure search bar or in the marketplace. This will open the creation wizard for a new ADF:<\/p>\n\n\n\nI just filled in the basics, and skipped the git integration for now. When the ADF is created, the first thing we’ll do is give this ADF permissions to Key Vault. We could do this later on in the pipeline creation wizard, but I’d like to show this manually here. What we need here is the managed identity object ID, and then give that permissions in key vault. To start, get this object ID from the properties:<\/p>\n\n\n\nThen open your key vault, and add an access policy. Look for the managed identity of your ADF by using the object ID, and give it secret list and get permissions.<\/p>\n\n\n\nImportant gotcha now, once you add the access policy, you can’t forget to actually save the access policy. Hit the save button before you move forward.<\/p>\n\n\n\nWith that setup, we can open up the ADF editor. Go back to your ADF, and hit the Author and Monitor button:<\/p>\n\n\n\nIn this window, we’ll start off by adding the linked services we need. We need our key vault and both storage account. To add a linked service, start by clicking the manage button, go to linked services, hit the add button and look for Azure Key Vault:<\/p>\n\n\n\nThen look for the Azure Key Vault we created earlier. As you can see, we could provide permissions to key vault for the identity of ADF here as well, but I wanted to actually show this manually so you know where to look for this.<\/p>\n\n\n\nNext up, add another linked service and look for Azure blob storage. In there, provide the following details:<\/p>\n\n\n\n
Authentication method: SAS URI<\/li> Select SAS URI<\/li> Provide the URL of the container in blob you want to monitor<\/li> Select Key Vault<\/li> Select the key vault we configured before<\/li> Provide the secret name as storageAccountName-SASDefinitionName<\/code>. <\/li><\/ul>\n\n\n\nYou can test this connection, and it should work.<\/p>\n\n\n\nDo the same for the second account, changing the account name and the secret name.<\/p>\n\n\n\nThat’s the setup of the data factory. Next step is to actually build the copy activity:<\/p>\n\n\n\n
Building the copy activity<\/h2>\n\n\n\n To start building the copy activity, select the copy data wizard in the ADF wizard getting started page.<\/p>\n\n\n\nThis is a guided wizard that will walk us through the copy activity. It’s pretty straightforward and well explained. Let me walk you through it:<\/p>\n\n\n\n
Step one is to provide the metadata of the copy activity. I configured mine to run once every 15 minutes.<\/p>\n\n\n\nNext, you’ll select the source. Select the Read blob connection we created earlier:<\/p>\n\n\n\nThen, we’ll provide additional details for the read blob connection. I configured the file loading behavior to be based on the LastModifiedDate<\/code> and to do an actual binary copy without compression.<\/p>\n\n\n\nNext up, we’ll select the write blob connection we configured earlier as the destination of this copy activity.<\/p>\n\n\n\nThis also asks you for additional configuration information. In my case, I only provided the target container to store the data into.<\/p>\n\n\n\nThen we’ll provide additional settings for the copy activity. I configured additional data consistency verification and provided a container in my writing storage account to write logs to. (If you followed along earlier in the blob, you’ll remember that the data consistency verification will actually cause my pipeline to fail because I don’t have read permissions in the target storage account. We’ll change this later on).<\/em><\/p>\n\n\n\nFinally, we’ll get a summary (no screenshot) and we can deploy the copy operation. This will create a pipeline in ADF.<\/p>\n\n\n\nTo have a file to copy, I uploaded a file to blob storage that will be picked up by ADF in 15 minutes when it runs. <\/p>\n\n\n\nAfter waiting a couple minutes, the ADF was triggered, and I could see the file appear in my destination store. (which is good) <\/p>\n\n\n\nHowever, checking the ADF logs showed an error:<\/p>\n\n\n\nThis shows that ADF couldn’t retrieve the sink file, meaning the file in the target store. Let’s have a look at solving the error:<\/p>\n\n\n\n
Solving the data consistency error (can’t read from sink)<\/h2>\n\n\n\n There are two ways to solve this error:<\/p>\n\n\n\n
Add read permissions to the SAS token.<\/li> Disable consistency checks.<\/li><\/ul>\n\n\n\nIn my case, I actually went ahead and added read permissions to the SAS token. I did this using PowerShell (this is part of the comments below in the GitHub script btw.)<\/p>\n\n\n\n
# Change SAS token to include read for second account\nRemove-AzKeyVaultManagedStorageSasDefinition -AccountName $staccname2 -VaultName $kvname `\n-Name $SASDefinitionName2\n\n\n$storageContext = New-AzStorageContext -StorageAccountName $staccname2 -Protocol Https -StorageAccountKey Key1 \n$start = [System.DateTime]::Now.AddDays(-1)\n$end = [System.DateTime]::Now.AddMonths(1)\n\n$sasToken = New-AzStorageAccountSasToken -Service blob -ResourceType Container,Object -Permission \"wlr\" -Protocol HttpsOnly -StartTime $start -ExpiryTime $end -Context $storageContext\n\n# Need to give this a new name due to default key-vault soft delete behavior\n$newSasName = $SASDefinitionName2 + \"bis\"\nSet-AzKeyVaultManagedStorageSasDefinition -AccountName $staccname2 -VaultName $kvname `\n-Name $newSasName -TemplateUri $sasToken -SasType 'account' -ValidityPeriod ([System.Timespan]::FromDays(1))\n\n$secret = Get-AzKeyVaultSecret -VaultName $kvname -Name \"$staccname2-$newSasName\"\n$secret.SecretValueText<\/code><\/pre>\n\n\n\nAs you can see in the script above, I actually needed to change the name of the SASDefinition. I believe this is due to default Key Vault soft delete behavior. Because I was a little lazy, I just gave it a new name. (in all honesty, I wasn’t 100% lazy. I tried hard deleted the soft deleted SAS definition, but it appears Az PowerShell doesn’t support this yet. I plan to open a GitHub once this blog is posted).<\/em><\/p>\n\n\n\nWith the new secret created, we need to change the storage account 2 definition. We’ll do that in the linked services. There we’ll provide the updated secret name:<\/p>\n\n\n\nAnd with that out of the way, let’s test our pipeline again. I uploaded a second file to our read container:<\/p>\n\n\n\nThis time, I don’t want to wait 15 minutes for the pipeline to trigger. In stead, I’ll head over to the designer, select our pipeline and hit the debug button:<\/p>\n\n\n\nAnd this time, the copy job succeeded succesfully:<\/p>\n\n\n\nSummary<\/h2>\n\n\n\n In this post we explored how we can use SAS tokens provided by Key Vault to move data between storage accounts. We configured Key Vault to manage 2 storage accounts, and configured fine grained control in the SAS token on which permissions were allowed on which account. We hit a small issue with the data consistency check, but were able to solve this by editing the write SAS policy.<\/p>\n","protected":false},"excerpt":{"rendered":"
I am working with a customer right now that is doing a lot of work with Azure Data Factory (ADF). ADF is a powerful cloud based data integration tool that lets you move data from a multitude of source, process that data and store it in a target data store. You can think of it […]<\/p>\n","protected":false},"author":1,"featured_media":1280,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[2],"tags":[8,148,75,74,149,67],"class_list":["post-1261","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","tag-azure","tag-azure-data-factory","tag-blob-storage","tag-data-engineering","tag-etl","tag-storage"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2020\/09\/image-16.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1261","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=1261"}],"version-history":[{"count":3,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1261\/revisions"}],"predecessor-version":[{"id":1290,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1261\/revisions\/1290"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/1280"}],"wp:attachment":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=1261"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=1261"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=1261"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"\/wp-content\/uploads\/2020\/09\/ADF-SAS.png\")
![\"\"](\"\/wp-content\/uploads\/2020\/09\/image.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-1-1024x748.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-2.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-3.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-4-1024x371.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-5-1024x473.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-6.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-7.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-8.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-9-1024x685.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-10.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-11.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-12.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-13.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-14-1024x426.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-15.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-16-1024x435.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-17.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-18.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-19.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-20.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-21.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-22-1024x548.png\")
<\/figure>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/09\/image-23.png\")
<\/figure>\n\n\n\n