ckaczor/azuredatastudio
1
0
Fork 0
mirror of https://github.com/ckaczor/azuredatastudio.git synced 2026-04-01 17:40:30 -04:00
Code Issues Packages Projects Releases Wiki Activity

Azure SQL Hybrid Cloud Toolkit Notebooks Extension Command (#13286)

Browse Source 
* added extension folder incomplete

* WIP extension progress

* notebook finally opens in side panel

* notebook now opens via notebook extension

* html file spaces restored

* package json fixed

* fixed vscode import issue

* more cleanup

* remove git stuff

* placeholder icon logos added

* fixed gulpfile

* cleanup changes

* vscode import fixed

* fixed main and yarn.lock

* added provided notebooks view

* formatting for package.json

* removed first command as its not necessary

* fixed notebook typo

* readded spaces
This commit is contained in:
Alex Ma
2020-11-11 13:50:36 -08:00
committed by GitHub
parent b32e5f8f25
commit fa608f9f80
87 changed files with 8661 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
extensions/azurehybridtoolkit/notebooks/hybridbook/content/data-portability/VisualBootstrapperNB.PNG Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 123 KiB

559
extensions/azurehybridtoolkit/notebooks/hybridbook/content/data-portability/export-sql-server.ipynb Normal file
View File

@@ -0,0 +1,559 @@
{
"metadata": {
"kernelspec": {
"name": "powershell",
"display_name": "PowerShell"
},
"language_info": {
"name": "powershell",
"codemirror_mode": "shell",
"mimetype": "text/x-sh",
"file_extension": ".ps1"
}
},
"nbformat_minor": 2,
"nbformat": 4,
"cells": [
{
"cell_type": "markdown",
"source": [
"# Export Existing Azure SQL Server Resources\r\n",
"\r\n",
"\r\n",
"<!-- Disable bullets to be shown for checkbox markup -->\r\n",
"<style type=\"text/css\">\r\n",
" ul { list-style-type: none }\r\n",
"</style>\r\n",
"## Steps\r\n",
"Gather input:\r\n",
"* [ ] Connect to Azure Subscription\r\n",
"* [ ] Choose Resource Group (Read access required)\r\n",
"* [ ] Choose Microsoft SQL Server Resources to export\r\n",
"* [ ] Choose/Create Migration Storage \r\n",
"* [ ] Install Application + Data Portability function (orchestrator service)\r\n",
"* [ ] Install ADP Azure Batch processing pipeline\r\n",
"* [ ] Store SqlPackage.exe in Migration Storage for orchestrator to hand to Az Batch\r\n",
"\r\n",
"Execute:\r\n",
"* [ ] Check all pre-requisites\r\n",
"* [ ] Kick off orchestrator service\r\n",
"\r\n",
"Monitor:\r\n",
"* [ ] Check export status."
],
"metadata": {
"azdata_cell_guid": "a9da248a-20f1-4574-bd04-7324e70c05a3"
}
},
{
"cell_type": "markdown",
"source": [
"## Set Variables for the Notebook"
],
"metadata": {
"azdata_cell_guid": "b72d138a-566f-4161-b7a6-7264487e446c"
}
},
{
"cell_type": "code",
"source": [
"# ADP Resource \r\n",
"$Env:BOOTSTRAP_Subscription = \"\" # Azure Subscription ID/Name for the ADP Resource Group # Both RG are assumed to be in the same subscription\r\n",
"$Env:BOOTSTRAP_ResourceGroup = \"\" # Azure Resource Group which contains the ADP Resources\r\n",
"\r\n",
"# SQL Server \r\n",
"$SourceResourceGroupName = \"\" # Azure ResourceGroup where the sql server to be exported exists\r\n",
"$LogicalSQLServerName = \"\" # Logical sql server name of the sql server to be exported\r\n",
"$StorageAccount = \"\" # target storage account to store exported files # any storage account, but must be in the same RG as the ADP resources.\r\n",
"\r\n",
"# Set Variables for ADP Resources\r\n",
"$Env:BOOTSTRAP_FUNC = $Env:BOOTSTRAP_ResourceGroup + \"Control\"\r\n",
"$Env:BOOTSTRAP_BATCH = $Env:BOOTSTRAP_ResourceGroup.ToLower() + \"batch\"\r\n",
"$Env:BOOTSTRAP_VNET = $Env:BOOTSTRAP_ResourceGroup + \"Vnet\""
],
"metadata": {
"azdata_cell_guid": "417edc0e-1107-4a27-a4cf-e921f79b3f6a",
"tags": []
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Notebook Functions\r\n",
"Defines logical functions for the rest of the notebook. Function blocks are combined in a single cell that can be collapsed for readability or expanded for further examination. Nothing is executed until called later in the notebook. As a result, this cell is a requirement for any of the other cells below it. "
],
"metadata": {
"azdata_cell_guid": "b70909ed-1863-4882-bacc-a9956993268e"
}
},
{
"cell_type": "code",
"source": [
"# Expand cell to view framework\r\n",
"\r\n",
"function Login-Azure\r\n",
"{ \r\n",
" # query azure locations to test for existing az login session exists with valid access tocken\r\n",
" $azureLocations = az account list-locations -o JSON 2>$null | ConvertFrom-Json\r\n",
" \r\n",
" if (!$azureLocations){ #If there are no az locations, there is no existing az login session\r\n",
" $subscriptions = az login -o JSON | ConvertFrom-Json # Login \r\n",
" }\r\n",
" else {\r\n",
" $subscriptions = az account list -o JSON | ConvertFrom-Json # getting subscriptions for the user to use in gridview\r\n",
" }\r\n",
"\r\n",
" if(![string]::IsNullOrWhiteSpace($Env:BOOTSTRAP_Subscription)) #If there is a subscription specified by user in the variables section\r\n",
" {\r\n",
" $specified_Subscription= az account show --subscription $Env:BOOTSTRAP_Subscription -o json |ConvertFrom-Json \r\n",
" if (!$specified_Subscription) #if specified subscription is not valid\r\n",
" { \r\n",
" $currentUser= az ad signed-in-user show --query \"{displayName:displayName,UPN:userPrincipalName}\" -o json|ConvertFrom-Json # get current logged in user infomration\r\n",
" Write-Host \"Refer below for the list of subscriptions for logged in account '$($currentUser.UPN)'`n\"\r\n",
" az account list --query \"[].{Name:name,SubscriptionID:id}\" -o table # list subscriptions under current logged in account\r\n",
" }\r\n",
" else { # if specified subscription is valid\r\n",
" Write-Output \"Using subscription... '$($specified_Subscription.name)' ... '$($specified_Subscription.id)'\" \r\n",
" }\r\n",
" }\r\n",
" else { # if no subscription is specified, users are given a gridview to select subscription from\r\n",
"\r\n",
" $selectedSubscription = $subscriptions | Select-Object -Property Name, Id | Out-GridView -PassThru\r\n",
" $SubscriptionId = $selectedSubscription.Id\r\n",
" $Subscription = $selectedSubscription.Name \r\n",
" $Env:BOOTSTRAP_Subscription = $subscription \r\n",
" Write-Output \"Using subscription... '$Env:BOOTSTRAP_Subscription' ... '$SubscriptionId'\" \r\n",
" } \r\n",
"}\r\n",
"\r\n",
"function Verify-ADPResources\r\n",
"{ \r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ADPResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$FunctionName, \r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BatchAccountName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$VNetName,\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$ApplicationName=\"SqlPackageWrapper\", \r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$ApplicationPackageVersionName=\"1\",\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$SubNetName=\"default\" \r\n",
" ) \r\n",
"\r\n",
"# validate Subscription\r\n",
"$specified_Subscription= az account show --subscription $Subscription -o json | ConvertFrom-Json\r\n",
"if(!$specified_Subscription){\r\n",
" $currentUser= az ad signed-in-user show --query \"{displayName:displayName,UPN:userPrincipalName}\" -o json|ConvertFrom-Json # get current logged in user information\r\n",
" Write-Host \"Refer below for the list of subscriptions for logged in account '$($currentUser.UPN)'`n\"\r\n",
" az account list --query \"[].{Name:name,SubscriptionID:id}\" -o table # list subscriptions under current logged in account\r\n",
" return } \r\n",
"# validate ResourceGroup \r\n",
"$specified_ResourceGroup= az group show -n $ADPResourceGroupName --subscription $Subscription -o json | ConvertFrom-Json\r\n",
"if(!$specified_ResourceGroup) { \r\n",
" return\r\n",
" } \r\n",
"\r\n",
"$Installed = [ordered]@{} # ordered hash to store status of installation\r\n",
"$countError=0\r\n",
"\r\n",
"#Verify if VNet exists \r\n",
"$specified_VNet= az network vnet show -n $VNetName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null |ConvertFrom-Json \r\n",
"if(!$specified_VNet) {\r\n",
" $Installed['VNET']=\"Not Found\"\r\n",
" $countError++\r\n",
"} \r\n",
"else { \r\n",
" $existingVnetSubnet = az network vnet subnet show -n $SubNetName --vnet-name $VNetName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null |ConvertFrom-Json\r\n",
" if(!$existingVnetSubnet){\r\n",
" $Installed['VNET']=\"Default Subnet under\"+ $VNetName + \"Not Found\"\r\n",
" $countError++\r\n",
" }\r\n",
" else {\r\n",
" $Installed['VNET']=\"Installed\"\r\n",
" }\r\n",
" }\r\n",
"\r\n",
"#Verify if FunctionApp Exists\r\n",
"$specified_FunctionApp = az functionapp show -n $FunctionName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"if(!$specified_FunctionApp)\r\n",
"{\r\n",
" $Installed['FunctionApp']=\"Not Installed\"\r\n",
" $countError++\r\n",
"}\r\n",
"else\r\n",
"{\r\n",
" $Installed['FunctionApp']=\"Installed\"\r\n",
"} \r\n",
"\r\n",
"#check if Batch account exists\r\n",
"$specified_BatchAccount = az batch account show -n $BatchAccountName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"if(!$specified_BatchAccount)\r\n",
"{\r\n",
" $Installed['Batch']=\"Not Installed\"\r\n",
" $countError++\r\n",
"}\r\n",
"else\r\n",
"{\r\n",
" $appPackageInstalled = az batch application package show --application-name $ApplicationName --version-name $ApplicationPackageVersionName -n $BatchAccountName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
" $connectedToStorage= $specified_BatchAccount.autoStorage \r\n",
" if($connectedToStorage -and $appPackageInstalled){ # BatchAccount connected to storageaccount and applicationpackage is installed\r\n",
" $Installed['Batch']=\"Installed\"\r\n",
" $Installed['Batch_ApplicationPackage']=\"Installed\"\r\n",
" $Installed['Batch_StorageAccount']=\"Connected to storage- \"+$connectedToStorage.storageAccountId.Split(\"/\")[-1]\r\n",
" }\r\n",
" if(!$connectedToStorage)\r\n",
" {\r\n",
" $Installed['Batch_StorageAccount']='Not Found'\r\n",
" $countError++\r\n",
" } \r\n",
" if(!$appPackageInstalled)\r\n",
" {\r\n",
" $Installed['Batch_ApplicationPackage']=\"Not Found\"\r\n",
" $countError++\r\n",
" } \r\n",
"}\r\n",
"\r\n",
"if ($countError -gt 0){\r\n",
" Write-Output \"ADP Resources are not installed correctly. Please refer the list below and use the Bootstrap NB to install ADP Resources\"\r\n",
"}\r\n",
"$Installed\r\n",
"if ($countError -eq 0){\r\n",
" Write-Output \"`nFound all ADP Resources.\"\r\n",
"}\r\n",
"}\r\n",
"\r\n",
"function Prepare-InputForExportFunction\r\n",
"{ \r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ADPResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$FunctionName, \r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BatchAccountName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$VNetName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$SourceRGName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$SqlServerName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$StorageAccountName\r\n",
" )\r\n",
" \r\n",
" $InputResult = @{}\r\n",
" # Build Header \r\n",
" ## get Function key\r\n",
" $FunctionAppID =az functionapp show -n $FunctionName -g $ADPResourceGroupName --subscription $Subscription --query \"[id]\" -o JSON 2>$null | ConvertFrom-Json\r\n",
" $DefaultHostKey = az rest --method post --uri \"$FunctionAppID/host/default/listKeys?api-version=2018-11-01\" --query \"[functionKeys.default]\" -o JSON 2>$null | ConvertFrom-Json\r\n",
" ## Build Json Object for Headers\r\n",
" $headers = @{\r\n",
" 'x-functions-key' = $DefaultHostKey\r\n",
" }\r\n",
" $InputResult['Header']=$headers\r\n",
"\r\n",
" # Build string for Function URL \r\n",
" $specified_Subscription= az account show --subscription $Subscription -o json |ConvertFrom-Json #Get SpecifiedSubscriptionID\r\n",
" $SubscriptionID= $specified_Subscription.id\r\n",
" $FunctionUrl = 'https://'+ $FunctionName +'.azurewebsites.net/api/subscriptions/'+ $SubscriptionID +'/resourceGroups/' + $ADPResourceGroupName + '/Export'\r\n",
" $InputResult['FunctionURL']=$FunctionUrl\r\n",
"\r\n",
" # Set parameter variables for Body\r\n",
" ## Get BatchAccountURL \r\n",
" $specified_Batch = az batch account show -n $BatchAccountName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
" $BatchAccountURL = 'https://' + $specified_Batch.accountEndpoint \r\n",
"\r\n",
" ## Get default SubNet ID for specified VNet\r\n",
" $specified_VNet_SubNet = az network vnet subnet show -g $ADPResourceGroupName --vnet-name $VNetName -n 'default' --subscription $Subscription -o JSON |ConvertFrom-Json\r\n",
" $VNetSubNetID = $specified_VNet_SubNet.id\n",
"\n",
" ## Create access token to source sql server\n",
" $sourceAccessToken = az account get-access-token --resource=https://database.windows.net --query accessToken\n",
" $sourceAccessToken\r\n",
"\r\n",
" ## Build JSon object for Body\r\n",
" $Body = @{\r\n",
" batchAccountUrl = $BatchAccountURL\r\n",
" storageAccountName = $StorageAccountName # any storage account, not neccessarily the one connected to the batch account\r\n",
" sourceSqlServerResourceGroupName = $SourceRGName \r\n",
" sourceSqlServerName = $SQLServerName \n",
" accessToken= $sourceAccessToken\r\n",
" VNetSubnetId= $VNetSubNetID\r\n",
" }\r\n",
" $json = $Body | ConvertTo-Json\r\n",
" $InputResult['Body']=$json\r\n",
"\r\n",
" $InputResult\r\n",
"}\r\n",
"\r\n",
"function Provision-FuncRBAC {\r\n",
" [CmdletBinding()]\r\n",
" param (\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$Subscription,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$ResourceGroupName,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$FunctionName,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$ScopeRGName,\r\n",
" [Parameter(Mandatory=$false)][ValidateNotNullOrEmpty()][string]$Role=\"Contributor\"\r\n",
" )\r\n",
"\r\n",
" # Get the scope resource group's ID\r\n",
" $scopeID = az group show --resource-group $ScopeRGName --subscription $Subscription --query \"[id]\" -o JSON | ConvertFrom-Json \r\n",
" if(!$scopeID) {\r\n",
" Write-Output \"Provision-FuncRBAC failed.\" \r\n",
" return }\r\n",
" else { Write-Output \"Found scope '$ScopeRGName' with ID... '$scopeID'\"\r\n",
" }\r\n",
"\r\n",
" # Get the az function principal id\r\n",
" $app_PrincipalID = az functionapp show -n $FunctionName --resource-group $ResourceGroupName --subscription $Subscription --query \"[identity.principalId]\" -o JSON | ConvertFrom-Json \r\n",
" if(!$app_PrincipalID) {\r\n",
" Write-Output \"Provision-FuncRBAC failed.\" \r\n",
" return }\r\n",
" else { Write-Output \"Found principal id of Azure function '$FunctionName'... '$app_PrincipalID'\"\r\n",
" }\r\n",
"\r\n",
" # Verify if a role assignment has been created for function\r\n",
" $app_RoleAssignmentDefinition= az role assignment list --subscription $Subscription --assignee $app_PrincipalID --scope $scopeID --query \"[].roleDefinitionName\" -o JSON 2>$null | ConvertFrom-Json\r\n",
"\r\n",
" if($app_RoleAssignmentDefinition -eq $Role)\r\n",
" {\r\n",
" Write-Output \"Found Role Assignment for Principal ID.. '$app_PrincipalID' with Role.. '$app_RoleAssignmentDefinition' . No work needed\"\r\n",
" }\r\n",
" else\r\n",
" {\r\n",
" # Continue to setup RBAC, once we verify an assignment is not setup and all the resources exist\r\n",
" Write-Output \"Creating new role assignment by running: 'az functionapp identity assign -n $FunctionName --role $Role -g $ResourceGroupName --scope $scopeID --subscription $Subscription'\"\r\n",
" Write-Warning \"If your account does not have the access to assign new roles as Owner or User Access Administrator for the resource group, than you will need to contact your Azure AD Administrator to assign a service principle using the commands above\"\r\n",
" az functionapp identity assign -n $FunctionName --role $Role -g $ResourceGroupName --scope $scopeID --subscription $Subscription \r\n",
" }\r\n",
"}\r\n",
"Write-Host \"Helper Functions Created successfully\" "
],
"metadata": {
"azdata_cell_guid": "af70490a-c6cb-4086-99ff-c3527b4315ed",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Connect to Azure Account\r\n",
"Run the below cell to login to an Azure account. <b>Be sure to check the Windows Taskbar for a login dialog box underneath the notebook or other windows or by pressing Alt+TAB.</b>"
],
"metadata": {
"azdata_cell_guid": "55f9b3b2-9ce0-4607-9a45-384279f5d16f"
}
},
{
"cell_type": "code",
"source": [
"Login-Azure "
],
"metadata": {
"azdata_cell_guid": "4a577303-83b9-48f8-95ba-e4f74c7440c9"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Verify ADP Resources \r\n",
"Verify if ADP resources exists in specified Resource Group"
],
"metadata": {
"azdata_cell_guid": "a7e4bef8-3db6-4e68-8eb8-5a1feb4b3ac5"
}
},
{
"cell_type": "code",
"source": [
"Verify-ADPResources -Subscription $Env:BOOTSTRAP_Subscription -ADPResourceGroupName $Env:BOOTSTRAP_ResourceGroup `\r\n",
" -BatchAccountName $Env:BOOTSTRAP_BATCH -FunctionName $Env:BOOTSTRAP_FUNC -VNetName $Env:BOOTSTRAP_VNET "
],
"metadata": {
"azdata_cell_guid": "8185f2ea-d368-42c5-9246-bc1871affc63"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Verify RBAC of Azure Function\r\n",
"Roles based access control is a function of Azure that assigns services to a role with a specific access scope (or area of access). The ADP Orchestrator function requires Contributor access over the Resource Group where the SQL Server to be exported exists. The function below will attempt to create the role assignment. Any user executing this notebook will need to have Owner or User Access Administrator permissions to the Resource Group to assign the permission. Otherwise, contact your Azure AD Administrator. "
],
"metadata": {
"azdata_cell_guid": "e919b0e3-34d6-40aa-8b29-cf92b943954c"
}
},
{
"cell_type": "code",
"source": [
"Provision-FuncRBAC -FunctionName $Env:BOOTSTRAP_FUNC -ScopeRGName $SourceResourceGroupName -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "7678701e-ec40-43d9-baff-fd1cdabba1cd"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Verify Access to Source Server\r\n",
"\r\n",
"Before scheduling the export, validate the source SQL Server is accessible."
],
"metadata": {
"azdata_cell_guid": "e8310d46-b7ed-4012-a993-1515c2d17e66"
}
},
{
"cell_type": "code",
"source": [
"$sqlServer = az sql server show --name $LogicalSQLServerName --resource-group $SourceResourceGroupName --subscription $Env:BOOTSTRAP_Subscription -o JSON | ConvertFrom-JSON\r\n",
"if ($sqlServer)\r\n",
"{\r\n",
" Write-Host \"Source SQL Server: \" $sqlServer.name\r\n",
" if ($sqlServer.state -eq \"Ready\")\r\n",
" {\r\n",
" Write-Host \"State: Ready\"\r\n",
" }\r\n",
" else\r\n",
" {\r\n",
" Write-Host \"ERROR: Source server is not in Ready state. Current state is: \" $sqlServer.state\r\n",
" }\r\n",
"\r\n",
" $sqlAzureAdmin = az sql server ad-admin list --server $LogicalSQLServerName --resource-group $SourceResourceGroupName --subscription $Env:BOOTSTRAP_Subscription -o JSON | ConvertFrom-JSON\r\n",
" if ($sqlAzureAdmin)\r\n",
" {\r\n",
" Write-Host \"Azure AD admin set to\" $sqlAzureAdmin.login\r\n",
" Write-Host \"SUCCESS: source server accessible and properly configured.\"\r\n",
" }\r\n",
" else\r\n",
" {\r\n",
" Write-Host \"ERROR: Source server has no Azure AD administrator configured. The Data Portability solution requires Azure Active Directory to be configured on source servers to provide secure access.\"\r\n",
" }\r\n",
"\r\n",
"}\r\n",
"else \r\n",
"{\r\n",
" Write-Host \"ERROR: Source server \" $sqlServer.name \"not found or current account lacks access to resource.\"\r\n",
" Write-Host \"Validate input settings:\"\r\n",
" Write-Host \"Resource group: \" $SourceResourceGroupName\r\n",
" Write-Host \"Subscription: \" $Env:BOOTSTRAP_Subscription\r\n",
"}"
],
"metadata": {
"azdata_cell_guid": "5e217942-b9b0-47d0-a9b7-79b5db93ea55"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Prepare input variable for Orchestrator Export Function"
],
"metadata": {
"azdata_cell_guid": "bc9acbcc-f5e1-4642-b1e3-677fca704dfc"
}
},
{
"cell_type": "code",
"source": [
"$InputForExportFunction = Prepare-InputForExportFunction -Subscription $Env:BOOTSTRAP_Subscription -ADPResourceGroupName $Env:BOOTSTRAP_ResourceGroup `\r\n",
" -BatchAccountName $Env:BOOTSTRAP_BATCH -FunctionName $Env:BOOTSTRAP_FUNC -VNetName $Env:BOOTSTRAP_VNET -SourceRGName $SourceResourceGroupName `\r\n",
" -SqlServerName $LogicalSQLServerName -StorageAccountName $StorageAccount\r\n",
"Write-Host \"Setting parameter variables for Export Function Call...\"\r\n",
"$InputForExportFunction.Header\r\n",
"$InputForExportFunction.FunctionURL\r\n",
"$InputForExportFunction.Body"
],
"metadata": {
"azdata_cell_guid": "9e6f142c-3295-4c12-9b9f-588c7c283b2d"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Start Export of SQL Server \r\n",
"Run the cell to start export operation of specified sql server"
],
"metadata": {
"azdata_cell_guid": "d6a9e5c3-bd3c-4f06-8adc-c7751e33b4cf"
}
},
{
"cell_type": "code",
"source": [
"$ExportResponse = Invoke-RestMethod -Method 'Post' -Headers $InputForExportFunction.Header -Uri $InputForExportFunction.FunctionURL -Body $InputForExportFunction.Body -ContentType 'application/json'\r\n",
"$ExportResponse"
],
"metadata": {
"azdata_cell_guid": "2e0167f6-b91c-4d65-a6e6-a1ef0bada436"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Get Status of Export Operation\r\n",
"Run the cell to get export operation status"
],
"metadata": {
"azdata_cell_guid": "34812090-faa7-4375-8e76-72defea6d57a"
}
},
{
"cell_type": "code",
"source": [
"$statusCheckResponse = Invoke-RestMethod -Method 'Get' -Uri $ExportResponse.statusQueryGetUri\r\n",
"\r\n",
"\r\n",
"Write-Host \"Orchestrator Request: \" $statusCheckResponse.name\r\n",
"Write-Host \"`tOrchestrator Status: \" $statusCheckResponse.runtimeStatus\r\n",
"\r\n",
"$outputParams = $statusCheckResponse.output \r\n",
"if ($outputParams)\r\n",
"{\r\n",
" $batchJobID = $outputParams.Item2[0]\r\n",
" $containerUrl = $outputParams.Item2[1]\r\n",
"\r\n",
" Write-Host \"`tCreated Export Batch Job ID: \" $batchJobId\r\n",
" Write-Host \"`tExport container URL: \" $containerUrl\r\n",
"\r\n",
" $azBatchLogin = az batch account login --name $Env:BOOTSTRAP_BATCH --resource-group $Env:BOOTSTRAP_ResourceGroup -o JSON | ConvertFrom-Json\r\n",
" $jobStatus = az batch job show --job-id $batchJobID -o JSON | ConvertFrom-Json\r\n",
" Write-Host \"Export Job running on Pool: \" $jobStatus.poolInfo.poolId\r\n",
" Write-Host \"`tExport Request Status: \" $jobStatus.state\r\n",
"\r\n",
" $taskList = az batch task list --job-id $batchJobId -o JSON | ConvertFrom-Json\r\n",
" if ($taskList)\r\n",
" {\r\n",
" foreach ($task in $taskList)\r\n",
" {\r\n",
" Write-Host \"`tDatabase Export Task ID: \" $task.id \r\n",
" Write-Host \"`t`tStatus: \" $task.state\r\n",
" $taskExecution = $task.executionInfo\r\n",
" if ($taskExecution)\r\n",
" {\r\n",
" Write-Host \"`t`tResult: \" $taskExecution.result\r\n",
" }\r\n",
" }\r\n",
" }\r\n",
"}"
],
"metadata": {
"azdata_cell_guid": "de949b83-195d-455b-ab63-5607cad7c9dd"
},
"outputs": [],
"execution_count": null
}
]
}

530
extensions/azurehybridtoolkit/notebooks/hybridbook/content/data-portability/import-sql-server.ipynb Normal file
View File

@@ -0,0 +1,530 @@
{
"metadata": {
"kernelspec": {
"name": "powershell",
"display_name": "PowerShell"
},
"language_info": {
"name": "powershell",
"codemirror_mode": "shell",
"mimetype": "text/x-sh",
"file_extension": ".ps1"
}
},
"nbformat_minor": 2,
"nbformat": 4,
"cells": [
{
"cell_type": "markdown",
"source": [
"# Import Existing Azure SQL Server Resources\r\n",
""
],
"metadata": {
"azdata_cell_guid": "baa1b762-f2a9-461f-a126-49f0d3b5e4f0"
}
},
{
"cell_type": "markdown",
"source": [
"The notebook will help accomplish the below steps as a part of Importing existing Azure SQL Server Resources\n",
"\n",
"- [ ] Connect to Azure Subscription\n",
"- [ ] Choose Resource Group (Read access required)\n",
"- [ ] Choose Microsoft SQL Server Resources to import\n",
"- [ ] Choose/Create Migration Storage\n",
"- [ ] Install Application + Data Portability function\n",
"- [ ] Install ADP Azure Batch processing pipeline\n",
"- [ ] Store SqlPackage.exe in Migration Storage for ADP function to hand to Az Batch\n",
"\n",
"Execute:\n",
"\n",
"- [ ] Check all prerequisites\n",
"- [ ] Kick off ADP service\n",
"\n",
"Monitor:\n",
"\n",
"- [ ] Check import status."
],
"metadata": {
"azdata_cell_guid": "1ccbf203-d568-408c-a641-7e5cfa93802a"
}
},
{
"cell_type": "markdown",
"source": [
"## Set Variables for the Notebook"
],
"metadata": {
"azdata_cell_guid": "09f648ab-67e8-4127-94bd-649d5e970321"
}
},
{
"cell_type": "code",
"source": [
"# ADP Resource \r\n",
"$Env:BOOTSTRAP_Subscription = \"\" # Azure Subscription ID/Name # The bacpac files and ADP Resources are assumed to be in the same subscription\r\n",
"$Env:BOOTSTRAP_ResourceGroup = \"\" # Azure Resource Group which contains the ADP Resources\r\n",
"\r\n",
"# SQL Server \r\n",
"$TargetResourceGroupName = \"\" # Azure ResourceGroup into which the sql server backup needs to be restored\r\n",
"$StorageAccountName = \"\"\r\n",
"$ContainerName = \"\"\r\n",
"$LogicalSQLServerName = \"\" # New sql server name\r\n",
"$LSqlServerPassword = \"\"\r\n",
"\r\n",
"# Set Variables for ADP Resources\r\n",
"$Env:BOOTSTRAP_FUNC = $Env:BOOTSTRAP_ResourceGroup + \"Control\" \r\n",
"$Env:BOOTSTRAP_BATCH = $Env:BOOTSTRAP_ResourceGroup.ToLower() + \"batch\"\r\n",
"$Env:BOOTSTRAP_VNET = $Env:BOOTSTRAP_ResourceGroup + \"Vnet\""
],
"metadata": {
"azdata_cell_guid": "01888595-0d1c-445b-ba85-dd12caa30192",
"tags": []
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Notebook Functions\r\n",
"Defines logical functions for the rest of the notebook. Function blocks are combined in a single cell that can be collapsed for readability or expanded for further examination. Nothing is executed until called later in the notebook. As a result, this cell is a requirement for any of the other cells below it. "
],
"metadata": {
"azdata_cell_guid": "4a35cf8e-6598-43e8-a0da-2c0c369df548"
}
},
{
"cell_type": "code",
"source": [
"# Expand cell to view framework\r\n",
"\r\n",
"function Login-Azure\r\n",
"{ \r\n",
" # query azure locations to test for existing az login session exists with valid access tocken\r\n",
" $azureLocations = az account list-locations -o JSON 2>$null | ConvertFrom-Json\r\n",
" \r\n",
" if (!$azureLocations){ #If there are no az locations, there is no existing az login session\r\n",
" $subscriptions = az login -o JSON | ConvertFrom-Json # Login \r\n",
" }\r\n",
" else {\r\n",
" $subscriptions = az account list -o JSON | ConvertFrom-Json # getting subscriptions for the user to use in gridview\r\n",
" }\r\n",
"\r\n",
" if(![string]::IsNullOrWhiteSpace($Env:BOOTSTRAP_Subscription)) #If there is a subscription specified by user in the variables section\r\n",
" {\r\n",
" $specified_Subscription= az account show --subscription $Env:BOOTSTRAP_Subscription -o json |ConvertFrom-Json \r\n",
" if (!$specified_Subscription) #if specified subscription is not valid\r\n",
" { \r\n",
" $currentUser= az ad signed-in-user show --query \"{displayName:displayName,UPN:userPrincipalName}\" -o json|ConvertFrom-Json # get current logged in user infomration\r\n",
" Write-Host \"Refer below for the list of subscriptions for logged in account '$($currentUser.UPN)'`n\"\r\n",
" az account list --query \"[].{Name:name,SubscriptionID:id}\" -o table # list subscriptions under current logged in account\r\n",
" }\r\n",
" else { # if specified subscription is valid\r\n",
" Write-Output \"Using subscription... '$($specified_Subscription.name)' ... '$($specified_Subscription.id)'\" \r\n",
" }\r\n",
" }\r\n",
" else { # if no subscription is specified, users are given a gridview to select subscription from\r\n",
"\r\n",
" $selectedSubscription = $subscriptions | Select-Object -Property Name, Id | Out-GridView -PassThru\r\n",
" $SubscriptionId = $selectedSubscription.Id\r\n",
" $Subscription = $selectedSubscription.Name \r\n",
" $Env:BOOTSTRAP_Subscription = $subscription \r\n",
" Write-Output \"Using subscription... '$Env:BOOTSTRAP_Subscription' ... '$SubscriptionId'\" \r\n",
" } \r\n",
"}\r\n",
"\r\n",
"function Verify-ADPResources\r\n",
"{ \r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ADPResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$FunctionName, \r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BatchAccountName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$VNetName,\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$ApplicationName=\"SqlPackageWrapper\", \r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$ApplicationPackageVersionName=\"1\",\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$SubNetName=\"default\" \r\n",
" ) \r\n",
"\r\n",
"# validate Subscription\r\n",
"$specified_Subscription= az account show --subscription $Subscription -o json | ConvertFrom-Json\r\n",
"if(!$specified_Subscription){\r\n",
" $currentUser= az ad signed-in-user show --query \"{displayName:displayName,UPN:userPrincipalName}\" -o json|ConvertFrom-Json # get current logged in user information\r\n",
" Write-Host \"Refer below for the list of subscriptions for logged in account '$($currentUser.UPN)'`n\"\r\n",
" az account list --query \"[].{Name:name,SubscriptionID:id}\" -o table # list subscriptions under current logged in account\r\n",
" return } \r\n",
"# validate ResourceGroup \r\n",
"$specified_ResourceGroup= az group show -n $ADPResourceGroupName --subscription $Subscription -o json | ConvertFrom-Json\r\n",
"if(!$specified_ResourceGroup) { \r\n",
" return\r\n",
" } \r\n",
"\r\n",
"$Installed = [ordered]@{} # ordered hash to store status of installation\r\n",
"$countError=0\r\n",
"\r\n",
"#Verify if VNet exists \r\n",
"$specified_VNet= az network vnet show -n $VNetName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null |ConvertFrom-Json \r\n",
"if(!$specified_VNet) {\r\n",
" $Installed['VNET']=\"Not Found\"\r\n",
" $countError++\r\n",
"} \r\n",
"else { \r\n",
" $existingVnetSubnet = az network vnet subnet show -n $SubNetName --vnet-name $VNetName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null |ConvertFrom-Json\r\n",
" if(!$existingVnetSubnet){\r\n",
" $Installed['VNET']=\"Default Subnet under\"+ $VNetName + \"Not Found\"\r\n",
" $countError++\r\n",
" }\r\n",
" else {\r\n",
" $Installed['VNET']=\"Installed\"\r\n",
" }\r\n",
" }\r\n",
"\r\n",
"#Verify if FunctionApp Exists\r\n",
"$specified_FunctionApp = az functionapp show -n $FunctionName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"if(!$specified_FunctionApp)\r\n",
"{\r\n",
" $Installed['FunctionApp']=\"Not Installed\"\r\n",
" $countError++\r\n",
"}\r\n",
"else\r\n",
"{\r\n",
" $Installed['FunctionApp']=\"Installed\"\r\n",
"} \r\n",
"\r\n",
"#check if Batch account exists\r\n",
"$specified_BatchAccount = az batch account show -n $BatchAccountName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"if(!$specified_BatchAccount)\r\n",
"{\r\n",
" $Installed['Batch']=\"Not Installed\"\r\n",
" $countError++\r\n",
"}\r\n",
"else\r\n",
"{\r\n",
" $appPackageInstalled = az batch application package show --application-name $ApplicationName --version-name $ApplicationPackageVersionName -n $BatchAccountName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
" $connectedToStorage= $specified_BatchAccount.autoStorage \r\n",
" if($connectedToStorage -and $appPackageInstalled){ # BatchAccount connected to storageaccount and applicationpackage is installed\r\n",
" $Installed['Batch']=\"Installed\"\r\n",
" $Installed['Batch_ApplicationPackage']=\"Installed\"\r\n",
" $Installed['Batch_StorageAccount']=\"Connected to storage- \"+$connectedToStorage.storageAccountId.Split(\"/\")[-1]\r\n",
" }\r\n",
" if(!$connectedToStorage)\r\n",
" {\r\n",
" $Installed['Batch_StorageAccount']='Not Found'\r\n",
" $countError++\r\n",
" } \r\n",
" if(!$appPackageInstalled)\r\n",
" {\r\n",
" $Installed['Batch_ApplicationPackage']=\"Not Found\"\r\n",
" $countError++\r\n",
" } \r\n",
"}\r\n",
"if ($countError -gt 0){\r\n",
" Write-Output \"ADP Resources are not installed correctly. Please refer the list below and use the Bootstrap NB to install ADP Resources\"\r\n",
"}\r\n",
"$Installed\r\n",
"if ($countError -eq 0){\r\n",
" Write-Output \"`nFound all ADP Resources.\"\r\n",
"}\r\n",
"}\r\n",
"\r\n",
"function Prepare-InputForImportFunction\r\n",
"{ \r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ADPResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$FunctionName, \r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BatchAccountName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BackupFiles_StorageAccount,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BackupFiles_ContainerName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$VNetName, \r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$TargetRGName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$SqlServerName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$SqlServerPassword\r\n",
" )\r\n",
" \r\n",
" $Result = @{}\r\n",
" # Build Header \r\n",
" ## get Function key\r\n",
" $FunctionAppID =az functionapp show -n $FunctionName -g $ADPResourceGroupName --subscription $Subscription --query \"[id]\" -o JSON 2>$null | ConvertFrom-Json\r\n",
" $DefaultHostKey = az rest --method post --uri \"$FunctionAppID/host/default/listKeys?api-version=2018-11-01\" --query \"[functionKeys.default]\" -o JSON 2>$null | ConvertFrom-Json\r\n",
" ## Build Json Object for Headers\r\n",
" $headers = @{\r\n",
" 'x-functions-key' = $DefaultHostKey\r\n",
" }\r\n",
" $Result['Header']=$headers\r\n",
"\r\n",
" # Build string for Function URL \r\n",
" $specified_Subscription= az account show --subscription $Subscription -o json |ConvertFrom-Json #Get SpecifiedSubscriptionID\r\n",
" $SubscriptionID= $specified_Subscription.id\r\n",
" $FunctionUrl = 'https://'+ $FunctionName +'.azurewebsites.net/api/subscriptions/'+ $SubscriptionID +'/resourceGroups/' + $ADPResourceGroupName + '/Import'\r\n",
" $Result['FunctionURL']=$FunctionUrl\r\n",
"\r\n",
" # Set parameter variables for Body\r\n",
" ## Get BatchAccountURL \r\n",
" $specified_Batch = az batch account show -n $BatchAccountName -g $ADPResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
" $BatchAccountURL = 'https://' + $specified_Batch.accountEndpoint\r\n",
"\r\n",
" ## Get default SubNet ID for specified VNet\r\n",
" $specified_VNet_SubNet = az network vnet subnet show -g $ADPResourceGroupName --vnet-name $VNetName -n 'default' --subscription $Subscription -o JSON |ConvertFrom-Json\r\n",
" $VNetSubNetID = $specified_VNet_SubNet.id\r\n",
"\r\n",
" ## Create access token to source sql server\r\n",
" $targetAccessToken = az account get-access-token --resource=https://database.windows.net --query accessToken\r\n",
" $targetAccessToken\r\n",
"\r\n",
" ## Build JSon object for Body\r\n",
" $Body = @{\r\n",
" batchAccountUrl = $BatchAccountURL\r\n",
" VNetSubnetId= $VNetSubNetID\r\n",
" storageAccountName = $BackupFiles_StorageAccount\r\n",
" containerName = $BackupFiles_ContainerName\r\n",
" targetSqlServerResourceGroupName = $TargetRGName\r\n",
" targetSqlServerName = $SQLServerName \r\n",
" userName = $SqlServerLogin \r\n",
" targetAccessToken = $targetAccessToken\r\n",
" sqlAdminPassword = $SqlServerPassword\r\n",
" }\r\n",
" $json = $Body | ConvertTo-Json\r\n",
" $Result['Body']=$json\r\n",
"\r\n",
" $Result\r\n",
" \r\n",
"}\r\n",
"\r\n",
"function Provision-FuncRBAC {\r\n",
" [CmdletBinding()]\r\n",
" param (\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$Subscription,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$ResourceGroupName,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$FunctionName,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$ScopeRGName,\r\n",
" [Parameter(Mandatory=$false)][ValidateNotNullOrEmpty()][string]$Role=\"Contributor\"\r\n",
" )\r\n",
"\r\n",
" # Get the scope resource group's ID\r\n",
" $scopeID = az group show --resource-group $ScopeRGName --subscription $Subscription --query \"[id]\" -o JSON | ConvertFrom-Json \r\n",
" if(!$scopeID) {\r\n",
" Write-Output \"Provision-FuncRBAC failed.\" \r\n",
" return }\r\n",
" else { Write-Output \"Found scope '$ScopeRGName' with ID... '$scopeID'\"\r\n",
" }\r\n",
"\r\n",
" # Get the az function principal id\r\n",
" $app_PrincipalID = az functionapp show -n $FunctionName --resource-group $ResourceGroupName --subscription $Subscription --query \"[identity.principalId]\" -o JSON | ConvertFrom-Json \r\n",
" if(!$app_PrincipalID) {\r\n",
" Write-Output \"Provision-FuncRBAC failed.\" \r\n",
" return }\r\n",
" else { Write-Output \"Found principal id of Azure function '$FunctionName'... '$app_PrincipalID'\"\r\n",
" }\r\n",
"\r\n",
" # Verify if a role assignment has been created for function\r\n",
" $app_RoleAssignmentDefinition= az role assignment list --subscription $Subscription --assignee $app_PrincipalID --scope $scopeID --query \"[].roleDefinitionName\" -o JSON 2>$null | ConvertFrom-Json\r\n",
"\r\n",
" if($app_RoleAssignmentDefinition -eq $Role)\r\n",
" {\r\n",
" Write-Output \"Found Role Assignment for Principal ID.. '$app_PrincipalID' with Role.. '$app_RoleAssignmentDefinition' . No work needed\"\r\n",
" }\r\n",
" else\r\n",
" {\r\n",
" # Continue to setup RBAC, once we verify an assignment is not setup and all the resources exist\r\n",
" Write-Output \"Creating new role assignment by running: 'az functionapp identity assign -n $FunctionName --role $Role -g $ResourceGroupName --scope $scopeID --subscription $Subscription'\"\r\n",
" Write-Warning \"If your account does not have the access to assign new roles as Owner or User Access Administrator for the resource group, than you will need to contact your Azure AD Administrator to assign a service principle using the commands above\"\r\n",
" az functionapp identity assign -n $FunctionName --role $Role -g $ResourceGroupName --scope $scopeID --subscription $Subscription \r\n",
" }\r\n",
"}\r\n",
"Write-Host \"Helper Functions Created successfully\" "
],
"metadata": {
"azdata_cell_guid": "4730aec5-7aa6-4a2a-baf4-696dc74aa898",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Connect to Azure Account\r\n",
"Run the below cell to login to an Azure account. Be sure to check the Windows Taskbar for a login dialog box underneath the notebook or other windows or by pressing Alt+TAB."
],
"metadata": {
"azdata_cell_guid": "5cd37536-c2c5-4b97-8383-fa761d7cbda3"
}
},
{
"cell_type": "code",
"source": [
"Login-Azure "
],
"metadata": {
"azdata_cell_guid": "e9cd7ac2-ff0b-43b4-baf8-b71d3128885c"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Verify ADP Resources \r\n",
"Verify if ADP resources exists in specified Resource Group"
],
"metadata": {
"azdata_cell_guid": "bec05e08-67ba-4071-8459-5b32dc7f876a"
}
},
{
"cell_type": "code",
"source": [
"Verify-ADPResources -Subscription $Env:BOOTSTRAP_Subscription -ADPResourceGroupName $Env:BOOTSTRAP_ResourceGroup `\r\n",
" -BatchAccountName $Env:BOOTSTRAP_BATCH -FunctionName $Env:BOOTSTRAP_FUNC -VNetName $Env:BOOTSTRAP_VNET "
],
"metadata": {
"azdata_cell_guid": "e89f6eb9-fcbc-4b7d-bcd1-37f1eb52cc02",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Verify RBAC of Azure Function\r\n",
"Roles based access control is a function of Azure that assigns services to a role with a specific access scope (or area of access). The ADP Orchestrator function requires Contributor access over the Resource Group where the SQL Server to be exported exists. The function below will attempt to create the role assignment. Any user executing this notebook will need to have Owner or User Access Administrator permissions to the Resource Group to assign the permission. Otherwise, contact your Azure AD Administrator. "
],
"metadata": {
"azdata_cell_guid": "0c95bb17-b3cf-4a8b-8aa6-690ac6139e37"
}
},
{
"cell_type": "code",
"source": [
"Provision-FuncRBAC -FunctionName $Env:BOOTSTRAP_FUNC -ScopeRGName $TargetResourceGroupName -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "c374e57c-51ec-4a3f-9966-1e50cefc8510"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Prepare input variable for Orchestrator Import Function"
],
"metadata": {
"azdata_cell_guid": "b517742f-fa3d-4a4f-9ec0-b320c71738d4"
}
},
{
"cell_type": "code",
"source": [
"$InputForImportFunction = Prepare-InputForImportFunction -Subscription $Env:BOOTSTRAP_Subscription -ADPResourceGroupName $Env:BOOTSTRAP_ResourceGroup `\r\n",
" -BatchAccountName $Env:BOOTSTRAP_BATCH -FunctionName $Env:BOOTSTRAP_FUNC -TargetRGName $TargetResourceGroupName `\r\n",
" -VNetName $Env:BOOTSTRAP_VNET -BackupFiles_StorageAccount $StorageAccountName -BackupFiles_ContainerName $ContainerName `\r\n",
" -SqlServerName $LogicalSQLServerName -SqlServerPassword $LSqlServerpassword\r\n",
"Write-Host \"Setting parameter variables for Import Function Call...\"\r\n",
"$InputForImportFunction.Header\r\n",
"$InputForImportFunction.FunctionURL\r\n",
"$InputForImportFunction.Body"
],
"metadata": {
"azdata_cell_guid": "bfba288e-3466-4c57-9f3c-5281753601cf",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Start Import of SQL Server \r\n",
"Run the cell to start import from specified backup files"
],
"metadata": {
"azdata_cell_guid": "8f615b19-1e1d-405f-9f4a-ad7cc303487a"
}
},
{
"cell_type": "code",
"source": [
"$importResponse = Invoke-RestMethod -Method 'Post' -Headers $InputForImportFunction.Header -Uri $InputForImportFunction.FunctionURL -Body $InputForImportFunction.Body -ContentType 'application/json'\r\n",
"$importResponse"
],
"metadata": {
"azdata_cell_guid": "7e251aa5-7a92-4212-8c81-394c2058f1fa",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Get Status of Import Operation\r\n",
"Run the cell to get import operation status"
],
"metadata": {
"azdata_cell_guid": "cb1988a9-9797-49f0-8ddb-1634df48f027"
}
},
{
"cell_type": "code",
"source": [
"$statusCheckResponse = Invoke-RestMethod -Method 'Get' -Uri $importResponse.statusQueryGetUri\r\n",
"\r\n",
"\r\n",
"Write-Host \"Orchestrator Request: \" $statusCheckResponse.name\r\n",
"Write-Host \"`tOrchestrator Status: \" $statusCheckResponse.runtimeStatus\r\n",
"\r\n",
"$outputParams = $statusCheckResponse.output \r\n",
"if ($outputParams)\r\n",
"{\r\n",
" $batchJobID = $outputParams.Item2[2]\r\n",
" $containerUrl = $outputParams.Item2[3]\r\n",
"\r\n",
" Write-Host \"`tCreated Import Batch Job ID: \" $batchJobId\r\n",
" $azBatchLogin = az batch account login --name $Env:BOOTSTRAP_BATCH --resource-group $Env:BOOTSTRAP_ResourceGroup -o JSON | ConvertFrom-Json\r\n",
" $jobStatus = az batch job show --job-id $batchJobID -o JSON | ConvertFrom-Json\r\n",
" Write-Host \"Import Job running on Pool: \" $jobStatus.poolInfo.poolId\r\n",
" Write-Host \"`Import Request Status: \" $jobStatus.state\r\n",
"\r\n",
" $taskList = az batch task list --job-id $batchJobId -o JSON | ConvertFrom-Json\r\n",
" if ($taskList)\r\n",
" {\r\n",
" foreach ($task in $taskList)\r\n",
" {\r\n",
" Write-Host \"`tDatabase Import Task ID: \" $task.id \r\n",
" Write-Host \"`t`tStatus: \" $task.state\r\n",
" $taskExecution = $task.executionInfo\r\n",
" if ($taskExecution)\r\n",
" {\r\n",
" Write-Host \"`t`tResult: \" $taskExecution.result\r\n",
" }\r\n",
" }\r\n",
" }\r\n",
"}\r\n",
""
],
"metadata": {
"azdata_cell_guid": "3c59dc7b-2648-46ee-b57e-d9e99580a093",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
}
]
}

15
extensions/azurehybridtoolkit/notebooks/hybridbook/content/data-portability/readme.md Normal file
View File

@@ -0,0 +1,15 @@
# Data Portability
[Home](../readme.md)
## Notebooks in this Chapter
- [Azure Data Portability Setup](bootstrap.ipynb) - Configure and install a custom Azure function to migrate data to and from Azure
- [Export Sql Server](export-sql-server.ipynb) - from SQL Azure to a standard SQL backup format
- [Import Sql Server](import-sql-server.ipynb) - from SQL backup format to Azure
The Notebooks in this chapter perform a data migration using a custom Azure function that can be deployed to an Azure subscription. It enables [Azure Batch](https://azure.microsoft.com/en-us/services/batch) computing of a complex SQL Server migration to and from a single Resource Group. Azure Batch is a process that runs large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. This greatly reduces the processing required locally which should prevent long execution times, timeouts and retries. Importing and exporting data to and from Azure is supported for multiple SQL database instances. Data is imported and exported to and from standard SQL backup formats (*.bacpac) which "encapsulates the database schema as well as the data stored in the database" ([Microsoft Docs](https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/data-tier-applications)).
1. The Azure function must first be deployed using the setup notebook
2. Open the notebook for the desired migration path
3. Configure and execute notebook
4. Monitor progress with periodic notebook queries
5. Verify data has been imported/exported by reviewing the storage account for the migrated Resource Group

643
extensions/azurehybridtoolkit/notebooks/hybridbook/content/data-portability/setup-adp.ipynb Normal file
View File

@@ -0,0 +1,643 @@
{
"metadata": {
"kernelspec": {
"name": "powershell",
"display_name": "PowerShell"
},
"language_info": {
"name": "powershell",
"codemirror_mode": "shell",
"mimetype": "text/x-sh",
"file_extension": ".ps1"
}
},
"nbformat_minor": 2,
"nbformat": 4,
"cells": [
{
"cell_type": "markdown",
"source": [
"# Application and Data Portability Setup Notebook\n",
"\n",
"This notebook provisions SQL ADP services to an Azure Subscription for migration purposes such as importing and exporting Azure SQL Server database(s). The following scenarios are covered:\n",
"\n",
"- [x] Azure to Azure migration\n",
"- [x] Azure to Air-gapped Azure (highly secured Azure offer)\n",
"- [x] Azure to !Azure (3rd party or on-premises)\n",
"\n",
"This bootstrapper is designed to be ran from a client workstation or laptop with minimal modification. The notebook will attempt to locate three different Azure Resources in a customer subscription. If not found it will install the following to a given Resource Group. If the resources group specified doesn't exist, it will create a new one:\n",
"\n",
"1. A **Custom Azure ADP Function** to orchestrate the Azure Batch and Storage (see \\[Github\\](........\\\\Program Files\\\\Azure Data Studio\\\\resources\\\\app\\\\out\\\\vs\\\\code\\\\electron-browser\\\\workbench\\\\workbench.html))\n",
"2. An Azure **Storage Account** for storage\n",
"3. An **Azure Batch** resource\n",
"\n",
"This notebook will help accomplish the below steps as a part of Bootstrapping an ADP Resource Group\n",
"\n",
"![](VisualBootstrapperNB.PNG)\n",
"\n",
"1. Set Environment Variables\n",
"2. Define Notebook Functions\n",
"3. MS Azure Login\n",
"4. Validate/ Create ADP Resource Group\n",
"5. Provision Virtual Network\n",
"6. Provision ADP Storage Account\n",
"7. Provision ADP Portability Function\n",
"8. Provision ADP Batch Account\n",
"9. Provision Contributor access (RBAC) to Portability function over ADP Resource Group\n",
"\n",
"Setup _will not_ perform tasks on existing Azure assets such as Microsoft SQL Servers, databases, VMs, etc. Other notebooks cover the <u><a href=\"import-sql-server.ipynb\" rel=\"nofollow\">import</a></u> or <u><a href=\"export-sql-server.ipynb\" rel=\"nofollow\">export</a></u> operations. This notebook will only check for the existence of these ADP resources and their setup."
],
"metadata": {
"azdata_cell_guid": "c5d49b77-973b-47bc-92ef-7bcfe05e4885"
}
},
{
"cell_type": "code",
"source": [
"# Setup client environment variables that the rest of the notebook will use\r\n",
"$Env:BOOTSTRAP_ResourceGroup = \"\" # Target Resource Group to bootstrap with ADP components - A new one will be created if the specified Resource Group doesn't exist\r\n",
"$Env:BOOTSTRAP_RG_REGION = \"eastus\" # Region/Location of the resource group to be bootstrapped\r\n",
"\r\n",
"# Derived settings\r\n",
"$Env:BOOTSTRAP_Subscription = \"\" # Target Azure Subscription Name or ID to bootstrap data portability resources\r\n",
"$Env:BOOTSTRAP_FUNC = $Env:BOOTSTRAP_ResourceGroup + \"Control\"\r\n",
"$Env:BOOTSTRAP_STORAGE = $Env:BOOTSTRAP_ResourceGroup.ToLower() + \"storage\"\r\n",
"$Env:BOOTSTRAP_BATCH = $Env:BOOTSTRAP_ResourceGroup.ToLower() + \"batch\"\r\n",
"$Env:BOOTSTRAP_VNET = $Env:BOOTSTRAP_ResourceGroup + \"VNet\"\r\n",
"\r\n",
"# Bootstrapper URLs - Update with the recommended toolkit version and build\r\n",
"$BaseToolkitUrl = \"https://hybridtoolkit.blob.core.windows.net/components\"\r\n",
"$ReleaseVersion = \"0.13\"\r\n",
"$BuildNumber = \"74938\"\r\n",
"$Env:BOOTSTRAP_URL_FUNC = \"$BaseToolkitUrl/$ReleaseVersion/ADPControl-$BuildNumber.zip\"\r\n",
"$Env:BOOTSTRAP_URL_WRAP = \"$BaseToolkitUrl/$ReleaseVersion/BatchWrapper-$BuildNumber.zip\"\r\n",
"\r\n",
"Write-Output \"Setting the Environment:\"\r\n",
"Get-ChildItem Env: | Where-Object Name -Match \"BOOTSTRAP\""
],
"metadata": {
"azdata_cell_guid": "5896a9a8-5ba8-4888-967a-10c3ceaccf12",
"tags": []
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Define Helper Functions\r\n",
"Defines logical functions for the rest of the notebook. Function blocks are combined in a single cell that can be collapsed for readability or expanded for further examination. Nothing is executed until called later in the notebook. As a result, this cell is a requirement for any of the other cells below it. "
],
"metadata": {
"azdata_cell_guid": "7df7c213-9e97-46e4-acb5-547993b9b03a"
}
},
{
"cell_type": "code",
"source": [
"# Expand cell to view framework\r\n",
"\r\n",
"function Get-ConnectionStringProperties\r\n",
"{\r\n",
" param([Parameter (Mandatory=$true)] [string]$ConnectionString)\r\n",
" $hash = @{}\r\n",
" $attributes = $ConnectionString.Split(\";\"); \r\n",
" foreach($at in $attributes)\r\n",
" {\r\n",
" $attributeKeyValue=$at.Split(\"=\",2) \r\n",
" $hash[$attributeKeyValue[0]]=$attributeKeyValue[1] \r\n",
" }$hash \r\n",
"}\r\n",
"\r\n",
"function Login-Azure\r\n",
"{ \r\n",
" # query azure locations to test for existing az login session exists with valid access tocken\r\n",
" $azureLocations = az account list-locations -o JSON 2>$null | ConvertFrom-Json\r\n",
" \r\n",
" if (!$azureLocations){ #If there are no az locations, there is no existing az login session\r\n",
" $subscriptions = az login -o JSON | ConvertFrom-Json # Login \r\n",
" }\r\n",
" else {\r\n",
" $subscriptions = az account list -o JSON | ConvertFrom-Json # getting subscriptions for the user to use in gridview\r\n",
" }\r\n",
"\r\n",
" if(![string]::IsNullOrWhiteSpace($Env:BOOTSTRAP_Subscription)) #If there is a subscription specified by user in the variables section\r\n",
" {\r\n",
" $specified_Subscription= az account show --subscription $Env:BOOTSTRAP_Subscription -o json |ConvertFrom-Json \r\n",
" if (!$specified_Subscription) #if specified subscription is not valid\r\n",
" { \r\n",
" $currentUser= az ad signed-in-user show --query \"{displayName:displayName,UPN:userPrincipalName}\" -o json|ConvertFrom-Json # get current logged in user infomration\r\n",
" Write-Host \"Refer below for the list of subscriptions for logged in account '$($currentUser.UPN)'`n\"\r\n",
" az account list --query \"[].{Name:name,SubscriptionID:id}\" -o table # list subscriptions under current logged in account\r\n",
" }\r\n",
" else { # if specified subscription is valid\r\n",
" Write-Output \"Using subscription... '$($specified_Subscription.name)' ... '$($specified_Subscription.id)'\" \r\n",
" }\r\n",
" }\r\n",
" else { # if no subscription is specified, users are given a gridview to select subscription from\r\n",
"\r\n",
" $selectedSubscription = $subscriptions | Select-Object -Property Name, Id | Out-GridView -PassThru\r\n",
" $SubscriptionId = $selectedSubscription.Id\r\n",
" $Subscription = $selectedSubscription.Name \r\n",
" $Env:BOOTSTRAP_Subscription = $subscription \r\n",
" Write-Output \"Using subscription... '$Env:BOOTSTRAP_Subscription' ... '$SubscriptionId'\" \r\n",
" } \r\n",
"}\r\n",
"\r\n",
"function Bootstrap-AzResourceGroup\r\n",
"{\r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupLocation \r\n",
" ) \r\n",
" \r\n",
" #Validating parameter $Subscription\r\n",
" $specified_Subscription = az account show --subscription $Subscription -o json |ConvertFrom-Json\r\n",
" if(!$specified_Subscription){\r\n",
" $currentUser= az ad signed-in-user show --query \"{displayName:displayName,UPN:userPrincipalName}\" -o json|ConvertFrom-Json # get current logged in user infomration\r\n",
" Write-Host \"Refer below for the list of subscriptions for logged in account '$($currentUser.UPN)'`n\"\r\n",
" az account list --query \"[].{Name:name,SubscriptionID:id}\" -o table # list subscriptions under current logged in account\r\n",
" return\r\n",
" }\r\n",
" #Validating parameter $ResourceGroupLocation\r\n",
" $specified_RGLocation = az account list-locations --query \"[?name == '$ResourceGroupLocation']\" -o JSON | ConvertFrom-Json\r\n",
" if(!$specified_RGLocation) {\r\n",
" Write-Host \"ResourceGroupLocation '$ResourceGroupLocation' not found. Refer below for the list of locations for the account:`n\" \r\n",
" az account list-locations --query \"[].{Locations: name}\" -o table\r\n",
" return\r\n",
" }\r\n",
" # check if Resource group already exists under specified subscription\r\n",
" $existingResourceGroup = az group show -n $ResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"\r\n",
" if ($existingResourceGroup){ #Found specified resource group \r\n",
" Write-Output \"Found Resource Group '$($existingResourceGroup.name)'. \" \r\n",
" } \r\n",
" else { \r\n",
" # Resource group not found under specified subscription. Create new Resource group \r\n",
" Write-Output \"Resource group '$($ResourceGroupName)' not found. `nCreating new resource group ... \"\r\n",
" $newResourceGroup = az group create -n $ResourceGroupName -l $ResourceGroupLocation --subscription $Subscription -o JSON | ConvertFrom-Json\r\n",
" Write-Output \"Created Resource Group '$($newResourceGroup.name)'\"\r\n",
" } \r\n",
"}\r\n",
"\r\n",
"function Bootstrap-AzVirtualNetwork\r\n",
"{ \r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$VNetName, \r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$SubNetName = \"default\" \r\n",
" )\r\n",
" \r\n",
" #Validating parameter $ResourceGroupName\r\n",
" $specified_RG = az group show -n $ResourceGroupName --subscription $Subscription 2>$null\r\n",
" if(!$specified_RG) {\r\n",
" Write-Host \"Invalid Input: ResourceGroupName. Resource Group $ResourceGroupName not found\"\r\n",
" return\r\n",
" } \r\n",
" \r\n",
" #Check if default subnet exists under specified Vnet \r\n",
" $existingVnetSubnet = az network vnet subnet show -n $SubNetName --vnet-name $VNetName -g $ResourceGroupName --subscription $Subscription -o JSON |ConvertFrom-Json\r\n",
" if ($existingVnetSubnet){ #Found default subnet under specified VNet\r\n",
" Write-Output \"Found default Subnet under specified VNet. `nSubNet ID '$($existingVnetSubnet.id)'\"\r\n",
" } \r\n",
" else { \r\n",
" #VNet or defaut subnet not found under specified resource group. Create new VNet with default Subnet /Add default subnet to existing VNet\r\n",
" Write-Output \"Creating new Virtual network with default Subnet ID ... \"\r\n",
" $newVNet = az network vnet create --name \"$Env:BOOTSTRAP_VNET\" --resource-group $Env:BOOTSTRAP_ResourceGroup --subscription $Env:BOOTSTRAP_Subscription --subnet-name $SubNetName -o JSON |ConvertFrom-Json #vnet create/Update command: Bug: In this command, the output variable is not getting converted to PS objects.\r\n",
" $newVNet = az network vnet subnet show -g $Env:BOOTSTRAP_ResourceGroup --vnet-name $Env:BOOTSTRAP_VNET -n $SubNetName --subscription $Env:BOOTSTRAP_Subscription -o JSON |ConvertFrom-Json # added this line due to above bug\r\n",
" Write-Output \"Created VNet with default Subnet - ID: '$($newVNet.id)'\"\r\n",
" }\r\n",
"}\r\n",
"\r\n",
"function Bootstrap-AzStorageAccount\r\n",
"{ \r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$StorageAccountName,\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$StorageAccountKind = \"StorageV2\",\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$StorageAccountSKU = \"Standard_LRS\" \r\n",
" )\r\n",
"\r\n",
" $ListStorageAccountKind = \"BlobStorage\", \"BlockBlobStorage\", \"FileStorage\", \"Storage\", \"StorageV2\"\r\n",
" $ListStorageAccountSKU = \"Premium_LRS\", \"Premium_ZRS\", \"Standard_GRS\", \"Standard_GZRS\", \"Standard_LRS\", \"Standard_RAGRS\", \"Standard_RAGZRS\", \"Standard_ZRS\"\r\n",
"\r\n",
" \r\n",
" #Validating parameter $ResourceGroupName\r\n",
" $specified_RG = az group show -n $ResourceGroupName --subscription $Subscription 2>$null\r\n",
" if(!$specified_RG) {\r\n",
" Write-Host \"Invalid Input: ResourceGroupName. Resource Group $ResourceGroupName not found\"\r\n",
" return\r\n",
" } \r\n",
" #Validating parameter $StorageAccountKind\r\n",
" if($StorageAccountKind -notin $ListStorageAccountKind) {\r\n",
" Write-Host \"StorageAccountKind '$StorageAccountKind' not valid. Accepted values: $ListStorageAccountKind\" \r\n",
" return\r\n",
" }\r\n",
" #Validating parameter $StorageAccountSKU\r\n",
" if($StorageAccountSKU -notin $ListStorageAccountSKU) {\r\n",
" Write-Host \"StorageAccountSKU '$StorageAccountSKU' not valid. Accepted values: $ListStorageAccountSKU\" \r\n",
" return\r\n",
" }\r\n",
" \r\n",
" #Check if storage account exists under specified resource group \r\n",
" $existingStorageAccount = az storage account show -n $StorageAccountName -g $ResourceGroupName --subscription $Subscription -o JSON| ConvertFrom-Json\r\n",
"\r\n",
" if ($existingStorageAccount){ #Found specified storage account\r\n",
" Write-Output \"Found storage account '$($existingStorageAccount.name)'\"\r\n",
" } \r\n",
" else { \r\n",
" #Storage account not found under specified resource group. Create new storage account\r\n",
" Write-Output \"Creating new storage account ... \"\r\n",
" $newStorageAccount = az storage account create -n $StorageAccountName --kind $StorageAccountKind --sku $StorageAccountSKU -g $ResourceGroupName --subscription $Subscription -o JSON | ConvertFrom-Json\r\n",
" Write-Output \"Created Storage Account '$($newStorageAccount.name)'\"\r\n",
" }\r\n",
"}\r\n",
"\r\n",
"function Bootstrap-AzFunctionApp\r\n",
"{\r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$FunctionName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$StorageAccountName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$FunctionAppPackageURL,\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$FunctionAppVersion=\"3\",\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ConsumptionPlanLocation \r\n",
" ) \r\n",
" #Validating parameter $ResourceGroupName\r\n",
" $specified_RG = az group show -n $ResourceGroupName --subscription $Subscription 2>$null\r\n",
" if(!$specified_RG) {\r\n",
" Write-Host \"Invalid Input: ResourceGroupName. Resource Group $ResourceGroupName not found\"\r\n",
" return\r\n",
" } \r\n",
" #Validating parameter StorageAccountName\r\n",
" $specified_storageAccount = az storage account show -n $StorageAccountName -g $ResourceGroupName --subscription $Subscription 2>$null\r\n",
" if(!$specified_storageAccount){ \r\n",
" Write-Host \"Invalid Input: StorageAccount name. Storage Account $StorageAccountName not found\" \r\n",
" return\r\n",
" }\r\n",
" #Validating parameter FunctionAppVersion\r\n",
" if($FunctionAppVersion -notin (2,3)){\r\n",
" Write-Host \"FunctionAppVersion '$FunctionAppVersion' not valid. Accepted values: 2,3\" \r\n",
" return\r\n",
" }\r\n",
" #Validating parameter ConsumptionPlanLocation\r\n",
" $specified_Location = az functionapp list-consumption-locations --query \"[?name == '$ConsumptionPlanLocation']\" -o JSON | ConvertFrom-Json \r\n",
" if(!$specified_Location) { \r\n",
" Write-Host \"ComsumptionPlanLocation '$ConsumptionPlanLocation' not valid. Refer below for available options: `n\" \r\n",
" az functionapp list-consumption-locations --query \"[].{ConsumptionPlanLocations: name}\" -o table\r\n",
" return\r\n",
" }\r\n",
" \r\n",
" #Check if functionapp exists within the specified resource group\r\n",
" $existingFunctionApp = az functionapp show -n $FunctionName -g $ResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"\r\n",
" if ($existingFunctionApp){ #ensure that function application is linked to the specified storage account\r\n",
"\r\n",
" #retrieve connection string of storage account linked to FunctionApp and check if the connected storage account is equal to the storage account passed to here\r\n",
" $funcApp_Storage_ConnString= az functionapp config appsettings list -n $FunctionName -g $ResourceGroupName --subscription $Subscription --query \"[?name == 'AzureWebJobsStorage'].value\" -o JSON 2>$null | ConvertFrom-Json\r\n",
" $connectionString = Get-ConnectionStringProperties -ConnectionString \"$funcApp_Storage_ConnString\"\r\n",
" $isConnected = $connectionString.AccountName -eq $StorageAccountName\r\n",
"\r\n",
" if($isConnected) { \r\n",
" Write-Output \"Found FunctionApp '$FunctionName'. `nConnected storage account is '$StorageAccountName'\"\r\n",
" } \r\n",
" else {\r\n",
" Write-Output \"Found FunctionApp '$FunctionName'. `nConnected storage account is NOT '$StorageAccountName', but '$($connectionString.AccountName)' \"\r\n",
" }\r\n",
" } \r\n",
" else { #Provision FunctionApp \r\n",
"\r\n",
" $newFuncApp= az functionapp create --name $FunctionName --resource-group $ResourceGroupName --subscription $Subscription --storage-account $StorageAccountName --functions-version $FunctionAppVersion --consumption-plan-location $ConsumptionPlanLocation -o JSON | ConvertFrom-JSON \r\n",
" Write-Output \"`nCreated Function Application '$($newFuncApp.name)'\"\r\n",
" $AppInsightsInstrumentationKey = az functionapp config appsettings list --name $FunctionName --resource-group $ResourceGroupName --subscription $Subscription --query \"[?name == 'APPINSIGHTS_INSTRUMENTATIONKEY'].value\" -o JSON | ConvertFrom-JSON \r\n",
" az functionapp config appsettings set --name $FunctionName --settings \"WEBSITE_RUN_FROM_PACKAGE=$FunctionAppPackageURL\" \"APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=$AppInsightsInstrumentationKey\" `\r\n",
" --resource-group $ResourceGroupName --subscription $Subscription -o none \r\n",
" Write-Output \"Set '$($newFuncApp.name)' to run from package '$FunctionAppPackageURL'\"\r\n",
" az functionapp identity assign -n $FunctionName -g $ResourceGroupName --subscription $Subscription -o none \r\n",
" Write-Output \"Enabled system assigned managed identity for '$($newFuncApp.name)'\" \r\n",
" }\r\n",
"}\r\n",
"\r\n",
"function Bootstrap-AzBatchAccount\r\n",
"{\r\n",
" [CmdletBinding()]\r\n",
" param(\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Subscription,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BatchAccountLocation,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$BatchAccountName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$StorageAccountName,\r\n",
" [Parameter (Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$ApplicationPackageURL,\r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$ApplicationName=\"SqlPackageWrapper\", \r\n",
" [Parameter (Mandatory=$false)] [ValidateNotNullOrEmpty()] [string]$ApplicationPackageVersionName=\"1\" \r\n",
" )\r\n",
" #Validating parameter $ResourceGroupName\r\n",
" $specified_RG = az group show -n $ResourceGroupName --subscription $Subscription 2>$null\r\n",
" if(!$specified_RG) {\r\n",
" Write-Host \"Invalid Input: ResourceGroupName. Resource Group $ResourceGroupName not found\"\r\n",
" return\r\n",
" } \r\n",
" #Validating parameter StorageAccountName\r\n",
" $specified_storageAccount = az storage account show -n $StorageAccountName -g $ResourceGroupName --subscription $Subscription 2>$null\r\n",
" if(!$specified_storageAccount){ \r\n",
" Write-Host \"Invalid Input: StorageAccount name. Storage Account $StorageAccountName not found\" \r\n",
" return\r\n",
" }\r\n",
" #Validating parameter $BatchAccountLocation\r\n",
" $specified_BLocation = az account list-locations --query \"[?name == '$BatchAccountLocation']\" -o JSON | ConvertFrom-Json\r\n",
" if(!$specified_BLocation) {\r\n",
" Write-Host \"BatchAccountLocation '$BatchAccountLocation' not found. Refer below for the list of locations for the account:`n\" \r\n",
" az account list-locations --query \"[].{Locations: name}\" -o table\r\n",
" return\r\n",
" } \r\n",
"\r\n",
" $SQLPackageWrapperLocal = \"$Env:USERPROFILE\" + \"\\Downloads\\batchApplicationPackage.zip\" #path to download application package\r\n",
" $downloaded= Test-Path $SQLPackageWrapperLocal \r\n",
"\r\n",
" #check if Batch account exists in the specified resource group\r\n",
" $existingBatchAccount = az batch account show -n $BatchAccountName -g $ResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"\r\n",
" if ($existingBatchAccount){ #Batch Account exists, \r\n",
" Write-output \"Found Batch Account '$BatchAccountName'\"\r\n",
"\r\n",
" #checking if storage account is set for Batch Account\r\n",
" #$existingStorageAccount = az storage account show -n $StorageAccountName -g $ResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json # check if storage account set is ADP Storage A/c\r\n",
" $connectedToStorage= $existingBatchAccount.autoStorage #.storageAccountId -eq $existingStorageAccount.id \r\n",
"\r\n",
" # check if latest application package is installed \r\n",
" $appPackageInstalled = az batch application package show --application-name $ApplicationName --version-name $ApplicationPackageVersionName -n $BatchAccountName -g $ResourceGroupName --subscription $Subscription -o JSON 2>$null | ConvertFrom-Json\r\n",
"\r\n",
" if($connectedToStorage -and $appPackageInstalled){ # BatchAccount connected to specified storageaccount and latest applicationpackage is installed\r\n",
" $connectedStorageAccName = $existingBatchAccount.autoStorage.storageAccountId.Split(\"/\")[-1]\r\n",
" Write-Output \"Already connected to Storage account '$connectedStorageAccName' `nFound latest application package installed. `nReusing existing Resources\"\r\n",
" }\r\n",
" else {\r\n",
" if(!$connectedToStorage){ # Connect to ADPStorageAccount \r\n",
"\r\n",
" az batch account set --storage-account $StorageAccountName -n $BatchAccountName --resource-group $ResourceGroupName --subscription $Subscription -o none | ConvertFrom-JSON \r\n",
" Write-Output \"No Storage Account Found. Connected to storage account '$StorageAccountName'\"\r\n",
" }\r\n",
" if(!$appPackageInstalled) { #application package is not installed, installing application package \r\n",
" if(!$downloaded) { Invoke-WebRequest -Uri $ApplicationPackageURL -OutFile $SQLPackageWrapperLocal } #downloads package to current user's downloads folder\r\n",
" az batch application package create --application-name $ApplicationName --version-name $ApplicationPackageVersionName --package-file $SQLPackageWrapperLocal --name $BatchAccountName --resource-group $ResourceGroupName --subscription $Subscription -o none\r\n",
" Write-Output \"No Application package found. Installed application package under '$ApplicationName'\"\r\n",
" }\r\n",
" Write-Output \"Batch Account '$BatchAccountName' is provisioned as required\"\r\n",
" }\r\n",
" } else { #Provision Batch Account\r\n",
"\r\n",
" $newBatchAccount = az batch account create -n $BatchAccountName -l $BatchAccountLocation --resource-group $ResourceGroupName --subscription $Subscription -o JSON | ConvertFrom-JSON\r\n",
" Write-Output \"Created Batch Account '$($newBatchAccount.name)'\" \r\n",
" az batch account set --storage-account $StorageAccountName -n $BatchAccountName --resource-group $ResourceGroupName --subscription $Subscription -o none \r\n",
" Write-Output \"Set Storage Account to '$StorageAccountName'\" \r\n",
" if(!$downloaded) { Invoke-WebRequest -Uri $ApplicationPackageURL -OutFile $SQLPackageWrapperLocal } #downloads package to current user's downloads folder\r\n",
" az batch application package create --application-name $ApplicationName --version-name $ApplicationPackageVersionName --package-file $SQLPackageWrapperLocal --name $BatchAccountName --resource-group $ResourceGroupName --subscription $Subscription -o none \r\n",
" Write-Output \"Installed application package '$ApplicationName'\" \r\n",
" \r\n",
" } \r\n",
"}\r\n",
"\r\n",
"function Bootstrap-FuncRBAC {\r\n",
" [CmdletBinding()]\r\n",
" param (\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$Subscription,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$ResourceGroupName,\r\n",
" [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()][string]$AzFunctionName,\r\n",
" [Parameter(Mandatory=$false)][ValidateNotNullOrEmpty()][string]$Role=\"Contributor\"\r\n",
" )\r\n",
"\r\n",
" # Get the resource group's scope\r\n",
" $rg = az group show --resource-group $ResourceGroupName --subscription $Subscription | ConvertFrom-Json\r\n",
" if (!$rg) {\r\n",
" Write-Error \"Failed to setup RBAC for $AzFunctionName... Resource Group '$ResourceGroupName' not found on subscription '$Subscription'\"\r\n",
" exit\r\n",
" }\r\n",
"\r\n",
" $scope = $rg.id\r\n",
" Write-Output \"Found scope... $scope\"\r\n",
" \r\n",
" # Get the az function principal id\r\n",
" $app = az functionapp show -n $AzFunctionName --resource-group $ResourceGroupName --subscription $Subscription | ConvertFrom-Json\r\n",
" if (!$app) {\r\n",
" Write-Error \"Failed to setup RBAC for $AzFunctionName... Az function not found!\"\r\n",
" exit\r\n",
" }\r\n",
"\r\n",
" $prid = $app.identity.principalId\r\n",
" Write-Output \"Found principal id of Azure function... $prid\"\r\n",
"\r\n",
" # Verify if a role assignment has been created\r\n",
" $assignments = az role assignment list --subscription $Subscription -g $ResourceGroupName | ConvertFrom-Json\r\n",
"\r\n",
" # Get the correct assignment\r\n",
" $svcPrincipal = $assignments | Where-Object {($_.scope -eq $scope) -and ($_.principalType -eq \"ServicePrincipal\") -and ($_.principalId -eq $prid)}\r\n",
" $id = $svcPrincipal.id\r\n",
" $def = $svcPrincipal.roleDefinitionName\r\n",
"\r\n",
" Write-Output \"Found service principal role... '$def'\" \r\n",
" # Check if the right role is assigned\r\n",
" if ($def -eq $Role)\r\n",
" { \r\n",
" Write-Output \"Service Principle '$prid' found with '$Role' role. No work needed\"\r\n",
" return\r\n",
" }\r\n",
"\r\n",
" # Continue to setup RBAC, once we verify an assignment is not setup and all the resources exist\r\n",
" Write-Output \"Creating new role assignment by running: 'az functionapp identity assign -g $ResourceGroupName -n $AzFunctionName --scope $scope --subscription $Subscription'\"\r\n",
" Write-Warning \"If your account does not have the access to assign new roles as Owner or User Access Administrator for the resource group, than you will need to contact your Azure AD Administrator to assign a service principle using the commands above\" \r\n",
" az functionapp identity assign -g $ResourceGroupName -n $AzFunctionName --scope $scope --subscription $Subscription \r\n",
"}\r\n",
"\r\n",
"Write-Host \"Helper Functions Created successfully\" "
],
"metadata": {
"azdata_cell_guid": "f524afff-ccca-4e19-94a1-dbe21aee2241",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Connect to Azure and Select Subscription\r\n",
"Run the below cell to login to an Azure account. <b>Be sure to check the Windows Taskbar for a subscription selection dialog box.</b>\r\n",
"\r\n",
"_Note: the dialog box window may appear behind the active Azure Data Studio window._"
],
"metadata": {
"azdata_cell_guid": "4952284c-dec7-4df3-9a31-044d795ef279"
}
},
{
"cell_type": "code",
"source": [
"Login-Azure"
],
"metadata": {
"azdata_cell_guid": "112d7a12-f66d-41c0-8a1c-0b234506d5ef",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Create Resource Group\r\n",
"The Data Portability Notebooks provision several resources to enable import and export of Azure SQL DB databases and Azure SQL Managed Instance databases. These resources include Storage, Azure Batch, Azure Function resources and their dependent Azure resources. All resources created by this notebook are associated with a common Azure Resource Group to enable easy management of resources related to Data Portability.\r\n",
"\r\n",
"_Note: this notebook will reuse the Resource Group specified if it already exists._"
],
"metadata": {
"azdata_cell_guid": "8298d187-3021-485c-b4d4-442da9aadd53"
}
},
{
"cell_type": "code",
"source": [
"Bootstrap-AzResourceGroup -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -ResourceGroupLocation $Env:BOOTSTRAP_RG_REGION -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "9beb8d22-4560-4c7e-917b-5a3c0d58e1a2",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Provision Virtual Network\r\n",
"\r\n",
"The Data Portability resources include Virtual Machines created to execute the database import and export operations. To provision this in a secure mode, this notebook will provision a Virtual Network to attach the Virtual Machines to. These VMs do not need to be directly accessed so no publicly addressable ports should be created."
],
"metadata": {
"azdata_cell_guid": "8fd75088-78fa-4cf3-b0f2-166813d0bed1"
}
},
{
"cell_type": "code",
"source": [
"Bootstrap-AzVirtualNetwork -VNetName $Env:BOOTSTRAP_VNET -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "d014a6a6-57ff-4de7-8210-b3360bf34daa"
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Provision Storage Account\r\n",
"\r\n",
"The Data Portability resources require a Storage Account to attach the Virtual Machines to. This storage account is used for maintaining log files and other Azure Batch related information. This Storage Account can be the same or different than the Storage Acount used during Import and Export operations to store the logical database (.bacpac) files."
],
"metadata": {
"azdata_cell_guid": "4e91a3f7-a676-4a4e-9d6f-3a343fcc0bc0"
}
},
{
"cell_type": "code",
"source": [
"Bootstrap-AzStorageAccount -StorageAccountName $Env:BOOTSTRAP_STORAGE -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "290498ee-3f31-4395-adab-a5fa93d28c80",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Provision Data Portability Orchestrator\r\n",
"\r\n",
"The Data Portability resources include an orchestration component which coordinates the import and export operations. The orchestrator is built as an Azure Function and provides the REST APIs that the Import and Export notebooks use to either import or export an Azure SQL DB Server and its databases."
],
"metadata": {
"azdata_cell_guid": "d1291c22-e5b7-42b3-bdd9-8a8d13fd3165"
}
},
{
"cell_type": "code",
"source": [
"Bootstrap-AzFunctionApp -FunctionName $Env:BOOTSTRAP_FUNC -StorageAccountName $Env:BOOTSTRAP_STORAGE -FunctionAppPackageURL $Env:BOOTSTRAP_URL_FUNC `\r\n",
" -ConsumptionPlanLocation $Env:BOOTSTRAP_RG_REGION -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "6fc2b5ec-c16f-4eb7-b2f9-c8c680d9a2df",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Provision Hosted SqlPackage Application\r\n",
"To Data Portability solution leverages SqlPackage and DacFx to perform the import and export of each database. To host SqlPackage the solution uses an Azure Batch application. Azure Batch provides the ability to create a pool of Virtual Machines that can perform the import and export operations. This notebook will provision a pool of 2 virutal machines. The machines are not actually allocated until the first Import or Export operation is performed using the Import or Export notebooks. "
],
"metadata": {
"azdata_cell_guid": "126a5abd-a3f4-46e0-b311-8cd1b8639127"
}
},
{
"cell_type": "code",
"source": [
"Bootstrap-AzBatchAccount -BatchAccountName $Env:BOOTSTRAP_BATCH -StorageAccountName $Env:BOOTSTRAP_STORAGE -BatchAccountLocation $Env:BOOTSTRAP_RG_REGION `\r\n",
" -ApplicationPackageURL $Env:BOOTSTRAP_URL_WRAP -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "489733c4-1162-479b-82b4-b0c18954b25b",
"tags": []
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"## Grant Data Portability Controller Access to Data Portability Resources\r\n",
"The Data Portability orchestrator requires Contributor access to the other Data Portability resources created. To do this, Azure RBAC (role based access control) is used to grant access to the orchestrator so that it can communicate with the hosted SqlPackage application. The function below will verify a Service Principal was created with the proper scope or attempt to create the role assignment. Any user executing this notebook will need to have Owner or User Access Administrator permissions to the Resource Group to assign the permission. Otherwise, contact your Azure AD Administrator. "
],
"metadata": {
"azdata_cell_guid": "3d03a987-4bec-4c0b-afdb-e6ae3c828252"
}
},
{
"cell_type": "code",
"source": [
"Bootstrap-FuncRBAC -AzFunctionName $Env:BOOTSTRAP_FUNC -ResourceGroupName $Env:BOOTSTRAP_ResourceGroup -Subscription $Env:BOOTSTRAP_Subscription"
],
"metadata": {
"azdata_cell_guid": "75882d3a-2004-4304-ab8f-e5146e14500c",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
}
]
}