Files
azuredatastudio/extensions/asde-deployment/notebooks/edge/deploy-sql-edge-azure-sshkey.ipynb
2021-03-12 10:53:02 -08:00

486 lines
25 KiB
Plaintext

{
"metadata": {
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"language_info": {
"name": "python",
"version": "3.6.6",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
}
},
"nbformat_minor": 2,
"nbformat": 4,
"cells": [
{
"cell_type": "markdown",
"source": [
"![Microsoft](https://raw.githubusercontent.com/microsoft/azuredatastudio/main/extensions/resource-deployment/images/microsoft-small-logo.png)\n",
"## Deploy Azure SQL Edge to an Azure VM via IoT hub\n",
"This notebook will walk you through the end to end setup of Azure SQL Edge.\n",
"1. Create an Azure Edge VM as a virtual IoT device, you can go the \"Default Settings\" cell and adjust the vm_size variable based on your needs. Available sizes and pricing information can be found [here](https://docs.microsoft.com/azure/virtual-machines/linux/sizes).\n",
"1. Create an Azure IoT hub, you can go to the \"Default Settings\" cell and adjust value of the following variables based on your needs: iot_hub_sku and iot_hub_units. Available SKUs and pricing information can be found [here](https://azure.microsoft.com/pricing/details/iot-hub/).\n",
"1. Add the device to the IoT hub\n",
"1. Deploy SQL Edge module to the device with optional package file\n",
"1. Enable connecting to the SQL Edge instance on the device\n",
"\n",
"### Dependencies\n",
"- Azure CLI. For more information, see [Azure CLI Installation](https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest).\n",
"\n",
"<span style=\"color:red\"><font size=\"3\">Please press the \"Run all\" button to run the notebook</font></span>"
],
"metadata": {
"azdata_cell_guid": "15b8cfc7-dd7f-4db8-9a3c-2151932fe7b5"
}
},
{
"cell_type": "markdown",
"source": [
"### Check dependencies"
],
"metadata": {
"azdata_cell_guid": "f78f4ff3-d4c9-4c3e-853f-4add05061eb0"
}
},
{
"cell_type": "code",
"source": [
"import sys,os,json,html,getpass,time,ntpath,uuid\n",
"\n",
"def run_command(command:str, displayCommand:str = \"\", returnObject:bool = False):\n",
" print(\"Executing: \" + (displayCommand if displayCommand != \"\" else command))\n",
" if returnObject:\n",
" output = os.popen(command).read()\n",
" print(f'Command successfully executed')\n",
" return json.loads(''.join(output))\n",
" else:\n",
" !{command}\n",
" if _exit_code != 0:\n",
" sys.exit(f'Command execution failed with exit code: {str(_exit_code)}.\\n')\n",
" else:\n",
" print(f'Command successfully executed')\n",
"\n",
"run_command(command='az --version')"
],
"metadata": {
"azdata_cell_guid": "70b9744f-eb59-44e8-9b35-db590ac4651d",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Install Azure IoT extension for Azure CLI"
],
"metadata": {
"azdata_cell_guid": "a7f15c68-1725-4caa-b4f7-ddc2b4934883"
}
},
{
"cell_type": "code",
"source": [
"extensions = run_command('az extension list', returnObject=True)\r\n",
"extensions = [ext for ext in extensions if ext['name'] == 'azure-cli-iot-ext']\r\n",
"if len(extensions) > 0:\r\n",
" run_command('az extension remove --name azure-cli-iot-ext')\r\n",
"run_command('az extension add --name azure-iot')"
],
"metadata": {
"azdata_cell_guid": "55bb2f96-6f7f-4aa0-9daf-d0f7f9d9243c",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Required information"
],
"metadata": {
"azdata_cell_guid": "b5dc5586-06e8-44d9-8bc3-2861d510efe5"
}
},
{
"cell_type": "code",
"source": [
"azure_subscription_id = os.environ[\"AZDATA_NB_VAR_ASDE_SUBSCRIPTIONID\"]\n",
"azure_resource_group = os.environ[\"AZDATA_NB_VAR_ASDE_RESOURCEGROUP\"]\n",
"azure_location = os.environ[\"AZDATA_NB_VAR_ASDE_AZURE_LOCATION\"]\n",
"sa_password = os.environ[\"AZDATA_NB_VAR_SA_PASSWORD\"]\n",
"vm_admin = os.environ[\"AZDATA_NB_VAR_ASDE_VM_ADMIN\"]\n",
"vm_password = os.environ[\"AZDATA_NB_VAR_ASDE_VM_PASSWORD\"]\n",
"package_path = os.environ[\"AZDATA_NB_VAR_ASDE_PACKAGE_PATH\"]\n",
"sql_port = os.environ[\"AZDATA_NB_VAR_ASDE_SQL_PORT\"]\n",
"new_rg_flag = os.environ[\"AZDATA_NB_VAR_ASDE_NEW_RESOURCEGROUP\"]\n",
"new_rg_name = os.environ[\"AZDATA_NB_VAR_ASDE_NEW_RESOURCEGROUP_NAME\"]\n",
"\n",
"if new_rg_flag == 'true':\n",
" azure_resource_group = new_rg_name\n",
"print(f'Subscription: {azure_subscription_id}')\n",
"print(f'Resource group: {azure_resource_group}')\n",
"print(f'Location: {azure_location}')\n",
"print(f'VM admin username: {vm_admin}')\n",
"print(f'VM admin password: ******')\n",
"print(f'SQL Server port: {sql_port}')\n",
"print(f'SQL Server sa password: ******')\n",
"print(f'Package path: {package_path}')"
],
"metadata": {
"azdata_cell_guid": "dde9388b-f623-4d62-bb74-36a05f5d2ea3",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Default settings"
],
"metadata": {
"azdata_cell_guid": "2a5755eb-85a7-4237-8d87-04cdab13cf40"
}
},
{
"cell_type": "code",
"source": [
"suffix = time.strftime(\"%y%m%d%H%M%S\", time.localtime())\n",
"iot_hub_name = f'hub{suffix}'\n",
"iot_hub_sku = 'S1'\n",
"iot_hub_units = 4\n",
"iot_device_id = f'vm{suffix}'\n",
"azure_storage_account = f'sa{suffix}'\n",
"storage_account_container = 'sqldatabasepackage'\n",
"sql_lcid = '1033'\n",
"sql_collation = 'SQL_Latin1_General_CP1_CI_AS'\n",
"vm_size = 'Standard_DS1_v2'"
],
"metadata": {
"azdata_cell_guid": "19ebeaf4-94c9-4d2b-bd9f-e3c6bf7f2dda",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Login to Azure"
],
"metadata": {
"azdata_cell_guid": "84f57c09-5772-4f7a-a270-4039b8d5b081"
}
},
{
"cell_type": "code",
"source": [
"run_command('az login')"
],
"metadata": {
"azdata_cell_guid": "f9e8ddee-aefa-4951-b767-b318d941d2cd",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Set active Azure subscription"
],
"metadata": {
"azdata_cell_guid": "59249fa6-f76c-4e5d-bee7-a9ebef6f873e"
}
},
{
"cell_type": "code",
"source": [
"if azure_subscription_id != \"\":\n",
" run_command(f'az account set --subscription {azure_subscription_id}')\n",
"else:\n",
" print('Using the default Azure subscription', {azure_subscription_id})\n",
"run_command(f'az account show')"
],
"metadata": {
"azdata_cell_guid": "6e085676-2cc5-4af8-819c-fa210244e6c3",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Create resource group"
],
"metadata": {
"azdata_cell_guid": "67dacbaa-92f4-4d06-90bb-8974964852aa"
}
},
{
"cell_type": "code",
"source": [
"rg_exists = run_command(f'az group exists --name {azure_resource_group}', returnObject=True)\n",
"\n",
"if rg_exists:\n",
" print(f'resource group \\\"{azure_resource_group}\\\" already exists.')\n",
"else:\n",
" run_command(f'az group create --location {azure_location} --name {azure_resource_group}')"
],
"metadata": {
"azdata_cell_guid": "f29b439e-cf05-4c35-aa47-1482ccd653bf",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Create IoT hub"
],
"metadata": {
"azdata_cell_guid": "e37a04c3-515d-4cb7-99b2-f8bc6167510e"
}
},
{
"cell_type": "code",
"source": [
"hub_list = run_command(f'az iot hub list --resource-group {azure_resource_group}', returnObject=True)\n",
"hub_list = [hub for hub in hub_list if hub['name'] == iot_hub_name]\n",
"if len(hub_list) == 0:\n",
" run_command(f'az iot hub create --name {iot_hub_name} --resource-group {azure_resource_group} --location {azure_location} --sku {iot_hub_sku} --unit {iot_hub_units}')\n",
"else:\n",
" print(f'IoT hub \\\"{iot_hub_name}\\\" already exists')"
],
"metadata": {
"azdata_cell_guid": "f9f5e4ec-82a5-45df-a408-ddb0fb21847c",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Create storage account and storage account container, then upload the package"
],
"metadata": {
"azdata_cell_guid": "90ec2b26-0c4a-4aa4-b397-f16b09b454ea"
}
},
{
"cell_type": "code",
"source": [
"storage_account_created = False\n",
"if package_path == \"\":\n",
" print(f'Package file not provided')\n",
" blob_sas = ''\n",
"else: \n",
" package_name = ntpath.basename(package_path)\n",
" storage_accounts = run_command(f'az storage account list --resource-group {azure_resource_group} --subscription {azure_subscription_id}', returnObject=True)\n",
" storage_accounts = [storage_account for storage_account in storage_accounts if storage_account['name'] == azure_storage_account]\n",
" if len(storage_accounts) == 0:\n",
" storage_account_created = True\n",
" run_command(f'az storage account create -n {azure_storage_account} -g {azure_resource_group} -l {azure_location} --sku Standard_LRS --kind Storage')\n",
" else:\n",
" print(f'storage account \\\"{azure_storage_account}\\\" already exists.')\n",
"\n",
" storage_account_key = run_command(f'az storage account keys list --account-name {azure_storage_account} --resource-group {azure_resource_group}', returnObject=True)[0]['value']\n",
" container_exists = run_command(f'az storage container exists --name {storage_account_container} --account-key {storage_account_key} --account-name {azure_storage_account} --auth-mode key --output json', returnObject=True)['exists']\n",
" if container_exists:\n",
" print(f'storage account container \\\"{storage_account_container}\\\" already exists.')\n",
" else:\n",
" run_command(f'az storage container create --name {storage_account_container} --account-key {storage_account_key} --account-name {azure_storage_account} --auth-mode key')\n",
"\n",
" blob_exists = run_command(f'az storage blob exists --container-name {storage_account_container} --name \\\"{package_name}\\\" --account-key {storage_account_key} --account-name {azure_storage_account} --auth-mode key', returnObject=True)['exists']\n",
" if blob_exists:\n",
" print(f'blob \\\"{package_name}\\\" already exists.')\n",
" else:\n",
" run_command(f'az storage blob upload --account-name {azure_storage_account} --container-name {storage_account_container} --name {package_name} --file \\\"{package_path}\\\" --account-key {storage_account_key} --auth-mode key')\n",
" now = time.localtime()\n",
" expiry = f'{(now.tm_year + 1)}-{now.tm_mon}-{now.tm_mday}'\n",
" blob_sas = run_command(f'az storage blob generate-sas --container-name {storage_account_container} --name \\\"{package_name}\\\" --account-name {azure_storage_account} --account-key {storage_account_key} --auth-mode key --full-uri --https-only --permissions r --expiry {expiry}', returnObject=True)"
],
"metadata": {
"azdata_cell_guid": "7ab2b3ec-0832-40b3-98c0-4aa87320e7ce",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Add the Edge device to the IoT hub"
],
"metadata": {
"azdata_cell_guid": "fbc5f4ac-dfe0-4543-ace1-49b796251910"
}
},
{
"cell_type": "code",
"source": [
"device_list = run_command(f'az iot hub device-identity list --edge-enabled true --hub-name {iot_hub_name} --resource-group {azure_resource_group}', returnObject=True)\n",
"device_list = [device for device in device_list if device['deviceId'] == iot_device_id]\n",
"if len(device_list) == 0:\n",
" run_command(f'az iot hub device-identity create --device-id {iot_device_id} --hub-name {iot_hub_name} --resource-group {azure_resource_group} --edge-enabled true')\n",
"else:\n",
" print(f'Edge device \\\"{iot_device_id}\\\" already exists.')\n",
"connection_string = run_command(f'az iot hub device-identity show-connection-string --device-id {iot_device_id} --hub-name {iot_hub_name} --resource-group {azure_resource_group}', returnObject=True)\n",
"connection_string = connection_string['connectionString']"
],
"metadata": {
"azdata_cell_guid": "c183c3e3-8699-4f29-993b-07bf848336e3",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Create an Edge enabled VM as an Edge device"
],
"metadata": {
"azdata_cell_guid": "41b10249-cd40-4053-b1b0-b02f562789f7"
}
},
{
"cell_type": "code",
"source": [
"iot_deploy_result = run_command((f\"az deployment group create \"\r\n",
"f\"--resource-group {azure_resource_group} \"\r\n",
"f\"--template-uri \\\"https://aka.ms/iotedge-vm-deploy\\\" \"\r\n",
"f\"--parameters vmSize={vm_size} \"\r\n",
"f\"--parameters dnsLabelPrefix={iot_device_id} \"\r\n",
"f\"--parameters adminUsername={vm_admin} \"\r\n",
"f\"--parameters deviceConnectionString={connection_string} \"\r\n",
"f\"--parameters authenticationType=sshPublicKey \"\r\n",
"f\"--parameters adminPasswordOrKey=\\\"{vm_password}\\\"\"), returnObject=True)\r\n",
"vm_resource = [resource for resource in iot_deploy_result['properties']['dependencies'] if resource['resourceType'] == 'Microsoft.Compute/virtualMachines']\r\n",
"if len(vm_resource) != 1:\r\n",
" sys.exit('Failed to deploy the IoT Edge VM')\r\n",
"vm_name = vm_resource[0]['resourceName']\r\n",
"nsg_name = vm_name.replace('vm-','nsg-')\r\n",
"ip_address = run_command(f'az vm show -d -g {azure_resource_group} -n {vm_name} --query publicIps', returnObject=True)\r\n",
"run_command(f'az network nsg rule create --name \\\"SQL\\\" --nsg-name {nsg_name} --priority 100 --resource-group {azure_resource_group} --access Allow --description \\\"Allow SQL\\\" --destination-address-prefixes \\\"*\\\" --destination-port-ranges {sql_port} --direction Inbound --source-address-prefixes Internet --protocol Tcp')"
],
"metadata": {
"azdata_cell_guid": "c8590c65-b274-460d-9659-97e81d2fd3ea",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### Deploy Azure SQL Edge to the device"
],
"metadata": {
"azdata_cell_guid": "ec46957f-0795-4c75-804d-f8a7ecb26382"
}
},
{
"cell_type": "code",
"source": [
"manifest = '{\\\"modulesContent\\\":{\\\"$edgeAgent\\\":{\\\"properties.desired\\\":{\\\"modules\\\":{\\\"AzureSQLEdge\\\":{\\\"settings\\\":{\\\"image\\\":\\\"mcr.microsoft.com/azure-sql-edge\\\",\\\"createOptions\\\":\\\"{\\\\\\\"HostConfig\\\\\\\":{\\\\\\\"CapAdd\\\\\\\":[\\\\\\\"SYS_PTRACE\\\\\\\"],\\\\\\\"Binds\\\\\\\":[\\\\\\\"sqlvolume:/sqlvolume\\\\\\\"],\\\\\\\"PortBindings\\\\\\\":{\\\\\\\"1433/tcp\\\\\\\":[{\\\\\\\"HostPort\\\\\\\":\\\\\\\"<SQL_Port>\\\\\\\"}]},\\\\\\\"Mounts\\\\\\\":[{\\\\\\\"Type\\\\\\\":\\\\\\\"volume\\\\\\\",\\\\\\\"Source\\\\\\\":\\\\\\\"sqlvolume\\\\\\\",\\\\\\\"Target\\\\\\\":\\\\\\\"/var/opt/mssql\\\\\\\"}]},\\\\\\\"User\\\\\\\":\\\\\\\"0:0\\\\\\\",\\\\\\\"Env\\\\\\\":[\\\\\\\"MSSQL_AGENT_ENABLED=TRUE\\\\\\\",\\\\\\\"ClientTransportType=AMQP_TCP_Only\\\\\\\",\\\\\\\"PlanId=asde-developer-on-iot-edge\\\\\\\"]}\\\"},\\\"type\\\":\\\"docker\\\",\\\"version\\\":\\\"1.0\\\",\\\"env\\\":{\\\"ACCEPT_EULA\\\":{\\\"value\\\":\\\"Y\\\"},\\\"SA_PASSWORD\\\":{\\\"value\\\":\\\"<Default_SQL_SA_Password>\\\"},\\\"MSSQL_LCID\\\":{\\\"value\\\":\\\"<SQL_LCID>\\\"},\\\"MSSQL_COLLATION\\\":{\\\"value\\\":\\\"<SQL_Collation>\\\"}<PACKAGE_INFO>},\\\"status\\\":\\\"running\\\",\\\"restartPolicy\\\":\\\"always\\\"}},\\\"runtime\\\":{\\\"settings\\\":{\\\"minDockerVersion\\\":\\\"v1.25\\\"},\\\"type\\\":\\\"docker\\\"},\\\"schemaVersion\\\":\\\"1.0\\\",\\\"systemModules\\\":{\\\"edgeAgent\\\":{\\\"settings\\\":{\\\"image\\\":\\\"mcr.microsoft.com/azureiotedge-agent:1.0\\\",\\\"createOptions\\\":\\\"\\\"},\\\"type\\\":\\\"docker\\\"},\\\"edgeHub\\\":{\\\"settings\\\":{\\\"image\\\":\\\"mcr.microsoft.com/azureiotedge-hub:1.0\\\",\\\"createOptions\\\":\\\"{\\\\\\\"HostConfig\\\\\\\":{\\\\\\\"PortBindings\\\\\\\":{\\\\\\\"443/tcp\\\\\\\":[{\\\\\\\"HostPort\\\\\\\":\\\\\\\"443\\\\\\\"}],\\\\\\\"5671/tcp\\\\\\\":[{\\\\\\\"HostPort\\\\\\\":\\\\\\\"5671\\\\\\\"}],\\\\\\\"8883/tcp\\\\\\\":[{\\\\\\\"HostPort\\\\\\\":\\\\\\\"8883\\\\\\\"}]}}}\\\"},\\\"type\\\":\\\"docker\\\",\\\"status\\\":\\\"running\\\",\\\"restartPolicy\\\":\\\"always\\\"}}}},\\\"$edgeHub\\\":{\\\"properties.desired\\\":{\\\"routes\\\":{},\\\"schemaVersion\\\":\\\"1.0\\\",\\\"storeAndForwardConfiguration\\\":{\\\"timeToLiveSecs\\\":7200}}},\\\"AzureSQLEdge\\\":{\\\"properties.desired\\\":{\\\"ASAJobInfo\\\":\\\"<Optional_ASA_Job_SAS_URL>\\\"}}}}'\n",
"package_info = '' if blob_sas == ''else ',\\\"MSSQL_PACKAGE\\\":{\\\"value\\\":\\\"'+blob_sas+'\\\"}'\n",
"manifest = manifest.replace('<PACKAGE_INFO>', package_info).replace('<Default_SQL_SA_Password>',sa_password).replace('<SQL_LCID>',sql_lcid).replace('<SQL_Port>',sql_port).replace('<SQL_Collation>',sql_collation)\n",
"file_name = f'{uuid.uuid4().hex}.json'\n",
"manifest_file = open(file_name, 'w')\n",
"manifest_file.write(manifest)\n",
"manifest_file.close()\n",
"run_command(f'az iot edge set-modules --device-id \\\"{iot_device_id}\\\" --hub-name \\\"{iot_hub_name}\\\" --content \\\"{file_name}\\\" --resource-group {azure_resource_group}')\n",
"os.remove(file_name)"
],
"metadata": {
"azdata_cell_guid": "81a86ff6-5a83-48be-8be7-654d152eea89",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"source": [
"### **Connect to Azure SQL Edge instance in Azure Data Studio**\n",
"Click the link below to connect to the Azure SQL Edge instance, it might take a couple minutes for the service to start."
],
"metadata": {
"azdata_cell_guid": "3bdfa537-a749-45c4-b219-57d296c22739"
}
},
{
"cell_type": "code",
"source": [
"from IPython.display import *\n",
"connectionParameter = '{\"serverName\":\"' + f'{ip_address},{sql_port}' + '\",\"providerName\":\"MSSQL\",\"authenticationType\":\"SqlLogin\",\"userName\": \"sa\",\"password\":' + json.dumps(sa_password) + '}'\n",
"display(HTML('<br/><a href=\"command:azdata.connect?' + html.escape(connectionParameter)+'\"><font size=\"3\">Click here to connect to the Azure SQL Edge instance</font></a><br/>'))\n",
"display(HTML('<br/><span style=\"color:red\"><font size=\"2\">NOTE: The Azure SQL Edge instance password is included in this link, you may want to clear the results of this code cell before saving the notebook.</font></span>'))"
],
"metadata": {
"azdata_cell_guid": "8bc29cce-96a7-4a78-89af-5c73a6431c24",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
},
{
"cell_type": "code",
"source": [
"if storage_account_created:\r\n",
" delete_storage_account_command = \"run_command(f'az storage account delete -n {azure_storage_account} -g {azure_resource_group} --yes')\"\r\n",
" display(HTML('<span style=\"color:red\"><font size=\"2\">NOTE: A storage account was created to host the package file, you can delete it after the database is created and populated successfully. To delete the storage account, copy the following code to a new code cell and run the cell.</font></span>'))\r\n",
" display(HTML('<span><font size=\"2\">'+delete_storage_account_command+'</font></span>'))"
],
"metadata": {
"azdata_cell_guid": "8b74ac43-a871-4d28-832d-e6da586f6d3a",
"tags": [
"hide_input"
]
},
"outputs": [],
"execution_count": null
}
]
}