azuredatastudio/extensions/resource-deployment/notebooks/bdc/2019/CTP3-1/deploy-bdc-aks.ipynb

{
    "metadata": {
        "kernelspec": {
            "name": "python3",
            "display_name": "Python 3"
        },
        "language_info": {
            "name": "python",
            "version": "3.6.6",
            "mimetype": "text/x-python",
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "pygments_lexer": "ipython3",
            "nbconvert_exporter": "python",
            "file_extension": ".py"
        }
    },
    "nbformat_minor": 2,
    "nbformat": 4,
    "cells": [
        {
            "cell_type": "markdown",
            "source": "![Microsoft](https://raw.githubusercontent.com/microsoft/azuredatastudio/master/src/sql/media/microsoft-small-logo.png)\n\n## Create Azure Kubernetes Service cluster and deploy SQL Server 2019 big data cluster\n\nThis notebook walks through the process of deploying a SQL Server 2019 big data cluster on Azure Kubernetes Service.\n\n* Please follow the instructions in the **Dependencies** cell to install the dependencies.\n* The **Required information** cell will prompt you for the required information to create a SQL Server 2019 big data cluster.\n* We have included some default settings in the **Azure settings** cell of this Notebook, Please review the default values and make changes accordingly.\n* If you are experiencing issue in the *Create AKS cluster* step you might need to re-run the cell, more information: https://docs.microsoft.com/en-us/azure/aks/kubernetes-service-principal#specify-a-service-principal-for-an-aks-cluster. \n\n",
            "metadata": {}
        },
        {
            "cell_type": "markdown",
            "source": "### **Dependencies**\n<table>\n<colgroup>\n<col style=\"width: 10%\" />\n<col style=\"width: 80%\" />\n<col style=\"width: 10%\" />\n</colgroup>\n<thead>\n<tr class=\"header\">\n<th>Tool</th>\n<th>Description</th>\n<th>Installation</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Azure CLI</strong></td>\n<td>Command-line interface for managing Azure services. Used with AKS big data cluster deployments (<a href=\"https://docs.microsoft.com/cli/azure/?view=azure-cli-latest\">More info</a>).</td>\n<td><a href=\"https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest\">Install</a></td>\n</tr>\n<tr>\n<td><strong>kubectl</strong></td>\n<td>Command-line tool for monitoring the underlying Kuberentes cluster (<a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/\">More info</a>).</td>\n<td><a href=\"https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-using-native-package-management\">Install</a></td>\n</tr>\n<tr>\n<td><strong>mssqlctl</strong></td>\n<td>Command-line tool for installing and managing a big data cluster.</td>\n<td><a href=\"https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-mssqlctl?view=sqlallproducts-allversions\">Install</a></td>\n</tr>\n</tbody>\n</table>\n<p>",
            "metadata": {}
        },
        {
            "cell_type": "markdown",
            "source": "### **Check dependencies**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "#Run command helper function\r\ndef run_command():\r\n    print(\"Executing: \" + cmd)\r\n    !{cmd}\r\n    if _exit_code != 0:\r\n        raise SystemExit(f'Shell command:\\n\\n\\t{cmd}\\n\\nreturned non-zero exit code: ' + str(_exit_code) + '.\\n')\r\n    print(f'Successfully executed: {cmd}')\r\n\r\ncmd = 'az --version'\r\nrun_command()\r\ncmd = 'kubectl version --client=true'\r\nrun_command()\r\ncmd = 'mssqlctl --version'\r\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 1
        },
        {
            "cell_type": "markdown",
            "source": "### **Required information**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "import getpass\ndef get_user_input(input_name, is_password = False, confirm_password = False):\n    if is_password:\n        user_input = getpass.getpass(prompt = input_name)\n        if confirm_password:\n            user_input_confirm = getpass.getpass(prompt = 'Confirm '+ input_name)\n            if user_input != user_input_confirm:\n                raise SystemExit(f'{input_name} does not match the confirmation password')\n        print(f'{input_name}: ******')\n    else:\n        user_input = input(input_name)\n        print(input_name + ': ' + user_input)\n    if user_input == \"\":\n        raise SystemExit(f'{input_name} is required')\n    return user_input;\n\nmssql_cluster_name = get_user_input('Cluster name')\nbdc_controller_username = get_user_input('Controller username')\nbdc_password = get_user_input('Controller password', True, True)\nprint('Knox and SQL Server will use the same password')\ndocker_username = get_user_input('Docker username')\ndocker_password = get_user_input('Docker password', True)",
            "metadata": {},
            "outputs": [],
            "execution_count": 2
        },
        {
            "cell_type": "markdown",
            "source": "### **Azure settings**\n*Subscription ID*: visit <a href=\"https://portal.azure.com/#blade/Microsoft_Azure_Billing/SubscriptionsBlade\">here</a> to find out the subscriptions you can use, if you leave it unspecified, the default subscription will be used.\n\n*VM Size*: visit <a href=\"https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes\">here</a> to find out the available VM sizes you could use. \n \n*Region*: visit <a href=\"https://azure.microsoft.com/en-us/global-infrastructure/services/?products=kubernetes-service\">here</a> to find out the Azure regions where the Azure Kubernettes Service is available.",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "azure_subscription_id = \"\"\nazure_vm_size = \"Standard_E4s_v3\"\nazure_region = \"eastus\"\nazure_vm_count = int(5)",
            "metadata": {},
            "outputs": [],
            "execution_count": 3
        },
        {
            "cell_type": "markdown",
            "source": "### **Default settings**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "azure_resource_group = mssql_cluster_name\naks_kubernetes_version = ''\naks_cluster_name = mssql_cluster_name\nmssqlctl_configuration_profile = 'aks-dev-test'\nmssqlctl_configuration_file = 'mssql-bdc-configuration'\n\n# Show parameter values\nprint('')\nprint(f'mssql_cluster_name = {mssql_cluster_name}')\nprint(f'azure_subscription_id = {azure_subscription_id}')\nprint(f'azure_vm_size = {azure_vm_size}')\nprint(f'azure_vm_count = {str(azure_vm_count)}')\nprint(f'azure_region = {azure_region}')\nprint(f'azure_resource_group = {azure_resource_group}')\nprint(f'aks_kubernetes_version = {aks_kubernetes_version}')\nprint(f'aks_cluster_name = {aks_cluster_name}')\nprint(f'docker_username = {docker_username}')\nprint(f'mssqlctl_configuration_profile = {mssqlctl_configuration_profile}')\nprint(f'mssqlctl_configuration_file = {mssqlctl_configuration_file}')\nprint(f'bdc_controller_username = {bdc_controller_username}')\nprint('')",
            "metadata": {},
            "outputs": [],
            "execution_count": 4
        },
        {
            "cell_type": "markdown",
            "source": "### **Login to Azure**\n\nThis will open a web browser window to enable credentials to be entered. If this cells is hanging forever, it might be because your Web browser windows is waiting for you to enter your Azure credentials!\n",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "cmd = f'az login'\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 5
        },
        {
            "cell_type": "markdown",
            "source": "\n### **Set active Azure subscription**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "if azure_subscription_id != \"\":\n    cmd = f'az account set --subscription {azure_subscription_id}'\n    run_command()\nelse:\n    print('Using the default Azure subscription', {azure_subscription_id})\ncmd = f'az account show'\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 6
        },
        {
            "cell_type": "markdown",
            "source": "### **Create Azure resource group**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "cmd = f'az group create --name {azure_resource_group} --location {azure_region}'\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 7
        },
        {
            "cell_type": "markdown",
            "source": "### **Create AKS cluster**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "if aks_kubernetes_version != \"\":\n    kubernetes_version_option=\"--kubernetes-version {aks_kubernetes_version}\"\nelse:\n    kubernetes_version_option = \"\"\ncmd = f'az aks create --name {aks_cluster_name} --resource-group {azure_resource_group} --generate-ssh-keys --node-vm-size {azure_vm_size} --node-count {azure_vm_count} {kubernetes_version_option}' \nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 8
        },
        {
            "cell_type": "markdown",
            "source": "### **Set the new AKS cluster as current context**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "cmd = f'az aks get-credentials --resource-group {azure_resource_group} --name {aks_cluster_name} --admin --overwrite-existing'\r\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 9
        },
        {
            "cell_type": "markdown",
            "source": "### **List the MSSQLCTL configuration profiles**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "import os\nos.environ[\"ACCEPT_EULA\"] = 'yes'\ncmd = f'mssqlctl bdc config list'\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 10
        },
        {
            "cell_type": "markdown",
            "source": "### **Create a MSSQLCTL configuration file**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "# Create a configuration file\ncmd = f'mssqlctl bdc config init --source {mssqlctl_configuration_profile} --target {mssqlctl_configuration_file} --force'\nrun_command()\n\n# Set the 'big data cluster' name\njsonPath = '\"metadata.name=''{0}''\"'.format(mssql_cluster_name)\ncmd = f'mssqlctl bdc config section set -c {mssqlctl_configuration_file} -j {jsonPath}'\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 11
        },
        {
            "cell_type": "markdown",
            "source": "### **Create SQL Server 2019 big data cluster**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "import os\nprint (f'Creating SQL Server 2019 big data cluster: {mssql_cluster_name} using configuration file {mssqlctl_configuration_file}')\nos.environ[\"CONTROLLER_USERNAME\"] = bdc_controller_username\nos.environ[\"CONTROLLER_PASSWORD\"] = bdc_password\nos.environ[\"MSSQL_SA_PASSWORD\"] = bdc_password\nos.environ[\"KNOX_PASSWORD\"] = bdc_password\nos.environ[\"DOCKER_USERNAME\"] = docker_username\nos.environ[\"DOCKER_PASSWORD\"] = docker_password\n\ncmd = f'mssqlctl bdc create -c {mssqlctl_configuration_file} --accept-eula yes'\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 12
        },
        {
            "cell_type": "markdown",
            "source": "### **Login to SQL Server 2019 big data cluster**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "cmd = f'mssqlctl login --cluster-name {mssql_cluster_name}'\nrun_command()",
            "metadata": {},
            "outputs": [],
            "execution_count": 13
        },
        {
            "cell_type": "markdown",
            "source": "### **Show SQL Server 2019 big data cluster endpoints**",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "import json\nimport pandas as pd\nfrom IPython.display import *\npd.set_option('display.max_colwidth', -1)\n\ndef formatColumnNames(column):\n    return ' '.join(word[0].upper() + word[1:] for word in column.split())\n\ndef show_results(results):\n    df = pd.DataFrame(results)\n    df.columns = [formatColumnNames(n) for n in results[0].keys()]\n    mydata = HTML(df.to_html(render_links=True))\n    display(mydata)\n\ncmd = f'mssqlctl bdc endpoint list'\nendpointsResults  = !{cmd}\nendpointsInJson = json.loads(''.join(endpointsResults))\nshow_results(endpointsInJson)",
            "metadata": {},
            "outputs": [],
            "execution_count": 14
        },
        {
            "cell_type": "markdown",
            "source": "### **Connect to master SQL Server instance in Azure Data Studio**\r\nClick the link below to connect to the master SQL Server instance of the SQL Server 2019 big data cluster.",
            "metadata": {}
        },
        {
            "cell_type": "code",
            "source": "filteredEndpoints = [x for x in endpointsInJson if x['name'] == 'sql-server-master']\r\nif filteredEndpoints and len(filteredEndpoints) == 1:\r\n    display(HTML(\"<h3><a href=\\\"command:azdata.connect?{&quot;serverName&quot;:&quot;\"+filteredEndpoints[0]['endpoint']+\"&quot;,&quot;providerName&quot;:&quot;MSSQL&quot;, &quot;authenticationType&quot;:&quot;SqlLogin&quot;,&quot;userName&quot;:&quot;sa&quot;,&quot;password&quot;:&quot;\"+bdc_password+\"&quot;}\\\">Click here to connect to SQL Server</a></h3>\"))",
            "metadata": {},
            "outputs": [],
            "execution_count": 15
        }
    ]
}