mirror of
https://github.com/ckaczor/azuredatastudio.git
synced 2026-01-13 17:22:15 -05:00
04622d4cb3
* azdata updates and relative formatting updates * bigdatacluster case fix * only upgrade pandas to 24.2 if it's lower * rename bigdatacluster to caps
390 lines
18 KiB
Plaintext
390 lines
18 KiB
Plaintext
{
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"name": "python3",
|
|
"display_name": "Python 3"
|
|
},
|
|
"language_info": {
|
|
"name": "python",
|
|
"version": "3.6.6",
|
|
"mimetype": "text/x-python",
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"pygments_lexer": "ipython3",
|
|
"nbconvert_exporter": "python",
|
|
"file_extension": ".py"
|
|
}
|
|
},
|
|
"nbformat_minor": 2,
|
|
"nbformat": 4,
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"\n",
|
|
"\n",
|
|
"# View the status of your SQL Server Big Data Cluster\n",
|
|
"This notebook allows you to see the status of the controller, master instance, and pools in your SQL Server Big Data Cluster.\n",
|
|
"\n",
|
|
"> ## **Important Instructions**\n",
|
|
"> ### **Before you begin, you will need:**\n",
|
|
">* Big Data Cluster name\n",
|
|
">* Controller username\n",
|
|
">* Controller password\n",
|
|
">* Controller endpoint \n",
|
|
"\n",
|
|
"You can find the controller endpoint from the SQL Big Data Cluster dashboard in the Service Endpoints table. The endpoint is listed as **Cluster Management Service.**\n",
|
|
"\n",
|
|
"If you do not know the credentials, ask the admin who deployed your cluster.\n",
|
|
"\n",
|
|
"### **Prerequisites**\n",
|
|
"Ensure the following tools are installed and added to PATH before proceeding.\n",
|
|
"\n",
|
|
"|Tools|Description|Installation|\n",
|
|
"|---|---|---|\n",
|
|
"|kubectl | Command-line tool for monitoring the underlying Kubernetes cluster | [Installation](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-using-native-package-management) |\n",
|
|
"|azdata | Command-line tool for installing and managing a big data cluster |[Installation](https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-install-azdata?view=sqlallproducts-allversions) |\n",
|
|
"|Pandas Package | Python package for data manipulation | Will be installed by the notebook if not present |\n",
|
|
"\n",
|
|
"\n",
|
|
"### **Instructions**\n",
|
|
"* For the best experience, click **Run Cells** on the toolbar above. This will automatically execute all code cells below and show the cluster status in each table.\n",
|
|
"* When you click **Run Cells** for this Notebook, you will be prompted at the *Log in to your Big Data Cluster* code cell to provide your login credentials. Follow the prompts and press enter to proceed.\n",
|
|
"* **You won't need to modify any of the code cell contents** in this Notebook. If you accidentally made a change, you can reopen this Notebook from the cluster dashboard.\n",
|
|
"\n",
|
|
"\n",
|
|
""
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": "### **Check azdata version**",
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"import sys, os\r\n",
|
|
"cmd = f'azdata --version'\r\n",
|
|
"cmdOutput = !{cmd}\r\n",
|
|
"azdataStr = '\\'azdata\\''\r\n",
|
|
"if len(cmdOutput) > 0 and ('command not found' in cmdOutput[1] or f'{azdataStr} is not recognized as an internal or external command' in cmdOutput[0]):\r\n",
|
|
" raise SystemExit('azdata not found! Please make sure azdata is installed and added to path' + '.\\n')\r\n",
|
|
"if '15.0' in cmdOutput[0]:\r\n",
|
|
" print('azdata version: ' + cmdOutput[0])\r\n",
|
|
""
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": "### **Install latest version of pandas**",
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"#install pandas\r\n",
|
|
"import pandas\r\n",
|
|
"pandas_version = pandas.__version__.split('.')\r\n",
|
|
"pandas_major = int(pandas_version[0])\r\n",
|
|
"pandas_minor = int(pandas_version[1])\r\n",
|
|
"pandas_patch = int(pandas_version[2])\r\n",
|
|
"if not (pandas_major > 0 or (pandas_major == 0 and pandas_minor > 24) or (pandas_major == 0 and pandas_minor == 24 and pandas_patch >= 2)):\r\n",
|
|
" pandasVersion = 'pandas==0.24.2'\r\n",
|
|
" cmd = f'{sys.executable} -m pip install {pandasVersion}'\r\n",
|
|
" cmdOutput = !{cmd}\r\n",
|
|
" print(f'\\nSuccess: Upgraded pandas to 0.24.2.')\r\n",
|
|
"else:\r\n",
|
|
" print('Pandas required version is already installed!') "
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"## **Log in to your Big Data Cluster**\r\n",
|
|
"To view cluster status, you will need to connect to your Big Data Cluster through azdata. \r\n",
|
|
"\r\n",
|
|
"When you run this code cell, you will be prompted for:\r\n",
|
|
"- Cluster name\r\n",
|
|
"- Controller username\r\n",
|
|
"- Controller password\r\n",
|
|
"\r\n",
|
|
"To proceed:\r\n",
|
|
"- **Click** on the input box\r\n",
|
|
"- **Type** the login info\r\n",
|
|
"- **Press** enter.\r\n",
|
|
"\r\n",
|
|
"If your cluster is missing a configuration file, you will be asked to provide your controller endpoint. (Format: **https://00.00.00.000:00000**) You can find the controller endpoint from the big data cluster dashboard in the Service Endpoints table. The endpoint is listed as **Cluster Management Service.**\r\n",
|
|
""
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"import os, getpass, json\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"from IPython.display import *\n",
|
|
"\n",
|
|
"def PromptForInfo(promptMsg, isPassword, errorMsg):\n",
|
|
" if isPassword:\n",
|
|
" promptResponse = getpass.getpass(prompt=promptMsg)\n",
|
|
" else:\n",
|
|
" promptResponse = input(promptMsg)\n",
|
|
" if promptResponse == \"\":\n",
|
|
" raise SystemExit(errorMsg + '\\n')\n",
|
|
" return promptResponse\n",
|
|
"\n",
|
|
"# Prompt user inputs:\n",
|
|
"cluster_name = PromptForInfo('Please provide your Cluster Name: ', False, 'Cluster Name is required!')\n",
|
|
"\n",
|
|
"controller_username = PromptForInfo('Please provide your Controller Username for login: ', False, 'Controller Username is required!')\n",
|
|
"\n",
|
|
"controller_password = PromptForInfo('Controller Password: ', True, 'Password is required!')\n",
|
|
"print('***********')\n",
|
|
"\n",
|
|
"!azdata logout\n",
|
|
"# Login in to your big data cluster \n",
|
|
"cmd = f'azdata login -n {cluster_name} -u {controller_username} -a yes'\n",
|
|
"print(\"Start \" + cmd)\n",
|
|
"os.environ['CONTROLLER_USERNAME'] = controller_username\n",
|
|
"os.environ['CONTROLLER_PASSWORD'] = controller_password\n",
|
|
"os.environ['ACCEPT_EULA'] = 'yes'\n",
|
|
"\n",
|
|
"loginResult = !{cmd}\n",
|
|
"if 'ERROR: Please check your kube config or specify the correct controller endpoint with: --controller-endpoint https://<ip>:<port>.' in loginResult[0] or 'ERROR' in loginResult[0]:\n",
|
|
" controller_ip = input('Please provide your Controller endpoint: ')\n",
|
|
" if controller_ip == \"\":\n",
|
|
" raise SystemExit(f'Controller IP is required!' + '\\n')\n",
|
|
" else:\n",
|
|
" cmd = f'azdata login -n {cluster_name} -e {controller_ip} -u {controller_username} -a yes'\n",
|
|
" loginResult = !{cmd}\n",
|
|
"print(loginResult)"
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"## **Status of Big Data Cluster**\r\n",
|
|
"After you successfully login to your bdc, you can view the overall status of each container before drilling down into each component."
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Helper methods for formatting\n",
|
|
"def formatColumnNames(column):\n",
|
|
" return ' '.join(word[0].upper() + word[1:] for word in column.split())\n",
|
|
"\n",
|
|
"pd.set_option('display.max_colwidth', -1)\n",
|
|
"def show_results(results):\n",
|
|
" strResult = ''.join(results)\n",
|
|
" jsonResults = json.loads(strResult)\n",
|
|
" results = jsonResults['result']\n",
|
|
" if isinstance(results, list):\n",
|
|
" for result in results:\n",
|
|
" if isinstance(result, list):\n",
|
|
" show_formattedArray(result)\n",
|
|
" else:\n",
|
|
" show_keys(result)\n",
|
|
" else:\n",
|
|
" show_keys(results)\n",
|
|
"\n",
|
|
"def show_keys(results):\n",
|
|
" listKeys = []\n",
|
|
" if isinstance(results, dict):\n",
|
|
" for key in results.keys():\n",
|
|
" if results[key] and not isinstance(results[key], list):\n",
|
|
" print('\\033[1m' + formatColumnNames(key) + ': \\033[0m' + results[key])\n",
|
|
" if results[key] and isinstance(results[key], list):\n",
|
|
" listKeys.append(key)\n",
|
|
" for key in listKeys:\n",
|
|
" show_formattedArray(results[key])\n",
|
|
" if isinstance(results, str):\n",
|
|
" print('\\033[1m' + results + ': \\033[0m')\n",
|
|
"\n",
|
|
"def show_formattedArray(results):\n",
|
|
" fomattedRow = []\n",
|
|
" if not isinstance(results, list):\n",
|
|
" show_formattedResults(results)\n",
|
|
" else:\n",
|
|
" for row in results:\n",
|
|
" if isinstance(row, str):\n",
|
|
" show_keys(row)\n",
|
|
" else:\n",
|
|
" fomattedRow.append({ k : v for k,v in row.items() if isinstance(v, str) or v is None})\n",
|
|
" df = pd.DataFrame(fomattedRow)\n",
|
|
" df.columns = [formatColumnNames(n) for n in fomattedRow[0].keys()]\n",
|
|
" mydata = HTML(df.to_html(render_links=True))\n",
|
|
" display(mydata)\n",
|
|
" nameKeys = [k for k in fomattedRow[0].keys() if 'Name' in k]\n",
|
|
" for key in results[0].keys():\n",
|
|
" if key not in fomattedRow[0].keys():\n",
|
|
" for result in results:\n",
|
|
" print('\\033[1m' + formatColumnNames(nameKeys[0]) + ': \\033[0m' + result[nameKeys[0]])\n",
|
|
" show_formattedArray(result[key])\n",
|
|
"\n",
|
|
"def show_formattedResults(input):\n",
|
|
" df = pd.DataFrame([input])\n",
|
|
" df.columns = [formatColumnNames(n) for n in [input][0].keys()]\n",
|
|
" mydata = HTML(df.to_html(render_links=True))\n",
|
|
" display(mydata)\n",
|
|
" \n",
|
|
"# Display status of big data cluster\n",
|
|
"results = !azdata bdc status show -o json\n",
|
|
"show_results(results)"
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"## **Cluster Status**\r\n",
|
|
"For each cluster component below, running each code cell will generate a table. This table will include:\r\n",
|
|
"\r\n",
|
|
"|Column Name|Description|\r\n",
|
|
"|---|---|\r\n",
|
|
"|**Kind** | Identifies if component is a pod or a set. |\r\n",
|
|
"|**LogsURL** | Link to [Kibana](https://www.elastic.co/guide/en/kibana/current/introduction.html) logs which is used for troubleshooting. |\r\n",
|
|
"|**Name** | Provides the specific name of the pod or set. |\r\n",
|
|
"|**NodeMetricsURL** | Link to [Grafana](https://grafana.com/docs/guides/basic_concepts/) dashboard to view key metrics of the node. |\r\n",
|
|
"|**SQLMetricsURL** | Link to [Grafana](https://grafana.com/docs/guides/basic_concepts/) dashboard to view key metrics of the SQL instance. |\r\n",
|
|
"|**State** | Indicates state of the pod or set. |\r\n",
|
|
"\r\n",
|
|
"----------------------------------------------------------------"
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"### **Controller status**\n",
|
|
"To learn more about the controller, [read here.](https://docs.microsoft.com/sql/big-data-cluster/concept-controller?view=sql-server-ver15)"
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Display status of controller\n",
|
|
"results = !azdata bdc control status show --all -o json\n",
|
|
"show_results(results)"
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"### **Master Instance status**\n",
|
|
"To learn more about the master instance, [read here.](https://docs.microsoft.com/sql/big-data-cluster/concept-master-instance?view=sqlallproducts-allversions)"
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"results = !azdata bdc sql status show --resource master --all -o json\n",
|
|
"show_results(results)"
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"### **Compute Pool status**\n",
|
|
"To learn more about compute pool, [read here.](https://docs.microsoft.com/sql/big-data-cluster/concept-compute-pool?view=sqlallproducts-allversions)"
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Display status of compute pool\n",
|
|
"results = !azdata bdc sql status show --resource compute-0 --all -o json\n",
|
|
"show_results(results)"
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"### **Storage Pool status**\n",
|
|
"To learn more about storage pool, [read here.](https://docs.microsoft.com/sql/big-data-cluster/concept-storage-pool?view=sqlallproducts-allversions)"
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Display status of storage pools\n",
|
|
"results = !azdata bdc sql status show --resource storage-0 --all -o json\n",
|
|
"show_results(results)"
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"### **Data Pool status**\n",
|
|
"To learn more about data pool, [read here.](https://docs.microsoft.com/sql/big-data-cluster/concept-data-pool?view=sqlallproducts-allversions)"
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Display status of data pools\n",
|
|
"results = !azdata bdc sql status show --resource data-0 --all -o json\n",
|
|
"show_results(results)"
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"### **Spark Pool status**\n",
|
|
"Displays status of spark pool if it exists. Otherwise, will show as \"No spark pool.\""
|
|
],
|
|
"metadata": {}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Display status of spark pool\n",
|
|
"results = !azdata bdc spark status show --all -o json\n",
|
|
"show_results(results)\n",
|
|
""
|
|
],
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"execution_count": 0
|
|
}
|
|
]
|
|
}
|