Files
sqltoolsservice/src/Microsoft.Kusto.ServiceLayer/DataSource/KustoQueryUtils.cs
Monica Gupta 148b6e398d Added new Kusto ServiceLayer (#1009)
* Copy smoModel some rename

* Copy entire service layer

* Building copy

* Fixing some references

* Launch profile

* Resolve namespace issues

* Compiling tests. Correct manifest.

* Fixing localization resources

* ReliableKustoClient

* Some trimming of extra code and Kusto code

* Kusto client creation in bindingContent

* Removing Smo and new Kusto classes

* More trimming

* Kusto schema hookup

* Solidying DataSource abstraction

* Solidifying further

* Latest refatoring

* More refactoring

* Building and launching Kusto service layer

* Working model which enumerates databases

* Refactoring to pass IDataSource to all tree nodes

* Removing some dependencies on the context

* Working with tables and schema

* Comment checkin

* Refactoring to give out select script

* Query created and sent back to ADS

* Fix query generation

* Fix listing of databases

* Tunneling the query through.

* Successful query execution

* Return only results table

* Deleting Cms

* Delete DacFx

* Delete SchemaCompare and TaskServices

* Change build definition to not stop at launch

* Fix error after merge

* Save Kusto results in different formats (#935)

* save results as csv etc

* some fixes

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* 2407 Added OrderBy clause in KustoDataSource > GetDatabaseMetaData and GetColumnMetadata (#959)

* 2405 Defaulted Options when setting ServerInfo in ConnectionService > GetConnectionCompleteParams (#965)

* 2747 Fixed IsUnknownType error for Kusto (#989)

* 2747 Removed unused directives in Kusto > DbColumnWrapper. Refactored IsUnknownType to handle null DataTypeName

* 2747 Reverted IsUnknownType change in DbColumnWrapper. Changed DataTypeName to get calue from ColumnType. Refactored SafeGetValue to type check before hard casting to reduce case exceptions.

* Added EmbeddedResourceUseDependentUponConvention to Microsoft.Kusto.ServiceLayer.csproj. Also renamed DACfx to match Microsoft.SqlTools.ServiceLayer. Added to compile Exclude="**/obj/**/*.cs"

* Srahman cleanup sql code (#992)

* Removed Management and Security Service Code.

* Remove FileBrowser service

* Comment why we are using SqlServer library

* Remove SQL specific type definitions

* clean up formatter service (#996)

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* Code clean up and Kusto intellisense (#994)

* Code clean up and Kusto intellisense

* Addressed few comments

* Addressed few comments

* addressed comments

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* Return multiple tables for Kusto

* Changes required for Kusto manage dashboard (#1039)

* Changes required for manage dashboard

* Addressed comments

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* 2728 Kusto function support (#1038)

* loc update (#914)

* loc update

* loc updates

* 2728 moved ColumnInfo and KustoResultsReader to separate files. Added Folder and Function to TreeNode.cs

* 2728 Added FunctionInfo. Added Folder to ColumnInfo. Removed partial class from KustoResultsReader. Set Function.IsAlwaysLeaf=true in TreeNode.cs. In KustoDataSource changed tableMetadata type to TableMetaData. Added folder and function dictionaries. Refactored GetSchema function. Renamed GenerateColumnMetadataKey to GenerateMetadataKey

* 2728 Added FunctionInfo. Added Folder to ColumnInfo. Removed partial class from KustoResultsReader. Set Function.IsAlwaysLeaf=true in TreeNode.cs. In KustoDataSource changed tableMetadata type to TableMetaData. Added folder and function dictionaries. Refactored GetSchema function. Renamed GenerateColumnMetadataKey to GenerateMetadataKey

* 2728 Created new SqlConnection within using block. Refactored KustoDataSource > columnmetadata to sort on get instead of insert.

* 2728 Added GetFunctionInfo function to KustoDataSource.

* 2728 Reverted change to Microsoft.Kusto.ServiceLayer.csproj from merge

* 2728 Reverted change to SqlTools.ServiceLayer\Localization\transXliff

* 2728 Reverted change to sr.de.xlf and sr.zh-hans.xlf

* 2728 Refactored KustoDataSource Function folders to support subfolders

* 2728 Refactored KustoDataSource to use urn for folders, functions, and tables instead of name.

* Merge remote-tracking branch 'origin/main' into feature-ADE

# Conflicts:
#	Packages.props

* 2728 Moved metadata files into Metadata subdirectory. Added GenerateAlterFunction to IDataSource and DataSourceBase.

* 2728 Added summary information to SafeAdd in SystemExtensions. Renamed local variable in SetTableMetadata

* 2728 Moved SafeAdd from SystemExtensions to KustoQueryUtils. Added check when getting database schema to return existing records before querying again. Added AddRange function to KustoQueryUtils. Created SetFolderMetadataForFunctions method.

* 2728 Added DatabaseKeyPrefix to only return tables to a database for the dashboard. Added logic to store all database tables within the tableMetadata dictionary for the dashboard.

* 2728 Created TableInfo and moved info objects into Models directory. Refactored KustoDataSource to lazy load columns for tables. Refactored logic to load tables using cslschema instead of schema.

* 2728 Renamed LoadColumnSchema to GetTableSchema to be consistent.

Co-authored-by: khoiph1 <khoiph@microsoft.com>

* Addressed comments

Co-authored-by: Shafiq Rahman <srahman@microsoft.com>
Co-authored-by: Monica Gupta <mogupt@microsoft.com>
Co-authored-by: Justin M <63619224+JustinMDotNet@users.noreply.github.com>
Co-authored-by: rkselfhost <rkselfhost@outlook.com>
Co-authored-by: khoiph1 <khoiph@microsoft.com>
2020-08-12 15:34:38 -07:00

116 lines
5.0 KiB
C#

// <copyright file="KustoQueryUtils.cs" company="Microsoft">
// Copyright (c) Microsoft. All Rights Reserved.
// </copyright>
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Text.RegularExpressions;
using System.Linq;
using Microsoft.Kusto.ServiceLayer.DataSource.Metadata;
namespace Microsoft.Kusto.ServiceLayer.DataSource
{
public static class KustoQueryUtils
{
public const string StatementSeparator = "\n | "; // Start each statement on a new line. Not required by Kusto, but doing this for readability of scripts generated from here.
/// <summary>
/// Escape table/column/database names for a Kusto query.
/// </summary>
/// <param name="name">The name to be escaped</param>
/// <param name="alwaysEscape">Always escape if this flag is set</param>
/// <returns>The escaped string</returns>
public static string EscapeName(string name, bool alwaysEscape = false)
{
if (name.StartsWith("[@") || name == "*") // Field already escaped. No escaping required for '*' operand
{
return name;
}
string result = name;
Regex rx = new Regex("[^_a-zA-Z0-9]");
string [] kustoKeywordList = {"and", "anomalychart", "areachart", "asc", "barchart", "between", "bool", "boolean", "by",
"columnchart", "consume", "contains", "containscs", "count", "date", "datetime", "default", "desc", "distinct",
"double", "dynamic", "endswith", "evaluate", "extend", "false", "filter", "find", "first", "flags", "float",
"getschema", "has", "hasprefix", "hassuffix", "in", "int", "join", "journal", "kind", "ladderchart", "last",
"like", "limit", "linechart", "long", "materialize", "mvexpand", "notcontains", "notlike", "of", "or", "order",
"parse", "piechart", "pivotchart", "print", "project", "queries", "real", "regex", "sample", "scatterchart",
"search", "set", "sort", "stacked", "stacked100", "stackedareachart", "startswith", "string", "summarize",
"take", "time", "timechart", "timeline", "timepivot", "timespan", "to", "top", "toscalar", "true", "union",
"unstacked", "viewers", "where", "withsource"}; // add more keywords here
var escapeName = rx.IsMatch(name) || kustoKeywordList.Any(name.Contains) || alwaysEscape;
if (escapeName)
{
if (name.IndexOf('"') > -1)
{
result = "[@'" + name + "']";
}
else
{
result = "[@\"" + name + "\"]";
}
}
return result;
}
public static bool IsClusterLevelQuery(string query)
{
string [] clusterLevelQueryPrefixes = {
".show databases",
".show schema"
};
return clusterLevelQueryPrefixes.Any(query.StartsWith);
}
/// <summary>
/// Adds an object of type DataSourceObjectMetadata to a dictionary<string, Dictionary<string, T>>. If the key exists then the item is added
/// to the list. If not then the key is created and then added.
/// </summary>
/// <param name="dictionary">The dictionary of the dictionary that the list should be added to.</param>
/// <param name="key">The key to be added.</param>
/// <param name="metadata">The metadata to be added to the list.</param>
/// <typeparam name="T"></typeparam>
public static void SafeAdd<T>(this Dictionary<string, Dictionary<string, T>> dictionary, string key,
T metadata) where T : DataSourceObjectMetadata
{
if (dictionary.ContainsKey(key))
{
if (dictionary[key].ContainsKey(metadata.Name))
{
return;
}
dictionary[key].Add(metadata.Name, metadata);
}
else
{
dictionary[key] = new Dictionary<string, T> {{metadata.Name, metadata}};
}
}
/// <summary>
/// Add a range to a dictionary of ConcurrentDictionary. Adds range to existing IEnumerable within dictionary
/// at the same key.
/// </summary>
/// <param name="dictionary"></param>
/// <param name="key"></param>
/// <param name="metadatas"></param>
/// <typeparam name="T"></typeparam>
public static void AddRange<T>(this ConcurrentDictionary<string, IEnumerable<T>> dictionary, string key,
List<T> metadatas) where T : DataSourceObjectMetadata
{
if (dictionary.ContainsKey(key))
{
metadatas.AddRange(dictionary[key]);
}
dictionary[key] = metadatas.OrderBy(x => x.PrettyName, StringComparer.OrdinalIgnoreCase).ToList();
}
}
}