* 3490 Injected OwnerUri into KustClient to store for token refreshing. Removed UpdateAzureToken from IDataSource, DataSourceBase, and KustoDataSource. Removed logic for retrying queries related to Unauthorized datasource in Batch and Query. Changed ScriptingService, ScriptingScriptOperation, and ScriptAsScriptingOperation to take DataSource in the constructor instead of datasourcefactory. Changed ScriptingService to inject ConnectionService through InitializeService function.
* 3490 Removed Catch block for DataSourceUnauthorizedException in ExecuteControlCommandAsync
* 3490 Removed OwnerUri from KustoClient and used azureAccountToken to refresh token in ConnectionService.
* 3490 Reverted unneeded changes.
* 3490 Split ExecuteQuery in KustoClient to execute first query then remaining queries after
* 3490 Passed OwnerUri down into KustoClient to refresh token.
* 3490 Removed DataSourceUnauthorizedException. Refactored ExecuteQuery to catch aggregate exception. Added RefreshAzureToken logic to ExecuteControlCommand
* 3490 Added logic to update ReliableDataSourceConnection azure token within ConnectionInfo.
* 3490 Add retry logic to ExecuteQuery and ExecuteControlCommand in KustoClient
* 3326 Removed sqlConnection.Open from CreateSqlConnection in ConnectionService.
* 3326 Removed unused variables and functions from ConnectedBindingContext, ConnectedBindingQueue, IBindingContext, and IConnectedBindingQueue
* 3326 Removed unused constant SqlAzureEdition and unused functions CreateSqlConnection and CreateServerConnection from ConnectionService. Removed ServerConnection, MetadataDisplayInfoProvider, Binder, and SmoMetaDataProvider from ConnectedBindingContext. Deleted SqlConnectionOpener and ISqlConnectionOpener.
* 3644 Created KustoClient to handle interaction with Kusto servers. Refactored KustoDataSource and moved execution logic into KustoClient.
* 3644 Added RequestSecurityTokenParams, RequestSecurityTokenResponse, and SecurityTokenRequest to Kusto. Moved intellisense functions from KustoDataSource to KustoIntellisenseHelper. Added SchemaState to readonly property on KustoDataSource. Added catch block to Batch.cs to catch KustoRequestExceptions.
* 3644 Removed unused reference from ConnectionDetails and ConnectedBindingContext. Added UpdateAzureToken function to IKustoClient, KustoClient, IDataSource, KustoDataSource, and KustoDataSource. Added dataSource.Dispose to ReliableDataSourceConnection > Close. Added RefreshAzureToken to ConnectionService to refresh azure token.
* 3644 Removed unused properties from RequestSecurityTokenParams and RequestSecurityTokenResponse
* 3644 Added default to DbColumnWrapper to UnknownTypeName when null. Moved database query logic from KustoIntellisenseHelper to KustoClient. Moved KustoIntellisenseHelper data objects out of class. Changed SchemaState to load through separate tasks.
* 3644 Changed ReRunQuery logic in Kusto Batch to flip back to false if the query fails a second time so it can be rejected
* 3644 Updated GetAutoCompleteSuggestions and GetHoverHelp with changes from main
* 3644 Added AccountId to RequestSecurityTokenParams and set value in ConnectionService. Added throw to Batch.cs to ensure exceptions that are non-Unauthorized bubble up to ADS.
* 3644 Changed KustoUnauthorizedException to take original exception as inner exception. Changed catch block to only throw KustoUnauthorizedException when FailureCode is 401
* 3644 Renamed KustoUnauthorizedException to DataSourceUnauthorizedException. Moved logic to throw exception down into KustoClient. Changed retryLogic in Batch.cs to a decrementing count
* 3644 Changed logic in Batch.cs for throwing InvalidOperationException
* Clearing first use of parameters
* restore space
* added finally block
* Added try catch block in correct position
* fixed spacing
* added space before try
* duplicate row check simplified by Alan
* Revert "duplicate row check simplified by Alan"
This reverts commit 445ac506bd96b353266778abd0cf9bad2be2a3c3.
* removed exception and changed message
* added exception back for test purposes
* added working executescalar
* Added "<=" for delete action that deletes nothing.
* spacing fix for check duplicate
* Added comments and changed function logic.
* added clarifying message
* Added new extended class
* small space fix
* 3715 Refactored GetDatabaseName to load database names from kusto server when no database name available in connection string.
* 3715 Removed variable name from GetDatabaseName
* 3643 Enhanced GetAutoCompleteSuggestions to escape spaces in the insertText when generating list of suggestions. Changed detail to include kind.
* 3643 Fixed formatting
* 3643 Changed variable in GetAutoCompleteSuggestions
* 3647 Renamed ParentMetadata to DatabaseMetadata in FolderMetadata.cs. Removed Refresh function from IDataSource and DataSourceBase since it's not used outside of KustoDataSource. Added unit tests for ServerNode.cs. Removed unused constructor from TreeNode and unused AddChild function.
* 3647 Refactored KustoDataSource. Moved Set logic for DatabaseMetadata and columns into separate functions
* 3647 Reverted Rename of ParentMetadata in FolderMetadata. Reverted removal of Refresh function from DataSourceBase and IDatasource
* 3647 Renamed SmoModel directories to DataSourceModel in Kusto projects.
* 3647 Removed reference to Kusto.ServiceLayer in SqlTools.ServiceLayer
* 3278 Added unit tests in MetadataFactoryTests and Microsoft.Kusto.ServiceLayer.UnitTests project
* 3278 Removed todo and changed unit test to validate megabytes
* 3278 Added file and unit tests in AutoCompleteHelperTests.cs
* 3278 Removed unused functions from Kusto > ScriptAsScriptingOperation
* 3278 Added unit tests for DataSourceFactory
* 3278 Refactored AdminService to pass in ConnectionService rather than through instance variable. Added unit test for AdminServiceTests
* 3278 Refactored DataSourceFactory to not have static functions for future unit tests
* 3278 Re-added properties that were flagged as unused but are being used by ADS in ReliableDataSourceConnection.cs
* 3278 Re-added properties that were flagged as unused but are being used by ADS in ReliableDataSourceConnection.cs
* adding pipeline to execute tests (#1062)
* 3278 Converted GetDefaultAutoComplete and GetDefaultSemanticMarkers to static functions in DataSourceFactory. Removed unused constructor in ScriptFile. Added positive unit tests for both functions
* undoing release version bump
* adding additional configs
* 3278 Minor refactors in ConnectionInfo, BindingQueue, DiagnosticsHelper, MetadataService, and HostLoader. Changed AssemblyInfo to only allow Kusto Unit Tests for internal access. Added lots of unit tests.
* 3278 Commented out bindingContext.IsConnected in AddConnectionContext_Sets_BindingContext
* 3278 Reversed order of unit tests in ConnectedBindingQueueTests and added throw to Catch block.
* 3278 Reverted change to ConnectedBindingQueue. Removed unit test from AddConnectionContext for NeedsMetaData True
Co-authored-by: Jorge Berumen <52225468+joberume@users.noreply.github.com>
Co-authored-by: joberume <jberumen3@miners.utep.edu>
* InsightsGenerator project template files
* Add insights projects to SLN
* Setting up siggen class (#1003)
* Setting up siggen class
* fixed the learn method
* Refactoring code
Fixed compile errors
* renamed results to result
* Basic transformation logic (#1004)
* Fix a couple bugs and add a simple test (#1007)
* Fix a couple bugs and add a simple test
* More tests and bug fix
* Nara/workflow (#1006)
* added a queue processor
* ordered using statements
* Armemon/analytics (#1008)
* Basic transformation logic
* changed some structure of siggen
* added sum and average method, as well as select rows by input name
* add insights to results
* min, max added
Co-authored-by: Karl Burtram <karlb@microsoft.com>
Co-authored-by: Aasim Khan <aasimkhan30@gmail.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>
* Added rules engine base implementation (#1005)
* Added rules engine base implementation
* update comments
* addressing comments
* adding template text to columnheaders object
* adding template text to columnheaders object
* fixing columnheaders class
* Added test
* Added Template Parser unit test in Test project
* Deleted unnecessary files and reverted the files that were modified by mistake
Co-authored-by: Jinjing Arima <jiarima@microsoft.com>
* Insights generator message handler placeholder (#1013)
* Aasim/insights/insight methods (#1014)
* Basic transformation logic
* changed some structure of siggen
* Added top and bottom insight functions
* Added top, bottom insights
Added tests for top, bottom insights
* Armemon/insights2 (#1011)
* max and min insightsperslice, and tests
* got rid of unneccesssary function
* get indexes
Co-authored-by: Arslan Memon <armemon@microsoft.com>
* Armemon/insights2 (#1012)
* max and min insightsperslice, and tests
* got rid of unneccesssary function
* get indexes
* learn for stringinputtyype
* add learn implentation
Co-authored-by: Arslan Memon <armemon@microsoft.com>
* Added Tests
Removed duplicate methods
Co-authored-by: Karl Burtram <karlb@microsoft.com>
Co-authored-by: arslan9955 <53170027+arslan9955@users.noreply.github.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>
* Armemon/insights2 (#1016)
* Basic transformation logic
* changed some structure of siggen
* Added top and bottom insight functions
* Added top, bottom insights
Added tests for top, bottom insights
* max and min insightsperslice, and tests
* got rid of unneccesssary function
* get indexes
* learn for stringinputtyype
* add learn implentation
* add unique inputs
* fix merge error
* add to result
Co-authored-by: Karl Burtram <karlb@microsoft.com>
Co-authored-by: Aasim Khan <aasimkhan30@gmail.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>
* Added all the templates (#1015)
* Added all the templates
Added a method to find matched template
* Added a function to replace # and ## values in a template
* Added ReplaceHashesInTemplate call
* Added comments
* Updated the template txt
* Updated GetTopHeadersWithHash function to add #toplist
* Updated the logic per offline discussion with Hermineh
* Update request handler contract to take array (#1020)
* added rulesengine findmatchingtemplate (#1019)
* Add support for getting DacFx deploy options from a publish profile (#995)
* add support for getting options from a publish profile
* update comments
* set values for default options if they aren't specified in the publish profile
* addressing comments
* Updating to latest DacFx for a bug fix (#1010)
* added rulesengine findmatchingtemplate
* Update DacFx deploy and generate script with options (#998)
* update deploy and generate script to accept deployment options
* add tests
* add test with option set to true
* merge
* merge
* incorporated FindMatchedTemplate
Co-authored-by: Kim Santiago <31145923+kisantia@users.noreply.github.com>
Co-authored-by: Udeesha Gautam <46980425+udeeshagautam@users.noreply.github.com>
* -Added logic for Insights Generator Service Handler (#1017)
* -Added logic for Insights Generator Service Handler
* Fixed some logic
* Adding workflow test
* Update transform and add tests (#1024)
* Jiarima/fix rules engine logic (#1021)
* Added all the templates
Added a method to find matched template
* Added a function to replace # and ## values in a template
* Added ReplaceHashesInTemplate call
* Added comments
* Updated the template txt
* Updated GetTopHeadersWithHash function to add #toplist
* Updated the logic per offline discussion with Hermineh
* Update with the fixes
* Updated template and foreach conditions
* Added distinct
* Updated tests according to the logic change (#1026)
* Nara/remove queing (#1023)
* loc update (#914)
* loc update
* loc updates
* Add support for getting DacFx deploy options from a publish profile (#995)
* add support for getting options from a publish profile
* update comments
* set values for default options if they aren't specified in the publish profile
* addressing comments
* Updating to latest DacFx for a bug fix (#1010)
* Update DacFx deploy and generate script with options (#998)
* update deploy and generate script to accept deployment options
* add tests
* add test with option set to true
* intermediate check in for merge, transformed not working
* intermediate check in for merge, transformed not working
* added test case
* merged
Co-authored-by: khoiph1 <khoiph@microsoft.com>
Co-authored-by: Kim Santiago <31145923+kisantia@users.noreply.github.com>
Co-authored-by: Udeesha Gautam <46980425+udeeshagautam@users.noreply.github.com>
* Output data types from transform (#1029)
* Fix bug process input_g (#1030)
* Fixed the insight generator service (#1028)
* Jiarima/added more testings (#1031)
* Added another test
Updated ReplaceHashesInTemplate function to return string instead of Template
* Added third test
* Merged
* Reverted the workflow file to match with the one in hack/insights
* Bugs fixes to hook insights up to ADS (#1033)
* Bug fixes for hack insights (#1032)
* Fixed the minColumn index bug in Data Transformation
Fixed the template matching logic.
* Adding changes from PR
* Try to fix Workflow tests
* Readd workflow tests
* Fix template load location
Co-authored-by: Aasim Khan <aasimkhan30@gmail.com>
Co-authored-by: Nara <NaraVen@users.noreply.github.com>
Co-authored-by: arslan9955 <53170027+arslan9955@users.noreply.github.com>
Co-authored-by: Arslan Memon <armemon@microsoft.com>
Co-authored-by: gadudhbh <68879970+gadudhbh@users.noreply.github.com>
Co-authored-by: Jinjing Arima <jiarima@microsoft.com>
Co-authored-by: jiarima <68882862+jiarima@users.noreply.github.com>
Co-authored-by: Kim Santiago <31145923+kisantia@users.noreply.github.com>
Co-authored-by: Udeesha Gautam <46980425+udeeshagautam@users.noreply.github.com>
Co-authored-by: khoiph1 <khoiph@microsoft.com>
* 3278 Added unit tests in MetadataFactoryTests and Microsoft.Kusto.ServiceLayer.UnitTests project
* 3278 Removed todo and changed unit test to validate megabytes
* 3278 Added file and unit tests in AutoCompleteHelperTests.cs
* 3278 Removed unused functions from Kusto > ScriptAsScriptingOperation
* 3278 Added unit tests for DataSourceFactory
* 3278 Refactored AdminService to pass in ConnectionService rather than through instance variable. Added unit test for AdminServiceTests
* 3278 Refactored DataSourceFactory to not have static functions for future unit tests
* 3278 Re-added properties that were flagged as unused but are being used by ADS in ReliableDataSourceConnection.cs
* 3278 Re-added properties that were flagged as unused but are being used by ADS in ReliableDataSourceConnection.cs
* adding pipeline to execute tests (#1062)
* 3278 Converted GetDefaultAutoComplete and GetDefaultSemanticMarkers to static functions in DataSourceFactory. Removed unused constructor in ScriptFile. Added positive unit tests for both functions
* undoing release version bump
* adding additional configs
* adressing feedback
* Correcting path in csproj
Co-authored-by: Jorge Berumen <52225468+joberume@users.noreply.github.com>
Co-authored-by: joberume <jberumen3@miners.utep.edu>
* INSERT VALUES has the limit of 1000 rows. Replace with derived table.
* Remove NOT NULL restriction from the generated table.
* Fix line endings in SQL Assessment source files.
* 3491 Added GenerateExecuteFunctionScript to IDataSource, DataSourceBase, and KustoDataSource. Added ExecutionFunction to Scripter and IScripter.
* 3491 Refactored GenerateScriptForFunction in ScriptAsScriptingOperation to handle execute and alter
* 3278 Moved functions related to Metadata from DataSourceFactory to MetadataFactory.cs
* 3278 Refactored DataSourceFactory to not be static. Added IDataSourceFactory interface.
* 3278 Refactored IKustoIntellisenseHelper.cs to not be static. Added IKustoIntellisenseHelper.cs interface.
* 3278 Removed unused functions from Scripter and deleted ScripterCore.cs because it was unused. Refactored Scripter.cs methods to not be static and created IScripter.cs
* 3278 Refactored datsasourceConnectionFactory in ConnectionService to use MEF through the InitializeService function
* 3278 Refactored IAutoCompleteHelper to not be static and added IAutoCompleteHelper interface.
* 3278 Removed unused classes DatabaseFullAccessException and FeatureWithFullDbAccess. Refactored ObjectExplorerService to use ImportingConstructor attribute. Removed commented out in KustoResultsReader. Removed unused functions from DatabaseLocksManager. Removed unused IDataSourceConnectionFactory from ConnectionService
* 3278 Moved SqlConnectionOpener class to new file. Added new interfaces IConnectedBindingQueue.cs and ISqlConnectionOpener.cs Added sqlConnectionOpener to dependency in ConnectedBindingQueue.
* 3278 Removed needsMetadata param from ConnectedBindingQueue. Added param to AddConnectionContext where it's used
* 3278 Refactored ConnectedBindingQueue to use MEF. Refactored usages to run against interface.
* 3278 Corrected ServiceHost namespace. Removed unused dependency to sqlToolsContext in LanguageService. Updated dependency in MetadataService and LanguageService to IProtocolEndpoint instead of ServiceHost
* 3278 Added back NextResult function and summary to KustoResultsReader. Renamed instance to _instance in ConnectionService and DatabaseLocksManager to stay consistent. Changed OpenDataSourceConnection to private in ConnectionService
* 3278 Converted helper methods back to static and removed backing interfaces
* 3278 Reverted AdminService > InitializeService to use ServiceHost as param
* Copy smoModel some rename
* Copy entire service layer
* Building copy
* Fixing some references
* Launch profile
* Resolve namespace issues
* Compiling tests. Correct manifest.
* Fixing localization resources
* ReliableKustoClient
* Some trimming of extra code and Kusto code
* Kusto client creation in bindingContent
* Removing Smo and new Kusto classes
* More trimming
* Kusto schema hookup
* Solidying DataSource abstraction
* Solidifying further
* Latest refatoring
* More refactoring
* Building and launching Kusto service layer
* Working model which enumerates databases
* Refactoring to pass IDataSource to all tree nodes
* Removing some dependencies on the context
* Working with tables and schema
* Comment checkin
* Refactoring to give out select script
* Query created and sent back to ADS
* Fix query generation
* Fix listing of databases
* Tunneling the query through.
* Successful query execution
* Return only results table
* Deleting Cms
* Delete DacFx
* Delete SchemaCompare and TaskServices
* Change build definition to not stop at launch
* Fix error after merge
* Save Kusto results in different formats (#935)
* save results as csv etc
* some fixes
Co-authored-by: Monica Gupta <mogupt@microsoft.com>
* 2407 Added OrderBy clause in KustoDataSource > GetDatabaseMetaData and GetColumnMetadata (#959)
* 2405 Defaulted Options when setting ServerInfo in ConnectionService > GetConnectionCompleteParams (#965)
* 2747 Fixed IsUnknownType error for Kusto (#989)
* 2747 Removed unused directives in Kusto > DbColumnWrapper. Refactored IsUnknownType to handle null DataTypeName
* 2747 Reverted IsUnknownType change in DbColumnWrapper. Changed DataTypeName to get calue from ColumnType. Refactored SafeGetValue to type check before hard casting to reduce case exceptions.
* Added EmbeddedResourceUseDependentUponConvention to Microsoft.Kusto.ServiceLayer.csproj. Also renamed DACfx to match Microsoft.SqlTools.ServiceLayer. Added to compile Exclude="**/obj/**/*.cs"
* Srahman cleanup sql code (#992)
* Removed Management and Security Service Code.
* Remove FileBrowser service
* Comment why we are using SqlServer library
* Remove SQL specific type definitions
* clean up formatter service (#996)
Co-authored-by: Monica Gupta <mogupt@microsoft.com>
* Code clean up and Kusto intellisense (#994)
* Code clean up and Kusto intellisense
* Addressed few comments
* Addressed few comments
* addressed comments
Co-authored-by: Monica Gupta <mogupt@microsoft.com>
* Return multiple tables for Kusto
* Changes required for Kusto manage dashboard (#1039)
* Changes required for manage dashboard
* Addressed comments
Co-authored-by: Monica Gupta <mogupt@microsoft.com>
* 2728 Kusto function support (#1038)
* loc update (#914)
* loc update
* loc updates
* 2728 moved ColumnInfo and KustoResultsReader to separate files. Added Folder and Function to TreeNode.cs
* 2728 Added FunctionInfo. Added Folder to ColumnInfo. Removed partial class from KustoResultsReader. Set Function.IsAlwaysLeaf=true in TreeNode.cs. In KustoDataSource changed tableMetadata type to TableMetaData. Added folder and function dictionaries. Refactored GetSchema function. Renamed GenerateColumnMetadataKey to GenerateMetadataKey
* 2728 Added FunctionInfo. Added Folder to ColumnInfo. Removed partial class from KustoResultsReader. Set Function.IsAlwaysLeaf=true in TreeNode.cs. In KustoDataSource changed tableMetadata type to TableMetaData. Added folder and function dictionaries. Refactored GetSchema function. Renamed GenerateColumnMetadataKey to GenerateMetadataKey
* 2728 Created new SqlConnection within using block. Refactored KustoDataSource > columnmetadata to sort on get instead of insert.
* 2728 Added GetFunctionInfo function to KustoDataSource.
* 2728 Reverted change to Microsoft.Kusto.ServiceLayer.csproj from merge
* 2728 Reverted change to SqlTools.ServiceLayer\Localization\transXliff
* 2728 Reverted change to sr.de.xlf and sr.zh-hans.xlf
* 2728 Refactored KustoDataSource Function folders to support subfolders
* 2728 Refactored KustoDataSource to use urn for folders, functions, and tables instead of name.
* Merge remote-tracking branch 'origin/main' into feature-ADE
# Conflicts:
# Packages.props
* 2728 Moved metadata files into Metadata subdirectory. Added GenerateAlterFunction to IDataSource and DataSourceBase.
* 2728 Added summary information to SafeAdd in SystemExtensions. Renamed local variable in SetTableMetadata
* 2728 Moved SafeAdd from SystemExtensions to KustoQueryUtils. Added check when getting database schema to return existing records before querying again. Added AddRange function to KustoQueryUtils. Created SetFolderMetadataForFunctions method.
* 2728 Added DatabaseKeyPrefix to only return tables to a database for the dashboard. Added logic to store all database tables within the tableMetadata dictionary for the dashboard.
* 2728 Created TableInfo and moved info objects into Models directory. Refactored KustoDataSource to lazy load columns for tables. Refactored logic to load tables using cslschema instead of schema.
* 2728 Renamed LoadColumnSchema to GetTableSchema to be consistent.
Co-authored-by: khoiph1 <khoiph@microsoft.com>
* Addressed comments
Co-authored-by: Shafiq Rahman <srahman@microsoft.com>
Co-authored-by: Monica Gupta <mogupt@microsoft.com>
Co-authored-by: Justin M <63619224+JustinMDotNet@users.noreply.github.com>
Co-authored-by: rkselfhost <rkselfhost@outlook.com>
Co-authored-by: khoiph1 <khoiph@microsoft.com>
* Get connection string call was changing the connection info (ref object) received from cache. Changing it to just get and make changes to only the returned string..
* Change to ensure PR validation works
* Fix for multiple delete bug
* minor space removal
* removed catch error in RowDelete
* small optimizations
* space adding
* WIP on creating test
* Some cleanup
* removed spaces
* Fix for verifytext
* Added check for command format correctness.
* tidying up
* added working test for command exception
* simplification of TestDbDataReader getint
* Corrections made
* spacing and naming issues
* minor space
* one more space issue
* Remove xunit dependency from testdriver
* swap expected/actual as needed
* Convert Test.Common to nunit
* port hosting unit tests to nunit
* port batchparser integration tests to nunit
* port testdriver.tests to nunit
* fix target to copy dependency
* port servicelayer unittests to nunit
* more unit test fixes
* port integration tests to nunit
* fix test method type
* try using latest windows build for PRs
* reduce test memory use
