The main goal of this feature is to enable a command that will
1) Generate a parameterized command for each edit that is in the session
2) Execute that command against the server
3) Update the cached results of the table/view that's being edited with the committed changes (including computed/identity columns)
There's some secret sauce in here where I cheated around worrying about gaps in the updated results. This was accomplished by implementing an IComparable for row edit objects that ensures deletes are the *last* actions to occur and that they occur from the bottom of the list up (highest row ID to lowest). Thus, all other actions that are dependent on the row ID are performed first, then the largest row ID is deleted, then next largest, etc. Nevertheless, by the end of a commit the associated ResultSet is still the source of truth. It is expected that the results grid will need updating once changes are committed.
Also worth noting, although this pull request supports a "many edits, one commit" approach, it will work just fine for a "one edit, one commit" approach.
* WIP
* Adding basic commit support. Deletions work!
* Nailing down the commit logic, insert commits work!
* Updates work!
* Fixing bug in DbColumnWrapper IsReadOnly setting
* Comments
* ResultSet unit tests, fixing issue with seeking in mock writers
* Unit tests for RowCreate commands
* Unit tests for RowDelete
* RowUpdate unit tests
* Session and edit base tests
* Fixing broken unit tests
* Moving constants to constants file
* Addressing code review feedback
* Fixes from merge issues, string consts
* Removing ad-hoc code
* fixing as per @abist requests
* Fixing a couple more issues
The goal of this make sure that test code is correctly organized to ensure that test suites aren't dependent on each other.
* UnitTests get their own project now (renaming Microsoft.SqlTools.ServiceLayer.Test to Microsoft.SqlTools.ServiceLayer.UnitTests) which is about 90% of the changes to the files.
* IntegrationTests no longer depends on UnitTests, only Test.Common
* Any shared components from TestObjects that spins up a "live" connection has been moved to IntegrationTests Utility/LiveConnectionHelper.cs
* The dictionary-based mock file stream factory has been moved to Test.Common since it is used by UnitTests and IntegrationTests
* Added a overload that doesn't take a dictionary for when we don't care about monitoring the storage (about 90% of the time)
* The RunIf* wrapper methods have been moved to Test.Common
* OwnerUri and StandardQuery constants have been moved to Test.Common Constants file
* Updating to latest SDK version available at https://www.microsoft.com/net/core#windowscmd
* Moving unit tests to unit test folder
* Changing namespaces to UnitTests
* Moving some constants and shared functionality into common project, making the UnitTests reference it
* Unit tests are working!
* Integration tests are working
* Updating automated test runs
* Fixing one last broken unit test
* Exposing internals for other projects
* Moving edit data tests to UnitTest project
* Applying refactor fixes to unit tests
* Fixing flaky test that wasn't awaiting completion
This is fixing a crashing bug in previous check-in so I'm pushing in. Please leave comments on the commit and email if there is feedback.
* Fix MEF loader issues with framework assemblies
* Fix broken test from changed behavior
* Fix braces to C# convention
* Initial credential service files
* Clean-up hostloader
* Update build scripts to build credentials archive
* Move hosting files to new assembly
* Add credentials files to MS.SqlTools.Credentials
* Remove duplicate files
* Update namespace in program.cs
* Fix test build breaks
* Update extensions visibility.
* Remove unused resource strings
* Add xproj files to SLN for appveyor builds
* Fix appveyor build break in test project
* Fix extensibility tests
* Fix various typos in latest iteration
* Add settings for Integration build
* Fix codecoverage.bat to use full pdb for new projects
* Fix bug when packing in folder with native images
* Fix typos in xproj
* Reset XLF to fix build.cmd
This is a very large change. I'll try to outline what's going on.
1. This adds the **EditDataService** which manages editing **Sessions**.
1. Each session has a **ResultSet** (from the QueryExecutionService) which has the rows of the table and basic metadata about the columns
2. Each session also has an **IEditTableMetadata** implementation which is derived from SMO metadata which provides more in-depth and trustworthy data about the table than SqlClient alone can.
3. Each session holds a list of **RowEditBase** abstract class implementations
1. **RowUpdate** - Update cells in a row (generates `UPDATE` statement)
2. **RowDelete** - Delete an entire row (generates `DELETE` statement)
3. **RowCreate** - Add a new row (generates `INSERT INTO` statement)
4. Row edits have a collection of **CellUpdates** that hold updates for individual cells (except for RowDelete)
1. Cell updates are generated from text
5. RowEditBase offers some baseline functionality
1. Generation of `WHERE` clauses (which can be parameterized)
2. Validation of whether a column can be updated
2. New API Actions
1. edit/initialize - Queries for the contents of a table/view, builds SMO metadata, sets up a session
2. edit/createRow - Adds a new RowCreate to the Session
3. edit/deleteRow - Adds a new RowDelete to the Session
4. edit/updateCell - Adds a CellUpdate to a RowCreate or RowUpdate in the Session
5. edit/revertRow - Removes a RowCreate, RowDelete, or RowUpdate from the Session
6. edit/script - Generates a script for the changes in the Session and stores to disk
7. edit/dispose - Removes a Session and releases the query
3. Smaller updates (unit test mock improvements, tweaks to query execution service)
**There are more updates planned -- this is just to get eyeballs on the main body of code**
* Initial stubs for edit data service
* Stubbing out update management code
* Adding rudimentary dispose request
* More stubbing out of update row code
* Adding complete edit command contracts, stubbing out request handlers
* Adding basic implementation of get script
* More in progress work to implement base of row edits
* More in progress work to implement base of row edits
* Adding string => object conversion logic and various cleanup
* Adding a formatter for using values in scripts
* Splitting IMessageSender into IEventSender and IRequestSender
* Adding inter-service method for executing queries
* Adding inter-service method for disposing of a query
* Changing edit contract to include the object to edit
* Fully fleshing out edit session initialization
* Generation of delete scripts is working
* Adding scripter for update statements
* Adding scripting functionality for INSERT statements
* Insert, Update, and Delete all working with SMO metadata
* Polishing for SqlScriptFormatter
* Unit tests and reworked byte[] conversion
* Replacing the awful and inflexible Dictionary<string, string>[][] with a much better test data set class
* Fixing syntax error in generated UPDATE statements
* Adding unit tests for RowCreate
* Adding tests for the row edit base class
* Adding row delete tests
* Adding RowUpdate tests, validation for number of key columns
* Adding tests for the unit class
* Adding get script tests for the session
* Service integration tests, except initialization tests
* Service integration tests, except initialization tests
* Adding messages to sr.strings
* Adding messages to sr.strings
* Fixing broken unit tests
* Adding factory pattern for SMO metadata provider
* Copyright and other comments
* Addressing first round of comments
* Refactoring EditDataService to have a single method for handling
session-dependent operations
* Refactoring Edit Data contracts to inherit from a Session and Row
operation params base class
* Copyright additions
* Small tweak to strings
* Updated unit tests to test the refactors
* More revisions as per pull request comments
- Fixed handling of PlaceCommasBeforeNextStatement so that it doesn't add whitespace unnecessarily. This ensures that repeated formats of the same text shouldn't keep adding whitespace
- Added tests covering a number of scenarios related to this
- Avoid incrementing indents for compound binary boolean expressions. Multiple 'AND x = Y' statements were each indenting instead of having the same indent level.
- Fixes https://github.com/Microsoft/vscode-mssql/issues/709
- Add telemetry point for format requests
- Update PeekDefinition telemetry so that it captures whether the attempt succeeded and if it was connected. This will help us understand whether our current behavior of just notifying success/failure to the output window was useful or not. It will also help us understand whether we're doing a good job implementing this / if we need to improve reliability and preformance.
Adding a new setting to query execution setting that will change the display value we generate for `BIT` columns. The new setting is `DefaultDisplayBitAsNumber`. If true, bit columns will be displayed as 1 or 0. If false, they'll be displayed as true or false. The default value is true, to keep parity with SSMS behavior.
Enables us to solve https://github.com/Microsoft/vscode-mssql/issues/690 and https://github.com/Microsoft/vscode-mssql/issues/513
- Fixes https://github.com/Microsoft/vscode-mssql/issues/705
- Updated the keyword list to include most common keyword ranges and added test to cover top 10 missing keywords
- Fixed a bug in TestUtilities.cs where text range was not defined correctly.
Fixes a bug where the GetOrOpenConnection method of the ConnectionService would throw if the connection didn't exist. It would yield a very unhelpful "key not found in collection" exception. The code has been updated to be much more helpful and fix the bug.
* Stage changes to other machine
* IntelliSense cache rebuild command
* Move to other machine
* Add a test for overwriting binding queue.
* Move event handler into lanaguageservice.cs
* Replacing the awful and inflexible Dictionary<string, string>[][] with a much better test data set class
* Applying all changes
* Removing unused RequestParamTests
- TSqlFormatterService with support for formatting document and text range inside document
- Settings support for all formatting options.
- Extensibility support so that the service can be initialized using MEF extensibility, and can find all necessary TSqlFormatters using the same process
Fix Initialize request error on startup
- Messages were being read from the input channel before all request handlers were registered
- In particular, the Initialize request which is key for any server to talk to the client was getting lost because the message reader thread begins consuming, and we take an extra few hundred milliseconds due to MEF startup before we register the handler
- The solution is to initialize the message handler so request handlers can register, but not actually start processing incoming messages until all handers are ready. This is a safer way to go and should improve reliability overall
Improvements from internal prototype:
- Normalizing baselines to handle the line ending differences on Mac & Linux vs. Windows
- Significantly shortened most lines by implementing base class methods to wrap common objects from Visitor.Context and removing unnecessary "this." syntax
- Refactored the SqlCommonTableExpressionFormatter and related classes to reduce code count significantly. This provides a pattern to follow when refactoring other classes for similar clarity. It's likely a lot of common logic could be found and reused across these.
- Reduced overall code size by adding utility methods
* locale set through CLI and tests updated
* adding cli locale support
* adding tests and test constants
* cleaning up new tests
* fixing test which only fails remotely
* adding support for MacOS/Linux
* triggering new build due to appveyor timeout
* updating usage printout
* Get db name from query connection
* Add comments
* Correct typos
* revert changes to .sln
* Add unit tests
* Fix typo
* Fix error due to a mistyped comment
Adding new methods for executing queries from other services (such as the upcoming edit data service). The code is written to avoid duplicating logic by using lambdas to perform custom logic.
Additionally, the service host protocol has been slightly modified to split the IMessageSender into IEventSender and IRequestSender. This allows us to use either a ServiceHost or any RequestContext<T> to send events. It becomes very convenient to use another service's request context to send the events for query execution.
**Breaking Change**: This removes the messages property for query dispose results and instead elects to use error for any errors encountered during query disposal. A result is only used when something is successful.
* Splitting IMessageSender into IEventSender and IRequestSender
* Adding inter-service method for executing queries
* Adding inter-service method for disposing of a query
* Adding null checking for the success/error handlers
This is a small API addition that allows us to execute queries directly as strings. This will make it easier to execute queries outside the confines of a workspace like VS Code.
* Refactor execution requests and events are now named less redundantly and moved into a separate namespace for organization. This is the bulk of the changes.
* QueryExecuteBatchNotification -> ExecuteRequests/BatchEvents
* QueryExecuteMessageNotification -> ExecuteRequests/MessageEvent
* QueryExecuteCompleteNotification -> ExecuteRequests/QueryCompleteEvent
* QueryExecuteResultSetCompleteNotification -> ExecuteRequests/ResultSetEvents
* QueryExecuteSubsetRequest -> SubsetRequest.cs
* Creating an inheritance pattern where
* `ExecuteRequestParamsBase` has execution options and ID for a query execute request
* `ExecuteDocumentSelectionParams` inherits from `ExecuteRequestParamsBase` and provides a document selection
* `ExecuteStringParams` inherits from `ExecuteRequestParamsBase` and provides the query text
* Adding a helper method to get SQL text based on request type
* Through the AWESOME POWER OF POLYMORPHISM, we are able to create a request for executing straight SQL basically for free.
* **Breaking change:** query/execute => query/executeDocumentSelection to make it more obvious what is expected.
* Adding unit tests for the code that gets SQL text
* Refactoring of execute contracts into their own namespace
* Refactoring application
* Adding new request for executing queries as strings
* Adding forgotten string request
* Changing the logic for checking the request param types
* Removing redundant declarations
* added a new tool to store SQL connections locally. Modified the peek definition tests to create test database before running test
* fixed failing test QueryExecutionPlanInvalidParamsTest
* Fixes based on code review comments
* fixed failing test GetSignatureHelpReturnsNotNullIfParseInfoInitialized
* Add codeGen for existing types
* Modify code gen logic to match current code
* Extend logic for new smo objects
* Add logic to retrieve token type from QuickInfo
* Remove duplicate types
* Add tests for new types
* Modify GetScript logic to use suggestions first
* Add more tests
* Modify codeGen to include quickInfo logic
* Cake build changes to run CodeGen
* CodeGen replace indentation
* Refactor GetScript and add more tests
* Refactor Resolver calls
* Fix TestDriver test for Definition
* Change quickInfo string case
* Revert change sto .sln file
* Fix typos in comments
* change String to string
* Rename test sql objects
* Add CancelTokenKey for uniquely identifying cancelations of Connections associated with an OwnerUri and ConnectionType string.
* Update ConnectionInfo to use ConcurrentDictionary of DbConnection instances. Add wrapper functions for the ConcurrentDictionary.
* Refactor Connect and Disconnect in ConnectionService.
* Update ConnectionService: Handle multiple connections per ConnectionInfo. Handle cancelation tokens uniquely identified with CancelTokenKey. Add GetOrOpenConnection() for other services to request an existing or create a new DbConnection.
* Add ConnectionType.cs for ConnectionType strings.
* Add ConnectionType string to ConnectParams, ConnectionCompleteNotification, DisconnectParams.
* Update Query ExecuteInternal to use the dedicated query connection and GetOrOpenConnection().
* Update test library to account for multiple connections in ConnectionInfo.
* Write tests ensuring multiple connections don’t create redundant data.
* Write tests ensuring database changes affect all connections of a given ConnectionInfo.
* Write tests for TRANSACTION statements and temp tables.
* Fix exception due to repeat disposal of stream
* Fix off by one index for saving selection
* Fix to overwrite file if file already exists
* Fix test for Json save selection
* Add Json Formatting
* experimental showplan implementation (tools side only)
* fix for redundant massages from showplan executions
* moved showplan batches to new variables to make it less confusing
* returns showplan as part of batch summary with in each result summary
* cleaned up showplan resultsets
* cleaning up code and making showplan var optional
* changes all var names to showplan
* adding estimated support
* small fixes
* updatin var names and adding EPOptions struct
* adding ssms execution plan logic based on server types
* adding special actions logic
* removing redundant name changes
* execution plan query handler added
* cleaning up functions and data structures
* seperated special actions into its own class
* cleaning up special actions
* cleaning up code
* small new line fixes
* commenting out pre-yukon code
* removing all pre yukon code
* last yukon code commented out
* fixes broken tests
* adding related unit tests; integration tests incoming
* finishing tests and cleaning up code
* semantic changes
* cleaning up semantics
* changes and test fixes, also adding new exceptions into RS
* fixing special actions and cleaning up request logic
* fixing comment to trigger new build
* triggering another build
* fixed up specialaction and added tests for it
This change is a reworking of the way that messages are sent to clients from the service layer. It is also a reworking of the protocol to ensure that all formulations of query send back events to the client in a deterministic ordering. To support the first change:
* Added a new event that will be sent when a message is generated
* Messages now indicate which Batch (if any) generated them
* Messages now indicate if they were error level
* Removed message storage in Batch objects and BatchSummary objects
* Batch objects no longer have error state
Add IntegrationTests project. Move all tests ifdef'd with LIVE_CONNECTION_TESTS to IntegrationTests project. Delete files that have no remaining code. Update codecoverage.bat to run integration tests
It's an overhaul of the Save As mechanism to utilize the file reader/writer classes to better align with the patterns laid out by the rest of the query execution. Why make this change? This change makes our code base more uniform and adherent to the patterns/paradigms we've set up. This change also helps with the encapsulation of the classes to "separate the concerns" of each component of the save as function.
* Replumbing the save as execution to pass the call down the query stack as QueryExecutionService->Query->Batch->ResultSet
* Each layer performs it's own parameter checking
* QueryExecutionService checks if the query exists
* Query checks if the batch exists
* Batch checks if the result set exists
* ResultSet checks if the row counts are valid and if the result set has been executed
* Success/Failure delegates are passed down the chain as well
* Determination of whether a save request is a "selection" moved to the SaveResultsRequest class to eliminate duplication of code and creation of utility classes
* Making the IFileStream* classes more generic
* Removing the requirements of max characters to store from the GetWriter method, and moving it into the constructor for the temporary buffer writer - the values have been moved to the settings and given defaults
* Removing the individual type writers from IFileStreamWriter
* Removing the individual type writers from IFIleStreamReader
* Adding a new overload for WriteRow to IFileStreamWriter that will write out data, given a row's worth of data and the list of columns
* Creating a new IFileStreamFactory that creates a reader/writer pair for reading from the temporary files and writing to CSV files
* Creating a new IFileStreamFactory that creates a reader/writer pair for reading from the temporary files and writing to JSON files
* Dramatically simplified the CSV encoding functionality
* Removed duplicated logic for saving in different types and condensed down to a single chain that only differs based on what type of factory is provided
* Removing the logic for managing the list of save as tasks, since the ResultSet now performs the actual saving work, there's no real need to expose the internals of the ResultSet
* Adding new strings to the sr.strings file for save as error messages
* Completely rewriting the unit tests for the save as mechanism. Very fine grained unit tests now that should cover majority of cases (aside from race conditions)
* Refactoring maxchars params into settings and out of file stream factory
* Removing write*/read* methods from file stream readers/writers
* Migrating the CSV save as to the resultset
* Tweaks to unit testing to eliminate writing files to disk
* WIP, moving to a base class for save results writers
* Everything is wired up and compiles
* Adding unit tests for CSV encoding
* Adding unit tests for CSV and Json writers
* Adding tests to the result set for saving
* Refactor to throw exceptions on errors instead of calling failure handler
* Unit tests for batch/query argument in range
* Unit tests
* Adding service integration unit tests
* Final polish, copyright notices, etc
* Adding NULL logic
* Fixing issue of unicode to utf8
* Fixing issues as per @kburtram code review comments
* Adding files that got broken?
This is a reworking of the unit tests to permit us to better test events from the service host. This new Event Flow Validator class allows creating a chain of events that can then be validated after execution of the request. Each event can have its own custom validation logic for verifying that the object sent via the service host is correct. It also allows us to validate that the order of events are correct.
The big drawback is that (at this time) the validator cannot support asynchronous events or non-determinant ordering of events. We don't need this for the query execution functionality despite messages being sent asynchronously because async messages aren't sent during unit tests (due to the db message event only being present on SqlDbConnection classes). If the need arises to do async or out of order event validation, then I have some ideas for how we can do that.
* Applying the event flow validator to the query execution service integration tests
* Undoing changes to events that were included in cherry-picked commit
* Cleaning up event flow validation to query execution
* Add efv to cancel tests
* Adding efv to dispose tests
* Adding efv to subset tests
* Adding efv to SaveResults tests
* Copyright
This basically is a replacement to the fix for Adding Milliseconds to DateTime fields. I didn't take into consideration that `DATE` columns would report as DateTime type. Date columns have a numeric scale set to 255, leading to the formatting string for the date time to include 255 millisecond places, which is invalid.
This change also reverses the change to store DateTime precision in the buffer file. Instead, the column metadata is now used when deserializing the rows from the db. `DATETIME` and `DATETIME2` columns are differentiated by their numeric scale while `DATE` columns are differentiated by their datatype name field.
More unit tests were added. Additionally, this fixes an unreported bug that `DATE` columns were being displayed with times, which is incorrect.
* Revert "Adding Milliseconds to DateTime fields (#173)"
This reverts commit 431dfa4156.
* Reworking the reader to use the column metadata for date types
* DbColumn -> DbColumnWrapper
* Fina tweaks to support DATETIME2(0)
* Removing the unused arguments
