This is a reworking of the unit tests to permit us to better test events from the service host. This new Event Flow Validator class allows creating a chain of events that can then be validated after execution of the request. Each event can have its own custom validation logic for verifying that the object sent via the service host is correct. It also allows us to validate that the order of events are correct.
The big drawback is that (at this time) the validator cannot support asynchronous events or non-determinant ordering of events. We don't need this for the query execution functionality despite messages being sent asynchronously because async messages aren't sent during unit tests (due to the db message event only being present on SqlDbConnection classes). If the need arises to do async or out of order event validation, then I have some ideas for how we can do that.
* Applying the event flow validator to the query execution service integration tests
* Undoing changes to events that were included in cherry-picked commit
* Cleaning up event flow validation to query execution
* Add efv to cancel tests
* Adding efv to dispose tests
* Adding efv to subset tests
* Adding efv to SaveResults tests
* Copyright
This basically is a replacement to the fix for Adding Milliseconds to DateTime fields. I didn't take into consideration that `DATE` columns would report as DateTime type. Date columns have a numeric scale set to 255, leading to the formatting string for the date time to include 255 millisecond places, which is invalid.
This change also reverses the change to store DateTime precision in the buffer file. Instead, the column metadata is now used when deserializing the rows from the db. `DATETIME` and `DATETIME2` columns are differentiated by their numeric scale while `DATE` columns are differentiated by their datatype name field.
More unit tests were added. Additionally, this fixes an unreported bug that `DATE` columns were being displayed with times, which is incorrect.
* Revert "Adding Milliseconds to DateTime fields (#173)"
This reverts commit 431dfa4156.
* Reworking the reader to use the column metadata for date types
* DbColumn -> DbColumnWrapper
* Fina tweaks to support DATETIME2(0)
* Removing the unused arguments
* DbColumn and ReliableConnection tests
* More retry connection tests
* More tests
* Fix broken peek definition integration tests
* Fix test bug
* Add a couple batch tests
* Add some more tests
* More tests for code coverage.
* Validation and Diagnostic tests
* A few more tests
* A few mote test changes.
* Update file path tests to run on Windows only
This is a slightly larger change than anticipated due the difference between `DATETIME`, `DATETIME2`, and `DateTime`. The `DATETIME` type always uses 3 decimal points of a second, while the `DATETIME2` type has 7 (although since `DATETIME2(7)` is default in SSMS suggesting that it is a variable precision type). Regardless of the db type, the engine returns `DateTime` C# type. The db types are only made visible via the column info, via the numeric precision and numeric scale. My findings were as such:
`DATETIME `: Precision = 23, Scale = 3
`DATETIME2`: Precision = 255, Scale = 7
The scale corresponds neatly with the number of second decimal points to show. The buffer file writer was modified to store both the scale and the number of ticks. Then the buffer file reader was modified to read in the precision and the number of ticks and generate the ToString version of the DateTime to add "f" as many times as there is scale, which corresponds to milliseconds.
* Code for writing milliseconds of datetime/datetime2 columns
* Adding unit tests
* Fixing potential bug with datetime2(0)
This change is part of the progressive results code. It will submit a notification from the service layer to indicate when execution of a batch has completed. This notification will contain the selection for batch, execution start time, and its ID. This will enable the extension to produce a header for the batch before the batch completes, in order to make it more clear to the user that execution is going on.
* Adding new event for batch start
* Unit tests
* Fixing comments as per @kevcunnane
The main change in this pull request is to add a new event that will be fired upon completion of a resultset but before the completion of a batch. This event will only fire if a resultset is available and generated.
Changes:
* ConnectionService - Slight changes to enable mocking, cleanup
* Batch - Moving summary generation into ResultSet class, adding generation of ordinals for resultset and locking of result set list (which needs further refinement, but would be outside scope of this change)
* Adding new event and associated parameters for completion of a resultset. Params return the resultset summary
* Adding logic for assigning the event a handler in the query execution service
* Adding unit tests for testing the new event /making sure the existing tests work
* Refactoring some private properties into member variables
* Refactor to remove SectionData class in favor of BufferRange
* Adding callback for batch completion that will let the extension know that a batch has completed execution
* Refactoring to make progressive results work as per async query execution
* Allowing retrieval of batch results while query is in progress
* reverting global.json, whoops
* Adding a few missing comments, and fixing a couple code style bugs
* Using SelectionData everywhere again
* One more missing comment
* Adding new notification type for result set completion
* Plumbing event for result set completion
* Unit tests for result set events
This includes a fairly substantial change to create a mock of the
ConnectionService and to create separate memorystream storage arrays. It
preserves more correct behavior with a integration test, fixes an issue
where the test db reader will return n-1 rows because the Reliable
Connection Helper steals a record.
* Adding locking to ResultSets for thread safety
* Adding/fixing unit tests
* Adding batch ID to result set summary
This is a fairly large set of changes to the unit tests that help isolate the effectiveness of the unit tests.
* Unit tests for query execution have been split into separate files for different classes.
* Unit tests have been added for the ResultSet class which previously did not have tests
* The InMemoryStreamWrapper has been improved to share memory, creating a simulated filesystem
* Creating a mock ConnectionService to decrease noisy exceptions and prevent "row stealing". Unfortunately this lowers code coverage. However, since the tests that touched the connection service were not really testing it, this helps keep us honest. But it will require adding more unit tests for connection service.
* Standardizing the await mechanism for query execution
* Cleaning up the mechanism for getting WorkspaceService mocks and mock FileStreamFactories
* Refactor the query execution tests into their own files
* Removing tests from ExecuteTests.cs that were moved to separate files
* Adding tests for ResultSet class
* Adding test for the FOR XML/JSON component of the resultset class
* Setting up shared storage between file stream readers/writers
* Standardizing on Workspace mocking, awaiting execution completion
* Adding comment for ResultSet class
The main feature of this pull request is a new callback that's added to the query class that is called when a batch has completed execution and retrieval of results. This callback will send an event to the extension with the batch summary information. After that, the extension can submit subset requests for the resultsets of the batch.
Other smaller changes in this pull request:
Refactor to assign a batch a id when its created instead of when returning the list of batch summaries
Passing the SelectionData around instead of extracting the values for it
Moving creation of BatchSummary into the Batch class
Retrieval of results is now permitted even if the entire query has not completed, as long as the batch requested has completed.
Also note, this does not break the protocol. It adds a new event that a queryRunner can listen to, but it doesn't require it to be listened to.
* Refactor to remove SectionData class in favor of BufferRange
* Adding callback for batch completion that will let the extension know that a batch has completed execution
* Refactoring to make progressive results work as per async query execution
* Allowing retrieval of batch results while query is in progress
* reverting global.json, whoops
* Adding a few missing comments, and fixing a couple code style bugs
* Using SelectionData everywhere again
* One more missing comment
Next round of code coverage test cases. Please review the commit for next iteration.
* Add connection retry tests
* More test coverage
* Update diagnostics end-to-end test
* Make save results asynchronous
* Prevent write share of file
* Lock objects in stages
* Create Save result objects
* refactor and write rows in batches
* CHange batchSize from test value
* Remove await in handler
* Removing the file reader as a member of the resultset
* Change Dispose to wait for save
* Change concurrentBag
* PascalCase variables
* Modify function signature and tests
* Safe file methods
* refactor ResultSets to Ilist and remove ToList
* Change dictionary key and prevent add to saveTasks during dispose
* Simplify row concatenation
* Fix prevent add
* Fix prevent add
* Add methods to expose saveTasks and isBeingDisposed
After much thinking, this change brings the message behavior into line with SSMS, including if the NOCOUNT is set. This is a somewhat non-obvious solution, but the StatementCompleted event handler will only be fired if there is a record count to return. We'll add the message for number of records if the StatementCompleted event is fired, otherwise we won't add any messages while processing the resultsets of the batch. If any messages are returned from the server, we'll capture those. Then, if at the end of the batch, we haven't collected any messages from StatementCompleted events or server messages, then we'll add the "completed successfully" message.
This matches behavior of SSMS that will only emit a "completed successfully" message if there were no other messages for the batch.
* Solution to issue, some unit tests needed to be tweaked
* Comments for the event handler
The two main changes in this pull request:
Launching query execution as an asynchronous task that performs a callback upon completion or failure of a query. (Which also sets us up for callbacks progressive results)
Moving away from using the Result of a query execution to return an error. Instead we'll use an error event to return an error
Additionally, some nice refactoring and cleaning up of the unit tests to take advantage of the cool RequestContext mock tooling by @kevcunnane
* Initial commit of refactor to run execution truely asynchronously
* Moving the storage of the task into Query class
Callbacks for completion of a query and failure of a query are setup as
events in the Query class. This actually sets us up for a very nice
framework for adding batch and resultset completion callbacks.
However, this also exposes a problem with cancelling queries and returning
errors -- we don't properly handle errors during execution of a query
(aside from DB errors).
* Wrapping things up in order to submit for code review
* Adding fixes as per comments
* Implemented connection cancellation
* Made connect requests return immediately and created a separate connection complete notification
* Fix spelling
* Fix sorting
* Add separate lock for cancellation source map
* Initial code for binding queue
* Fix-up some of the timeout wait code
* Add some initial test code
* Add missing test file
* Update the binding queue tests
* Add more test coverage and refactor a bit. Disable reliabile connection until we can fix it..it's holding an open data reader connection.
* A few more test updates
* Initial integrate queue with language service.
* Hook up the connected binding queue into al binding calls.
* Cleanup comments and remove dead code
* More missing comments
* Fix build break. Reenable ReliabileConnection.
* Revert all changes to SqlConnectionFactory
* Resolve merge conflicts
* Cleanup some more of the timeouts and sync code
* Address code review feedback
* Address more code review feedback
Fixing a bug where in various situations, the files used for temporary storage of query results would be leftover. In particular, the following changes were made:
* When the dispose query request is submitted, the corresponding query is now disposed in addition from being removed from the list of active queries
* When a query is cancelled, it is disposed after it is cancelled
* If a query already exists for a given ownerURI, the existing query is disposed before creating a new query
* All queries are disposed when the query execution service is disposed (ie, at shutdown of the service)
A unit test to verify the action of the dispose method for a ResultSet was added.
* Ensuring queries are disposed
Adding logic to dispose any queries when:
* URI that already has a query executes another query
* A request to dispose a query is submitted
* A request to cancel a query is submitted
* Small tweaks for cleanup of query execution service
* added support for timestamps
* fixed tests
* Moved message class to own file; added 'z' to end of date strings
* added default time constructor
* removed unnecessary z
* added time string format info in comment
* changed from utc time to using local time
* Do not use ReliableCommand in the query execution service.
* Fixing the logic to remove InfoMessage handlers from ReliableSqlConnection
* Adding test to query UDT
* inital pipe of line numbers and getting text from workspace services
* tests compile
* Fixed bug regarding tests using connections on mac
* updated tests
* fixed workspace service and fixed tests
* integrated feedback
* WIP for ability to localize cell values
* Changing how DateTimeOffsets are stored, getting unit tests going
* Reworking BufferFileStreamWriter to use dictionary approach
* Plumbing the DbCellValue type the rest of the way through
* Removing unused components to simplify contract
* Cleanup and making sure byte[] appears in parity with SSMS
* CR comments, small tweaks for optimizing LINQ
* Strings sweep for connection service
* String sweep for credentials service
* String sweep for hosting
* String sweep for query execution service
* String sweep for Workspace service
* Renaming utility namespace to match standards
Renaming Microsoft.SqlTools.EditorServices.Utility to
Microsoft.SqlTools.ServiceLayer.Utility to match the naming changes done a
while back. Also renaming them on the files that use them
* Namespace change on reliable connection
* Adding the new resx and designer files
* Final bug fixes for srgen
Fixing flakey moq package name
* Removing todo as per @kevcunnane
* Adding using statements as per @llali's comment
* Fixing issues from broken unit tests
Note: This feature contains changes that will break the contract for
saving as CSV and JSON. On success, null is returned as a message instead
of "Success". Changes will be made to the vscode component to handle this
change.
For whatever reason, launching a the HandleRequestSubsetRequest causes the
unit test to hang at the process of starting a new Task. This only appears
to happen on our build server (which runs Win2K12R2), which suggests a bug
with dotnet or XUnit. But, I've worked around it by changing that specific
unit test to be async/await and not use .Wait calls.
* Code changes to save results as csv
* changes to save resultset as csv
* removed csvHelper
* Retrieve right resultSet after batch execution]
* code clean up
* encode column names and use string.Join
* code review changes - property fix and rowBuilder removal
* changes to fix execution tests
* Code clean up
* Code clean up
* Test save as CSV
* Fix tests for Mac/Linux
* Add doc comment
* Add doc comments
* Delete file if exception occurs
* WIP for buffering in temporary file
* Adding support for writing to disk for buffering
* WIP - Adding file reader, factory for reader/writer
* Making long list use generics and implement IEnumerable
* Reading/Writing from file is working
* Removing unused 'skipValue' logic
* More tweaks to file buffer
Adding logic for cleaning up the temp files
Adding fix for empty/null column names
* Adding comments and cleanup
* Unit tests for FileStreamWrapper
* WIP adding more unit tests, and finishing up wiring up the output writers
* Finishing up initial unit tests
* Fixing bugs with long fields
* Squashed commit of the following:
commit df0ffc12a46cb286d801d08689964eac08ad71dd
Author: Benjamin Russell <beruss@microsoft.com>
Date: Wed Sep 7 14:45:39 2016 -0700
Removing last bit of async for file writing.
We're seeing a 8x improvement of file write speeds!
commit 08a4b9f32e825512ca24d5dc03ef5acbf7cc6d94
Author: Benjamin Russell <beruss@microsoft.com>
Date: Wed Sep 7 11:23:06 2016 -0700
Removing async wrappers
* Rolling back test code for Program.cs
* Changes as per code review
* Fixing broken unit tests
* More fixes for codereview
* CredentialService initial impl with Win32 support
- Basic CredentialService APIs for Save, Read, Delete
- E2E unit tests for Credential Service
- Win32 implementation with unit tests
* Save Password support on Mac v1
- Basic keychain support on Mac using Interop with the KeyChain APIs
- All but 1 unit test passing. This will pass once API is changed, but checking this in with the existing API so that if we decide to alter behavior, we have a reference point.
* Remove Username from Credentials API
- Removed Username option from Credentials as this caused conflicting behavior on Mac vs Windows
* Cleanup Using Statements and add Copyright
* Linux CredentialStore Prototype
* Linux credential store support
- Full support for Linux credential store with tests
* Plumbed CredentialService into Program init
* Addressing Pull Request comments
Query cancellation support is added via CancellationToken mechanisms that
were implemented previously. This change adds a new request type
"query/cancel" that will issue the cancellation token. Unit tests were
also added.
