* add package reference type for system db references
* Add radio buttons
* hookup
* add test
* handle changing reference type radio buttons
* cleanup
* update strings
* change style to type
* change more places with style to type
* blank lines
* Adding exclude folder and base for move folder
* checkpoint
* rename
* Fixing up tests
* Adding exclude test to projController
* Adding tests
* fixing order of service.moveX() calls
* Updating move() order in sqlproj service
* PR feedback
* unskipping
* reskipping test
* Fixing folder move conditional
* updating comments
* Move call to STS move api into project.ts (#22071)
* move call to STS move api into project.ts
* remove undefined
* Remove convert to sdk style code (#22081)
* remove convert to sdk style code
* remove from package.json
* Merging latest from main (#22097)
* [SQL-Migration] Login migrations telemetry (#22038)
This PR enhances telemetry for login migrations (and in the following ways:
Add details for starting migration (number of logins migrating, type of logins)
Log Migration result (number of errors per step, duration of each step, type of logins, if system error occurred)
Add sql-migration extension to our telemetry
Adds details when trying to connect to target
Tracks clicking "done" from the wizard
Fixes bucketizing for navigating telemetry in the login migration wizard
Sample usage of kusto query for new telemetry:
RawEventsADS
| where EventName contains 'sql-migration'
| extend view = tostring(Properties['view'])
| extend action = tostring(Properties['action'])
| extend buttonPressed = tostring(Properties['buttonpressed'])
| extend pageTitle = tostring(Properties['pagetitle'])
| extend adsVersion = tostring(Properties['common.adsversion'])
| extend targetType = tostring(Properties['targettype'])
| extend tenantId = tostring(Properties['tenantid'])
| extend subscriptionId = tostring(Properties['subscriptionid'])
| where view contains "login"
//| where adsVersion contains "1.42.0-insider"
| where ClientTimestamp >= ago(18h)
| project EventName, ClientTimestamp, SessionId, view, pageTitle, action, buttonPressed, targetType
, tenantId, subscriptionId
, adsVersion, OSVersion, Properties
* Add Secure Enclaves dropdown with customizable Advanced options (#22019)
* Update extension READMEs (#22079)
* Fix query-history README images (#22084)
* [Loc] update to mssql and sql-migration xlf files (#22087)
* [Loc] small fix to Portuguese lcl file (#22088)
* [Loc] small fix to Portuguese lcl file
* remove newline
* Adding None bindings to the sqlProjects service (#22085)
* Adding None bindings
* updating names of None bindings
---------
Co-authored-by: AkshayMata <akam520@gmail.com>
Co-authored-by: Cheena Malhotra <13396919+cheenamalhotra@users.noreply.github.com>
Co-authored-by: Charles Gagnon <chgagnon@microsoft.com>
Co-authored-by: Alex Ma <alma1@microsoft.com>
* Swap add and update sqlcmd variables in sql projects to use STS apis (#22086)
* delete sqlcmd variable working
* undo add
* remove variable from add and update sqlcmd variable apis
* hookup add and edit sqlcmd variable
* update vscode-mssql.d.ts
* move add and edit to project.ts
* update STS and tests
* move delete sqlcmd variable to project.ts (#22104)
* add test for add and edit sqlcmd variable (#22107)
* Swapping property access to STS (#22115)
* checkpoint
* Adding sqlproj property bindings
* Swapping out property loading and setting
* consolidating to this.sqlProjService
* Update dacpac reference to use STS api (#22116)
* Update add dacpac reference to use STS api
* remove changes for project ref
* validate unset settings from add database reference dialog
* update one more place getting sqlprojservice
* addressing comments
* fix path for dacpac reference (#22143)
* Swap add project reference to call STS (#22148)
* initial changes for swapping add project reference
* fix include path
* move common logic to helper function
* read sqlcmd variables from STS (#22147)
* Swapping .sqlproj- and crossplat compatibility-related functions to use STS (#22146)
* Supporting roundtrip
* Updating sqlproj style checks and cross-platform compatibility to use STS
* removing unnecessary awaits
* Fixing assertions
* Adding roundtrip update confirmations
* test cleanup
* cleaning up comment; localizing error
* Swap add system db reference (#22171)
* swap adding system database references
* fix tests
* remove only in test
* Read database references from STS (#22179)
* Read database references from STS
* fix system dacpac names
* fix project reference name
* Swap changeTargetPlatform to call STS (#22172)
* swap changeTargetPlatform to call STS
* Address comments
* De-duplicating enum for SystemDatabase (#22196)
* Deudping SystemDatabase enum
* simplifying enum refs
* Removing the now-unused imports code from SqlProjects (#22199)
* Removing unused importTargets entries
* whitespace; also to retrigger github checks on correct branch
* Hooking in Move() for Nones and PublishProfiles (#22200)
* Swap delete database reference to call STS (#22215)
* initial changes
* update contracts
* remove unnecessary info from SystemDatabaseReferenceProjectEntry
* uppercase master and msdb
* cleanup
* update test
* update comment
* undo change in projectController.ts
* remove unused system dacpac helper functions (#22204)
* more cleanup of project.ts (#22242)
* fix a couple database reference tests (#22248)
* Organizing sqlcmd variable and db reference code (#22284)
* organize database references and sqlcmd variable functions
* separate database reference tests
* Script and folder get + add support (#22166)
* Initial sqlobjectscripts
* adding mock filter
* test fixing
* another test passing
* swapping pre/post/none checkpoint
* awaiters
* convert addExistingItem
* swapping folders
* removing print
* stripping out project folder and file readers
* adding some regions
* Updating sqlproj style checks and cross-platform compatibility to use STS
* Updating sqlproj style checks and cross-platform compatibility to use STS
* added type property to tree items
* projectController swapovers
* removing imported targets
* Deleting the last of the TS XML parsing!
* Removing old functions
* renamed readNoneScripts to readNoneItems
* fixing path passed to STS calls
* remove system dacpac tests that were moved to DacFx (#22299)
* fix error when opening file after adding a file to sql project (#22309)
* fix error when opening file after adding a file to sql project
* remove unused import
* fix exclude for table and externalStreamingJob (#22310)
* add reload project (#22313)
* set DSP from STS (#22320)
* fix adding post-deployment script and existing item (#22317)
* Test cleanup for .sqlproj content operations (#22330)
* Fixing up tests
* sqlproj content operations tests
* remove only
* Cleanup
* Correcting collation
* cleanup constants.ts (#22336)
* fix folders not showing in project tree (#22319)
* Fix project controller tests (#22327)
* fixing ProjectController tests after swap
* remove only from database reference tests
* change system dbs back to lowercase in sql projects (#22353)
* Bump tools service
* Updated yarn.lock file
* pass SystemDacpacsLocation when building legacy style sql projects (#22329)
* Benjin/fix types (#22365)
* Updated yarn.lock file
* Fixing types
* fix projectController tests (#22375)
* Fixing the deletion flow for files and folders (#22366)
* checkpoint
* PR feedback
* Fixing up SDK-style operations project test group (#22377)
* Fixing up SDK-style project test group
* Removing .only
* Fixing up database reference tests (#22380)
* Fixing DB reference test group
* Extra cleanup
* removing only
* Consolidating None and PublishProfile; lighting up test (#22382)
* Lighting up project property tests (#22395)
* Checkpoint
* Lighting up project property tests
* removing timeout
* Fixing buildHelper test (#22400)
* Unskipping up roundtrip test (#22397)
* Refactoring database references to split literalVariable from databaseName (#22412)
* refactoring database references to split databaseVariableLiteralValue out from databaseName
* renaming more properties
* Removing branch in entry population
* removing only
* Fixing baselines for delete test
* PR feedback
* Fixing up ProjectTree tests (#22419)
* Fixing up projectTree tests
* remove only
* Updating projectController exclude test (#22413)
* Updating test
* moving filtering for external folders to readFolders() method
* Removing EntryType import
* fix ups (#22435)
* adding extra info for test failure
* hide exclude folder from context menu until it's supported (#22454)
* Adding current test name to generated folder to avoid conflicts (#22478)
* Adding current test name to generated folder to avoid conflicts
* passing correct test parameter in
* Adding trimming and entropy
* Deleting unused baselines (#22497)
* Replacing addToProject() with addSqlObjectScripts() (#22489)
* checkpoint
* Fixing test
* Updating file scraper function to filter only to .sql files (no folders, no txt)
* changing var names to reflect that the lists only contain .sql scripts
---------
Co-authored-by: Kim Santiago <31145923+kisantia@users.noreply.github.com>
Co-authored-by: AkshayMata <akam520@gmail.com>
Co-authored-by: Cheena Malhotra <13396919+cheenamalhotra@users.noreply.github.com>
Co-authored-by: Charles Gagnon <chgagnon@microsoft.com>
Co-authored-by: Alex Ma <alma1@microsoft.com>
* Read publish profiles stored in sqlproj file and present it in the projects tree
* Save publish profile and add it to sqlproj file, and present it in the tree
* Fix context menu operations
* Add tests
* Address comments
* Revert "Revert "Fix sql projects not using OutputPath in sqlproj for publishing (#19987)" (#20127)"
This reverts commit dc7522c661.
* read configuration from sqlproj
* Add more tests
* use last OutputPath in sqlproj
* Add sqlproj property to trace the origin of the project.
As part of the database migration process (schema conversion, in particular) we want to be able to tell when converted schemas are being built/deployed to the actual database server. Given that we rely on the SQL Database Projects ADS extension for the compilation/deployment, we don't have too many options other than updating the said extension.
The suggested approach is to make the following changes:
1) Add new property to the sqlproj file (called `DatabaseSource`), which will maintain the origin(s) of the project. The property can contain multiple values (separated by semicolon), in case same project contains objects produced by multiple sources (extract schema, convert from another database, etc.).
2) During build and deploy actions, send the well-known values from the newly added property to the telemetry. We don't want to send any random value of the property, as it may raise some privacy concerns. Instead we define a list of the well-known values that we know do not carry any personal information and send those, if they are specified.
This change adds all necessary APIs to the SQl Database projects extension which will be consumed by our migration extensions to populate new `DatabaseSource` property.
* Use `undefined` instead of `null`
Co-authored-by: Kim Santiago <kisantia@microsoft.com>
* add support for removing file in new style project
* fix test
* only load files, not whole project when checking if a <Build Remove> needs to be added
* merge changes
* fixes after merge
* use glob to get files for new style msbuild sdk sqlproj
* add tests
* cleanup
* fix test
* don't show bin and obj files and folders
* handle other glob patterns
* fix duplicate entries getting added for glob patterns in project's folder
* don't show SSDT update warning for new style projects
* update strings
* add checks for the 2 other ways to specify msbuild sdk
* add link to docs on how to use project sdk
Co-authored-by: Kim Santiago <kisantia@Kims-MacBook-Pro.local>
* Automatically add intermediate folders for SQL project items.
While using the SQL database projects through the API, I noticed that project may end up in somewhat inconsistent state, where files will be added to the project, but their parent folders will not. This in turn resulted in failure to remove these folders from project - they will show up in the UI tree, but deleting them will cause an error. In order to align with how Visual Studio manages the projects, this change will ensure that all intermediate folders are present in the project, when new files or folders are added.
While this change improves project "correctness" when accessing it through SQL projects extension APIs, there is still a possibility that someone will open an "incorrect" previously created project. This change does not address it and folder removal may still fail.
* Update the code to never throw on duplicate items when adding files and folders to project.
After a conversation with the sqlproj owners, we agreed that there are no scenarios that would prompt us to throw an error, if duplicate item is being added to the project. Ultimately, the goal of such a request would be to have an item in the project file, which is already present, therefore the call becomes a no-op.
This allowed me to simplify the new code that was ensuring all intermediate folders are present in the project when adding files and folders.
* Expose default database collation through 'sql-database-projects' extension API.
For the purpose of schema conversion we would need to know whether target database is configured as CI or CS. This will be used to produce a warning, if we detect a case-sensitive identifier, but database is configured as CI. In order to support this scenario we need to access `<DefaultCollation/>` property of the project.
This change adds new method to the `ISqlProject` interface that allows to read the value of the project property. There already was similar method for the SQL version/platform (`<DSP/>` property) and while working on the change, I took an opportunity to refactor the way project properties are extracted from the XML. Original code was hardcoded in the `getProjectTargetVersion` and I extracted it into separate `evaluateProjectPropertyValue` helper, that can be used in the future by any getter or access method that is supposed to return a value of the single property. This also allows us to improve the way properties are retrieved from the XML. Today the logic is very rudimentary - we read the first matching XML element with the required name. This is not correct as it does not verify the parent to be `<PropertyGroup/>`, neither it evaluates the `Condition` attributes nor property value itself. I did not invest in this, but added a TODO there explaining that the method may not perform as expected.
After the helper method was added, I updated the existing `getProjectTargetVersion` method to leverage it. The only complication here was the error throwing logic, as it was using custom error message. I preserved that, as there were tests verifying it already. For the new accessor method I did not introduce a special error message and rely on generic one I defined for use within the helper method. Additionally, for the collation we return default value of `SQL_Latin1_General_CP1_CI_AS`, if project does not have the property defined. This is what SSDT for Visual Studio shows in the UI when property is missing and I decided to align with that.
Finally, I added tests for both - original `getProjectTargetVersion` and new collation extraction method to make sure they work as expected. While working on the tests, I've noticed complaints that some rejected promises were not awaited. I added missing `await`s.
This change introduces new boolean flag for `addToProject`, `addScriptItem` and `addFolderItem` APIs that allows to skip files/folders if they are already included in the project file. Original behavior was to throw an error if duplicate entry is encountered, so I kept that as a default (new flag is not set).
I started by fixing the original behavior, as it was not always working as expected. In our extension that relies on `addToProject` API we've hit an issue where duplicate items were added and no errors were thrown. There was a test for this scenario, but unfortunately the test had few bugs in it as well, so I addressed that first. First issue with the test code was missing `await` on the `testUtils.shouldThrowSpecificError` calls, so test was not actually testing anything. After adding missing keywords, I hit an issue where exception was not thrown, but this turned out to be different issue, compared to what we were hitting. In the test code, it was using the very first folder from the generate list of test entires. This folder wass actually the root of the project (where sqlproj file is located), and `addToProject` API had a special case to ignore the root. This means neither first call nor second call was actually adding anything to the project and no errors were produced. I fixed this problem by using next available folder in the generated files list.
After addressing the test code I could not reproduce the issue that we were seeing with duplicate entires being added, everything was working as expected and errors were thrown. I started adding more tests that better resemble our production scenario - add files in subfolders, add files to existing project, rather than a new one. Finally I was able to reproduce the problem in tests when adding a file in a subfolder to an existing project. After investigation this turned out to be an issue with mismatch in how `relativePath` is maintained within the `FileProjectEntry`. When loading an existing project, `relativePath` is populated based on the value of the `Include` attribute of the `Build` item. This attribute is normalized to Windows-style path, using `\`, so for nested file you will have `folder\file.sql`. When adding new item to the project, one could pass either Windows or Unix-style path (`folder/file.sql`), so the path comparison between loaded Windows-style path and newly added Unix-style path was failing, resulting in them being treated as different items. In fact, `addToProject` API that we were using was relying on `Uri` helpers to extract relative path, thus the path was forced to be Unix-style and path was never the same as the loaded one. After this discovery I added a dedicated test to validate the round-trip of the `relativePath` for serialized and desirialized project file.
In order to address this problem, I updated the factory method `createFileProjectEntry` to always run `utils.convertSlashesForSqlProj` on the relative path to ensure we have Windows-style path stored in there. I also optimized the helper code slightly to not do split/join, if there are no split points in the input string, which should eliminate unnecessary array instantiation. It is worth mentioning that I had to normalize the input relative paths in the `addScriptItem` and `addFolderItem` APIs, because there is no guarantee that they will be Windows-style when we try to compare them to `relativePath` of the existing project items.
Finally I was able to add a simply flag and update the condition to return existing record, if duplicates were allowed. I also updated typings file for the extension and added tests to cover this scenario.
* don't allow adding the same file or folder to the sqlproj if it has already been added
* add a couple more checks in test
* toLowerCase when comparing
* expose addToProject in dataworkspace.d.ts
* remove changes in data workspace extension
* add sqldbproj.d.ts
* change list to be Uris instead of strings
* don't add files/folders if any don't exist
* fix test on windows
* Fix project context menu actions (#12541)
* delete works again
* make fewer changes
* update all sql db project commands
* cleanup
* Remove old projects view (#12563)
* remove old projects view from file explorer view
* fix tests failing
* remove projects in open folder opening up in old view
* Update db reference dialog to show projects in the workspace (#12580)
* update database reference dialog to show projects in the workspace in the project dropdown
* remove workspace stuff from sql projects extension
* undo change
* add class that implements IExtension
* undo a change
* update DataWorkspaceExtension to take workspaceService as a parameter
* add type
* Update sql database project commands (#12595)
* remove sql proj's open and create new project from comman palette
* hook up create project from database to data workspace
* rename the remaining import databases to create project from database
* remove open, new, and close commands
* expose addProjectsToWorkspace() in IExtension instead of calling command
* Addressing comments
* fix failing sql project tests (#12651)
* update SSDT projects opened in projects viewlet (#12669)
* fix action not refreshing the tree issue (#12692)
* fix adding project references in new projects viewlet (#12688)
* Remove old projects tree provider (#12702)
* Remove old projects tree provider and fix tests
* formatting
* update refreshProjectsTree() to accept workspaceTreeItem()
* Cleanup ProjectsController (#12718)
* remove openProject from ProjectController and some cleanup
* rename
* add project and open project dialogs (#12729)
* empty dialogs
* wip
* new project dialog implementation
* revert gitattributes
* open project dialog
* implement add project
* remove icon helper
* refactor
* revert script change
* adjust views
* more updates
* make data-workspace a builtin extension
* show the view only when project provider is detected (#12819)
* only show the view when proj provider is available
* update
* fix sql project tests after merge (#12793)
* Update dialogs to be closer to mockups (#12879)
* small UI changes to dialogs
* center radio card group text
* Create workspace if needed when opening/new project (#12930)
* empty dialogs
* wip
* new project dialog implementation
* revert gitattributes
* open project dialog
* implement add project
* remove icon helper
* refactor
* revert script change
* create workspace
* initial changes
* create new workspace working
* fix tests
* cleanup
* remove showWorkspaceRequiredNotification()
* Add test for no workspace open
* update blue buttons
* move loading temp project to activate() instead of workspaceService constructor
* move workspace creation warning message to before project is created
* pass uri to createWorkspace
* add tests
Co-authored-by: Alan Ren <alanren@microsoft.com>
* Additional create workspace changes (#13004)
* Dialogs workspace updates (#13010)
* adding workspace text boxes
* match new project dialog to mockups
* Add validation error message for workspace file
* add enterWorkspace api
* add warning message for opening workspace
* cleanup
* update commands to remove project so they're more generic
* remove 'empty' from string
* Move default project location setting to data workspace extension (#13022)
* remove project location setting and notification from sql database projects extension
* add default project location setting to data workspace extension
* fix typo
* Add back project name incrementing
* other merge fixes
* fix strings from other PR
* default to last opened directory instead of home directory if no specified default location
* A few small updates (#13092)
* fix build error
* update title for inputboxes
* add missing file
* Add tests for data workspace dialogs (#13324)
* add tests for dialogs
* create helper functions
* New project dialog workspace inputbox fixes (#13407)
* workspace inputbox fixes
* fix folder icons
* Update package.jsons and readme (#13451)
* update package.jsons
* update readme
* add workspace information to open existing dialog (#13455)
Co-authored-by: Alan Ren <alanren@microsoft.com>
* Added Tools Service call for ValidateStreamingJob
* Partial addition of ESJ
* adding test mocks
* Validation working
* Modifying command visibility logic to submatch ESJs in addition to files
* Changed string literal to constant, corrected attribute order
* Added tests
* correcting casing that's causing test failures on linux
* Swapping Thenable for Promise
* excluded validate from command palette
