mirror of
https://github.com/ckaczor/azuredatastudio.git
synced 2026-02-16 10:58:30 -05:00
Refresh master with initial release/0.24 snapshot (#332)
* Initial port of release/0.24 source code * Fix additional headers * Fix a typo in launch.json
This commit is contained in:
Karl Burtram
@@ -1,64 +1,46 @@
|
||||
# VS Code Smoke Testing
|
||||
# VS Code Smoke Test
|
||||
|
||||
- Run `npm install`
|
||||
- Start the tests: `npm test -- --latest "path/to/binary"`.
|
||||
## How to run
|
||||
|
||||
If you want to include 'Data Migration' area tests use `npm test -- --latest path/to/binary --stable path/to/currentStable` respectively.
|
||||
|
||||
Detailed prerequisites and running steps are described [in our smoke test wiki](https://github.com/Microsoft/vscode/wiki/Smoke-Test#automated-smoke-test).
|
||||
|
||||
# Architecture
|
||||
* `main.js` is used to prepare all smoke test dependencies (fetching key bindings and 'Express' repository, running `npm install` there).
|
||||
* `mocha-runner.js` launches Mocha programmatically. It is spawned in Node environment from main.js to ensure that it is possible to listen on `stderr`s (primary `process.stderr` is not readable otherwise). This is accomplished because WebDriverIO command deprecation warnings need to be redirected to a separate log. Those warnings are coming from WebDriverIO because ChromeDriver has not migrated from JsonWire to W3C WebDriver protocol.
|
||||
* `test.ts` contains the main smoke test suite calling the tests that are bundled in areas and defined in `./tests/`. It includes all tests separated into mocha `describe()` groups that represent each of the areas of [Smoke Test document](https://github.com/Microsoft/vscode/wiki/Smoke-Test).
|
||||
|
||||
* `./areas/` folder contains a `.ts` file per each area of the document. E.g. `'Search'` area goes under `'search.ts'`. Every area file contains a list of methods with the name that represents the action that can be performed in the corresponding test. This reduces the amount of test suite code and means that if the UI changes, the fix need only be applied in one place. The name of the method reflects the action the tester would do if he would perform the test manually. See [Selenium Page Objects Wiki](https://github.com/SeleniumHQ/selenium/wiki/PageObjects) and [Selenium Bot Style Tests Wiki](https://github.com/SeleniumHQ/selenium/wiki/Bot-Style-Tests) for a good explanation of the implementation. Every smoke test area contains methods that are used in a bot-style approach in `main.ts`.
|
||||
* `./spectron/` wraps the Spectron, with WebDriverIO API wrapped in `client.ts` and instance of Spectron Application is wrapped in `application.ts`.
|
||||
|
||||
* `./test_data/` folder contains temporary data used by smoke test (cloned express repository, temporary user-data-dir/extensions-dir).
|
||||
* `./test_data/screenshots` has action screenshots captured by a smoke test when performing actions during runtime. Screenshots are split in folders per each test.
|
||||
|
||||
# Adding new area
|
||||
To contribute a new smoke test area, add `${area}.ts` file under `./areas/`. All related tests to the area should go to the alike named file under `./tests/${area}.ts`. This has to follow the bot-style approach described in the links mentioned above. Methods should be calling WebDriverIO API through `SpectronClient` class. If there is no existing WebDriverIO method, add it to the class.
|
||||
|
||||
# Adding new test
|
||||
To add new test, `./test/${area}.ts` should be updated. The same instruction-style principle needs to be followed with the called area method names that reflect manual tester's actions.
|
||||
|
||||
# Debugging
|
||||
1. Add the following configuration to launch.json, specifying binaries in `args`:
|
||||
```json
|
||||
{
|
||||
"type": "node",
|
||||
"request": "launch",
|
||||
"name": "Launch Smoke Test",
|
||||
"program": "${workspaceRoot}/test/smoke/out/main.js",
|
||||
"cwd": "${workspaceRoot}/test/smoke",
|
||||
"timeout": 240000,
|
||||
"port": 9999,
|
||||
"args": [
|
||||
"-l",
|
||||
"path/to/Code.exe"
|
||||
],
|
||||
"outFiles": [
|
||||
"${cwd}/out/**/*.js"
|
||||
]
|
||||
}
|
||||
```
|
||||
2. In main.js add `--debug-brk=9999` as a first argument to the place where `out/mocha-runner.js` is spawned.
|
||||
# Dev
|
||||
npm run smoketest
|
||||
|
||||
# Screenshots
|
||||
Almost on every automated test action it captures a screenshot. These help to determine an issue, if smoke test fails. The normal workflow is that you understand what code is doing and then try to match it up with screenshots obtained from the test.
|
||||
# Specific build
|
||||
npm run smoketest -- --build "path/to/code"
|
||||
|
||||
# Running "Out of Sources"
|
||||
If you did a fix in VS Code that you need in order for the smoke test to succeed, here is how you can run the smoke test against the sources of VS Code:
|
||||
* Set related environment variables in the console:
|
||||
* `export NODE_ENV=development`
|
||||
* `export VSCODE_DEV=1`
|
||||
* `export VSCODE_CLI=1`
|
||||
* open `application.ts`
|
||||
* pass in the vscode folder as argument to the application
|
||||
* e.g. instead of `args: args` type `args: ['/Users/bpasero/Development/vscode', ...args]`
|
||||
* `cd test/smoke`
|
||||
* `npm install`
|
||||
* `npm test -- --latest <path to electron>`
|
||||
* e.g. on macOS: `npm test -- --latest <path to vscode>/.build/electron/Code\ -\ OSS.app/Contents/MacOS/Electron`
|
||||
# Data Migration tests
|
||||
npm run smoketest -- --build "path/to/code-insiders" --stable "path/to/code"
|
||||
```
|
||||
|
||||
The script calls mocha, so all mocha arguments should work fine. For example, use `-f Git` to only run the `Git` tests.
|
||||
|
||||
By default, screenshots are not captured. To run tests with screenshots use the argument `--screenshots`.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Beware of **state**. The tests within a single suite will share the same state.
|
||||
|
||||
- Beware of **singletons**. This evil can, and will, manifest itself under the form of FS paths, TCP ports, IPC handles. Whenever writing a test, or setting up more smoke test architecture, make sure it can run simultaneously with any other tests and even itself. All test suites should be able to run many times in parallel.
|
||||
|
||||
- Beware of **focus**. **Never** depend on DOM elements having focus using `.focused` classes or `:focus` pseudo-classes, since they will lose that state as soon as another window appears on top of the running VS Code window. A safe approach which avoids this problem is to use the `waitForActiveElement` API. Many tests use this whenever they need to wait for a specific element to _have focus_.
|
||||
|
||||
- Beware of **timing**. You need to read from or write to the DOM... yeah I know. But is it the right time to do that? Can you 100% promise that that `input` box will be visible and in the DOM at this point in time? Or are you just hoping that it will be so? Every time you want to interact with the DOM, be absolutely sure that you can. Eg. just because you triggered Quick Open, it doesn't mean that it's open; you must wait for the widget to be in the DOM and for its input field to be the active element.
|
||||
|
||||
- Beware of **waiting**. **Never** wait longer than a couple of seconds for anything, unless it's justified. Think of it as a human using Code. Would a human take 10 minutes to run through the Search viewlet smoke test? Then, the computer should even be faster. **Don't** use `setTimeout` just because. Think about what you should wait for in the DOM to be ready, then wait for that instead.
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Certain keys don't appear in input boxes (eg: <kbd>Space</kbd>)
|
||||
|
||||
This is a **waiting** issue. Everytime you send keys to Code, you must be aware that the keybinding service can handle them. Even if you're sure that input box is focused.
|
||||
|
||||
Here's an example: when opening quick open, focus goes from its list to its input. We used to simply wait for the input to have focus and then send some text to be typed, like `Workbench: Show Editor`; yet, only `Workbench:ShowEditor` would be rendered in the input box. This happened due to the fact that the [`ListService` takes 50ms to unset the context key which indicates a list is focused](https://github.com/Microsoft/vscode/blob/c8dee4c016d3a3d475011106e04d8e394d9f138c/src/vs/platform/list/browser/listService.ts#L59). The fix was to [wait 50ms as well on the smoke test](https://github.com/Microsoft/vscode/blob/b82fa8dcb06bbf9c85c1502d0d43322e2e9d1a59/test/smoke/src/areas/quickopen/quickopen.ts#L65).
|
||||
|
||||
### I type in a Monaco editor instance, but the text doesn't appear to be there
|
||||
|
||||
This is a **waiting** issue. When you type in a Monaco editor instance, you're really typing in a `textarea`. The `textarea` is then polled for its contents, then the editor model gets updated and finally the editor view gets updated. It's a good idea to always wait for the text to appear rendered in the editor after you type in it.
|
||||
|
||||
### I type in a Monaco editor instance, but the text appears scrambled
|
||||
|
||||
This is an issue which is **not yet fixed**. Unfortunately this seems to happen whenever the CPU load of the system is high. Rerunning the test will often result in a successful outcome.
|
||||
Reference in New Issue
Block a user