Hook up Notebook execution edits (#17943)

* Start rerouting VSCode cell execution APIs.

* Add more conversion code.

* Convert VSCode notebook registrations into ADS equivalents.

* Update vscode notebook provider kernels when notebook controller's supportedLanguages are set.

* Update an error message.

* Add another session argument.

* Add base classes for converting notebook serializers.

* Disable some vscode notebook methods.

* Disable more vscode APIs.

* Disable more stuff.

* Start implementing serializer notebook data conversions.

* Use direct references to extension host notebook methods, rather than azdata ones.

* Add a comment.

* Remove a space.

* Use import type to fix module loading errors.

* Use internal cancellation token class.

* Start adding cell output conversion.

* Convert data from byte array to a string.

* More output work.

* Use a Set for proxy filtering.

* Start adding tests.

* Include metadata in cell conversion. Fix other test failures.

* Fix serialize tests.

* Add more tests.

* Remove wildcard characters from vscode filenames.

* Start implementing session details.

* Add more kernel info.

* Add kernel spec.

* Add Future callback wrapper class.

* Start implementing execute conversion.

* Pass notebook URI to requestExecute.

* Start working on CellExecution methods.

* Move some code around to fix layering issues.

* Use proxy to access browser code, rather than direct imports.

* Move files around to fix layering issues.

* Remove unused imports.

* Start implementing some notebook cell execution behaviors.

* Revert some unnecessary extHost API changes.

* Check for nbformat.

* Also handle nbformat in serialize case.

* Active notebook extensions when resolving NotebookInput.

* Fix nbformat handling.

* Disable VSCode notebooks code.

* Filter out notebook services from registration assertion.

* Wait for providers to load before calling canResolve.

* Use controller's viewType for notebook provider ID, instead of controller ID.

* Start adding extHostNotebook tests for new APIs.

* Re-order proxy calls.

* Remove commented code.

* Move vscode provider files to browser folder. Fix RPC serialization issues by using readonly field instead of getter for providerId.

* Add a comment.

* Remove unnecessary dispose call.

* Handle disposable from registerExecuteProvider.

* Remove a comment.

* Remove unnecessary provider fields.

* Remove reference to notebook service to fix circular reference issue in stringify.

* Add object types for methods in ADSNotebookController.

* Wait for controller languages to be ready before marking session manager as ready.

* Add correct promise.

* Add undefined return type for optional supportedLanguages property.

* Refine promise logic.

* Move vscode functionality back to ExtHostNotebook, since the NotebookService can't be passed back over RPC (some kind of circular reference error).

* Fix remaining issues from last commit.

* Replace "not implemented" methods with placeholder return types in order to enable testing.

* Also wait for execution handler to be set before marking session manager as ready.

* Fix usage of NotebookRegistry when updating provider description languages.

* Refine file extension conversion.

* Fix file extension conversion to match ADS extension behavior.

* Emit new provider registration event when adding supported languages.

* Remove checks for duplicate file providers and kernels.

* Fix a test failure.

* Fix file extension parsing.

* Use default executeManager if one isn't defined for provider in notebookModel.

* Add descriptors for waiting on standardKernels registration.

* Increase timeout

* Add an error message.

* Start working on retrieving default kernel from registered providers, rather than always falling back to SQL.

* Revert "Start working on retrieving default kernel from registered providers, rather than always falling back to SQL."

This reverts commit 1916ea1ce3a0072f51bec683116dc7bb6c7aefdc.

* Emit activation events after provider registration.

* Wait on standard kernels availability when getting an execute provider.

* Throw an error if session manager isn't ready yet.

* Actually resolve language promise correctly.

* Add some checks for undefined notebook data objects.

* Create kernel spec data correctly.

* Add extension changes for local testing only.

* Clean up test class.

* Add a reminder comment.

* Undo commented out notebook stuff

* Temporarily hard code default kernel.

* Retrieve default kernel in notebookModel if it's not already provided.

* Revert an import change.

* Remove unnecessary method from extHostNotebook.

* Move an interface around.

* wip

* Check for proposed API for some VSCode extHost methods.

* Remove a comment.

* Fix notebookUtils tests.

* Fix notebookModel tests.

* Fix notebookFindModel tests.

* Fix notebookViewsExtension tests.

* Fix remaining notebookView tests.

* Refactor output conversion functionality into separate methods.

* Update some unit tests for output conversion.

* Move a method.

* Rename conversion methods to fit acronym styling.

* Add another conversion test case.

* Revert local testing changes.

* Remove old method.

* cleanup

* Remove some comments.

* Move localized string to locConstants.

* Add a space to loc string.

* Add comments to new SQL Carbon Edit tags.

* Create constants for nbformat and nbformat_minor.

* Move some vscode-only fields to proposed APIs.

* Check for valid state

* Properly null check

* Adding logging for provider wait timeouts.

* wip update

* Fix compile

* Switch to cell edits

* Update docs

* Remove custom output type

* cleanup

* fix

* cleanup

* more cleanup

* Fixes

* Fix tests and lint errors

Co-authored-by: Cory Rivera <corivera@microsoft.com>
This commit is contained in:
Charles Gagnon
2022-01-04 16:35:16 -08:00
committed by GitHub
parent c7bb58ceb9
commit 4a2b31f3ba
24 changed files with 304 additions and 97 deletions

View File

@@ -9,6 +9,11 @@ import { INotebookKernelDto2 } from 'vs/workbench/api/common/extHost.protocol';
import { Emitter, Event } from 'vs/base/common/event';
import * as extHostTypeConverters from 'vs/workbench/api/common/extHostTypeConverters';
import { Deferred } from 'sql/base/common/promise';
import { ExtHostNotebookDocumentsAndEditors } from 'sql/workbench/api/common/extHostNotebookDocumentsAndEditors';
import { URI } from 'vs/base/common/uri';
import { VSCodeContentManager } from 'sql/workbench/api/common/vscodeSerializationProvider';
import { NotebookCellExecutionTaskState } from 'vs/workbench/api/common/extHostNotebookKernels';
import { asArray } from 'vs/base/common/arrays';
type SelectionChangedEvent = { selected: boolean, notebook: vscode.NotebookDocument; };
type MessageReceivedEvent = { editor: vscode.NotebookEditor, message: any; };
@@ -34,6 +39,7 @@ export class ADSNotebookController implements vscode.NotebookController {
private _viewType: string,
private _label: string,
private _addLanguagesHandler: (providerId, languages) => void,
private _extHostNotebookDocumentsAndEditors: ExtHostNotebookDocumentsAndEditors,
private _handler?: ExecutionHandler,
preloads?: vscode.NotebookRendererScript[]
) {
@@ -135,7 +141,7 @@ export class ADSNotebookController implements vscode.NotebookController {
}
public createNotebookCellExecution(cell: vscode.NotebookCell): vscode.NotebookCellExecution {
return new ADSNotebookCellExecution(cell);
return new ADSNotebookCellExecution(cell, this._extHostNotebookDocumentsAndEditors);
}
public dispose(): void {
@@ -157,7 +163,8 @@ export class ADSNotebookController implements vscode.NotebookController {
class ADSNotebookCellExecution implements vscode.NotebookCellExecution {
private _executionOrder: number;
constructor(private readonly _cell: vscode.NotebookCell) {
private _state = NotebookCellExecutionTaskState.Init;
constructor(private readonly _cell: vscode.NotebookCell, private readonly _extHostNotebookDocumentsAndEditors: ExtHostNotebookDocumentsAndEditors) {
this._executionOrder = this._cell.executionSummary?.executionOrder ?? -1;
}
@@ -178,30 +185,69 @@ class ADSNotebookCellExecution implements vscode.NotebookCellExecution {
}
public start(startTime?: number): void {
// No-op
this._state = NotebookCellExecutionTaskState.Started;
}
public end(success: boolean, endTime?: number): void {
// No-op
this._state = NotebookCellExecutionTaskState.Resolved;
}
public async clearOutput(cell?: vscode.NotebookCell): Promise<void> {
// No-op
this.verifyStateForOutput();
const targetCell = typeof cell === 'number' ? this._cell.notebook.cellAt(cell) : (cell ?? this._cell);
const editor = this._extHostNotebookDocumentsAndEditors.getEditor(URI.from(targetCell.notebook.uri).toString());
await editor.clearOutput(editor.document.cells[targetCell.index]);
}
public async replaceOutput(out: vscode.NotebookCellOutput | vscode.NotebookCellOutput[], cell?: vscode.NotebookCell): Promise<void> {
// No-op
this.verifyStateForOutput();
return this.updateOutputs(out, cell, false);
}
public async appendOutput(out: vscode.NotebookCellOutput | vscode.NotebookCellOutput[], cell?: vscode.NotebookCell): Promise<void> {
// No-op
this.verifyStateForOutput();
return this.updateOutputs(out, cell, true);
}
public async replaceOutputItems(items: vscode.NotebookCellOutputItem | vscode.NotebookCellOutputItem[], output: vscode.NotebookCellOutput): Promise<void> {
// No-op
this.verifyStateForOutput();
return this.updateOutputItems(items, output, false);
}
public async appendOutputItems(items: vscode.NotebookCellOutputItem | vscode.NotebookCellOutputItem[], output: vscode.NotebookCellOutput): Promise<void> {
// No-op
this.verifyStateForOutput();
return this.updateOutputItems(items, output, true);
}
private async updateOutputs(outputs: vscode.NotebookCellOutput | vscode.NotebookCellOutput[], cell: vscode.NotebookCell | number | undefined, append: boolean): Promise<void> {
this.verifyStateForOutput();
const targetCell = typeof cell === 'number' ? this._cell.notebook.cellAt(cell) : (cell ?? this._cell);
const editor = this._extHostNotebookDocumentsAndEditors.getEditor(URI.from(targetCell.notebook.uri).toString());
await editor.edit(builder => {
const adsOutputs = VSCodeContentManager.convertToADSCellOutput(outputs);
builder.updateCell(targetCell.index, { outputs: adsOutputs }, append);
});
}
private async updateOutputItems(items: vscode.NotebookCellOutputItem | vscode.NotebookCellOutputItem[], output: vscode.NotebookCellOutput, append: boolean): Promise<void> {
this.verifyStateForOutput();
const editor = this._extHostNotebookDocumentsAndEditors.getEditor(URI.from(this._cell.notebook.uri).toString());
await editor.edit(builder => {
const adsOutput = VSCodeContentManager.convertToADSCellOutput({ id: output.id, items: asArray(items) }, undefined);
builder.updateCellOutput(this._cell.index, { outputs: adsOutput }, append);
});
}
/**
* Verify that the execution is in a state where it's valid to modify the output (currently executing).
*/
private verifyStateForOutput() {
if (this._state === NotebookCellExecutionTaskState.Init) {
throw new Error('Must call start before modifying cell output');
}
if (this._state === NotebookCellExecutionTaskState.Resolved) {
throw new Error('Cannot modify cell output after calling end');
}
}
}

View File

@@ -17,6 +17,7 @@ import { VSCodeSerializationProvider } from 'sql/workbench/api/common/vscodeSeri
import { IExtensionDescription } from 'vs/platform/extensions/common/extensions';
import { ADSNotebookController } from 'sql/workbench/api/common/adsNotebookController';
import { VSCodeExecuteProvider } from 'sql/workbench/api/common/vscodeExecuteProvider';
import { ExtHostNotebookDocumentsAndEditors } from 'sql/workbench/api/common/extHostNotebookDocumentsAndEditors';
type Adapter = azdata.nb.NotebookSerializationProvider | azdata.nb.SerializationManager | azdata.nb.NotebookExecuteProvider | azdata.nb.ExecuteManager | azdata.nb.ISession | azdata.nb.IKernel | azdata.nb.IFuture;
@@ -27,7 +28,7 @@ export class ExtHostNotebook implements ExtHostNotebookShape {
private _adapters = new Map<number, Adapter>();
// Notebook URI to manager lookup.
constructor(_mainContext: IMainContext) {
constructor(_mainContext: IMainContext, private _extHostNotebookDocumentsAndEditors: ExtHostNotebookDocumentsAndEditors) {
this._proxy = _mainContext.getProxy(SqlMainContext.MainThreadNotebook);
}
@@ -264,7 +265,7 @@ export class ExtHostNotebook implements ExtHostNotebookShape {
createNotebookController(extension: IExtensionDescription, id: string, viewType: string, label: string, handler?: (cells: vscode.NotebookCell[], notebook: vscode.NotebookDocument, controller: vscode.NotebookController) => void | Thenable<void>, rendererScripts?: vscode.NotebookRendererScript[]): vscode.NotebookController {
let addLanguagesHandler = (id, languages) => this._proxy.$updateProviderDescriptionLanguages(id, languages);
let controller = new ADSNotebookController(extension, id, viewType, label, addLanguagesHandler, handler, extension.enableProposedApi ? rendererScripts : undefined);
let controller = new ADSNotebookController(extension, id, viewType, label, addLanguagesHandler, this._extHostNotebookDocumentsAndEditors, handler, extension.enableProposedApi ? rendererScripts : undefined);
let executeProvider = new VSCodeExecuteProvider(controller);
this.registerExecuteProvider(executeProvider);
return controller;

View File

@@ -12,15 +12,9 @@ import { readonly } from 'vs/base/common/errors';
import { MainThreadNotebookDocumentsAndEditorsShape } from 'sql/workbench/api/common/sqlExtHost.protocol';
import { ExtHostNotebookDocumentData } from 'sql/workbench/api/common/extHostNotebookDocumentData';
import { CellRange, ISingleNotebookEditOperation, ICellRange } from 'sql/workbench/api/common/sqlExtHostTypes';
import { CellRange, INotebookEditOperation, ICellRange, NotebookEditOperationType } from 'sql/workbench/api/common/sqlExtHostTypes';
import { HideInputTag } from 'sql/platform/notebooks/common/outputRegistry';
export interface INotebookEditOperation {
range: azdata.nb.CellRange;
cell: Partial<azdata.nb.ICellContents>;
forceMoveMarkers: boolean;
}
export interface INotebookEditData {
documentVersionId: number;
edits: INotebookEditOperation[];
@@ -64,7 +58,7 @@ export class NotebookEditorEdit {
replace(location: number | CellRange, value: Partial<azdata.nb.ICellContents>): void {
let range: CellRange = this.getAsRange(location);
this._pushEdit(range, value, false);
this._pushEdit(NotebookEditOperationType.ReplaceCells, range, value);
}
private getAsRange(location: number | CellRange): CellRange {
@@ -99,7 +93,7 @@ export class NotebookEditorEdit {
value.metadata.tags.push(HideInputTag);
}
}
this._pushEdit(new CellRange(index, index), value, true);
this._pushEdit(NotebookEditOperationType.InsertCell, new CellRange(index, index), value);
}
deleteCell(index: number): void {
@@ -114,15 +108,24 @@ export class NotebookEditorEdit {
throw new Error('Unrecognized index');
}
this._pushEdit(range, null, true);
this._pushEdit(NotebookEditOperationType.DeleteCell, range, null);
}
private _pushEdit(range: azdata.nb.CellRange, cell: Partial<azdata.nb.ICellContents>, forceMoveMarkers: boolean): void {
updateCell(index: number, updatedContent: Partial<azdata.nb.ICellContents>, append: boolean): void {
this._pushEdit(NotebookEditOperationType.UpdateCell, new CellRange(index, index + 1), updatedContent, append);
}
updateCellOutput(cellIndex: number, updatedContent: Partial<azdata.nb.ICellContents>, append: boolean): void {
this._pushEdit(NotebookEditOperationType.UpdateCellOutput, new CellRange(cellIndex, cellIndex + 1), updatedContent, append);
}
private _pushEdit(type: NotebookEditOperationType, range: azdata.nb.CellRange, cell: Partial<azdata.nb.ICellContents>, append?: boolean): void {
let validRange = this._document.validateCellRange(range);
this._collectedEdits.push({
type: type,
range: validRange,
cell: cell,
forceMoveMarkers: forceMoveMarkers
append: append
});
}
}
@@ -188,7 +191,7 @@ export class ExtHostNotebookEditor implements azdata.nb.NotebookEditor, IDisposa
return this._proxy.$changeKernel(this._id, kernel);
}
public edit(callback: (editBuilder: azdata.nb.NotebookEditorEdit) => void, options?: { undoStopBefore: boolean; undoStopAfter: boolean; }): Thenable<boolean> {
public edit(callback: (editBuilder: NotebookEditorEdit) => void, options?: { undoStopBefore: boolean; undoStopAfter: boolean; }): Thenable<boolean> {
if (this._disposed) {
return Promise.reject(new Error('NotebookEditor#edit not possible on closed editors'));
}
@@ -228,11 +231,12 @@ export class ExtHostNotebookEditor implements azdata.nb.NotebookEditor, IDisposa
}
// prepare data for serialization
let edits: ISingleNotebookEditOperation[] = editData.edits.map((edit) => {
let edits: INotebookEditOperation[] = editData.edits.map((edit) => {
return {
type: edit.type,
range: toICellRange(edit.range),
cell: edit.cell,
forceMoveMarkers: edit.forceMoveMarkers
append: edit.append
};
});

View File

@@ -89,8 +89,8 @@ export function createAdsApiFactory(accessor: ServicesAccessor): IAdsExtensionAp
const extHostDashboard = rpcProtocol.set(SqlExtHostContext.ExtHostDashboard, new ExtHostDashboard(rpcProtocol));
const extHostModelViewDialog = rpcProtocol.set(SqlExtHostContext.ExtHostModelViewDialog, new ExtHostModelViewDialog(rpcProtocol, extHostModelView, extHostBackgroundTaskManagement));
const extHostQueryEditor = rpcProtocol.set(SqlExtHostContext.ExtHostQueryEditor, new ExtHostQueryEditor(rpcProtocol));
const extHostNotebook = rpcProtocol.set(SqlExtHostContext.ExtHostNotebook, new ExtHostNotebook(rpcProtocol));
const extHostNotebookDocumentsAndEditors = rpcProtocol.set(SqlExtHostContext.ExtHostNotebookDocumentsAndEditors, new ExtHostNotebookDocumentsAndEditors(rpcProtocol));
const extHostNotebook = rpcProtocol.set(SqlExtHostContext.ExtHostNotebook, new ExtHostNotebook(rpcProtocol, extHostNotebookDocumentsAndEditors));
const extHostExtensionManagement = rpcProtocol.set(SqlExtHostContext.ExtHostExtensionManagement, new ExtHostExtensionManagement(rpcProtocol));
const extHostWorkspace = rpcProtocol.set(SqlExtHostContext.ExtHostWorkspace, new ExtHostWorkspace(rpcProtocol));
return {

View File

@@ -19,7 +19,7 @@ import { ITaskHandlerDescription } from 'sql/workbench/services/tasks/common/tas
import {
IItemConfig, IComponentShape, IModelViewDialogDetails, IModelViewTabDetails, IModelViewButtonDetails,
IModelViewWizardDetails, IModelViewWizardPageDetails, IExecuteManagerDetails, INotebookSessionDetails,
INotebookKernelDetails, INotebookFutureDetails, FutureMessageType, INotebookFutureDone, ISingleNotebookEditOperation,
INotebookKernelDetails, INotebookFutureDetails, FutureMessageType, INotebookFutureDone, INotebookEditOperation,
NotebookChangeKind,
ISerializationManagerDetails
} from 'sql/workbench/api/common/sqlExtHostTypes';
@@ -990,7 +990,7 @@ export interface MainThreadNotebookDocumentsAndEditorsShape extends IDisposable
$trySetTrusted(_uri: UriComponents, isTrusted: boolean): Thenable<boolean>;
$trySaveDocument(uri: UriComponents): Thenable<boolean>;
$tryShowNotebookDocument(resource: UriComponents, options: INotebookShowOptions): Promise<string>;
$tryApplyEdits(id: string, modelVersionId: number, edits: ISingleNotebookEditOperation[], opts: IUndoStopOptions): Promise<boolean>;
$tryApplyEdits(id: string, modelVersionId: number, edits: INotebookEditOperation[], opts: IUndoStopOptions): Promise<boolean>;
$runCell(id: string, cellUri: UriComponents): Promise<boolean>;
$runAllCells(id: string, startCellUri?: UriComponents, endCellUri?: UriComponents): Promise<boolean>;
$clearOutput(id: string, cellUri: UriComponents): Promise<boolean>;

View File

@@ -633,10 +633,47 @@ export class CellRange {
}
}
export interface ISingleNotebookEditOperation {
export const enum NotebookEditOperationType {
/**
* Inserts a new cell with the specified content at the specified position.
*/
InsertCell = 0,
/**
* Deletes a single cell.
*/
DeleteCell = 1,
/**
* Replace the specified cell range with a new cell made from the specified content.
*/
ReplaceCells = 2,
/**
* Update a cell with the specified new values. Currently only supports updating cell output.
*/
UpdateCell = 3,
/**
* Updates a cell outputs with the specified new values.
*/
UpdateCellOutput = 4
}
// TODO This should be split up into separate edit operation types
export interface INotebookEditOperation {
/**
* The type of edit operation this is
*/
type: NotebookEditOperationType;
/**
* The range of cells that this edit affects
*/
range: ICellRange;
/**
* The cell metadata to use for the edit operation (only for some edit operations)
*/
cell: Partial<nb.ICellContents>;
forceMoveMarkers: boolean;
/**
* Whether to append the content to the existing content or replace it.
*/
append?: boolean;
}
export class ConnectionProfile {

View File

@@ -130,6 +130,7 @@ class VSCodeKernel implements azdata.nb.IKernel {
let executePromise: Promise<void>;
if (this._controller.executeHandler) {
let cell = <vscode.NotebookCell>{
index: content.cellIndex,
document: <vscode.TextDocument>{
uri: content.notebookUri,
languageId: this._kernelSpec.language,

View File

@@ -9,27 +9,27 @@ import { VSBuffer } from 'vs/base/common/buffer';
import { NotebookCellKind } from 'vs/workbench/api/common/extHostTypes';
import { CancellationTokenSource } from 'vs/base/common/cancellation';
import { OutputTypes } from 'sql/workbench/services/notebook/common/contracts';
import { asArray } from 'vs/base/common/arrays';
import { NBFORMAT, NBFORMAT_MINOR } from 'sql/workbench/common/constants';
/**
* A Notebook Content Manager that is used as part of converting VS Code notebook extension APIs into ADS equivalents.
*/
export class VSCodeContentManager implements azdata.nb.ContentManager {
constructor(private readonly _serializer: vscode.NotebookSerializer) {
}
public static convertToADSCellOutput(output: vscode.NotebookCellOutput, executionOrder?: number): azdata.nb.IExecuteResult {
let outputData = {};
for (let item of output.items) {
outputData[item.mime] = VSBuffer.wrap(item.data).toString();
}
return {
output_type: 'execute_result',
data: outputData,
execution_count: executionOrder,
metadata: output.metadata,
id: output.id
};
public static convertToADSCellOutput(outputs: vscode.NotebookCellOutput | vscode.NotebookCellOutput[], executionOrder?: number): azdata.nb.IDisplayResult[] {
return asArray(outputs).map(output => {
let outputData = {};
for (let item of output.items) {
outputData[item.mime] = VSBuffer.wrap(item.data).toString();
}
return {
output_type: 'execute_result',
data: outputData,
execution_count: executionOrder,
metadata: output.metadata,
id: output.id
};
});
}
public async deserializeNotebook(contents: string): Promise<azdata.nb.INotebookContents> {
@@ -45,7 +45,7 @@ export class VSCodeContentManager implements azdata.nb.ContentManager {
language: cell.languageId
},
execution_count: executionOrder,
outputs: cell.outputs?.map<azdata.nb.IExecuteResult>(output => VSCodeContentManager.convertToADSCellOutput(output, executionOrder))
outputs: cell.outputs ? VSCodeContentManager.convertToADSCellOutput(cell.outputs, executionOrder) : undefined
};
}),
metadata: notebookData.metadata ?? {},