Merge vscode source through release 1.79.2 (#23482)

* log when an editor action doesn't run because of enablement

* notebooks create/dispose editors. this means controllers must be created eagerly (😢) and that notebooks need a custom way of plugging comparision keys for session. works unless creating another session for the same cell of a duplicated editor

* Set offSide to sql lang configuration to true (#183461)

* Fixes #181764 (#183550)

* fix typo

* Always scroll down and focus the input (#183557)

* Fixes #180386 (#183561)

* cli: ensure ordering of rpc server messages (#183558)

* cli: ensure ordering of rpc server messages

Sending lots of messages to a stream would block them around the async
tokio mutex, which is "fair" so doesn't preserve ordering. Instead, use
the write_loop approach I introduced to the server_multiplexer for the
same reason some time ago.

* fix clippy

* update for May endgame

* testing: allow invalidateTestResults to take an array (#183569)

* Document `ShareProvider` API proposal (#183568)

* Document `ShareProvider` API proposal

* Remove mention of VS Code from JSDoc

* Add support for rendering svg and md in welcome message (#183580)

* Remove toggle setting more eagerly (#183584)

* rm message abt macOS

* Change text (#183589)

* Change text

* Accidentally changed the wrong file

* cli: improve output for code tunnel status (#183571)

* testing: allow invalidateTestResults to take an array

* cli: improve output for code tunnel status

Fixes #183570

* [json/css/html] update services (#183595)

* Add experimental setting to enable this dialog

* Fix exporting chat model to JSON before it is initialized (#183597)

* minimum scrolling to reveal the next cell on shift+enter (#183600)

do minimum scrolling to reveal the next cell on Execute cell and select next

* Fixing Jupyter notebook issue 13263 (#183527)

fix for the issue, still need to understand why there is strange focusing

* Tweak proposed API JSDoc (#183590)

* Tweak proposed API JSDoc

* workbench -> workspace

* fix ? operator

* Use active editor and show progress when sharing (#183603)

Use active editor and show progress

* use scroll setting variable correctly

* Schedule welcome widget to show once between typing. (#183606)

* Schedule dialog to show once between typing

* Don't re-render if already displayed once

* Add F10 keybinding for debugger step, even on Web. (#183510)

Fixes #181792.
Previously, for Web the keyboard shortcut was Alt-F10, because it was
believed that F10 could not be bound on browsers. This turned out to be
incorrect, so we make the shortcut consistent (F10) with desktop VSCode
which is also what many other debuggers use.
We keep Alt-F10 on web as a secondary keybinding to keep the experience
some web users may have gotten used to by now.

* Also pass process.env

* Restore missing chat clear commands (#183651)

* chore: update electron@22.5.4 (#183716)

* Show remote indicator in web when remoteAuthority is set (#183728)

* feat: .vuerc as json file (#153017)

Co-authored-by: Martin Aeschlimann <martinae@microsoft.com>

* Delete --compatibility=1.63 code from the server (#183738)

* Copy vscode.dev link to tunnel generates an invalid link when an untitled workspace is open (#183739)

* Recent devcontainer display string corrupted on Get Started page (#183740)

* Improve "next codeblock" navigation (#183744)

* Improve "next codeblock" navigation
Operate on the current focused response, or the last one, and scroll to the selected item

* Normalize command title

* Git - run git status if similarityThreshold changes (#183762)

* fix aria-label issue in kb editor

fixes A11y_GradeB_VSCode_Keyboard shortcut reads words together - Blind: Arrow key navigation to row Find the binding keys and  "when" cell data are read together resulting in a word " CTRL + FeditorFocus  instead of CTRL + F editorFocus" #182490

* Status - fix compact padding (#183768)

* Remove angle brackets from VB brackets (#183782)

Fixes #183359

* Update language config schema with more details about brackets. (#183779)

* fix comment (#183812)

* Support for `Notebook` CodeAction Kind (#183457)

* nb kind support -- wip

* allow notebook codeactions around single cell edit check

* move notebook code action type out of editor

---------

Co-authored-by: rebornix <penn.lv@gmail.com>

* cli: fix connection default being applied (#183827)

* cli: bump to openssl 1.1.1u (#183828)

* Implement "delete" action for chat history (#183609)

* Use desired file name when generating new md pasted file paths (#183861)

Fixes #183851

* Default to filename for markdown new file if empty (#183864)

Fixes #183848

* Fix small typo (#183865)

Fixes #183819

* Noop when moving a symbol into the file it is already in (#183866)

Fixes #183793

* Adjust codeAction validation to account for notebook kind (#183859)

* Make JS/TS `go to configuration` commands work on non-`file:` file systems (#183688)

Make `go to project` commands work on non-`file:` file systems

Fixes #183685

* Can't do regex search after opening notebook (#183884)

Fixes #183858

* Default to current dir for `move to file` select (#183875)

Fixes #183870

`showOpenDialog` seems to ignore `defaultUri` if the file doesn't exist

* Use `<...>` style markdown links when needed (#183876)

Fixes #183849

* Remove check for context keys

* Update xterm package

* Enable updating a chat model without triggering incremental typing (#183894)

* Enable chat "move" commands on empty sessions (#183895)

* Enable chat "move" commands on empty sessions
and also imported sessions

* Fix command name

* Fix some chat keybindings on windows (#183896)

* "Revert File" on inactive editors are ignored (fix #177557) (#183903)

* Empty reason while switching profile (fix #183775) (#183904)

* fix https://github.com/microsoft/vscode-internalbacklog/issues/4278 (#183910)

* fix https://github.com/microsoft/vscode/issues/183770 (#183914)

* code --status displays a lot of errors before actual status output (fix #183787) (#183915)

* joh/icy manatee (#183917)

* Use idle value for widget of interactive editor controller

https://github.com/microsoft/vscode/issues/183820

* also make preview editors idle values

https://github.com/microsoft/vscode/issues/183820

* Fix #183777 (#183929)

* Fix #182309 (#183925)

* Tree checkbox item -> items (#183931)

Fixes #183826

* Fixes #183909 (#183940)

* Fix #183837 (#183943)

fix #183837

* Git - fix #183941 (#183944)

* Update xterm.css

Fixes #181242

* chore: add @ulugbekna and @aiday-mar to my-endgame notebook (#183946)

* Revert "When snippet mode is active, make `Tab` not accept suggestion but advance placeholder"

This reverts commit 50a80cdb61511343996ff1d41d0b676c3d329f48.

* revert not focusing completion list when quick suggest happens during snippet

* change `snippetsPreventQuickSuggestions` default to false

* Fix #181446 (#183956)

* fix https://github.com/microsoft/vscode-internalbacklog/issues/4298 (#183957)

* fix: remove extraneous incorrect context keys (#183959)

These were actually getting added in getTestItemContextOverlay, and the test ID was using the extended ID which extensions do not know about.

Fixes #183612

* Fixes https://github.com/microsoft/monaco-editor/issues/3920 (#183960)

* fix https://github.com/microsoft/vscode-internalbacklog/issues/4324 (#183961)

* fix #183030

* fix #180826 (#183962)

* make message more generic for interactive editor help

* .

* fix #183968

* Keep codeblock toolbar visible when focused

* Fix when clause on "Run in terminal" command

* add important info to help menu

* fix #183970

* Set `isRefactoring` for all TS refactoring edits (#183982)

* consolidate

* Disable move to file in TS versions < 5.2 (#183992)

There are still a few key bugs with refactoring. We will  ship this as a preview for TS 5.2+ instead of for 5.1

* Polish query accepting (#183995)

We shouldn't send the same request to Copilot if the query hasn't changed. So if the query is the same, we short circut.

Fixes https://github.com/microsoft/vscode-internalbacklog/issues/4286

Also, when we open in chat, we should use the last accepted query, not what's in the input box.

Fixes https://github.com/microsoft/vscode-internalbacklog/issues/4280

* Allow widget to have focus (#184000)

So that selecting non-code text works.

Fixes https://github.com/microsoft/vscode-internalbacklog/issues/4294

* Fix microsoft/vscode-internalbacklog#4257. Mitigate zindex for zone widgets. (#184001)

* Change welcome dialog contribution to Eventually

* Misc fixes

* Workspace folder picker entry descriptions are suboptimal for some filesystems (fix #183418) (#184018)

* cli - ignore std error unless verbose (#183787) (#184031)

* joh/inquisitive meerkat (#184034)

* only stash sessions that are none empty

https://github.com/microsoft/vscode-internalbacklog/issues/4281

* only unstash a session once - unless new exchanges are made,

https://github.com/microsoft/vscode-internalbacklog/issues/4281

* account for all exchange types

* Improve declared components (#184039)

* make sure to read setting (#184040)

d'oh, related to https://github.com/microsoft/vscode/issues/173387#issuecomment-1571696644

* [html] update service (#184049)

[html] update service. FIxes #181176

* reset context keys on reset/hide (#184042)

fixes https://github.com/microsoft/vscode-internalbacklog/issues/4330

* use `Lazy`, not `IdleValue` for the IE widget held by the eager controller (#184048)

https://github.com/microsoft/vscode/issues/183820

* fix https://github.com/microsoft/vscode-internalbacklog/issues/4333 (#184067)

* use undo-loop instead of undo-edit when discarding chat session (#184063)

* use undo-loop instead of undo-edit when discarding chat session

fixes https://github.com/microsoft/vscode-internalbacklog/issues/4118

* fix tests, wait for correct state

* Add logging to node download (#184070)

Add logging to node download. For #182951

* re-enable default zone widget revealing when showing (#184072)

fixes https://github.com/microsoft/vscode-internalbacklog/issues/4332, also fixes https://github.com/microsoft/vscode-internalbacklog/issues/3784

* fix #178202

* Allow APIs in stable (#184062)

* Fix microsoft/vscode-internalbacklog#4206. Override List view whitespace css for monaco editor (#184087)

* Fix JSDoc grammatical error (#184090)

* Pick up TS 5.1.3 (#184091)

Fixes #182931

* Misc fixes

* update distro (#184097)

* chore: update electron@22.5.5 (#184116)

* Extension host veto is registered multiple times on restart (fix #183778) (#184127)

Extension host veto is registered multiple times on restart (#183778)

* Do not auto start the local web worker extension host (#184137)

* Allow embedders to intercept trustedTypes.createPolicy calls (#184136)

Allow embedders to intercept trustedTypes.createPolicy calls (#184100)

* fix: reading from console output for --status on windows and linux (#184138)

fix: reading from console output for --status on windows and linux (#184118)

* Misc fixes

* code --status displays a lot of errors before actual status output (fix #183787) (#184200)

fix 183787

* (cherry-pick to 1.79 from main) Handle galleryExtension failure in featuredExtensionService (#184205)

Handle galleryExtension failure in featuredExtensionService (#184198)

Handle galleryExtension failure

* Fix #184183. Multiple output height updates are skipped. (#184188)

* Post merge init fixes

* Misc build issues

* disable toggle inline diff of `alt` down

https://github.com/microsoft/vscode-internalbacklog/issues/4342

* Take into account already activated extensions when computing running locations (#184303)

Take into account already activated extensions when computing running locations (fixes #184180)

* Avoid `extensionService.getExtension` and use `ActivationKind.Immediate` to allow that URI handling works while resolving (#184310)

Avoid `extensionService.getExtension` and use `ActivationKind.Immediate` to allow that URI handling works while resolving (fixes #182217)

* WIP

* rm fish auto injection

* More breaks

* Fix Port Attributes constructor (#184412)

* WIP

* WIP

* Allow extensions to get at the exports of other extensions during resolving (#184487)

Allow extensions to get at the exports of other extensions during resolving (fixes #184472)

* do not auto finish session when inline chat widgets have focus

re https://github.com/microsoft/vscode-internalbacklog/issues/4354

* fix compile errors caused by new base method

* WIP

* WIP

* WIP

* WIP

* Build errors

* unc - fix path traversal bypass

* Bump version

* cherry-pick prod changes from main

* Disable sandbox

* Build break from merge

* bump version

* Merge pull request #184739 from max06/max06/issue184659

Restore ShellIntegration for fish (#184659)

* Git - only add --find-renames if the value is not the default one (#185053)

Git - only add --find-renames if the value is not the default one (#184992)

* Cherry-pick: Revert changes to render featured extensions when available (#184747)

Revert changes to render featured extensions when available.  (#184573)

* Lower timeouts for experimentation and gallery service

* Revert changes to render extensions when available

* Add audio cues

* fix: disable app sandbox when --no-sandbox is present (#184913)

* fix: disable app sandbox when --no-sandbox is present (#184897)

* fix: loading minimist in packaged builds

* Runtime errors

* UNC allow list checks cannot be disabled in extension host (fix #184989) (#185085)

* UNC allow list checks cannot be disabled in extension host (#184989)

* Update src/vs/base/node/unc.js

Co-authored-by: Robo <hop2deep@gmail.com>

---------

Co-authored-by: Robo <hop2deep@gmail.com>

* Add notebook extension

* Fix mangling issues

* Fix mangling issues

* npm install

* npm install

* Issues blocking bundle

* Fix build folder compile errors

* Fix windows bundle build

* Linting fixes

* Fix sqllint issues

* Update yarn.lock files

* Fix unit tests

* Fix a couple breaks from test fixes

* Bump distro

* redo the checkbox style

* Update linux build container dockerfile

* Bump build image tag

* Bump native watch dog package

* Bump node-pty

* Bump distro

* Fix documnetation error

* Update distro

* redo the button styles

* Update datasource TS

* Add missing yarn.lock files

* Windows setup fix

* Turn off extension unit tests while investigating

* color box style

* Remove appx

* Turn off test log upload

* update dropdownlist style

* fix universal app build error (#23488)

* Skip flaky bufferContext vscode test

---------

Co-authored-by: Johannes <johannes.rieken@gmail.com>
Co-authored-by: Henning Dieterichs <hdieterichs@microsoft.com>
Co-authored-by: Julien Richard <jairbubbles@hotmail.com>
Co-authored-by: Charles Gagnon <chgagnon@microsoft.com>
Co-authored-by: Megan Rogge <merogge@microsoft.com>
Co-authored-by: meganrogge <megan.rogge@microsoft.com>
Co-authored-by: Rob Lourens <roblourens@gmail.com>
Co-authored-by: Connor Peet <connor@peet.io>
Co-authored-by: Joyce Er <joyce.er@microsoft.com>
Co-authored-by: Bhavya U <bhavyau@microsoft.com>
Co-authored-by: Raymond Zhao <7199958+rzhao271@users.noreply.github.com>
Co-authored-by: Martin Aeschlimann <martinae@microsoft.com>
Co-authored-by: Aaron Munger <aamunger@microsoft.com>
Co-authored-by: Aiday Marlen Kyzy <amarlenkyzy@microsoft.com>
Co-authored-by: rebornix <penn.lv@gmail.com>
Co-authored-by: Ole <oler@google.com>
Co-authored-by: Jean Pierre <jeanp413@hotmail.com>
Co-authored-by: Robo <hop2deep@gmail.com>
Co-authored-by: Yash Singh <saiansh2525@gmail.com>
Co-authored-by: Ladislau Szomoru <3372902+lszomoru@users.noreply.github.com>
Co-authored-by: Ulugbek Abdullaev <ulugbekna@gmail.com>
Co-authored-by: Alex Ross <alros@microsoft.com>
Co-authored-by: Michael Lively <milively@microsoft.com>
Co-authored-by: Matt Bierner <matb@microsoft.com>
Co-authored-by: Andrea Mah <31675041+andreamah@users.noreply.github.com>
Co-authored-by: Benjamin Pasero <benjamin.pasero@microsoft.com>
Co-authored-by: Sandeep Somavarapu <sasomava@microsoft.com>
Co-authored-by: Daniel Imms <2193314+Tyriar@users.noreply.github.com>
Co-authored-by: Tyler James Leonhardt <me@tylerleonhardt.com>
Co-authored-by: Alexandru Dima <alexdima@microsoft.com>
Co-authored-by: Joao Moreno <Joao.Moreno@microsoft.com>
Co-authored-by: Alan Ren <alanren@microsoft.com>
This commit is contained in:
Karl Burtram
2023-06-27 15:26:51 -07:00
committed by GitHub
parent 7975fda6dd
commit 01e66ab3e6
4335 changed files with 252586 additions and 164604 deletions

View File

@@ -1,47 +0,0 @@
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import type * as MarkdownIt from 'markdown-it';
import type * as MarkdownItToken from 'markdown-it/lib/token';
import type { RendererContext } from 'vscode-notebook-renderer';
interface MarkdownItRenderer {
extendMarkdownIt(fn: (md: MarkdownIt) => void): void;
}
export async function activate(ctx: RendererContext<void>) {
const markdownItRenderer = (await ctx.getRenderer('vscode.markdown-it-renderer')) as MarkdownItRenderer | any;
if (!markdownItRenderer) {
throw new Error(`Could not load 'vscode.markdown-it-renderer'`);
}
markdownItRenderer.extendMarkdownIt((md: MarkdownIt) => {
const original = md.renderer.rules.image;
md.renderer.rules.image = (tokens: MarkdownItToken[], idx: number, options, env, self) => {
const token = tokens[idx];
const src = token.attrGet('src');
const attachments: Record<string, Record<string, string>> = env.outputItem.metadata?.custom?.attachments; // this stores attachment entries for every image in the cell
if (attachments && src) {
const imageAttachment = attachments[src.replace('attachment:', '')];
if (imageAttachment) {
// objEntries will always be length 1, with objEntries[0] holding [0]=mime,[1]=b64
// if length = 0, something is wrong with the attachment, mime/b64 weren't copied over
const objEntries = Object.entries(imageAttachment);
if (objEntries.length) {
const [attachmentKey, attachmentVal] = objEntries[0];
const b64Markdown = 'data:' + attachmentKey + ';base64,' + attachmentVal;
token.attrSet('src', b64Markdown);
}
}
}
if (original) {
return original(tokens, idx, options, env, self);
} else {
return self.renderToken(tokens, idx, options);
}
};
});
}

View File

@@ -3,12 +3,12 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import { ExtensionContext, NotebookDocument, NotebookDocumentChangeEvent, workspace, WorkspaceEdit } from 'vscode';
import { ExtensionContext, NotebookDocument, NotebookDocumentChangeEvent, NotebookEdit, workspace, WorkspaceEdit } from 'vscode';
import { v4 as uuid } from 'uuid';
import { getCellMetadata } from './serializers';
import { CellMetadata } from './common';
import { getNotebookMetadata } from './notebookSerializer';
import * as nbformat from '@jupyterlab/nbformat';
import type * as nbformat from '@jupyterlab/nbformat';
/**
* Ensure all new cells in notebooks with nbformat >= 4.5 have an id.
@@ -34,7 +34,7 @@ function onDidChangeNotebookCells(e: NotebookDocumentChangeEvent) {
// Don't edit the metadata directly, always get a clone (prevents accidental singletons and directly editing the objects).
const updatedMetadata: CellMetadata = { ...JSON.parse(JSON.stringify(cellMetadata || {})) };
updatedMetadata.id = id;
edit.replaceNotebookCellMetadata(cell.notebook.uri, cell.index, { ...(cell.metadata), custom: updatedMetadata });
edit.set(cell.notebook.uri, [NotebookEdit.updateCellMetadata(cell.index, { ...(cell.metadata), custom: updatedMetadata })]);
workspace.applyEdit(edit);
});
});

View File

@@ -3,7 +3,7 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as nbformat from '@jupyterlab/nbformat';
import type * as nbformat from '@jupyterlab/nbformat';
/**
* Metadata we store in VS Code cell output items.
@@ -44,7 +44,7 @@ export interface CellOutputMetadata {
/**
* Metadata we store in VS Code cells.
* This contains the original metadata from the Jupyuter cells.
* This contains the original metadata from the Jupyter cells.
*/
export interface CellMetadata {
/**

View File

@@ -3,4 +3,9 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as vscode from 'vscode';
export const defaultNotebookFormat = { major: 4, minor: 2 };
export const ATTACHMENT_CLEANUP_COMMANDID = 'ipynb.cleanInvalidImageAttachment';
export const JUPYTER_NOTEBOOK_MARKDOWN_SELECTOR: vscode.DocumentSelector = { notebookType: 'jupyter-notebook', language: 'markdown' };

View File

@@ -3,7 +3,7 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as nbformat from '@jupyterlab/nbformat';
import type * as nbformat from '@jupyterlab/nbformat';
import { extensions, NotebookCellData, NotebookCellExecutionSummary, NotebookCellKind, NotebookCellOutput, NotebookCellOutputItem, NotebookData } from 'vscode';
import { CellMetadata, CellOutputMetadata } from './common';
@@ -149,21 +149,29 @@ function convertJupyterOutputToBuffer(mime: string, value: unknown): NotebookCel
}
}
function getNotebookCellMetadata(cell: nbformat.IBaseCell): CellMetadata {
function getNotebookCellMetadata(cell: nbformat.IBaseCell): {
[key: string]: any;
} {
const cellMetadata: { [key: string]: any } = {};
// We put this only for VSC to display in diff view.
// Else we don't use this.
const propertiesToClone: (keyof CellMetadata)[] = ['metadata', 'attachments'];
const custom: CellMetadata = {};
propertiesToClone.forEach((propertyToClone) => {
if (cell[propertyToClone]) {
custom[propertyToClone] = JSON.parse(JSON.stringify(cell[propertyToClone]));
}
});
if (cell['metadata']) {
custom['metadata'] = JSON.parse(JSON.stringify(cell['metadata']));
}
if ('id' in cell && typeof cell.id === 'string') {
custom.id = cell.id;
}
return custom;
cellMetadata.custom = custom;
if (cell['attachments']) {
cellMetadata.attachments = JSON.parse(JSON.stringify(cell['attachments']));
}
return cellMetadata;
}
function getOutputMetadata(output: nbformat.IOutput): CellOutputMetadata {
// Add on transient data if we have any. This should be removed by our save functions elsewhere.
const metadata: CellOutputMetadata = {
@@ -284,7 +292,7 @@ export function jupyterCellOutputToCellOutput(output: nbformat.IOutput): Noteboo
function createNotebookCellDataFromRawCell(cell: nbformat.IRawCell): NotebookCellData {
const cellData = new NotebookCellData(NotebookCellKind.Code, concatMultilineString(cell.source), 'raw');
cellData.outputs = [];
cellData.metadata = { custom: getNotebookCellMetadata(cell) };
cellData.metadata = getNotebookCellMetadata(cell);
return cellData;
}
function createNotebookCellDataFromMarkdownCell(cell: nbformat.IMarkdownCell): NotebookCellData {
@@ -294,7 +302,7 @@ function createNotebookCellDataFromMarkdownCell(cell: nbformat.IMarkdownCell): N
'markdown'
);
cellData.outputs = [];
cellData.metadata = { custom: getNotebookCellMetadata(cell) };
cellData.metadata = getNotebookCellMetadata(cell);
return cellData;
}
function createNotebookCellDataFromCodeCell(cell: nbformat.ICodeCell, cellLanguage: string): NotebookCellData {
@@ -313,7 +321,7 @@ function createNotebookCellDataFromCodeCell(cell: nbformat.ICodeCell, cellLangua
const cellData = new NotebookCellData(NotebookCellKind.Code, source, cellLanguageId);
cellData.outputs = outputs;
cellData.metadata = { custom: getNotebookCellMetadata(cell) };
cellData.metadata = getNotebookCellMetadata(cell);
cellData.executionSummary = executionSummary;
return cellData;
}

View File

@@ -0,0 +1,142 @@
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
export function deepClone<T>(obj: T): T {
if (!obj || typeof obj !== 'object') {
return obj;
}
if (obj instanceof RegExp) {
// See https://github.com/microsoft/TypeScript/issues/10990
return obj as any;
}
const result: any = Array.isArray(obj) ? [] : {};
Object.keys(<any>obj).forEach((key: string) => {
if ((<any>obj)[key] && typeof (<any>obj)[key] === 'object') {
result[key] = deepClone((<any>obj)[key]);
} else {
result[key] = (<any>obj)[key];
}
});
return result;
}
// from https://github.com/microsoft/vscode/blob/43ae27a30e7b5e8711bf6b218ee39872ed2b8ef6/src/vs/base/common/objects.ts#L117
export function objectEquals(one: any, other: any) {
if (one === other) {
return true;
}
if (one === null || one === undefined || other === null || other === undefined) {
return false;
}
if (typeof one !== typeof other) {
return false;
}
if (typeof one !== 'object') {
return false;
}
if ((Array.isArray(one)) !== (Array.isArray(other))) {
return false;
}
let i: number;
let key: string;
if (Array.isArray(one)) {
if (one.length !== other.length) {
return false;
}
for (i = 0; i < one.length; i++) {
if (!objectEquals(one[i], other[i])) {
return false;
}
}
} else {
const oneKeys: string[] = [];
for (key in one) {
oneKeys.push(key);
}
oneKeys.sort();
const otherKeys: string[] = [];
for (key in other) {
otherKeys.push(key);
}
otherKeys.sort();
if (!objectEquals(oneKeys, otherKeys)) {
return false;
}
for (i = 0; i < oneKeys.length; i++) {
if (!objectEquals(one[oneKeys[i]], other[oneKeys[i]])) {
return false;
}
}
}
return true;
}
/**
* A helper to delay/debounce execution of a task, includes cancellation/disposal support.
* Pulled from https://github.com/microsoft/vscode/blob/3059063b805ed0ac10a6d9539e213386bfcfb852/extensions/markdown-language-features/src/util/async.ts
*/
export class Delayer<T> {
public defaultDelay: number;
private _timeout: any; // Timer
private _cancelTimeout: Promise<T | null> | null;
private _onSuccess: ((value: T | PromiseLike<T> | undefined) => void) | null;
private _task: ITask<T> | null;
constructor(defaultDelay: number) {
this.defaultDelay = defaultDelay;
this._timeout = null;
this._cancelTimeout = null;
this._onSuccess = null;
this._task = null;
}
dispose() {
this._doCancelTimeout();
}
public trigger(task: ITask<T>, delay: number = this.defaultDelay): Promise<T | null> {
this._task = task;
if (delay >= 0) {
this._doCancelTimeout();
}
if (!this._cancelTimeout) {
this._cancelTimeout = new Promise<T | undefined>((resolve) => {
this._onSuccess = resolve;
}).then(() => {
this._cancelTimeout = null;
this._onSuccess = null;
const result = this._task && this._task?.();
this._task = null;
return result;
});
}
if (delay >= 0 || this._timeout === null) {
this._timeout = setTimeout(() => {
this._timeout = null;
this._onSuccess?.(undefined);
}, delay >= 0 ? delay : this.defaultDelay);
}
return this._cancelTimeout;
}
private _doCancelTimeout(): void {
if (this._timeout !== null) {
clearTimeout(this._timeout);
this._timeout = null;
}
}
}
export interface ITask<T> {
(): T;
}

View File

@@ -4,8 +4,10 @@
*--------------------------------------------------------------------------------------------*/
import * as vscode from 'vscode';
import { ensureAllNewCellsHaveCellIds } from './cellIdService';
import { NotebookSerializer } from './notebookSerializer';
import { ensureAllNewCellsHaveCellIds } from './cellIdService';
import { notebookImagePasteSetup } from './notebookImagePaste';
import { AttachmentCleaner } from './notebookAttachmentCleaner';
// From {nbformat.INotebookMetadata} in @jupyterlab/coreutils
type NotebookMetadata = {
@@ -33,9 +35,13 @@ export function activate(context: vscode.ExtensionContext) {
transientOutputs: false,
transientCellMetadata: {
breakpointMargin: true,
custom: false
custom: false,
attachments: false
},
cellContentMetadata: {
attachments: true
}
}));
} as vscode.NotebookDocumentContentOptions));
vscode.languages.registerCodeLensProvider({ pattern: '**/*.ipynb' }, {
provideCodeLenses: (document) => {
@@ -77,12 +83,21 @@ export function activate(context: vscode.ExtensionContext) {
await vscode.window.showNotebookDocument(document);
}));
context.subscriptions.push(notebookImagePasteSetup());
const enabled = vscode.workspace.getConfiguration('ipynb').get('pasteImagesAsAttachments.enabled', false);
if (enabled) {
const cleaner = new AttachmentCleaner();
context.subscriptions.push(cleaner);
}
// Update new file contribution
vscode.extensions.onDidChange(() => {
vscode.commands.executeCommand('setContext', 'jupyterEnabled', vscode.extensions.getExtension('ms-toolsai.jupyter'));
});
vscode.commands.executeCommand('setContext', 'jupyterEnabled', vscode.extensions.getExtension('ms-toolsai.jupyter'));
return {
exportNotebook: (notebook: vscode.NotebookData): string => {
return exportNotebook(notebook, serializer);
@@ -94,7 +109,7 @@ export function activate(context: vscode.ExtensionContext) {
}
const edit = new vscode.WorkspaceEdit();
edit.replaceNotebookMetadata(resource, {
edit.set(resource, [vscode.NotebookEdit.updateNotebookMetadata({
...document.metadata,
custom: {
...(document.metadata.custom ?? {}),
@@ -103,7 +118,7 @@ export function activate(context: vscode.ExtensionContext) {
...metadata
},
}
});
})]);
return vscode.workspace.applyEdit(edit);
},
};

View File

@@ -0,0 +1,391 @@
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as vscode from 'vscode';
import { ATTACHMENT_CLEANUP_COMMANDID, JUPYTER_NOTEBOOK_MARKDOWN_SELECTOR } from './constants';
import { deepClone, objectEquals, Delayer } from './helper';
interface AttachmentCleanRequest {
notebook: vscode.NotebookDocument;
document: vscode.TextDocument;
cell: vscode.NotebookCell;
}
interface IAttachmentData {
[key: string /** mimetype */]: string;/** b64-encoded */
}
interface IAttachmentDiagnostic {
name: string;
ranges: vscode.Range[];
}
export enum DiagnosticCode {
missing_attachment = 'notebook.missing-attachment'
}
export class AttachmentCleaner implements vscode.CodeActionProvider {
private _attachmentCache:
Map<string /** uri */, Map<string /** cell fragment*/, Map<string /** attachment filename */, IAttachmentData>>> = new Map();
private _disposables: vscode.Disposable[];
private _imageDiagnosticCollection: vscode.DiagnosticCollection;
private readonly _delayer = new Delayer(750);
constructor() {
this._disposables = [];
this._imageDiagnosticCollection = vscode.languages.createDiagnosticCollection('Notebook Image Attachment');
this._disposables.push(this._imageDiagnosticCollection);
this._disposables.push(vscode.commands.registerCommand(ATTACHMENT_CLEANUP_COMMANDID, async (document: vscode.Uri, range: vscode.Range) => {
const workspaceEdit = new vscode.WorkspaceEdit();
workspaceEdit.delete(document, range);
await vscode.workspace.applyEdit(workspaceEdit);
}));
this._disposables.push(vscode.languages.registerCodeActionsProvider(JUPYTER_NOTEBOOK_MARKDOWN_SELECTOR, this, {
providedCodeActionKinds: [
vscode.CodeActionKind.QuickFix
],
}));
this._disposables.push(vscode.workspace.onDidChangeNotebookDocument(e => {
this._delayer.trigger(() => {
e.cellChanges.forEach(change => {
if (!change.document) {
return;
}
if (change.cell.kind !== vscode.NotebookCellKind.Markup) {
return;
}
const metadataEdit = this.cleanNotebookAttachments({
notebook: e.notebook,
cell: change.cell,
document: change.document
});
if (metadataEdit) {
const workspaceEdit = new vscode.WorkspaceEdit();
workspaceEdit.set(e.notebook.uri, [metadataEdit]);
vscode.workspace.applyEdit(workspaceEdit);
}
});
});
}));
this._disposables.push(vscode.workspace.onWillSaveNotebookDocument(e => {
if (e.reason === vscode.TextDocumentSaveReason.Manual) {
this._delayer.dispose();
e.waitUntil(new Promise((resolve) => {
if (e.notebook.getCells().length === 0) {
return;
}
const notebookEdits: vscode.NotebookEdit[] = [];
for (const cell of e.notebook.getCells()) {
if (cell.kind !== vscode.NotebookCellKind.Markup) {
continue;
}
const metadataEdit = this.cleanNotebookAttachments({
notebook: e.notebook,
cell: cell,
document: cell.document
});
if (metadataEdit) {
notebookEdits.push(metadataEdit);
}
}
const workspaceEdit = new vscode.WorkspaceEdit();
workspaceEdit.set(e.notebook.uri, notebookEdits);
resolve(workspaceEdit);
}));
}
}));
this._disposables.push(vscode.workspace.onDidCloseNotebookDocument(e => {
this._attachmentCache.delete(e.uri.toString());
}));
this._disposables.push(vscode.workspace.onWillRenameFiles(e => {
const re = /\.ipynb$/;
for (const file of e.files) {
if (!re.exec(file.oldUri.toString())) {
continue;
}
// transfer cache to new uri
if (this._attachmentCache.has(file.oldUri.toString())) {
this._attachmentCache.set(file.newUri.toString(), this._attachmentCache.get(file.oldUri.toString())!);
this._attachmentCache.delete(file.oldUri.toString());
}
}
}));
this._disposables.push(vscode.workspace.onDidOpenTextDocument(e => {
this.analyzeMissingAttachments(e);
}));
this._disposables.push(vscode.workspace.onDidCloseTextDocument(e => {
this.analyzeMissingAttachments(e);
}));
vscode.workspace.textDocuments.forEach(document => {
this.analyzeMissingAttachments(document);
});
}
provideCodeActions(document: vscode.TextDocument, _range: vscode.Range | vscode.Selection, context: vscode.CodeActionContext, _token: vscode.CancellationToken): vscode.ProviderResult<(vscode.CodeAction | vscode.Command)[]> {
const fixes: vscode.CodeAction[] = [];
for (const diagnostic of context.diagnostics) {
switch (diagnostic.code) {
case DiagnosticCode.missing_attachment:
{
const fix = new vscode.CodeAction(
'Remove invalid image attachment reference',
vscode.CodeActionKind.QuickFix);
fix.command = {
command: ATTACHMENT_CLEANUP_COMMANDID,
title: 'Remove invalid image attachment reference',
arguments: [document.uri, diagnostic.range],
};
fixes.push(fix);
}
break;
}
}
return fixes;
}
/**
* take in a NotebookDocumentChangeEvent, and clean the attachment data for the cell(s) that have had their markdown source code changed
* @param e NotebookDocumentChangeEvent from the onDidChangeNotebookDocument listener
* @returns vscode.NotebookEdit, the metadata alteration performed on the json behind the ipynb
*/
private cleanNotebookAttachments(e: AttachmentCleanRequest): vscode.NotebookEdit | undefined {
if (e.notebook.isClosed) {
return;
}
const document = e.document;
const cell = e.cell;
const markdownAttachmentsInUse: { [key: string /** filename */]: IAttachmentData } = {};
const cellFragment = cell.document.uri.fragment;
const notebookUri = e.notebook.uri.toString();
const diagnostics: IAttachmentDiagnostic[] = [];
const markdownAttachmentsRefedInCell = this.getAttachmentNames(document);
if (markdownAttachmentsRefedInCell.size === 0) {
// no attachments used in this cell, cache all images from cell metadata
this.saveAllAttachmentsToCache(cell.metadata, notebookUri, cellFragment);
}
if (this.checkMetadataHasAttachmentsField(cell.metadata)) {
// the cell metadata contains attachments, check if any are used in the markdown source
for (const [currFilename, attachment] of Object.entries(cell.metadata.attachments)) {
// means markdown reference is present in the metadata, rendering will work properly
// therefore, we don't need to check it in the next loop either
if (markdownAttachmentsRefedInCell.has(currFilename)) {
// attachment reference is present in the markdown source, no need to cache it
markdownAttachmentsRefedInCell.get(currFilename)!.valid = true;
markdownAttachmentsInUse[currFilename] = attachment as IAttachmentData;
} else {
// attachment reference is not present in the markdown source, cache it
this.saveAttachmentToCache(notebookUri, cellFragment, currFilename, cell.metadata);
}
}
}
for (const [currFilename, attachment] of markdownAttachmentsRefedInCell) {
if (attachment.valid) {
// attachment reference is present in both the markdown source and the metadata, no op
continue;
}
// if image is referenced in markdown source but not in metadata -> check if we have image in the cache
const cachedImageAttachment = this._attachmentCache.get(notebookUri)?.get(cellFragment)?.get(currFilename);
if (cachedImageAttachment) {
markdownAttachmentsInUse[currFilename] = cachedImageAttachment;
this._attachmentCache.get(notebookUri)?.get(cellFragment)?.delete(currFilename);
} else {
// if image is not in the cache, show warning
diagnostics.push({ name: currFilename, ranges: attachment.ranges });
}
}
this.updateDiagnostics(cell.document.uri, diagnostics);
if (cell.index > -1 && !objectEquals(markdownAttachmentsInUse, cell.metadata.attachments)) {
const updateMetadata: { [key: string]: any } = deepClone(cell.metadata);
updateMetadata.attachments = markdownAttachmentsInUse;
const metadataEdit = vscode.NotebookEdit.updateCellMetadata(cell.index, updateMetadata);
return metadataEdit;
}
return;
}
private analyzeMissingAttachments(document: vscode.TextDocument): void {
if (document.uri.scheme !== 'vscode-notebook-cell') {
// not notebook
return;
}
if (document.isClosed) {
this.updateDiagnostics(document.uri, []);
return;
}
let notebook: vscode.NotebookDocument | undefined;
let activeCell: vscode.NotebookCell | undefined;
for (const notebookDocument of vscode.workspace.notebookDocuments) {
const cell = notebookDocument.getCells().find(cell => cell.document === document);
if (cell) {
notebook = notebookDocument;
activeCell = cell;
break;
}
}
if (!notebook || !activeCell) {
return;
}
const diagnostics: IAttachmentDiagnostic[] = [];
const markdownAttachments = this.getAttachmentNames(document);
if (this.checkMetadataHasAttachmentsField(activeCell.metadata)) {
for (const [currFilename, attachment] of markdownAttachments) {
if (!activeCell.metadata.attachments[currFilename]) {
// no attachment reference in the metadata
diagnostics.push({ name: currFilename, ranges: attachment.ranges });
}
}
}
this.updateDiagnostics(activeCell.document.uri, diagnostics);
}
private updateDiagnostics(cellUri: vscode.Uri, diagnostics: IAttachmentDiagnostic[]) {
const vscodeDiagnostics: vscode.Diagnostic[] = [];
for (const currDiagnostic of diagnostics) {
currDiagnostic.ranges.forEach(range => {
const diagnostic = new vscode.Diagnostic(range, `The image named: '${currDiagnostic.name}' is not present in cell metadata.`, vscode.DiagnosticSeverity.Warning);
diagnostic.code = DiagnosticCode.missing_attachment;
vscodeDiagnostics.push(diagnostic);
});
}
this._imageDiagnosticCollection.set(cellUri, vscodeDiagnostics);
}
/**
* remove attachment from metadata and add it to the cache
* @param notebookUri uri of the notebook currently being edited
* @param cellFragment fragment of the cell currently being edited
* @param currFilename filename of the image being pulled into the cell
* @param metadata metadata of the cell currently being edited
*/
private saveAttachmentToCache(notebookUri: string, cellFragment: string, currFilename: string, metadata: { [key: string]: any }): void {
const documentCache = this._attachmentCache.get(notebookUri);
if (!documentCache) {
// no cache for this notebook yet
const cellCache = new Map<string, IAttachmentData>();
cellCache.set(currFilename, this.getMetadataAttachment(metadata, currFilename));
const documentCache = new Map();
documentCache.set(cellFragment, cellCache);
this._attachmentCache.set(notebookUri, documentCache);
} else if (!documentCache.has(cellFragment)) {
// no cache for this cell yet
const cellCache = new Map<string, IAttachmentData>();
cellCache.set(currFilename, this.getMetadataAttachment(metadata, currFilename));
documentCache.set(cellFragment, cellCache);
} else {
// cache for this cell already exists
// add to cell cache
documentCache.get(cellFragment)?.set(currFilename, this.getMetadataAttachment(metadata, currFilename));
}
}
/**
* get an attachment entry from the given metadata
* @param metadata metadata to extract image data from
* @param currFilename filename of image being extracted
* @returns
*/
private getMetadataAttachment(metadata: { [key: string]: any }, currFilename: string): { [key: string]: any } {
return metadata.attachments[currFilename];
}
/**
* returns a boolean that represents if there are any images in the attachment field of a cell's metadata
* @param metadata metadata of cell
* @returns boolean representing the presence of any attachments
*/
private checkMetadataHasAttachmentsField(metadata: { [key: string]: unknown }): metadata is { readonly attachments: Record<string, unknown> } {
return !!metadata.attachments && typeof metadata.attachments === 'object';
}
/**
* given metadata from a cell, cache every image (used in cases with no image links in markdown source)
* @param metadata metadata for a cell with no images in markdown source
* @param notebookUri uri for the notebook being edited
* @param cellFragment fragment of cell being edited
*/
private saveAllAttachmentsToCache(metadata: { [key: string]: unknown }, notebookUri: string, cellFragment: string): void {
const documentCache = this._attachmentCache.get(notebookUri) ?? new Map();
this._attachmentCache.set(notebookUri, documentCache);
const cellCache = documentCache.get(cellFragment) ?? new Map<string, IAttachmentData>();
documentCache.set(cellFragment, cellCache);
if (metadata.attachments && typeof metadata.attachments === 'object') {
for (const [currFilename, attachment] of Object.entries(metadata.attachments)) {
cellCache.set(currFilename, attachment);
}
}
}
/**
* pass in all of the markdown source code, and get a dictionary of all images referenced in the markdown. keys are image filenames, values are render state
* @param document the text document for the cell, formatted as a string
*/
private getAttachmentNames(document: vscode.TextDocument) {
const source = document.getText();
const filenames: Map<string, { valid: boolean; ranges: vscode.Range[] }> = new Map();
const re = /!\[.*?\]\(<?attachment:(?<filename>.*?)>?\)/gm;
let match;
while ((match = re.exec(source))) {
if (match.groups?.filename) {
const index = match.index;
const length = match[0].length;
const startPosition = document.positionAt(index);
const endPosition = document.positionAt(index + length);
const range = new vscode.Range(startPosition, endPosition);
const filename = filenames.get(match.groups.filename) ?? { valid: false, ranges: [] };
filenames.set(match.groups.filename, filename);
filename.ranges.push(range);
}
}
return filenames;
}
dispose() {
this._disposables.forEach(d => d.dispose());
this._delayer.dispose();
}
}

View File

@@ -0,0 +1,326 @@
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as vscode from 'vscode';
import { JUPYTER_NOTEBOOK_MARKDOWN_SELECTOR } from './constants';
import { basename, extname } from 'path';
enum MimeType {
bmp = 'image/bmp',
gif = 'image/gif',
ico = 'image/ico',
jpeg = 'image/jpeg',
png = 'image/png',
tiff = 'image/tiff',
webp = 'image/webp',
uriList = 'text/uri-list',
}
const imageMimeTypes: ReadonlySet<string> = new Set<string>([
MimeType.bmp,
MimeType.gif,
MimeType.ico,
MimeType.jpeg,
MimeType.png,
MimeType.tiff,
MimeType.webp,
]);
const imageExtToMime: ReadonlyMap<string, string> = new Map<string, string>([
['.bmp', MimeType.bmp],
['.gif', MimeType.gif],
['.ico', MimeType.ico],
['.jpe', MimeType.jpeg],
['.jpeg', MimeType.jpeg],
['.jpg', MimeType.jpeg],
['.png', MimeType.png],
['.tif', MimeType.tiff],
['.tiff', MimeType.tiff],
['.webp', MimeType.webp],
]);
function getImageMimeType(uri: vscode.Uri): string | undefined {
return imageExtToMime.get(extname(uri.fsPath).toLowerCase());
}
class DropOrPasteEditProvider implements vscode.DocumentPasteEditProvider, vscode.DocumentDropEditProvider {
private readonly id = 'insertAttachment';
private readonly defaultPriority = 5;
async provideDocumentPasteEdits(
document: vscode.TextDocument,
_ranges: readonly vscode.Range[],
dataTransfer: vscode.DataTransfer,
token: vscode.CancellationToken,
): Promise<vscode.DocumentPasteEdit | undefined> {
const enabled = vscode.workspace.getConfiguration('ipynb', document).get('pasteImagesAsAttachments.enabled', true);
if (!enabled) {
return;
}
const insert = await this.createInsertImageAttachmentEdit(document, dataTransfer, token);
if (!insert) {
return;
}
const pasteEdit = new vscode.DocumentPasteEdit(insert.insertText, this.id, vscode.l10n.t('Insert Image as Attachment'));
pasteEdit.priority = this.getPastePriority(dataTransfer);
pasteEdit.additionalEdit = insert.additionalEdit;
return pasteEdit;
}
async provideDocumentDropEdits(
document: vscode.TextDocument,
_position: vscode.Position,
dataTransfer: vscode.DataTransfer,
token: vscode.CancellationToken,
): Promise<vscode.DocumentDropEdit | undefined> {
const insert = await this.createInsertImageAttachmentEdit(document, dataTransfer, token);
if (!insert) {
return;
}
const dropEdit = new vscode.DocumentDropEdit(insert.insertText);
dropEdit.id = this.id;
dropEdit.priority = this.defaultPriority;
dropEdit.additionalEdit = insert.additionalEdit;
dropEdit.label = vscode.l10n.t('Insert Image as Attachment');
return dropEdit;
}
private getPastePriority(dataTransfer: vscode.DataTransfer): number {
if (dataTransfer.get('text/plain')) {
// Deprioritize in favor of normal text content
return -5;
}
// Otherwise boost priority so attachments are preferred
return this.defaultPriority;
}
private async createInsertImageAttachmentEdit(
document: vscode.TextDocument,
dataTransfer: vscode.DataTransfer,
token: vscode.CancellationToken,
): Promise<{ insertText: vscode.SnippetString; additionalEdit: vscode.WorkspaceEdit } | undefined> {
const imageData = await getDroppedImageData(dataTransfer, token);
if (!imageData.length || token.isCancellationRequested) {
return;
}
const currentCell = getCellFromCellDocument(document);
if (!currentCell) {
return undefined;
}
// create updated metadata for cell (prep for WorkspaceEdit)
const newAttachment = buildAttachment(currentCell, imageData);
if (!newAttachment) {
return;
}
// build edits
const additionalEdit = new vscode.WorkspaceEdit();
const nbEdit = vscode.NotebookEdit.updateCellMetadata(currentCell.index, newAttachment.metadata);
const notebookUri = currentCell.notebook.uri;
additionalEdit.set(notebookUri, [nbEdit]);
// create a snippet for paste
const insertText = new vscode.SnippetString();
newAttachment.filenames.forEach((filename, i) => {
insertText.appendText('![');
insertText.appendPlaceholder(`${filename}`);
insertText.appendText(`](${/\s/.test(filename) ? `<attachment:${filename}>` : `attachment:${filename}`})`);
if (i !== newAttachment.filenames.length - 1) {
insertText.appendText(' ');
}
});
return { insertText, additionalEdit };
}
}
async function getDroppedImageData(
dataTransfer: vscode.DataTransfer,
token: vscode.CancellationToken,
): Promise<readonly ImageAttachmentData[]> {
// Prefer using image data in the clipboard
const files = coalesce(await Promise.all(Array.from(dataTransfer, async ([mimeType, item]): Promise<ImageAttachmentData | undefined> => {
if (!imageMimeTypes.has(mimeType)) {
return;
}
const file = item.asFile();
if (!file) {
return;
}
const data = await file.data();
return { fileName: file.name, mimeType, data };
})));
if (files.length) {
return files;
}
// Then fallback to image files in the uri-list
const urlList = await dataTransfer.get('text/uri-list')?.asString();
if (token.isCancellationRequested) {
return [];
}
if (urlList) {
const uris: vscode.Uri[] = [];
for (const resource of urlList.split(/\r?\n/g)) {
try {
uris.push(vscode.Uri.parse(resource));
} catch {
// noop
}
}
const entries = await Promise.all(uris.map(async (uri) => {
const mimeType = getImageMimeType(uri);
if (!mimeType) {
return;
}
const data = await vscode.workspace.fs.readFile(uri);
return { fileName: basename(uri.fsPath), mimeType, data };
}));
return coalesce(entries);
}
return [];
}
function coalesce<T>(array: ReadonlyArray<T | undefined | null>): T[] {
return <T[]>array.filter(e => !!e);
}
function getCellFromCellDocument(cellDocument: vscode.TextDocument): vscode.NotebookCell | undefined {
for (const notebook of vscode.workspace.notebookDocuments) {
if (notebook.uri.path === cellDocument.uri.path) {
for (const cell of notebook.getCells()) {
if (cell.document === cellDocument) {
return cell;
}
}
}
}
return undefined;
}
/**
* Taken from https://github.com/microsoft/vscode/blob/743b016722db90df977feecde0a4b3b4f58c2a4c/src/vs/base/common/buffer.ts#L350-L387
*/
function encodeBase64(buffer: Uint8Array, padded = true, urlSafe = false) {
const base64Alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
const base64UrlSafeAlphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_';
const dictionary = urlSafe ? base64UrlSafeAlphabet : base64Alphabet;
let output = '';
const remainder = buffer.byteLength % 3;
let i = 0;
for (; i < buffer.byteLength - remainder; i += 3) {
const a = buffer[i + 0];
const b = buffer[i + 1];
const c = buffer[i + 2];
output += dictionary[a >>> 2];
output += dictionary[(a << 4 | b >>> 4) & 0b111111];
output += dictionary[(b << 2 | c >>> 6) & 0b111111];
output += dictionary[c & 0b111111];
}
if (remainder === 1) {
const a = buffer[i + 0];
output += dictionary[a >>> 2];
output += dictionary[(a << 4) & 0b111111];
if (padded) { output += '=='; }
} else if (remainder === 2) {
const a = buffer[i + 0];
const b = buffer[i + 1];
output += dictionary[a >>> 2];
output += dictionary[(a << 4 | b >>> 4) & 0b111111];
output += dictionary[(b << 2) & 0b111111];
if (padded) { output += '='; }
}
return output;
}
interface ImageAttachmentData {
readonly fileName: string;
readonly data: Uint8Array;
readonly mimeType: string;
}
function buildAttachment(
cell: vscode.NotebookCell,
attachments: readonly ImageAttachmentData[],
): { metadata: { [key: string]: any }; filenames: string[] } | undefined {
const cellMetadata = { ...cell.metadata };
const tempFilenames: string[] = [];
if (!attachments.length) {
return undefined;
}
if (!cellMetadata.attachments) {
cellMetadata.attachments = {};
}
for (const attachment of attachments) {
const b64 = encodeBase64(attachment.data);
const fileExt = extname(attachment.fileName);
const filenameWithoutExt = basename(attachment.fileName, fileExt);
let tempFilename = filenameWithoutExt + fileExt;
for (let appendValue = 2; tempFilename in cellMetadata.attachments; appendValue++) {
const objEntries = Object.entries(cellMetadata.attachments[tempFilename]);
if (objEntries.length) { // check that mime:b64 are present
const [mime, attachmentb64] = objEntries[0];
if (mime === attachment.mimeType && attachmentb64 === b64) { // checking if filename can be reused, based on comparison of image data
break;
} else {
tempFilename = filenameWithoutExt.concat(`-${appendValue}`) + fileExt;
}
}
}
tempFilenames.push(tempFilename);
cellMetadata.attachments[tempFilename] = { [attachment.mimeType]: b64 };
}
return {
metadata: cellMetadata,
filenames: tempFilenames,
};
}
export function notebookImagePasteSetup(): vscode.Disposable {
const provider = new DropOrPasteEditProvider();
return vscode.Disposable.from(
vscode.languages.registerDocumentPasteEditProvider(JUPYTER_NOTEBOOK_MARKDOWN_SELECTOR, provider, {
pasteMimeTypes: [
MimeType.png,
MimeType.uriList,
],
}),
vscode.languages.registerDocumentDropEditProvider(JUPYTER_NOTEBOOK_MARKDOWN_SELECTOR, provider, {
dropMimeTypes: [
...Object.values(imageExtToMime),
MimeType.uriList,
],
})
);
}

View File

@@ -3,7 +3,7 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as nbformat from '@jupyterlab/nbformat';
import type * as nbformat from '@jupyterlab/nbformat';
import * as detectIndent from 'detect-indent';
import * as vscode from 'vscode';
import { defaultNotebookFormat } from './constants';
@@ -84,7 +84,7 @@ export class NotebookSerializer implements vscode.NotebookSerializer {
public serializeNotebookToString(data: vscode.NotebookData): string {
const notebookContent = getNotebookMetadata(data);
// use the preferred language from document metadata or the first cell language as the notebook preferred cell language
const preferredCellLanguage = notebookContent.metadata?.language_info?.name ?? data.cells[0].languageId;
const preferredCellLanguage = notebookContent.metadata?.language_info?.name ?? data.cells.find(cell => cell.kind === vscode.NotebookCellKind.Code)?.languageId;
notebookContent.cells = data.cells
.map(cell => createJupyterCellFromNotebookCell(cell, preferredCellLanguage))
@@ -93,7 +93,7 @@ export class NotebookSerializer implements vscode.NotebookSerializer {
const indentAmount = data.metadata && 'indentAmount' in data.metadata && typeof data.metadata.indentAmount === 'string' ?
data.metadata.indentAmount :
' ';
// ipynb always ends with a trailing new line (we add this so that SCMs do not show unnecesary changes, resulting from a missing trailing new line).
// ipynb always ends with a trailing new line (we add this so that SCMs do not show unnecessary changes, resulting from a missing trailing new line).
return JSON.stringify(sortObjectPropertiesRecursively(notebookContent), undefined, indentAmount) + '\n';
}
}

View File

@@ -3,9 +3,9 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as nbformat from '@jupyterlab/nbformat';
import type * as nbformat from '@jupyterlab/nbformat';
import { NotebookCell, NotebookCellData, NotebookCellKind, NotebookCellOutput } from 'vscode';
import { CellMetadata, CellOutputMetadata } from './common';
import { CellOutputMetadata } from './common';
import { textMimeTypes } from './deserializers';
const textDecoder = new TextDecoder();
@@ -55,8 +55,14 @@ export function sortObjectPropertiesRecursively(obj: any): any {
}
export function getCellMetadata(cell: NotebookCell | NotebookCellData) {
return cell.metadata?.custom as CellMetadata | undefined;
return {
// it contains the cell id, and the cell metadata, along with other nb cell metadata
...(cell.metadata?.custom ?? {}),
// promote the cell attachments to the top level
attachments: cell.metadata?.custom?.attachments ?? cell.metadata?.attachments
};
}
function createCodeCellFromNotebookCell(cell: NotebookCellData, preferredLanguage: string | undefined): nbformat.ICodeCell {
const cellMetadata = getCellMetadata(cell);
let metadata = cellMetadata?.metadata || {}; // This cannot be empty.
@@ -274,7 +280,7 @@ function convertStreamOutput(output: NotebookCellOutput): JupyterOutput {
.filter((opit) => opit.mime === CellOutputMimeTypes.stderr || opit.mime === CellOutputMimeTypes.stdout)
.map((opit) => textDecoder.decode(opit.data))
.forEach(value => {
// Ensure each line is a seprate entry in an array (ending with \n).
// Ensure each line is a separate entry in an array (ending with \n).
const lines = value.split('\n');
// If the last item in `outputs` is not empty and the first item in `lines` is not empty, then concate them.
// As they are part of the same line.
@@ -326,16 +332,17 @@ function convertOutputMimeToJupyterOutput(mime: string, value: Uint8Array) {
} else if (mime.toLowerCase().includes('json')) {
const stringValue = textDecoder.decode(value);
return stringValue.length > 0 ? JSON.parse(stringValue) : stringValue;
} else if (mime === 'image/svg+xml') {
return splitMultilineString(textDecoder.decode(value));
} else {
const stringValue = textDecoder.decode(value);
return stringValue;
return textDecoder.decode(value);
}
} catch (ex) {
return '';
}
}
function createMarkdownCellFromNotebookCell(cell: NotebookCellData): nbformat.IMarkdownCell {
export function createMarkdownCellFromNotebookCell(cell: NotebookCellData): nbformat.IMarkdownCell {
const cellMetadata = getCellMetadata(cell);
const markdownCell: any = {
cell_type: 'markdown',

View File

@@ -3,10 +3,10 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
const path = require('path');
const testRunner = require('../../../../test/integration/electron/testrunner');
import * as path from 'path';
import * as testRunner from '../../../../test/integration/electron/testrunner';
const options: any = {
const options: import('mocha').MochaOptions = {
ui: 'tdd',
color: true,
timeout: 60000

View File

@@ -3,10 +3,11 @@
* Licensed under the Source EULA. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import * as nbformat from '@jupyterlab/nbformat';
import type * as nbformat from '@jupyterlab/nbformat';
import * as assert from 'assert';
import * as vscode from 'vscode';
import { jupyterCellOutputToCellOutput, jupyterNotebookModelToNotebookData } from '../deserializers';
import { createMarkdownCellFromNotebookCell, getCellMetadata } from '../serializers';
function deepStripProperties(obj: any, props: string[]) {
for (let prop in obj) {
@@ -52,6 +53,71 @@ suite('ipynb serializer', () => {
assert.deepStrictEqual(notebook.cells, [expectedCodeCell, expectedMarkdownCell]);
});
test('Serialize', async () => {
const markdownCell = new vscode.NotebookCellData(vscode.NotebookCellKind.Markup, '# header1', 'markdown');
markdownCell.metadata = {
attachments: {
'image.png': {
'image/png': 'abc'
}
},
custom: {
id: '123',
metadata: {
foo: 'bar'
}
}
};
const cellMetadata = getCellMetadata(markdownCell);
assert.deepStrictEqual(cellMetadata, {
id: '123',
metadata: {
foo: 'bar',
},
attachments: {
'image.png': {
'image/png': 'abc'
}
}
});
const markdownCell2 = new vscode.NotebookCellData(vscode.NotebookCellKind.Markup, '# header1', 'markdown');
markdownCell2.metadata = {
custom: {
id: '123',
metadata: {
foo: 'bar'
},
attachments: {
'image.png': {
'image/png': 'abc'
}
}
}
};
const nbMarkdownCell = createMarkdownCellFromNotebookCell(markdownCell);
const nbMarkdownCell2 = createMarkdownCellFromNotebookCell(markdownCell2);
assert.deepStrictEqual(nbMarkdownCell, nbMarkdownCell2);
assert.deepStrictEqual(nbMarkdownCell, {
cell_type: 'markdown',
source: ['# header1'],
metadata: {
foo: 'bar',
},
attachments: {
'image.png': {
'image/png': 'abc'
}
},
id: '123'
});
});
suite('Outputs', () => {
function validateCellOutputTranslation(
outputs: nbformat.IOutput[],