worker: add wpt tests for Web Workers

Author: avivkeller

doc/api/globals.md

@@ -1316,8 +1316,98 @@ with the [`--no-experimental-websocket`][] CLI flag.
 added: REPLACEME
 -->
 
-A browser-compatible implementation of Web Workers. Enable this API
-with the [`--experimental-web-worker`][] CLI flag.
+> Stability: 1 - Experimental. Enable this API with the
+> [`--experimental-web-worker`][] CLI flag.
+
+A browser-compatible implementation of Web Workers of the [HTML Standard][],
+implemented on top of [`node:worker_threads`][]. Threads created with it
+are given the {DedicatedWorkerGlobalScope} API (`self`,
+`name`, `location`, `navigator`, `postMessage()`, `close()`, and
+`importScripts()`), in addition to the usual Node.js globals, such as `process`.
+
+```js
+// worker.js
+addEventListener('message', (event) => {
+  postMessage(`${event.data} from ${name}!`);
+});
+```
+
+```js
+// main.js
+const worker = new Worker('./worker.js', { name: 'greeter' });
+
+worker.addEventListener('message', (event) => {
+  console.log(event.data); // Prints: Hello from greeter!
+  worker.terminate();
+});
+
+worker.postMessage('Hello');
+```
+
+Because their lifetime and sharing model depend on origins and
+browsing contexts, Node.js does not currently implement `SharedWorker`.
+
+### Loading worker scripts
+
+Worker scripts are read synchronously from the local file system or from
+memory rather than fetched over the network, which changes which URLs are
+accepted and how failures are reported:
+
+* `new Worker()` and `importScripts()` accept only `file:`, `data:`, and
+  `blob:` URLs. Any other scheme makes `new Worker()` throw a
+  `NotSupportedError` and `importScripts()` throw a `NetworkError`.
+* A script that cannot be read makes `importScripts()` throw a `NetworkError`;
+  for `new Worker()` it fires an `error` event at the `Worker` object.
+* Redirects, the `nosniff` check, and HTTP MIME type validation do not apply.
+  MIME types are validated only for `data:` and `blob:` URLs. The
+  `credentials` option is validated for API compatibility but has no effect,
+  since no network request is made.
+* On the main thread, relative script URLs are resolved against the current
+  working directory, because there is no document base URL. Within a worker
+  they are resolved against the worker's own URL (as is done in the spec).
+* For `blob:` URLs, the script must be held in memory, so blobs backed by a file,
+  such as those returned by [`fs.openAsBlob()`][], cannot be used.
+
+### Differences from the HTML Standard
+
+Besides script loading, mentioned above:
+
+* Node.js has no origin model, so same-origin and cross-origin distinctions do
+  not exist and `location.origin` is `'null'` for every supported scheme.
+* `close()` terminates the worker immediately instead of following the
+  specification's "closing flag" algorithm, so code remaining in the current
+  task after `close()` is not executed.
+* The worker global is the normal Node.js global object with
+  `DedicatedWorkerGlobalScope` inserted into its prototype chain, rather than
+  a fresh global created from the interface. Node.js globals such as
+  `process`, `Buffer`, and `require()` remain available to worker scripts.
+* `ErrorEvent`s dispatched at `Worker` instances include `message` and
+  `error`, but `filename`, `lineno`, and `colno` are always `''`, `0`, and
+  `0`. An uncaught exception terminates the worker thread, and an unhandled
+  `error` event is not propagated further: it neither reaches the parent's
+  global scope nor affects the exit code of the process.
+* The following {WorkerGlobalScope} events are never dispatched, although
+  their handler properties exist: `languagechange`, `online`, and `offline`,
+  since these concepts do not exist in Node.js; `rejectionhandled` and
+  `unhandledrejection`, since Node.js exposes the equivalent does not
+  implement the `PromiseRejectionEvent` interface or the per-rejection
+  `preventDefault()` behavior required by the HTML Standard.
+
+### Web Workers and `node:worker_threads`
+
+Every Web Worker is backed by a [`node:worker_threads`][] {Worker}, so the
+two APIs share their threading, structured clone, and transfer semantics.
+Inside a worker, \[`worker_threads.parentPort`]\[] is the port behind
+`self.postMessage()` and the worker's `message` events, `isMainThread` is
+`false`, and `workerData` is `undefined`.
+
+As a rule of thumb, use [`node:worker_threads`][] directly when a program
+needs `workerData`, a custom `env` or `execArgv`, resource limits, stdio
+redirection, the `'online'` and `'exit'` events, or `worker.threadId`;
+`Worker` accepts only the `name`, `type`, and `credentials` options and,
+per the specification, its `terminate()` returns `undefined`, rather than
+a promise. Threads started through [`node:worker_threads`][] are ordinary
+Node.js threads and do not get the worker global scope APIs.
 
 ## Class: `WritableStream`
 
@@ -1364,6 +1454,7 @@ A browser-compatible implementation of [`WritableStreamDefaultWriter`][].
 [CommonJS module]: modules.md
 [CommonJS modules]: modules.md
 [ECMAScript module]: esm.md
+[HTML Standard]: https://html.spec.whatwg.org/multipage/workers.html
 [Navigator API]: https://html.spec.whatwg.org/multipage/system-state.html#the-navigator-object
 [RFC 5646]: https://www.rfc-editor.org/rfc/rfc5646.txt
 [Web Crypto API]: webcrypto.md
@@ -1420,9 +1511,11 @@ A browser-compatible implementation of [`WritableStreamDefaultWriter`][].
 [`console`]: console.md
 [`exports`]: modules.md#exports
 [`fetch()`]: https://developer.mozilla.org/en-US/docs/Web/API/Window/fetch
+[`fs.openAsBlob()`]: fs.md#fsopenasblobpath-options
 [`globalThis`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/globalThis
 [`localStorage`]: https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage
 [`module`]: modules.md#module
+[`node:worker_threads`]: worker_threads.md
 [`perf_hooks.performance`]: perf_hooks.md#perf_hooksperformance
 [`process.nextTick()`]: process.md#processnexttickcallback-args
 [`process` object]: process.md#process

test/common/wpt.js

@@ -8,6 +8,7 @@ const path = require('path');
 const events = require('events');
 const os = require('os');
 const { inspect } = require('util');
+const { pathToFileURL } = require('url');
 const { Worker } = require('worker_threads');
 
 const workerPath = path.join(__dirname, 'wpt/worker.js');
@@ -194,6 +195,24 @@ class ResourceLoader {
       fixtures.path('wpt', base, url);
   }
 
+  /**
+   * Map a URL that a test would have fetched from the WPT server (an
+   * absolute path, or a path relative to the test file) to a file: URL
+   * into the fixtures directory. URLs that already have a scheme (data:,
+   * blob:, http:, ...) are returned unchanged.
+   * @param {string} from the path of the file loading this resource,
+   *   relative to the WPT folder.
+   * @param {string|URL} url the url of the resource being loaded.
+   * @returns {string}
+   */
+  mapServerURL(from, url) {
+    url = `${url}`;
+    if (/^[a-zA-Z][a-zA-Z0-9+.-]*:|^\/\//.test(url)) {
+      return url;
+    }
+    return pathToFileURL(this.toRealFilePath(from, url)).href;
+  }
+
   /**
    * Load a resource in test/fixtures/wpt specified with a URL
    * @param {string} from the path of the file loading this resource,
@@ -389,13 +408,15 @@ class WPTTestSpec {
    * @returns {{ script?: string[]; variant?: string[]; [key: string]: string }} parsed META tags of a spec file
    */
   getMeta() {
-    const matches = this.getContent().match(/\/\/ META: .+/g);
+    // Like upstream, tolerate missing whitespace around "META:".
+    // Refs: https://github.com/web-platform-tests/wpt/blob/master/tools/manifest/sourcefile.py
+    const matches = this.getContent().match(/\/\/\s*META:\s*.+/g);
     if (!matches) {
       return {};
     }
     const result = {};
     for (const match of matches) {
-      const parts = match.match(/\/\/ META: ([^=]+?)=(.+)/);
+      const parts = match.match(/\/\/\s*META:\s*([^=]+?)=(.+)/);
       const key = parts[1];
       const value = parts[2];
       if (key === 'script' || key === 'variant') {
@@ -570,7 +591,11 @@ class WPTRunner {
     this.resource = new ResourceLoader(path);
     this.concurrency = concurrency;
 
-    this.flags = [];
+    // Since we need to prepare the Web Worker APIs
+    // in the harness that runs on all WPT workers,
+    // we enable the API globally. This has no practical
+    // effect on the non-web-worker tests, however.
+    this.flags = ['--experimental-web-worker'];
     this.globalThisInitScripts = [];
     this.initScript = null;
 
@@ -596,7 +621,7 @@ class WPTRunner {
    * @param {string[]} flags
    */
   setFlags(flags) {
-    this.flags = flags;
+    this.flags = this.flags.concat(flags);
   }
 
   /**
@@ -679,23 +704,41 @@ class WPTRunner {
       const absolutePath = spec.getAbsolutePath();
       const relativePath = spec.getRelativePath();
       const harnessPath = fixtures.path('wpt', 'resources', 'testharness.js');
-
-      // Scripts specified with the `// META: script=` header
-      const scriptsToRun = meta.script?.map((script) => {
+      // *.worker.js tests are dedicated worker tests by definition.
+      // Multi-global (*.any.js) tests whose global scopes include a
+      // dedicated worker but not a window also run inside an actual Web
+      // Worker, like the .any.worker.html variant generated by the WPT
+      // server does. Refs:
+      // https://web-platform-tests.org/writing-tests/testharness.html#multi-global-tests
+      const isAnyTest = /\.any\.js$/.test(spec.filename);
+      const globalScopes = isAnyTest ?
+        (meta.global?.split(',').map((s) => s.trim()) ??
+          ['window', 'dedicatedworker']) : [];
+      const isWebWorkerTest = /\.worker\.js$/.test(spec.filename) ||
+        (isAnyTest && !globalScopes.includes('window') &&
+         (globalScopes.includes('worker') ||
+          globalScopes.includes('dedicatedworker')));
+
+      // Scripts specified with the `// META: script=` header. For tests
+      // that run inside a Web Worker they are imported by the worker
+      // instead.
+      const scriptsToRun = isWebWorkerTest ? [] : meta.script?.map((script) => {
         const obj = {
           filename: this.resource.toRealFilePath(relativePath, script),
           code: this.resource.read(relativePath, script),
         };
         this.scriptsModifier?.(obj);
         return obj;
       }) ?? [];
-      // The actual test
-      const obj = {
-        code: content,
-        filename: absolutePath,
-      };
-      this.scriptsModifier?.(obj);
-      scriptsToRun.push(obj);
+      if (!isWebWorkerTest) {
+        // The actual test
+        const obj = {
+          code: content,
+          filename: absolutePath,
+        };
+        this.scriptsModifier?.(obj);
+        scriptsToRun.push(obj);
+      }
 
       run(async () => {
         const worker = new Worker(workerPath, {
@@ -710,6 +753,15 @@ class WPTRunner {
               filename: harnessPath,
             },
             scriptsToRun,
+            // Set when the test runs inside an actual Web Worker.
+            webWorker: isWebWorkerTest ? {
+              path: absolutePath,
+              isAnyTest,
+              scripts: meta.script?.map(
+                (script) => this.resource.toRealFilePath(relativePath, script),
+              ) ?? [],
+              skippedTests: spec.skippedTests,
+            } : undefined,
             needsGc: !!meta.script?.find((script) => script === '/common/gc.js'),
             skippedTests: spec.skippedTests,
           },

test/common/wpt/webworker.js

@@ -0,0 +1,112 @@
+'use strict';
+
+// Runs a WPT test file inside a Web Worker
+// Refs: https://web-platform-tests.org/writing-tests/testharness.html
+
+const { pathToFileURL } = require('url');
+
+globalThis.onmessage = ({ data }) => {
+  // Let the test install its own handler.
+  globalThis.onmessage = null;
+
+  const { ResourceLoader } = require(data.wptRunner);
+  const resource = new ResourceLoader(data.wptPath);
+
+  // Pretend the worker was served from the URL the WPT server would have
+  // used
+  const fakePath = (data.isAnyTest ?
+    data.testRelativePath.replace(/\.any\.js$/, '.any.worker.js') :
+    data.testRelativePath).replace(/\\/g, '/');
+  const fakeURL = new URL(`/${fakePath}`, 'http://wpt');
+  // eslint-disable-next-line no-undef
+  const fakeLocation = { __proto__: WorkerLocation.prototype };
+  for (const key of ['href', 'origin', 'protocol', 'host', 'hostname',
+                     'port', 'pathname', 'search', 'hash']) {
+    Object.defineProperty(fakeLocation, key, {
+      value: fakeURL[key],
+      enumerable: true,
+    });
+  }
+  Object.defineProperty(fakeLocation, 'toString', {
+    value: function toString() { return fakeURL.href; },
+    enumerable: true,
+  });
+  Object.defineProperty(globalThis, 'location', {
+    value: fakeLocation,
+    enumerable: true,
+    configurable: true,
+  });
+
+  const testharnessPath =
+    pathToFileURL(resource.toRealFilePath(data.testRelativePath,
+                                          '/resources/testharness.js')).href;
+
+  // If there are skip patterns, wrap the test functions to prevent
+  // execution of matching tests. This must happen after testharness.js is
+  // loaded but before the test scripts run.
+  function applySkips() {
+    if (!data.skippedTests?.length) {
+      return;
+    }
+    function isSkipped(name) {
+      for (const matcher of data.skippedTests) {
+        if (typeof matcher === 'string') {
+          if (name === matcher) return true;
+        } else if (matcher.test(name)) {
+          return true;
+        }
+      }
+      return false;
+    }
+    for (const fn of ['test', 'async_test', 'promise_test']) {
+      const original = globalThis[fn];
+      globalThis[fn] = function(func, name, ...rest) {
+        if (typeof name === 'string' && isSkipped(name)) {
+          // eslint-disable-next-line no-undef
+          postMessage({ type: 'skip', name });
+          return;
+        }
+        return original.call(this, func, name, ...rest);
+      };
+    }
+  }
+
+  // Tests fetch scripts and nested worker scripts from the WPT server; map
+  // those URLs into the fixtures directory.
+  const realImportScripts = globalThis.importScripts;
+  globalThis.importScripts = function importScripts(...urls) {
+    const mapped = urls.map(
+      (url) => resource.mapServerURL(data.testRelativePath, url));
+    const result = realImportScripts.apply(this, mapped);
+    if (mapped.includes(testharnessPath)) {
+      applySkips();
+    }
+    return result;
+  };
+  const RealWorker = globalThis.Worker;
+  globalThis.Worker = class Worker extends RealWorker {
+    constructor(url, options) {
+      super(resource.mapServerURL(data.testRelativePath, url), options);
+    }
+  };
+
+  if (data.isAnyTest) {
+    // Emulate the generated .any.worker.js wrapper script.
+    // Refs: https://github.com/web-platform-tests/wpt/blob/master/tools/serve/serve.py
+    globalThis.GLOBAL = {
+      isWindow() { return false; },
+      isWorker() { return true; },
+      isShadowRealm() { return false; },
+    };
+    globalThis.importScripts('/resources/testharness.js');
+    for (const script of data.scripts) {
+      globalThis.importScripts(pathToFileURL(script).href);
+    }
+    globalThis.importScripts(pathToFileURL(data.path).href);
+    // eslint-disable-next-line no-undef
+    done();
+  } else {
+    // *.worker.js tests import testharness.js and call done() themselves.
+    globalThis.importScripts(pathToFileURL(data.path).href);
+  }
+};