A 2024 Wishlist for Node's Test Runner

When the Node.js test runner was initially created, it was supposed to be very minimal. Of course, developers are never happy with simple and/or minimal, so the test runner has already grown far beyond what it was ever intended to be. Since that trend seems likely to continue, I have put together a short wishlist of improvements that I would love to see in Node's test runner even though I have moved on from Node.js myself.

Improve Filtering

When I say filtering here, I am referring to only tests and the --test-name-pattern flag. I am grouping these features together because they require the same fix - build the complete test tree before executing any tests. This would unlock other potential features such as test randomization (I am not necessarily advocating for that though).

Currently, the test runner executes tests as soon as it encounters them. I think it would be ideal if the test runner detected certain options that require walking the test tree twice (--test-name-pattern, --test-only, etc.), and changed its behavior based on their presence. That would improve the experience for users of those features, while keeping the default behavior fast. I do not think creating the full test tree should be the default behavior just to remove the --test-only flag. Sorry if you disagree with me there.

It's worth noting that I don't think this functionality would ever work perfectly for subtests. The reason is that to know what is inside of a test, the test runner has to execute the test. If the test runner has to execute all of the tests anyway, then there is no point in filtering at all. Statically analyzing the source code is another possibility, but that has its own pitfalls.

Module Mocking

Module mocking is something that I actually already implemented in a git branch. However, mocking ESM, requires using Node's experimental module customization hooks. These experimental APIs were changing pretty drastically while I was working on module mocking, and in my opinion, they are still not adequate for building a stable module mocking API in Node's test runner. Once the remaining issues with customization hooks are addressed, a mocking API could look like this:

import { test } from 'node:test';
import assert from 'node:assert';

test('module mocking test', async (t) => {
  const fixture = './module-mocking/basic-esm.mjs';
  const original = await import(fixture);

  assert.strictEqual(original.string, 'original esm string');
  assert.strictEqual(original.fn, undefined);

  t.mock.module(fixture, {
    exports: { fn() { return 42; } },
  });
  const mocked = await import(fixture);

  assert.notStrictEqual(original, mocked);
  assert.notStrictEqual(mocked, await import(fixture));
  assert.strictEqual(mocked.string, undefined);
  assert.strictEqual(mocked.fn(), 42);
  t.mock.reset();
  assert.strictEqual(original, await import(fixture));
});

An Assertion Library Integration API

Node's test runner allows you to use any assertion library that throws exceptions when an assertion fails. A number of other test runners ship their own assertionn library (sometimes this is just an existing assertion library integrated into the test framework). Some of these assertion libraries don't throw exceptions, but instead keep track of passes and failures and report it as part of the test results. This tight integration also allows them to support test plans (essentially another assertion for tracking the number of assertions a test should execute).

I think it would be great if node:test grew an API that allowed users to define their own assertion library. I had plans to work on this, but never got around to it. Pseudocode might look something like this:

import { test, createAssertion } from 'node:test';
import assert from 'node:assert';

createAssertion('myStrictEqual', (context, ...args) => {
  context.count++;

  try {
    assert.strictEqual(args[0], args[1]);
  } catch (err) {
    context.failures++;
    context.error = err;
  }
});

test('sample test', (t) => {
  t.plan(2);
  t.assert.myStrictEqual(1, 1);
  t.assert.myStrictEqual(1, 2);
});

Snapshot Testing

In my opinion, snapshot testing is an assertion library feature, not a test runner feature. However, because assertion libraries and testing frameworks are so closely associated, I have included it in my wishlist.

The node:assert module included experimental snapshot functionality briefly. However, the persistence mechanism of that implementation was considered too fragile, and it was ultimately removed.

Source Map Support

Node's test runner has experimental support for code coverage. Source map support is probably the biggest blocker to stabilizing code coverage. It would be nice to see this implemented even though I don't personally transpile backend code.

Conclusion

That is my node:test wishlist. Of course there are other bugs and feature requests that could be addressed, but those are the biggest ones to me. I hope someone is willing to fight the uphill battle of adding these features to core.