Home

Upcoming Changes to the Test Runner in Node 22

Colin J. Ihrig

It's almost that time again. Node 22 is currently scheduled to be released in about a week and a half. I'd like to briefly mention some of the changes that have gone into Node's test runner (node:test) recently because they are unlikely to get called out in the official release notes.

  • PR 52003. Ensure that the before hook is always run. Prior to this change, the before hook would not run if there were no tests. It was updated to always run in order to match the behavior of after.
  • PR 52115. When a test is skipped, the beforeEach and afterEach hooks should also be skipped. This change implements that behavior.
  • PR 52239. This PR changes the order that afterEach hooks are run in. Prior to this change, a parent's afterEach hooks would run before the current test's afterEach hooks. This behavior has now been fixed so that the current test's afterEach hooks run first. It is worth noting that afterEach was the only hook impacted by this bug.
  • PR 51996. Abort tests that create uncaughtException events. Prior to this change, the test runner would clean up after the test. However, the test was never actually aborted, so the test runner would continue waiting for it to finish. Now, the uncaughtException handler aborts the test so that everything proceeds normally.
  • PR 52020. This changes focuses on making the reported total execution time of the test runner more accurate. In some cases it was previously possible to overwrite the overall start time. This would lead to an inaccurate total execution time. The test runner no longer allows the start time to be overwritten.
  • PR 52010 and PR 52036. These changes are focused on better handling of test locations. When a test fails (or if you consume the test events directly), the test runner reports the filename, line number, and column number where the test is located. After these changes, the location is correctly reported when Node's --enable-source-maps flag is used. The test runner and reporters were also updated to handle undefined test locations, which can occur if you are using the test runner in the REPL.
  • PR 52060. The test runner's experimental code coverage feature now respects source maps. Prior to this change, code coverage did not handle source maps at all.
  • PR 52117. When a test or suite is marked as TODO, it should never impact whether or not the test is run. It should only impact the way failures are interpreted. Prior to this change, it was possible for a suite to be incorrectly skipped when marked as TODO. This change corrects that behavior. By the way, if you were ever curious what TODO tests are all about, PR 52204 added a section to the API docs explaining how they work.
  • PR 52130. This change allows code coverage to be reported multiple times when using watch mode.
  • PR 52127. This change adds a suite() function and makes the existing describe() function an alias of it. Now, you can use test() and it() to create tests, and suite() and describe() to create suites.
  • PR 52038. This change adds a new --test-force-exit CLI flag and forceExit option to run(). This forces the test runner to exit once it has processed all tests, regardless of any active handles keeping the event loop alive. I personally dislike this change very much because you should figure out why the event loop is still alive and fix the underlying problem. However, I do understand the use case and have been on teams that really do need this functionality... so here we are.
  • PR 52287. This change disables the highWatermark on the stream that generates the test reporter events. This stream is an object mode stream, so the default highWatermark was only 16. This limit gets hit fairly quickly, resulting in a lot of overhead. However, the highWatermark doesn't actually make sense for this particular stream, so it is safe to disable. This lead to significant performance improvements in the test runner.
  • PR 52185. This is a small performance improvement. Prior to this change, every top level test awaited the same Promise. Now, only the first top level test incurs that small overhead.
  • PR 52408. This is another performance improvement. This one improves the performance of AbortSignals, which are used a good bit in the test runner.
  • PR 52221, PR 52326, and PR 52488. This change completely removes filtered tests from the test runner output. In this context, filtering refers to tests that are not run because of the --test-only or --test-name-pattern flags. Prior to this change, filtered tests would simply be treated as skipped tests. However, for large test suites this would result in a lot of unhelpful output. Now, these tests are treated as if they never existed at all. This is technically a breaking change, but I would personally be in favor of backporting it to older release lines.
  • PR 52092 and PR 52296. These changes are the first steps toward addressing a major pain point with Node's test runner - only tests. In order to run only tests, you need to attach the only flag to the appropriate tests and pass the --test-only CLI flag. You would also need to set the only flag on any suites containing the tests you want to run. After these changes, you no longer need to take that last step. Hopefully, if PR 51579 ever lands, you won't need to pass --test-only anymore either unless you're using test isolation (not to mention the addition performance improvements).

More on performance

I mentioned some performance enhancements. In Node 22, the test runner should indeed be faster (although Amdahl's law still applies). Let's run a very quick performance test on my laptop comparing Node's main branch to v21.7.3. This is not a perfect comparison, but it's good enough for our purposes here. Here is the code:

'use strict'
const { test } = require('node:test');

for (let i = 0; i < 100_000; ++i) {
  test(`test ${i + 1}`);
}

On my laptop, Node v21.7.3 took roughly 86 seconds. Node's main branch took roughly 8 seconds. Note, these numbers were obtained by running node test.js without the --test CLI flag. When --test is used, there is additional overhead and these numbers jump to 95 and 18 seconds, respectively.

Filtering should also be significantly faster. The same code run with --test-only took 84 seconds on Node v21. It makes sense that the old filtering implementation took about as long as a no-op test since filtered tests were really just a no-op skipped test. Node's main branch took about 5 seconds in this benchmark.

Conclusion

I am excited for some of the improvements coming with the test runner. I am also really excited for some of the other changes currently in open pull requests, such as the option to run all tests in the same process. Of course, I'm also still keeping an eye toward my 2024 Wishlist for Node's Test Runner.