Setting Expectations for the Node.js Test Runner

Node.js began shipping an experimental test runner starting in version 18. A year later, in Node 20, the test runner was marked stable. Since its introduction, I would say the test runner has been generally well received. I have read blog posts, interacted with users on the Node.js issue tracker, listened to conference talks and YouTube videos, and even had conversations with people in real life like the old days. I have noticed a few points that keep coming up, so I decided to write this blog post to help people set appropriate expectations for Node's test runner.

Before going any further, I want to include a few disclaimers:

This post only represents my personal opinion. It does not represent the opinion of the Node.js project, any companies, etc.
I am, in no way, claiming that Node's test runner is perfect, or otherwise bug free. It is still under active development, and has some rough edges that can be improved.
I am not going to be bashing any other test frameworks here. You can use any test framework you want.

Now, let's talk about the test runner.

"Why did Node.js add a test runner in the first place?"

There was already a wide selection of testing frameworks for Node.js. When it was announced that Node would ship a builtin test runner, some people wanted to know why. Here are my reasons for wanting a test runner in Node.js:

Testing should be a first class citizen. Every complex piece of code needs tests. There are varying opinions on what constitutes good tests, but most reasonable people agree that tests should exist.
npm is not safe. Malware and supply chain security are big problems. Big enough that entire companies exist to combat the problem. Test runners are complex enough that they can easily have hundreds of dependencies. As we have seen over and over again, it only takes one rogue package to cause a major issue. If testing is a widespread activity, then compromising a test runner could have a big payout for an attacker. Even though test runners don't generally need to run in production environments, I don't want my personal laptop hacked either.
New languages and runtimes are including more built in tooling, including testing tools. Node.js has historically had a "small core philosophy" that deferred most non-essential functionality to userland. Some people still prefer the small core philosophy, but over time I have observed an increasing number of users asking for more batteries included.

"Just add these 15 command line flags like test framework X has!"

Using node:test means that you don't need to install a test runner! After all, it is built into Node.js. However, being built in comes with other tradeoffs. In userland, package authors are free to pull in arbitrary dependencies and add as many CLI flags or niche features as they want. It is not uncommon for userland test runners to be several megabytes in size. Userland test runners usually are not shipped to production environments either.

Since Day 1 keeping the test runner fairly minimal has been a design goal. Node's test runner will never have the API surface or number of command-line flags as a runner like Jest. With the very recent exception of glob support (which has use cases beyond the test runner), Node's test runner has been built from scratch using features already built into the runtime. We cannot bloat the Node.js executable with megabytes of test runner code and dependencies. You are still welcome to make feature requests, but please realize that you are likely to receive pushback.

I am biased, but I think Node's test runner currently strikes a good balance between powerful but still relatively minimal. If you think it is too minimal, compared to something like Jest, then the built in test runner is unlikely to ever be the right choice for you.

"There are less built in assertions!"

This is related to the previous point, but worth addressing separately. The Node.js test runner is assertion library agnostic. Any code that throws to indicate a failed assertion can be used. That means you can use node:assert, chai, @hapi/code, or even throw errors yourself.

When people say Node's test runner doesn't have a lot of assertions, they usually mean that node:assert doesn't have a lot of assertions. This is probably because most node:test examples are shown with node:assert as the assertion library, or because testing without any third party dependencies usually means using node:assert.

Personally, I prefer using node:assert as my assertion library. In general, I prefer simpler APIs over more complex ones. I used to maintain @hapi/code, which was originally derived from chai. Several years ago, I had a change of heart. I realized that large assertion APIs make me spend more time checking the documentation for exact syntax, or debugging quirky edge cases like strings, arrays, and objects having different special behaviors. This is also why I designed node:test's mocking API with a much smaller surface area than something like sinon.

All of the pieces can be used separately

Assertions aren't the only decoupled piece of Node's test runner. The CLI (e.g. the --test flag) can be used without the node:test API. In other words, the following 'test.js' file can be run via node --test test.js:

// test.js
throw new Error('a random error');

As long as the file sets its exit code to non-zero to indicate failure, the test runner CLI will be perfectly happy. Admittedly, you will get more nicely formatted output using the node:test module, but it is not required.

Similarly, node:test can be used without the CLI. The Node core test suite uses a test runner written in Python. This runner, which predates Node's test runner by years, also relies on process exit codes to indicate success and failure. Several contributors have been writing Node core tests with the node:test API for a while now.

The coverage and mocking functionality can also be swapped out. I have personally used c8 with the Node test runner. I have never used a different mocking module, but there is nothing that would prevent it from working (and others have reported doing so successfully).

Always check the docs for your version of Node

Despite becoming stable, Node's test runner is still under pretty active development. This means that blog posts, conference talks, and YouTube videos on the test runner become outdated quickly. This past week, I attended Node.TLV 2023, which was a great conference and overall great experience. There was one talk dedicated to Node's test runner, and another comparing several testing frameworks, including node:test. The next release of Node.js should introduce mock timers and glob support in the test runner - making both of those talks slightly outdated.

Knowing exactly what the test runner supports is made even more difficult by Node's release schedule. It can take weeks for a merged feature to show up in a Current release, and it can take several more months to show up in a Long-Term Support (LTS) release. As the test runner matures, the rate of development will inevitably slow down, and there will be fewer differences between the release lines.

Dogfooding and conclusion

As previously mentioned, node:test has seen increasing use in Node's own test suite. I have also been increasingly using the builtin test runner at Platformatic for production projects that support Node 18 and 20. These codebases are very involved, making heavy use of the Fastify web framework, worker threads, the file system, CommonJS and ESM modules, ESM loaders, REST and GraphQL APIs, Prometheus metrics, various databases, and more. The test runner has handled all of these things well!

In conclusion, I feel confident saying a few things:

Like any other piece of software, Node's test runner is not perfect. There are bugs, rough edges, and missing features. This situation will continue to improve over time.
If you pick a test framework based solely on the number of features (whether you actually need them or not), then Node's test runner is not for you.
Node's test runner is ready for use in real world projects.