Home

Developer Initiates I/O Operation. You Won't Believe What Happens Next.

Colin J. Ihrig

As Node.js developers, we are extremely spoiled. The level of abstraction provided to us by JavaScript allows us to focus on creating interesting applications instead of wrestling with low level system concepts such as threads and synchronization. But, whether we like to think about it or not, our JavaScript code sits on top of a lot of low level code, mostly written in C/C++, as shown in the following figure. This article will trace a simple JavaScript function call as it traverses various layers of this figure. A basic understanding of the event loop is assumed. If you need an introduction or refresher, check out this guide.

Node.js Application Architecture

A JavaScript File System Request

To get a better idea of what is under the hood of Node.js, let's take a look at what happens when we execute fs.stat() in the following code. We'll trace the execution through Node core and libuv, the asynchronous I/O library used by Node. Many links to code on GitHub have been included throughout this post. Those links have been pinned to the commit of the Node 7.3.0 release, so they should be accurate over time, even when the Node codebase changes.

'use strict';
const fs = require('fs');

fs.stat(__filename, function statCb (err, stats) {
  if (err) {
    return console.error(err);
  }

  console.log(stats);
});

On calling fs.stat(), we first enter Node core's fs module. The source code for fs.stat(), shown below, is fairly straightforward. The first two lines ensure that callback is a function, and that the provided path does not contain any null bytes. Next, an instance of FSReqWrap is created, and callback is attached as the oncomplete property. FSReqWrap is a C++ class that is accessible in JavaScript via Node's binding layer. The name FSReqWrap is short for "file system request wrap," as it encapsulates file system operation requests to libuv. The oncomplete field is meant to hold the callback function that is invoked once the operation is complete. Once the request is set up, the C++ function associated with binding.stat(), Stat(), is called.

fs.stat = function(path, callback) {
  callback = makeCallback(callback);
  if (!nullCheck(path, callback)) return;
  var req = new FSReqWrap();
  req.oncomplete = callback;
  binding.stat(pathModule._makeLong(path), req);
};

Entering the Node.js Binding Layer

At this point, we have left the JavaScript layer and entered the C/C++ layer. Here be dragons unicorn velociraptors. The interesting part of the Stat() function is the if statement shown below. This conditional determines if the call should be synchronous (fs.statSync()) or asynchronous (fs.stat()). In our case, binding.stat() was called with the FSReqWrap instance as the second argument. Therefore, the args[1]->IsObject() check will be true, and the ASYNC_CALL path is taken.

if (args[1]->IsObject()) {
  ASYNC_CALL(stat, args[1], UTF8, *path)
} else {
  SYNC_CALL(stat, *path, *path)
  args.GetReturnValue().Set(
    BuildStatsObject(env,
                     static_cast<const uv_stat_t*>(SYNC_REQ.ptr)));
}

Note that ASYNC_CALL is a C++ macro, and not a function call. Furthermore, ASYNC_CALL is expanded further using the ASYNC_DEST_CALL macro. At compile time the ASYNC_CALL macro will be expanded to the following C++ code. The most important piece of this code is the call to uv_fs_stat(). This submits our stat request to libuv, tied to our Node application's event loop. When the operation finishes, the After function is called.

Environment* env = Environment::GetCurrent(args);
CHECK(request->IsObject());
FSReqWrap* req_wrap = FSReqWrap::New(env, request.As<Object>(),
                                     "stat", dest, encoding);
int err = uv_fs_stat(env->event_loop(),
                     req_wrap->req(),
                     __VA_ARGS__,
                     After);
req_wrap->Dispatched();
if (err < 0) {
  uv_fs_t* uv_req = req_wrap->req();
  uv_req->result = err;
  uv_req->path = nullptr;
  After(uv_req);
  req_wrap = nullptr;
} else {
  args.GetReturnValue().Set(req_wrap->persistent());
}

Exiting Node.js, Entering libuv

We are leaving Node's C++ layer, and moving to the libuv C code. libuv is able to provide asynchronous file system operations by offloading work to a pool of independent worker threads. To remain as platform independent as possible, libuv often has to implement the same features in both Unix and Windows. For the purposes of this article, we're going to assume that you're running on Linux, and jump to the appropriate implementation of uv_fs_stat(). If you're running on Windows, you can follow along in a similar fashion, starting at the Windows uv_fs_stat() implementation.

On non-Windows systems, uv_fs_stat(), whose source code is shown below, is comprised of three macros, INIT, PATH, and POST. INIT, sets up the file system request and sets the callback appropriately, while PATH sets the file path of the operation. The POST macro submits the request to the threadpool, or executes the task, if the request is synchronous. In our case, we're following the asynchronous code path, which passes the file system request to uv__work_submit(). The function uv__fs_work() is passed to handle the request. This is the same function that executes the work directly in the synchronous scenario. Additionally, the uv__fs_done() function is passed as a callback that is executed when the request is complete.

int uv_fs_stat(uv_loop_t* loop,
               uv_fs_t* req,
               const char* path,
               uv_fs_cb cb) {
  INIT(STAT);
  PATH;
  POST;
}

Inside of uv__work_submit(), the request is submitted to the thread pool's work queue (queue implementation here) via the post() function. Notice that post() uses a mutex to synchronize access to the work queue, wq. This allows the main thread and all of the worker threads to share access to the same queue in a safe manner. At this point, it is up to a worker thread to remove the request from the queue.

By default, there are four thread pool workers, implemented as a worker() function that processes items from the work queue as they become available. When there is no work available, the threads just wait. Once the worker dequeues a piece of work, it executes its work() method. Looking back to uv__work_submit(), the work() method is actually uv__fs_work().

Inside of uv__fs_work(), the type of file system request is checked. In our case, the type is STAT. This triggers a call to uv__fs_stat(), shown below, which calls the C stat(2) function. We aren't going to trace through any lower level code, but this request goes through the C standard library and operating system. If the stat() call succeeds, uv__to_stat() is called to copy the result object into a libuv uv_stat_t data structure. By using its own stat structure, libuv can provide a more consistent interface across platforms. The result of uv__fs_stat() is then attached to the original file system request.

static int uv__fs_stat(const char *path, uv_stat_t *buf) {
  struct stat pbuf;
  int ret;

  ret = stat(path, &pbuf);
  if (ret == 0)
    uv__to_stat(&pbuf, buf);

  return ret;
}

Calling Back with Results

At this point, the stat() operation is complete, and the result can begin making its way back to the JavaScript layer. Control flow returns to the thread's worker() function, where uv_async_send() is called. uv_async_send() is responsible for waking up the event loop, which in Node's case runs on the main thread. This is done in uv__async_send() via a write to a file descriptor that the main thread is watching.

In the main thread, uv__io_poll() picks up the notification of work and calls the IO event's callback function, uv__async_io(), which, in turn, calls the async event's callback, uv__async_event(). This calls uv__work_done(), which calls the work's done() method. Recall that the done() method is actually uv__fs_done() via the POST macro and uv__work_submit().

The uv__fs_done() callback unregisters the request with the event loop and calls the request's callback. Recall that the INIT macro set the request's callback function to the callback passed to uv_fs_stat(). This callback takes us back to the Node.js binding layer. Again, recall that uv_fs_stat() was called with After() as its callback.

Inside of After(), assuming the operation succeeded, the uv_stat_t structure, containing the uv_fs_stat() result is passed to the BuildStatsObject() function. This creates a JavaScript object containing the information from the uv_stat_t structure. Once the JavaScript object is built, MakeCallback() is called, and the request object can be cleaned up. MakeCallback() is responsible for calling back into the JavaScript runtime with the result of fs.stat(). At this point, your JavaScript code can finally continue executing.

Conclusion

This article has traced a file system request from Node's JavaScript layer, through the C++ binding layer, and libuv's threadpool. We've seen how the various layers interact with one another, as well as how synchronous and asynchronous calls can share the same code paths. If you haven't explored the world below Node's JavaScript layer, don't feel bad if this seems a bit overwhelming. There is a lot going on under the hood, and it's kind of amazing that anything works at all!

Thank you to Saúl Ibarra Corretgé, of the libuv core team, for reviewing this post.