samedi 2 août 2014

Develop and test a Node.js HTTP client

Call an external HTTP service is a very common task when we develop a Node.js application. In production, an external service can be instable and it is essential that this instability does not broke your application. Whatever the project or the external service, cases to take into account are always the same.

From a simple use case, we will see in this article what are the different steps to follow in order to create a robust HTTP client in Node.js.

Our example

Our goal is to create a method countEmployees returning the result of the following HTTP request :

GET http://localhost:3000/employees/count
{"count":1200}

Step 1 : nominal case


In this step, we will write a first implementation of our client and test the case where the external service responds correctly.

To begin, using mocha, let's create a test which verifies that the value returned by the client is directly the value returned by the external service.

describe('employees count service', function() {
  it('should return 1200', function() {
 
    client.countEmployees().then(function(count) {
      count.should.equal(1200);
    }).done();
  });
});

Then, write a first implementation of the countEmployees method :

var Q = require('Q'),
  request = require('request'),
  config = require('./config');

exports.countEmployees = function() {
  var deferred = Q.defer();

  var options = {
    url: config.employeeCountUrl,
    json: true
  };

  request(options, function (error, response, body) {
    var employeesCount = body.count;
    deferred.resolve(employeesCount);
  });

  return deferred.promise;
};

We use in this implementation two modules : request for the HTTP requests and Q for the promises.

Here our test do a real HTTP call to the external service. To make the test pass, the HTTP service must be available and always return the same value. So it becomes essential to simulate this service. To do that, we will use nock module which allows to mock HTTP access in Node.js.

Starting with nock is easy thanks to its recorder. Firstly let's add the following instruction in our test :

beforeEach(function() {
  nock.recorder.rec();
});

During the test execution, we discover these lines in the console :

<<<<<<-- cut here -->>>>>>
nock('http://localhost:3000')
  .get('/employees/count')
  .reply(200, {"count":1200}, { 'x-powered-by': 'Express', 'content-type': 'application/json; 
charset=utf-8', 'content-length': '14', etag: 'W/"e-2043423703"', date: 'Wed, 09 Jul 2014 20:59:20 GMT',
connection: 'keep-alive' });
<<<<<<-- cut here -->>>>>>

You just have to copy past these lines in your test to simulate the execution of the request http://localhost:3000/employees/count. Don't hesitate to clean what you don't need, like for example the response header.

Another useful nock instruction is this one : nock.disableNetConnect() which allows to forbid all HTTP access. Our test becomes :

describe('employees count service', function() {
  beforeEach(function() {
    nock.disableNetConnect();
  });
 
  it('should return employees count', function() {
    nock('http://localhost:3000')
      .get('/employees/count')
      .reply(200, {count:1986});
 
    client.countEmployees().then(function(count) {
      count.should.equal(1986);
    }).done();
  });
});

And now, when the client executes the request, it is not the service which answers but nock. Moreover, we have the guarantee that we don't do extra HTTP requests. Now, shutdown the HTTP service and relaunch the test : it passes !

Our first goal is reached : we have a first implementation of our client and a test covering the nominal case.

Step 2 : service in error


Using nock, it is now easy to test the error cases. In this second step, we want to handle the case where the service returns an HTTP error. Let's add a new test and simulate the case where the service returns an error 500.

it('should return an error if http service is in error', function(done) {
  nock('http://localhost:3000')
    .get('/employees/count')
    .reply(500, {});
 
  client.countEmployees().then(function(count) {
    done(new Error('method should return an error'));
  }).catch(function() {
    done();
  });
});

The test doesn't pass. Indeed, the client doesn't return an error but a null result. Don't panic, we can easily handle this case and return an error if the http status code is not in the 200 range :

if (response.statusCode >= 300) {
  deferred.reject(new Error('Service has an invalid status code : ' + response.statusCode));
}

Step 2 goal is reached : our client can now handle HTTP errors.

Step 3 : service returns unexpected data


In this third step, we want to check that the client works correctly if the service returns unexpected data. In a new test, we will simulate that the service returns no count field but another field. With nock again, it is easy :

whenEmployeesCountIsCalled().reply(200, {nb:1986});

Please note that we factorized the nock call with the following method :

var whenEmployeesCountIsCalled = function() {
  return nock('http://localhost:3000')
    .get('/employees/count');
};

The test fails. To fix it, you can test that the expected field is in the HTTP request :

if (!employeesCount) {
  deferred.reject(new Error('Service did not return employees count'));
}

Step 3 goal is reached : we know how to handle unexpected data.

Step 4 : service is slow


In this step, we want to test the case where the external service is too slow. We can also do that with nock :

whenEmployeesCountIsCalled()
  .delayConnection(1500)
  .reply(200, {count:1986});

DelayConnection instruction allows to delay the HTTP answer of 1500 ms. Here we want that our service interrupts the connection after, for example 500 ms. With request module, you can configure a timeout :

var options = {
  url: config.employeeCountUrl,
  json: true,
  timeout: 500
};

Then you have to test if the error object is defined :

if (error) {
  deferred.reject(error);
  return;
}

The test passes. Step 4  is finished : our client is protected from slow access.

Step 5 : service is unavailable


In this last step, we want to check our client behavior with an unavailable service. Current version of nock cannot help to test this case. However we just need to do a real HTTP access on an unexisting host :

it('should return an error if service is unavailable', function(done) {
  client = rewire('../client/employees.client');
  client.__set__('config', {
    configEmployeeCountUrl: 'http://doesnotexist.localhost:3000/employees/count'
  });
 
  client.countEmployees().then(function() {
    done(new Error('method should return an error'));
  }).catch(function() { 
    done();
  });
});

Here we use rewire module to override the config object and give an unexisting url. The test passes. Indeed the fix from step 4 allows also to handle this error. Our goal is reached, we finalized the implementation of our HTTP client !

Conclusion 

Unavailable, slow, broken... production hazards are large and it is important that your application stays stable in these cases. Thanks to nock, we can easily reproduced these errors and build a robust code.

We covered only a few features of nock. To go further, don't hesitate to read the full documentation here : https://github.com/pgte/nock.

To finish, you will find the complete code on the following github respository : https://github.com/jsebfranck/node-http-example.






Translated from a Xebia article I wrote (in french)