Digital (dis)content: Node.js, part 5: Zero the hero!

And you thought I was on holiday?

Time to hack our Node app again and introduce another great piece of software called zeromq. On top of having one of the coolest logos, zeromq is a high-performance, low-latency messaging system that you simply cannot afford to ignore when you need to build highly scalable apps (IMHO). Other contenders would be RabbitMQ (awesome and rock-solid, but not really lightweight) and NSQ (written by the bit.ly folks).

The basic idea will be to have our web app call a remote server through zeromq. The remote server will perform a super fancy calculation and return it back to the web app. I can feel that you're excited already, so let's get started :*)

Both our zeromq machines (the client and the server) require a little bit of setup: on top of Node.js and npm (as described in previous posts), we need to install the zeromq library. Once again, the one available in the Ubuntu repo is way too old, so let's build a fresh version:

$ wget http://download.zeromq.org/zeromq-3.2.3.tar.gz

$ tar xvfz zeromq-3.2.3.tar.gz

$ cd zeromq-3.2.3

$ ./configure
$ make

$ sudo make install

Now, let's add the zeromq bindings for Node.js, namely the zeromq.node library aka zmq:

$ sudo apt-get install build-essential (required by the zmq install)

$ npm install zmq

Now we're ready to roll. Let's keep it simple and use a nice request-reply pattern: the web app will send a request to the remote server and get a response back. Zeromq supports many other patters, you'll find more info in the official documentation. Definitely worth a read, zeromq is a superb piece of software engineering art. Lightweight, simple, elegant... ah, love at first sight.

But I digress (that fine bottle of Leffe, probably). Let's look at our zeromq server first.

BTW, since I guess it's time to stop mourning SCCS ;), I've created a proper Github account where you'll find all pieces of code posted on this blog. Can't say I love Git dearly, but a little progress every now and then is okay, I guess.

Nice and short, huh? Create a 'reply' socket, listen on port 3000/TCP and send back twice was we got. And yes, everything in life is an 'int', so there.

Now, let's look at the corresponding part in the client, i.e. our web app:

Just as simple. Create a 'request' socket, bind it to our remote server and, in a separate function, fire some data, read the answer and display it.

Here's the full code:

// Web stuff
var http = require('http');
var url = require('url');
// Memcache
var mc = require('mc');
// Winston (logging)
var winston = require('winston');
require('winston-syslogudp').Syslog;
// Zero MQ
var zmq = require('zmq')
var sock = zmq.socket('req');

function onRequest(request, response) {
        // No id parameter -->  do nothing
        // id=0 --> list all documents
        // id=SOME_OBJECT_ID --> show the 'x' value for this document

var query = url.parse(request.url,true).query;
        winston.info("Request received, id="+query.id);

if (query.id == null) {
                // No id parameter -->  do nothing
                response.end();
                return;
        }

if (query.id == "0") {
                // id=0 --> list all documents
                collection.find().toArray(function (err, items) {
                        if (err) {
                                winston.error("Could not find all documents, err="+err);
                                response.end();
                                return;
                        }

winston.info("Finding all documents");
                        response.writeHead(200, {"Content-Type": "text/html"});
                        response.write("<html>");
for (var i = 0; i < items.length; i++) {
                                response.write(JSON.stringify(items[i])+"<br>");
                        }
                        response.write("</HTML>");
                        response.end();
                });
        }

else {
                // Check that the ObjectId is valid
                try {
                        oid = new require('mongodb').ObjectID.createFromHexString(query.id);
                }
                catch (err) {
                        winston.info("Invalid OID, err="+err);
                        response.end();
                        return;
                }

// id=SOME_OBJECT_ID --> show the 'x' value for this document
                // Check the cache first
                MemcacheClient.get(query.id, function(err, result) {
                if (!err) {
                        // Key found
                        winston.info("Cache hit,  key="+query.id+", value="+result[query.id]);
                        // Call zeromq service and display
                        callMyFancyService(result[query.id], response);
                }
                else {
                        // Key not found, fetch value from DB
                        winston.info("Cache miss, key "+query.id+". Querying...");
                        collection.findOne({"_id": oid}, function (err, item) {
                                if ((err) || (item == null)) {
                                        winston.error("Could not query id "+query.id);
                                        response.end();
                                        return;
                                }
                                winston.info("Item found: "+JSON.stringify(item));

// Call zeromq service and display
                                callMyFancyService(item.x, response);

// Store value in cache with a 60 second TTL
                                MemcacheClient.set(query.id, item.x, { flags: 0, exptime: 60}, function(err, status) {
                                        if (!err) {
                                                winston.info("Stored key="+query.id+", value="+item.x);
                                        }
                                        else {
                                                winston.error("Couldn't store key="+query.id+", error="+err);
                                        }
                                });
                        });
                }
                });
        }
}

// Program start

// Don't log to console
winston.remove(winston.transports.Console);
// Log to local file
winston.add(winston.transports.File, { filename: 'log.txt' });
// Log to remote syslog
winston.add(winston.transports.Syslog, {"host":"10.34.200.246", "port":"5140", "localhost":"10.37.166.19"});

winston.info("*** NODE APP STARTED ***");

// Connect to the memcached server with its private AWS IP address
MemcacheClient = new mc.Client(["memcache.9vficl.0001.euw1.cache.amazonaws.com", "memcache.9vficl.0002.euw1.cache.amazonaws.com"]);

MemcacheClient.connect(function() {
        winston.info("Connected to memcache");
});

// Connect to the MongoDB server hosted at MongoLab
var collection;
require('mongodb').MongoClient.connect("mongodb://USER:PASSWORD@ds051067.mongolab.com:51067/mongolab-test", function(err, db) {
        if (!err) {
                winston.info("Connected to database");
                collection = db.collection("collection1");
        }
        else {
                winston.error("Dude, where's my DB?");
                process.exit(err);
        }
});

// Connect to our fancy service through zeromq
sock.connect('tcp://10.34.206.103:3000');
winston.info("Connected to zeromq server");

// Start the web server on port 8080 and wait for incoming requests
http.createServer(onRequest).listen(8080);
winston.info("Web server started");

Logs (or it didn't happen, right?): display all entries and get one id twice

local0.info<134>: Aug 16 17:29:52 10.37.166.19 {"message":"*** NODE APP STARTED ***"}
local0.info<134>: Aug 16 17:29:52 10.37.166.19 {"message":"Connected to zeromq server"}
local0.info<134>: Aug 16 17:29:52 10.37.166.19 {"message":"Web server started"}
local0.info<134>: Aug 16 17:29:52 10.37.166.19 {"message":"Connected to memcache"}
local0.info<134>: Aug 16 17:29:52 10.37.166.19 {"message":"Connected to database"}
local0.info<134>: Aug 16 17:30:04 10.37.166.19 {"message":"Request received, id=0"}
local0.info<134>: Aug 16 17:30:04 10.37.166.19 {"message":"Finding all documents"}
local0.info<134>: Aug 16 17:30:12 10.37.166.19 {"message":"Request received, id=51e3ce08915082db3df32c08"}
local0.info<134>: Aug 16 17:30:12 10.37.166.19 {"message":"Cache miss, key 51e3ce08915082db3df32c08. Querying..."}
local0.info<134>: Aug 16 17:30:12 10.37.166.19 {"message":"Item found: {\"_id\":\"51e3ce08915082db3df32c08\",\"x\":25}"}
local0.info<134>: Aug 16 17:30:12 10.37.166.19 {"message":"Zeromq request: 25"}
local0.info<134>: Aug 16 17:30:12 10.37.166.19 {"message":"Zeromq response: 50"}
local0.info<134>: Aug 16 17:30:12 10.37.166.19 {"message":"Stored key=51e3ce08915082db3df32c08, value=25"}
local0.info<134>: Aug 16 17:30:15 10.37.166.19 {"message":"Request received, id=51e3ce08915082db3df32c08"}
local0.info<134>: Aug 16 17:30:15 10.37.166.19 {"message":"Cache hit, key=51e3ce08915082db3df32c08, value=25"}
local0.info<134>: Aug 16 17:30:15 10.37.166.19 {"message":"Zeromq request: 25"}
local0.info<134>: Aug 16 17:30:15 10.37.166.19 {"message":"Zeromq response: 50"}

Let's take a few steps back and look at what we built over the last posts:

a simple Node.js web app...
fetching data from an Elasticache cluster...
fed by a MongoDB server...
logging remotely to a syslog server...
calling a remote server through zeromq...
all of it open-source...
all of it cloud-based...
each part easily scalable, independently from the others.

And above everything, very little code: 159 lines (comments included), with the help of a few well-chosen libraries.

Makes me want to be a developer again. Food for thought ;)

Till next time, my friends.

PS: In case you were wondering "Zero the hero" is a Sabbath song, although one could argue that the Cannibal Corpse cover is actually quite better.

Digital (dis)content

Aug 16, 2013

Node.js, part 5: Zero the hero!

No comments:

Post a Comment