Digital (dis)content: July 2013

Jul 29, 2013

The billion dollar app (Node.js! MongoDB! Clouds!)

Today, I'd like to show you how to build a basic web app running on EC2 and connecting to a MongoDB instance running on MongoLab. We used Python so far, so why not use Node.js this time around: Javascript and JSON should work with MongoDB, huh?

We'll start from the same Ubuntu VM running in EC2. Only a little bit of extra setup is required.

First, make sure you open a port for your web app (say, 8080): in the AWS console, go to the "Security group" section and add a custom rule opening port 8080/TCP. Don't forget to click on "Apply rules changes" ;) No need to restart your VM, the change is effective immediately.

Then, 'ssh' into to your VM and install the packages required by Node.js: the environment itself plus its package manager:

ubuntu@ip-10-48-45-182:~$ sudo apt-get install nodejs npm

Now, let's install the native MongoDB driver used by Node.js:

ubuntu@ip-10-48-45-182:~$ npm install mongodb

This generated errors on Ubuntu 12.04, but the package still got installed. YMMV.

All right, now to the good stuff. Here's our web server in Node.js:

connect to our MongoDB instance (once again, make sure you use your own URI, username and password). Then, select 'collection1', containing 25 documents ({"_id", "x"})
create a web server running on port 8080 and wait for incoming requests: they will be served by the onRequest function, which issues a global find() on the collection and prints out all documents. JSON.stringify() is a life-saving browser function, it should hopefully work everywhere. If not, use Chrome ;)

Everything will run asynchronously, which is pretty much the point of Node.js, but this is out of scope for today. I strongly recommend the Node Beginner website if you'd like to know more.

Pretty cool, yeah? Let's start this program on our VM:

ubuntu@ip-10-48-45-182:~$ node www.js
Web server started
Connected to database

Our web server is now running. Yes, really. Now, let's connect to our instance from another machine. I'll use 'curl' but your web browser will work too (feel free to actually try, my VM may be running):

$ curl http://ec2-54-216-192-183.eu-west-1.compute.amazonaws.com:8080/

And this, ladies and gentlemen, is a super scalable, state of the hype, cloud-based web application. Anybody wants to buy it for a billion dollars? :D

Until next time, read, learn and write some code!

Jul 27, 2013

Mass rename on Synology

Sorting through my digital "archives" (*cough*)... A while ago, I showed you how to mass encode audio files from FLAC to MP3 using only the default tools available on a Synology box.

Now, I've got another problem: a large collection of files with weird characters in the filename, showing up as '?' in the Synology GUI... and as plain crap on a MacOS SMB mount (e.g 6KE7HSG.pdf) . Some character encoding issue, I suppose.

So, how do you rename hundreds of files in one ago, filtering crappy and/or unwanted characters in the process? In this case, I'd like to rename [Squadron-Signal] Aircraft in Action n?032 - F-14 Tomcat.pdf (yeah, I'm an aircraft buff, sue me) into n032 - F-14 Tomcat.pdf

On Linux, I'd just use rename(1), but it's not installed on my Synology box, so let's ssh and do it the old way. This should be easy to adapt to your own needs.

sh and sed: what else?

Jul 25, 2013

From one cloud to another (or is it the same one?)

As previously described, MongoLab offers multiple cloud options for your instances: AWS, Azure, etc. What about my now legendary Python program? ;) Shouldn't I also run it in the cloud? Yeah, thought so.

I've been using AWS for a while, mostly to store a ton of personal junk in S3 and Glacier, so the obvious choice is to launch an EC2 instance. I'm quite fond of micro-sized instances based on ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20130411.1 (ami-ce7b6fba): 1 core, 600 MB of RAM, 8 GB of disk... and free :) Perfect for experimentation!

If you need to setup your own account, here's where to start. It's a straightforward process, no need to cover it in detail. Just some quick advice:

Once you've created and downloaded the key pair associated to the instance, don't forget to chmod it to 400 (or 'ssh' will complain that it's not safe, bla bla bla)
DO NOT LOSE THE KEYPAIR FILE, you won't be able to download it again.
When the instance is running, make sure you note its public name (e.g. ec2-54-217-8-54.eu-west-1.compute.amazonaws.com), that's the one you'll use from the outside world.
The 'ssh' port (22) is not open by default. "Security Groups" is where you need to look, just add a rule opening 22/TCP.

Now, let's connect to the instance with 'ssh':
$ ssh -i keypair.pem ubuntu@ec2-54-217-8-54.eu-west-1.compute.amazonaws.com

ubuntu@ip-10-48-247-163:~$

Congratulations, you're on AWS. Now, let's add some more software to this instance:

ubuntu@ip-10-48-247-163:~$ sudo apt-get update
ubuntu@ip-10-48-247-163:~$ sudo apt-get install mongodb-clients

Can't wait to connect to your db? All right, let's give it a try with the 'mongo' client:

ubuntu@ip-10-48-247-163:~$ mongo -u USERNAME -p PASSWORD ds051067.mongolab.com:51067/mongolab-test

MongoDB shell version: 2.0.4

connecting to: ds051067.mongolab.com:51067/mongolab-test

> show collections

collection1

objectlabs-system

objectlabs-system.admin.collections

system.indexes

system.users

This MongoDB instance is hosted by MongoLab in AWS, so we didn't travel too far. A nice mix of PaaS and IaaS, all based on the same infrastructure :)

Now let's add our Python tools:

ubuntu@ip-10-48-247-163:~$ sudo apt-get install build-essential python-dev

ubuntu@ip-10-48-247-163:~$ sudo apt-get install python-pip

ubuntu@ip-10-48-247-163:~$ sudo pip install pymongo

Let's copy our Python program to our EC2 instance:
$ scp -i keypair.pem mongo.py ubuntu@ec2-54-217-8-54.eu-west-1.compute.amazonaws.com:~
mongo.py

And now:
ubuntu@ip-10-48-247-163:~$ python mongo.py
{u'x': 1.0, u'_id': ObjectId('51e3ce08915082db3df32bf0')}
{u'x': 2.0, u'_id': ObjectId('51e3ce08915082db3df32bf1')}
{u'x': 3.0, u'_id': ObjectId('51e3ce08915082db3df32bf2')}
output removed from brevity
{u'x': 25.0, u'_id': ObjectId('51e3ce08915082db3df32c08')}

What's the point of all this? Simply to quickly demonstrate:

the insane amount of software technology available to anyone willing to read a few books or wikis
how easy it is to get started and build fun stuff. You need nothing else on your own machine than 'ssh'!
it's all free (or very very unexpensive)

None of this was possible 10 years ago. Back in the day.... well, who cares. You kids sure have it too easy, so stop wasting your time on Facebook or on reality TV (absolute scum, will it ever stop?), learn some proper engineering skills, get smart and start solving some problems, ok? There ain't no shortage of them!

Old man over and out.

Jul 24, 2013

MongoDB and Python gang thegither!

After a couple of recent posts on MongoLab and Python, I guess it was pretty obvious to mix both :) The only thing you need to add to your system is PyMongo, the Python driver for MongoDB. If you're using Linux or MacOS, it should be as easy as:


$ sudo easy_install pymongo

Searching for pymongo
Best match: pymongo 2.5.2
Processing pymongo-2.5.2-py2.7-macosx-10.8-intel.egg

Adding pymongo 2.5.2 to easy-install.pth file

Using /Library/Python/2.7/site-packages/pymongo-2.5.2-py2.7-macosx-10.8-intel.egg
Processing dependencies for pymongo
Finished processing dependencies for pymongo

Let's try some basic stuff first. You'll just need to use your own username, password, URI, database and collection :)

# Python 2.7
from pymongo import *
connection = MongoClient("mongodb://USERNAME:PASSWORD@ds051067.mongolab.com:51067/mongolab-test", 51067)
db = connection["mongolab-test"]
c = db["collection1"]
posts = c.find()
for post in posts:
print (post)

Output:
{u'x': 1.0, u'_id': ObjectId('51e3ce08915082db3df32bf0')}
{u'x': 2.0, u'_id': ObjectId('51e3ce08915082db3df32bf1')}
{u'x': 3.0, u'_id': ObjectId('51e3ce08915082db3df32bf2')}
output removed from brevity
{u'x': 25.0, u'_id': ObjectId('51e3ce08915082db3df32c08')}

OK, now let's go a little bit crazeee:

# Python 2.7
from pymongo import *
connection = MongoClient("mongodb://USERNAME:PASSWORD@ds051067.mongolab.com:51067/mongolab-test", 51067)
db = connection["mongolab-test"]
c = db["collection1"]
posts = c.find({"x":{"$gte":5,"$lte":10}}).sort([("x",DESCENDING)])
print (c.count())
for post in posts:
print (post)
c.insert({"y":post["x"]})
print (c.count())

Output:
25
{u'x': 10.0, u'_id': ObjectId('51e3ce08915082db3df32bf9')}
{u'x': 9.0, u'_id': ObjectId('51e3ce08915082db3df32bf8')}
{u'x': 8.0, u'_id': ObjectId('51e3ce08915082db3df32bf7')}
{u'x': 7.0, u'_id': ObjectId('51e3ce08915082db3df32bf6')}
{u'x': 6.0, u'_id': ObjectId('51e3ce08915082db3df32bf5')}
{u'x': 5.0, u'_id': ObjectId('51e3ce08915082db3df32bf4')}

See how easy this is (again)? The Python syntax is pretty much identical to the MongoDB syntax and their dictionaries are a perfect match. No conversion needed (DALs and ORMs, rot in Hell!).

Robert Burns, the famous 18th-century Scottish poet, once wrote: "Freedom an' whisky gang thegither!". With all due respect for such a wise man, please let me add: "So do MongoDB and Python, especially with a glass of 18-year old Glenlivet". Open Source (on the) rocks :D

That's it for today. Now write some code, will ya?

Jul 22, 2013

The $1 fix

I've got remote-controlled electric shutters at home. Very nice.

I've got kids too, who love to play with remote controls (read: throw, smash, stomp, lose, etc). Very nice too, in their opinion at least.

And so, here I am with two broken remote controls and thus, two shutters that I can't close. All right, no biggie: let's get in touch with a couple of resellers, ask them for a quote on two new remote controls. Problem solved.

Nope. The answer, after a couple of weeks "because the manufacturer is slow to answer" (yeah, right): "each remote control is unique and it's matched to the receiver inside the shutter. You have to change both the remote control and the receiver. About 200 euros for one shutter (about $260)".

WTF.

Option 1: "thank you for your kind answer, please steal my money, thank you"
Option 2: sell my kids on Ebay for 400 euros (tempting)
Option 3: "GFY, you thief. I'm fixing it myself"

Option 3, then. I've got a degree in Electrical Engineering, how hard can it be? Let's take the remote controls apart and figure it out. Ah ah, same problem on both: one component has broken loose from the PCB. Looks like a large condenser, but maybe not. Unfortunately, it can't be soldered back on, because the pins are super short and have broken at the base of the component. I need to find replacement parts.

Hmm. The only hint is a cryptic "1000J" printed on the component. Googling it reveals nothing. Neither do a number of searches on remote controls for my brand of shutters. Ah, the loneliness of the long distance family man...

Think, think, think. What is this thing you're trying to fix. A... ? A ... remote control. Which means it's sending a radio signal. And so, according to vague memories from the last century, it has to have some kind of resonator / oscillator, which when broken would prevent the remote control from working. And since everything else on the PCB is either a passive component or an IC, this must be it.

Ah, the sweet taste of adrenaline! Quick, let's ask Google a better question, tweak the results a bit and... yes, victory !This nasty little fellow is a CSBF1000J 1MHz resonator, manufactured by Murata. Easily found on Ebay. Received in 72 hours (German sellers: can't beat them). Soldered in 1 minute.

Problem fixed for a mighty 0.70 euro a pop (aka the $1 fix).

The moral of the story is: learn to fix things yourself... or be forever prey to greedy middlemen. Every now and then, you WILL need an expert, but most of the time, you can do it on your own fast, nice and cheap. And you'll have the IMMENSE satisfaction of being a little more self-reliant :)

Jul 20, 2013

Fooling around with Python on a Friday night

Browsing and listening to my LP collection... and updating the corresponding Excel file because, well, I have quite a lot of stuff and I can't remember it all when I'm hitting the record stores :D

And then I asked myself, why not write a short program to parse and search this file? A totally pointless little hack, the way I like them. It's too warm to sleep anyway... and "can you still do it, old man?" says the devil on my shoulder.

CSV file, one line per record: sounds like a Python job, baby.

A little while - and quite a bit of profanity - later...

import codecs

def loadRecords(filename, delimiter):

"Load record collection into a dictionary of dictionaries"

records = {}

# The most painful line in this program. F..k you Excel!

reader = codecs.open(filename, 'r', encoding='ISO-8859-1')

key=0

for line in reader:

row=line.split(delimiter)

# Some dates are missing or are not integers. Oops :)

try:

year = int(row[5])

except:

# 1970 is the year Black Sabbath released their 1st album

# How's that for a default date, huh?

year = 1970

records[key] = {'band':row[0], 'album':row[1], 'year':year}

key=key+1

return records

def findRecords(collection, band, startYear=1900, endYear=2013):

"Find records by band and year range"

for key in collection:

current = collection[key]

if (current['band'] == band) & ((current['year']) in range(startYear,endYear)):

print (band, current['album'], current['year'])

myCollection = loadRecords("records.csv",";")

findRecords(myCollection, "ANATHEMA")

findRecords(myCollection, "AMON AMARTH", 2002)

findRecords(myCollection, "OPETH", 2004, 2008)

You want to see the output, huh? All right:

ANATHEMA The crestfallen E.P. 1992
ANATHEMA They die 1992
ANATHEMA We are the bible 1994
AMON AMARTH Fate of norns 2004
AMON AMARTH Twilight of the thunder god 2008
AMON AMARTH Versus the world 2002
OPETH Ghost reveries 2005
OPETH Orchid 2004

Python is definitely a cool language. I'm sure this code could be perfected and I've also got more feature ideas now... but that's it for today. And yeah, I can still do it ;)

Jul 19, 2013

VirtualBox: ssh to guest VMs in 30 seconds

Playing with VirtualBox again... but how the hell do you ssh into NAT guests? Looks like I forgot how to do it :D So, for all of us memory-challenged people out there, here's how to set it up painlessly, using port forwarding from host to guest.

In this example, I'm using a Linux Ubuntu 12.04 guest running in VirtualBox 4.2.16 on a MacOS X host, but this should be pretty generic.

First of all, make sure your VM is stopped.

In VirtualBox:

Go to "Settings / Network / Advanced / Port Forwarding"
Add a new rule by clicking on the "+" icon

Name: guestssh
Protocol: TCP
Host IP: 127.0.0.1
Host port: 2222
Guest IP : leave blank
Guest port 22: 22

Alternatively, or if you have a large number of VMs to tweak, you can script this:

On the host:

$ /usr/bin/VBoxManage list vms

"Ubuntu 12.04" {e0cdb5d7-8079-452e-9eeb-e68824532173}

$ /usr/bin/VBoxManage modifyvm "Ubuntu 12.04" --natpf1 "guestssh,tcp,127.0.0.1,2222,,22"

That's it for VirtualBox. Now, start your VM. You need to have openssh-server installed on the guest. If it's not there, just type:

$ sudo apt-get install openssh-server

Now, back on the host:

$ ssh -p 2222 julien@127.0.0.1

The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.

RSA key fingerprint is 2d:23:96:0b:c2:56:02:40:0b:f3:fb:17:78:3b:a6:48.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '[127.0.0.1]:2222' (RSA) to the list of known hosts.

julien@127.0.0.1's password:

Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-36-generic x86_64)

* Documentation: https://help.ubuntu.com/

Last login: Fri Jul 19 10:22:55 2013

julien@julien-VirtualBox:~$

And, as we say in France: "voila" :)

Jul 15, 2013

Your MongoDB instance for here or to go?

In this post, I'll show you how to install MongoDB either on your local machine or in the cloud.

Let's look at the local installation first. This is as easy as it gets. Here's the Linux/MacOS version:


$ curl http://downloads.mongodb.org/osx/mongodb-osx-x86_64-2.4.5.tgz > mongodb.tgz

$ tar xvfz mongodb.tgz

$ ln -s mongodb-osx-x86_64-2.4.5 mongodb

$ sudo mkdir -p /data/db

$ sudo chown `id -u` /data/db

Done! Really.

Starting MongoDB requires a single command:

$ ./mongodb/bin/mongod

Now, let's open the MongoDB shell and type some commands from the MongoDB tutorial:

$ ./mongodb/bin/mongo
MongoDB shell version: 2.4.5
connecting to: test
> db
test
> use mydb
switched to db mydb

> j = {name:"name"}
{ "name" : "name" }

> k = {x:3}
{ "x" : 3 }

> db.testData.insert(j)
> db.testData.insert(k)
> show collections
system.indexes
testData
> db.testData.find()
{ "_id" : ObjectId("51e3c14987d8afffb5a6f97c"), "name" : "name" }
{ "_id" : ObjectId("51e3c15487d8afffb5a6f97d"), "x" : 3 }
> exit

Now, let's do the same thing in the cloud with MongoLab. You can get a single instance for free, as long as it doesn't exceed 500 MB of data. Pretty cool. Another option is MongoHQ who offer a similar service. I'm not affiliated with any, so feel free to check out both :)

Sign up at www.mongolab.com
Create a database ('mongolab-test' in my case)
Pick your hosting location: AWS, Google, etc. If you're simply testing, this doesn't really matter, but when you'll be deploying for real, you'll want to make sure your DB is the same cloud as your app, I suppose ;)
Select the 'Sandbox' plan
Create a first user (or you won't be able to connect to your DB, duh)

Done. You should now see something similar to this:

Click on the database name to view the connection information:

Add a collection, 'collection1' in my case. Obviously, it will be empty.

Let's use the MongoDB shell again and add some data:

$ ./mongo ds051067.mongolab.com:51067/mongolab-test -u DBUSER -p DBPASSWORD
MongoDB shell version: 2.4.5
connecting to: ds051067.mongolab.com:51067/mongolab-test
> use mongolab-test
switched to db mongolab-test
> show collections
collection1
system.indexes
system.users
> for (var i = 1; i <= 25; i++) db.collection1.insert( { x : i } )
> db.collection1.find()
{ "_id" : ObjectId("51e3ce08915082db3df32bf0"), "x" : 1 }
{ "_id" : ObjectId("51e3ce08915082db3df32bf1"), "x" : 2 }
{ "_id" : ObjectId("51e3ce08915082db3df32bf2"), "x" : 3 }
[Output removed for brevity]
> db.collection1.find({x:18})
{ "_id" : ObjectId("51e3ce08915082db3df32c01"), "x" : 18 }

All right, we wrote some stuff. Let's create an index:

> db.collection1.ensureIndex({"x":1})

> db.collection1.getIndexes()

[

{

"v" : 1,

"key" : {

"_id" : 1

"ns" : "mongolab-test.collection1",

"name" : "_id_"

{

"v" : 1,

"key" : {

"x" : 1

"ns" : "mongolab-test.collection1",

"name" : "x_1"

}

]

> db.collection1.find({x:18}).explain()

{

"cursor" : "BtreeCursor x_1",

"isMultiKey" : false,

"n" : 1,

"nscannedObjects" : 1,

"nscanned" : 1,

"nscannedObjectsAllPlans" : 1,

"nscannedAllPlans" : 1,

"scanAndOrder" : false,

"indexOnly" : false,

"nYields" : 0,

"nChunkSkips" : 0,

"millis" : 0,

"indexBounds" : {

"x" : [

[

18,

]

"server" : "h000432.mongolab.com:51067"

}

The index works as intended, that's a relief :-P

Going back to the MongoLab console, you should see the right number of documents:

That's it for today. I've just scratched the surface but it should get you started. See how easy this was? Happy learning!

Jul 3, 2013

Bad Company

Hey, it's been too long since I posted some music. Get a beer from the fridge, sit back, relax and enjoy a quick round-trip to the 70s :)