Upcoming meetup: Lyon Data Science, January 7

Busy day for meetups :) The kind folks at Lyon Data Science just announced their next meetup on January 7. It will be my pleasure to speak about Amazon Redshift and Amazon Machine Learning. See you there!

Upcoming meetup: Docker Marseille, December 16

I will speak about our managed Docker service, aka AWS EC2 Container Service at the next Docker meetup in Marseille on December 16. See you there!

Filtering the AWS CLI with jq

The JSON-formatted output of the AWS CLI may sometimes feel daunting, especially with a large number of objects. Running 'aws ec2 describe-instances' with more than 10 instances? Bad idea :)

 Fortunately, jq - the self-proclaimed 'sed for JSON data' - is a pretty simple way to filter this output. This will especially come in handy when writing automation scripts :)

Installing jq is one command away:
$ sudo apt-get install jq  (Debian / Ubuntu)
$ yum install jq  (CentOS)
$ brew install jq  (MacOS)

Let's fire up a few EC2 instances and try the following examples.

Show only one field, OwnerId:
$ aws ec2 describe-instances | jq '.Reservations[].OwnerId'

Show all information on the first instance: 
$ aws ec2 describe-instances | jq '.Reservations[].Instance[0]'

Show the instance id and the IP address of the first instance:

$ aws ec2 describe-instances | jq '.Reservations[].Instances[0] | {InstanceId, PublicIpAddress}'

{
  "InstanceId": "i-2181b498",
  "PublicIpAddress": "52.31.231.110"
}

Show the instance id and public IP address of all instances:

$ aws ec2 describe-instances | jq '.Reservations[].Instances[] | {InstanceId, PublicIpAddress}'

{
  "InstanceId": "i-2181b498",
  "PublicIpAddress": "52.31.231.110"
}
{
  "InstanceId": "i-2081b499",
  "PublicIpAddress": "52.31.208.25"
}
{
  "InstanceId": "i-2581b49c",
  "PublicIpAddress": "52.31.207.29"
}
{
  "InstanceId": "i-2781b49e",
  "PublicIpAddress": "52.31.228.234"
}
{
  "InstanceId": "i-2681b49f",
  "PublicIpAddress": "52.31.230.63"
}

 

Fields can also be concatenated, like so:
$ aws ec2 describe-instances | jq '.Reservations[].Instances[] | .InstanceId + " " + .PublicIpAddress'
"i-2181b498 52.31.231.110"
"i-2081b499 52.31.208.25"
"i-2581b49c 52.31.207.29"
"i-2781b49e 52.31.228.234"
"i-2681b49f 52.31.230.63"

Show the instance id and launch time of all instances, sorted by IP address:
$ aws ec2 describe-instances | jq '.Reservations[].Instances[] | .PublicIpAddress + " " + .InstanceId + " " +.LaunchTime' | sort
"52.31.207.29 i-2581b49c 2015-11-16T19:14:57.000Z"
"52.31.208.25 i-2081b499 2015-11-16T19:14:57.000Z"
"52.31.228.234 i-2781b49e 2015-11-16T19:14:57.000Z"
"52.31.230.63 i-2681b49f 2015-11-16T19:14:57.000Z"
"52.31.231.110 i-2181b498 2015-11-16T19:14:57.000Z"

All, let's add an 'environment' tag on all instances, set to either 'dev' or 'prod'.

Now, how about the instance id and IP address of all instances with a 'prod' tag:
$ aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(.Tags[].Value=="prod") | .InstanceId + " " + .PublicIpAddress'
"i-2581b49c 52.31.207.29"
"i-2781b49e 52.31.228.234"
"i-2681b49f 52.31.230.63"

We barely scratched the surface, but I'm sure you get the idea. Make sure you add this nifty tool to your collection!

Set up the AWS CLI on MacOS in 60 seconds

I had to do this AGAIN on a new Mac, so here goes:

1) Install Homebrew

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 

2) Install python and pip

$ brew install python 
$ curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" 
$ sudo python get-pip.py 

3) Install Iterm2 (c'mon, how can you live without it?)

4) Install zsh and oh-my-zsh

$ sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

5) Install and configure AWS CLI

$ sudo pip install awscli
$ sudo pip install awsebcli (if you work with Elastic Beanstalk)
$ aws configure 

6) Enable auto-completion for the AWS CLI in zsh

Open ~/.zshrc  and add aws to the list of plugins, e.g.: plugins=(git aws)

$ source ~/.zshrc

7) Test :)

$ aws s3 ls (or any other AWS command that you like!)
 

Talk @ Velocity Conf 2015 (Amsterdam)

Hi, my colleague Antoine and myself recently gave a talk at the Velocity conference in Amsterdam.

The topic was our all-in move to AWS. The session was great fun, with plenty of questions from the audience. Thanks to everyone who attended.

Here are the slides.

A real-life account of moving 100% to a public cloud from Julien SIMON