Dec 27, 2012

Viking laws for tech teams, part 6: be direct

This post is part of the "Viking laws for tech teams" series.

5 more days till 2013. Here's my contribution to the holiday cheer :)

Whether you like it or not, my nerd brothers, our job involves more than breaking building cool applications and platforms in solitary confinement. OK, maybe some of you ARE confined (and rightfully so), but the rest of us have to interact with people. Here are a few examples : team members (ok to speak to, hopefully), technical colleagues (mostly ok), non-technical colleagues (possibly nice but clueless), managers (political bastards), top managers (now it gets really bad), customers (why me? I'm just an engineer), suppliers (clueless AND political), etc.

Endless combination of personalities, skills, experience, goals, benevolence, etc. The richness of human interaction.... yeah, right (snarls). I stopped long ago trying to adjust: too complicated, too frustrating, too slow.  Overall, a very inefficient way of solving problems and getting anything done.

I'm not saying that diplomacy, patience and kindness are not desirable attributes. They certainly are, if they're genuine and not an excuse for cowardliness and indecision.

So, when in doubt, let's just be direct, dammit! Say what you think, tell it like it is, spit it all out. Have the guts to stand for what you believe in. Try to stick to facts. Respect the right of others to speak up as well (yes, it's very unpleasant at times). And should there be some yellin' in the process, well... so be it. High stakes and high performers will generate sparks every now and then. Go work for a wishy-washy loser company if you can't take it.

Face it, people almost always know what the real problem is. Let them speak openly. Let them bring bad news to the table. Let them disagree with each other and with you. Create an environment where it's ok for everyone to do all of this and walk away, not only safely but with confidence that issues ARE understood and tackled. 

I don't know of any better way to define, crystallize and solve problems. The good news is that it doesn't take brains to do this in your team: just a little bit of strength, courage and honesty. It's within reach, try it.

And if all else fails, do it one last time on your way out: "During times of universal deceit, telling the truth becomes a revolutionary act" - Orwell

Sep 5, 2012

Viking laws for tech teams, part 5: agree on important points

This post is part of the "Viking laws for tech teams" series.


"Agree on important points": this sounds like a no-brainer, right? Obviously, a team won't get much done if members keep arguing and disagreeing over the same issues over and over again. Nothing to write about, then.

Really? Quick exercise: ask each team member to describe the team's operation in 5 bullet points. How many different items do you get? 5 total or 5 times the number of team members? Do the items look like the Agile Manifesto or like a checklist for code reviews? By the way, when was the last time the team actually discussed what these items should be? 

So item #0 should always be: "Talk to each other, goddammit!". No, the code is not the documentation. No, it's not enough to send e-mails (it's actually the last thing you should do). No, it's not enough to assign tickets. You guys need to talk face to face (that's IRL to you Gen Z). And debate. And argue, if needed. Whatever it takes to come to a clear and shared agreement on WTF the subject was.

Short, frequent discussions are the way to go. You don't HAVE to do a formal Scrum daily meeting, but starting the day with a quick exchange of information goes a long way. If nothing else, it creates an culture of sharing ideas, raising issues, asking for help, etc. If your only opportunity to do any of this is during that dreadful 3-hour long "Weekly Project Review" (you know what I'm talking about), then doom on you.

Okay, now that we've established that you need to talk to each other, what should you agree on? Well, that's probably the first thing you guys should talk about :) Here are a few pointers:
  1. what is the role of the team in the company?
  2. who are its customers?
  3. what do they expect from the team? Ask them!
  4. external input required by the team: what? how? how often? from whom?
  5. team deliverables: what? how? how often? to whom?
'lo and behold! 5 items :)

That IS enough. If you know who you are and what's expected from you, all other decisions will follow. Tools, methodology, resources, roles, etc.

"We're the lead drakkar of our clan. Our mission is to raid the Eastern coast of Scotland for food, livestock and slaves. Loot quotas are set every semester by the King himself and he expects no less than one delivery per month. We're free to choose our own targets: a list is agreed on by the clan before setting off to sea, the actual target is picked by the crew at the last minute depending of weather and tide conditions".

Sure beats your crappy corporate mission statement, huh?

Please share your comments or anecdotes. See you next time.

Sep 3, 2012

Viking laws for tech teams, part 4: keep your weapons in good condition

This post is part of the "Viking laws for tech teams" series.


As much as I regret it, software engineering doesn't actually involve two-handed swords and battle axes... although managing teams sometimes does :) Still, every trade had its tools and no matter if you're chopping heads off or squashing bugs, proper tools are a key element in achieving high productivity.

First things first: the workstation. Yes, you need to give your team nice machines to work on. It's quite likely that your guys have fancy hardware at home and they won't settle for a 5-year old setup in the office. If nothing else, see this as a (de)motivation factor. Truth be told, modern software development environments *are* resource hogs and a slow workstation is a silent productivity killer (build time, freezes, inability to debug properly, etc).  Dual screens are super useful too. Throw in a fancy keyboard or mouse, too. Whatever keeps your guys working on their code is a good investment. And no, it's not going to cost a lot of money: slow teams & delayed projects do.

What about team tools? Version control system, continuous integration, unit test framework, ticketing system, wiki, etc: absolutely mandatory from day 1. Yes, even if you're a one-man team. If hosted solutions are your thing, then Sourceforge or Github are just a few clicks away. One way or the other, there's just no excuse for not setting things up properly. 'nuff said.

Next, get the best debugging / profiling tools possible and spend some time learning them (yes, you need to go beyond debugging HelloWorld). Make sure that everybody in the team is proficient BEFORE a real emergency arises. The last thing you want is having to read tutorials and installation documentation while your production systems are on fire, with management asking for progress reports every 30 minutes... You should also consider 3rd-party tools, whether Open Source or commercial: your IDE debugger will only take you so far, especially if you work on complex, dynamic systems (web servers, databases, etc).

Remember that the strongest castle will easily fall if no sentries man its perimeter. Real-time monitoring is your first line of defense, so you'd better set up the right probes in your code and make sure alerts are actually sent when something goes wrong. If you think error logs or e-mails are enough, you'll sure get plenty of opportunity to read them when the SHTF: whether you'll find the actual problem or not is a different story. You should invest a lot of time watching your systems and understanding their dynamics. It's particularly important to study how they fail (silently vs loudly, progressively vs abruptly, early symptoms, etc). You should definitely consider breaking them in a controlled way to confirm your observations and then write short documentation on how to detect and repair the most common issues.

A couple more things:
  1. tools are not enough, you need the right organization and the right tactics. This will be covered in a further post :)
  2. I hear some of you saying: "it's all good and true, but in my company, workstations are renewed every 5 years at best, we have 14-inch CRT screens, it's company policy to use CVS and management won't ever spend a dime on 3rd-party tools". My answer is: "these guys have zero respect for your work. Get the fuck out & find new battle comrades". Yes, it's THAT simple. Do it or stop whining.
Bottom line: one day or the other, barbarians WILL show up at the gates, preferably when you least expect it. If you have neither beefed up the armory nor trained hard enough, you will be overrun and Odin's crows will feast over your carcass. If you've done both, THEIR heads will decorate your walls and warn the next assaillants that your team is not to be fucked with.

Your choice.

Please share your comments or anecdotes. See you next time.

Aug 31, 2012

Viking laws for tech teams, part 3: find good battle comrades

This post is part of the "Viking laws for tech teams" series.

It boils down to this: you need to join the right company.

Many of you will disagree, I'm sure: "the daily job is more important", "there are good teams in bad companies", "yes, it's a shit company, but I'm still learning a lot of stuff", etc. All very valid points, but sooner or later, reality will catch up with you. And most of all, why settle for less? Maybe there are good teams in bad companies, but there's bound to be better teams in good companies :)

What is the "right" company anyway? Everyone will have their own, personal criteria. Discussing mine in detail would be of no interest. However, having worked for (and in some cases, survived) a number of very different companies, I'm convinced that there are some mandatory questions that need to be answered before making career changes. And yes, I failed to do it on a couple of occasions, sometimes stupidly bravely ignoring some obvious red lights. Bad idea, lesson learned :)

Researching a company that you'll join for many years sounds like a necessary precaution. However, the interview process is not the place to ask tricky (and possibly nasty) questions: if you think it will make you look clever,  you're mistaken, it's actually quite likely to ruin your chances. You must do your own homework.  Most people don't: fresh meat for the grinder.

I realize that it may be difficult to know what rug to look under, especially if you have little work experience. Hopefully, my battle-tested checklist will help you out.
  • Company
    • How old is the company?
    • Is the company the market leader? a challenger? a startup?
    • Who is the competition? Smaller or bigger is not the point, maybe THEY are the right company you should join (true story).
    • Does the company have any revenue (don't laugh)? If not, how long before cash runs out?
    • Is it company profitable? What's the trend?
    • If the company is public, read the latest annual report. You will learn A LOT, believe me. If you don't understand financials at all, find a friend who does and ask him/her to run through the numbers. If you do understand financials, would you buy this company's stock? No? Then why the hell would you join them? You are your most precious asset!
    • Where are the headquarters and will you be working there? In other words, will you be close to the decision center or not?
    • last but not least, is technology really the core business of the company? Or is it just a tool? The old IT vs R&D debate...
  • Founders (unless you're interviewing with Boeing or General Electric...)
    • Who are they?
    • What's their background (technical or business)? 
    • What's their track record prior to starting the company?
    • What's their reputation? Ask around, the world is VERY small and bad news spread faster than good ones.
    • Are still involved in the company?
    • Are they still driving the company (which is a different thing)?
    • Did you meet them during the interview process? 
  • Governance
    • Who are the shareholders: individuals? Family? VC? another company?
    • CEO/CMO/CFO/Managing Directors: background, track record, reputation, etc. You might not interact with them on a daily basis, but their strategy will either drive the company through the roof or into the ground. If the company is assigning weak profiles to critical positions, what does this tell you?
    • Top technical job (CTO or equivalent): dig even deeper :) This guy would be your boss. Does he have street cred or is he just a MBA graduate with fancy Powerpoint skills?
    • Hiring policy: how strict is it? Does anybody get in? Hopefully not (as discussed in a previous post).
I could go on, but this should get you started. Gathering all this information is actually pretty easy: corporate website, LinkedIn, financial & industry websites, blogs and of course your own personal contacts. There's just no excuse for not doing it, except if you're not really motivated (which kind of solves the problem, doesn't it?).

Once that you have all information, assign your own weight to each item and take a long, hard look. How does this company compare to the other ones you've looked at? What are its strengths and weaknesses? How will they impact your job? If there's any red light, what's the compelling reason to ignore it?

Most important of all, trust your instinct and don't try to rationalize: if it stinks, then it's probably shit. Pass and keep looking.

Still, there is no silver bullet. Maybe you'll miss something, maybe things will change dramatically in a few months, but running your own due diligence will help you decide on facts, not on hear-say or hype. And you might avoid an awful work experience.

It's your life. Spend it well.

Please share your comments or anecdotes. See you next time.

Aug 30, 2012

Viking laws for tech teams, part 2: keep yourself in shape

This post is part of the "Viking laws for tech teams" series.

Skills, attitude, stamina: the iron triangle of survival in challenging environments.

No skills? You're dead weight, no one will ever want you on their team. Rusty or outdated skills? If you're lucky, you may get a job maintaining legacy systems… until they're shut down or outsourced. Bottom line: if you think you were done learning when you left school, think again. You should never stop learning: blogs, books, personal projects, conferences, anything goes. Don't let your edge grow dull, keep it razor sharp.

Example: 40-year old embedded software engineer who decides in 2001 that he doesn't like object-oriented languages and that he will keep writing assembly and C language for the rest of his life. Fast forward 5 years: ALL projects now include embedded Java applications, which junior engineers can write and debug with their eyes closed (for half the salary). Game over. The guy barely saves his ass by accepting Quality & Documentation tasks: not what he had in mind, but still useful and way better than being out of a job at 45 with two kids.


Poor attitude? You might fool the team for a while, but you'll end up being thrown overboard. You need to truly believe in what you're doing and give it your best shot. It has to be more than "just a job". No, this isn't about lapping up corporate bullshit ("work hard, have fun", you know what I mean). It's about standing your ground, soldiering on, delivering the goods, being someone that your buddies can count on in good and bad times alike. Ask yourself: when the time comes to leave, will I be able to look proudly at what I've helped build over the years? Two years later, will I be remembered as the tough-as-nails engineer who got shit done, no matter what? Will my code still run in production? Yes? Job done then, with plenty of good memories and war stories to share. Nothing else matters.

Example: that last guy who joined, left after 6 months and that no one remotely remembers. Which is the way it should be.


No stamina? You might be the sharpest, nicest guy around but you won't last. It's not about long hours, nights or weekends (yes, they're sometimes needed). It's about enduring the daily insanity of fast-paced projects: 180-degree turns, showstopping bugs 2 days before the release, supplier issues, customer whims, management stupidity… The list is endless. If you blow a fuse at every roadblock, you're bound to waste energy, grow tired and lose motivation, so whatever you need to do to vent off, do it regularly. Also, never ever lose your sense of humor: unless your code runs in nuclear plants, airplanes or hospital life-support systems,  it can't be THAT bad, so stop being a miserable bastard and just try to laugh it off (even if it's damn hard sometimes). Remember: it's not a sprint, it's a marathon, a 40-year long marathon. You won't make it without nerves of steel and a healthy dose of self-derision.

Example: the burnt-out, negative, cynical guy in your team who used to be a super achiever and a great guy. Do him a favor and let him go.


Bottom line: it all starts or ends with you. Take care of yourself, but keep pushing your own envelope. Don't pretend, do it for real: the more you sweat in training, the less you bleed in combat.

Please share your comments or anecdotes. See you next time.

Aug 28, 2012

Viking laws for tech teams

Viking laws... An oxymoron to most people, I'm sure. That's not surprising given that the Men from the North have long been depicted as drunken primitive barbarians, only interested in pillaging and looting (what's wrong with that anyway?). After all, History is written by the victors, in this case Christian victors who probably had no fascination for paganism and polytheism. Most of what we really know about the Vikings comes from the famous Sagas, which were only written at the very end of their civilization (12th-14th centuries).

According to these Sagas, Vikings *were* civilized and organized. Actually, the oldest parliament in the world is the Icelandic parliament founded in 930AD in Þingvellir: a stunning location with cliffs and waterfalls, straight out of Wagner's Ring (do not miss it if you ever visit Iceland). Free men met there regularly to solve disputes and take decisions according to law, which was memorized and recited by the "law speaker". And for the next three hundred years, "one man, one vote" was the rule.

Some of these laws, or at least their underlying principles, have been documented around 1250 in the "King's Mirror", an ancient Norwegian text meant for the education of the King's son on commerce, military tactics, etc. A modern, business-oriented summary of these principles appeared some years ago in a book called "The Viking's Guide to Good Business" (ISBN-10: 997985622X). It's a nice, short read and you should be able to grab a copy for $0.99 :)

Another set of Viking laws is floating around, although I can't seem to pinpoint their origin:

BE BRAVE AND AGGRESSIVE 
BE PREPARED 
BE A GOOD MERCHANT 
  • Find out what the market needs 
  • Do not make promises you cannot keep 
  • Do not demand over-payment 
  • Arrange things so that you can return 
KEEP THE CAMP IN ORDER 
  • Keep things tidy and organised 
  • Arrange enjoyable activities which strengthen the group 
  • Make sure everybody does useful work 
  • Consult all members of the group for advice

And so, in my next posts, I will indeed go medieval on you and discuss how this wisdom from the past may apply to today's agile, high-performing tech warriors. I might even use some real-life examples (PH34R!), although names will be changed to protect the stupid and incompetent.

Oh, I'm afraid none it of will be remotely PMP, Prince or ITIL compliant. It will be fun, then :)

Jul 27, 2012

Under the hood of Real-Time Bidding

Originally written for boom, the latest in performance display advertising.

Within the display advertising environment, Real-Time Bidding has evolved from curiosity to mainstay in the last 18 months.  Though real-time bidding can appear to be a pretty simple value proposition – advertisers bid, win and display on an impression-by-impression basis –don’t make the mistake of thinking it’s unsophisticated.   The technology behind RTB is extraordinary.   At Criteo, we’ve built out our real-time capabilities from scratch in the last two years, to handle 7 billion daily RTB requests on 16 exchanges covering more than 30 countries.

Big challenge, big solution.  Let’s take a look under the hood, shall we?  
In order to participate on a RTB exchange, media buyers first need to implement proper bidding protocols. Apart from the nascent OpenRTB initiative, there is not yet an industry-wide standard, so development work is required in each case.  Getting everything set is usually quite straightforward, especially with the right software architecture (and the right team … another story in itself).  However, since the ecosystem is made up of many companies figuring this out on the fly – no names – things can occasionally get tricky and surprises can happen.
Now, what about the bidding process?  In a trivial case, where an advertiser might simply enter a bid ten cents across the board, the process is not a big deal.  But in the non-trivial case, a bid should be the end result of the system applying sophisticated algorithms to a combination of data sent in the request as well as the media buyer’s own data.  This data can include things like:
  • User properties: user id (cookie matching required), user data stored by your platform
  • Zone properties: size, positioning e.g. above or below the fold
  • Website properties: domain, URL, site content.
  • Business rules: block lists, ad verification for brand safety.

For the winning bidder, it’s then time to deliver a banner, which could again range from a simple, static banner to a unique, fully personalized ad incorporating multiple products, offers, layouts, etc. 

Up to now, all of this is actually the easy part!  And here’s the real catch: once a request has been sent by the exchange, the bid must be received within a very short amount of time - generally under 100 milliseconds. Since typical one-way coast-to-coast latency in the US being 75 milliseconds, infrastructure becomes the alpha and omega of RTB. Get it wrong and your most clever algorithm won’t even get a chance to bid! 
The next challenge is then to introduce as much algorithmic intelligence as possible in whatever precious time is left. This leads to another problem: namely, storing and retrieving data quickly enough within the bidding process.  Then there are the issues of logging and crunching huge amounts of data, and detecting and fixing technical issues.   Quickly.  Not to mention the unannounced specification changes breaking the protocol, but I said “no names.”   Oh, and did I mention 24/7/365 availability?  The list goes on and on. 
RTB is far from mundane, and it represents a tremendous technical challenge.  But it’s also the future of media buying, and so we keep working at it.

Jul 18, 2012

Learning about data centers

A lot of my time was spent visiting data centers lately, with more visits on the horizon. Supersonic growth, you see :)

Not that I'm complaining, mind you. I find data centers totally fascinating. Aiming for perfect, 100% availability. Designed to survive pretty much any natural or man-made disaster.  Plain and simple at first sight, yet so complex and intricate when you scratch the surface. Secret, cold and lonely places, but throbbing with invisible energy. Are server racks a modern-day Stonehenge, erected to worship faceless gods? Mystical considerations aside, data centers are the alpha and the omega of modern civilization: our everyday life - and sometimes, the difference between life and death - depends on their availability. Face it: in the 21st century, Norns are weaving the web of fate with fiber optic cables.

For any self-respecting engineer, the hunger to understand how data centers work should be irresistible. There's so much to learn: construction, power, cooling, cabling, network, servers, etc. Where do you start? Actually, the question really is: where CAN you start? The data center industry is quite opaque. The veil is rarely lifted and some questions are hardly ever answered.

Of course, you should visit as many data centers as you can. I never refuse any opportunity, even when I'm not actively looking for hosting space. The more you see, the more you learn... and the more you realize how much there is still to learn. Keep your eyes open, use your common sense (water pipes above the racks? Hmm), ask a million questions. Try to get in touch with the guys actually running the site. They may not look like much, but with a bit of social engineering, they will give you the real lowdown on the site. Chances are you'll meet VERY colorful characters too, so don't miss out.

In addition to visits, there are a few, precious engineering resources that you may find useful.

Books: most of them are a complete waste of time and money (especially if they say "Green IT" in the title). However, I can vouch for the quality of these two:
  • "Administering Data Centers: Servers, Storage, And Voice over Ip" by K. Jayaswal, (John Wiley & Sons, 2005, ISBN-13: 978-0471771838): a very good place to start. All introductory concepts are present and newcomers will learn a lot on data center infrastructure and on IT inftastructure in general (servers, network, redundancy, etc). 
  •  "Maintaining Mission Critical Systems in a 24/7 Environment", by. P. Curtis (Wiley-Blackwell, 2nd edition, 2011, ISBN-13: 978-0470650424): fantastic book, but definitely not recommended as an introduction. This book focuses on daily maintenance of data center infrastructure (not IT infrastructure) and it goes into A LOT of detail on how things work and what to do to keep them going. Quick example: 9 pages on how to check the quality of fuel deliveries for your generators... you get the idea :) A mine of technical information, definitely the bedtime book for data center staff.
Online resources: sadly, most of the material out there is either watered-down to braindead level, or just infected with marketing/PR/sales bullshit (maybe that's the same thing, hey?). If you want technical information, you'll find it here. I'm not affiliated with any vendor, in case you were wondering :)
  • Energy University @ Schneider Electric: lots of free courses on data center technology (power, cooling, etc). Tested and approved.
  • Online courses @ Siemens: power only (breakers, transformers, etc). Very good stuff too.
So there you go. Learn, learn, learn. Till next time, get rackin', boy!

Jun 18, 2012

New Criteo office in Paris

Only one word: AWESOME.

Publisher / Scalability / Production teams settling down
Our own private 5th-floor garden
One of our meeting rooms... the name says it's MINE ;)
The garden again, Eiffel tower in the background
The mother of all lightwells
Lunch area

May 20, 2012

Audio mass encode on Synology NAS (FLAC to MP3)

Quick Sunday hack while waiting for the damn rain to stop :-(

 I was looking for an easy solution to mass-convert my music library from FLAC to MP3 (because iP*ds don't play FLAC...). Don't worry, I'm still keeping the FLACs for home playing :)

My library is stored on a Synology NAS and the prospect of transferring all of it back and forth through my LAN didn't sound really attractive, so I figured maybe I could find a way to reencode everything on the NAS itself, using whatever tools are available on a vanilla box (i.e. no additional package).

And voila!

This script is pretty self-explanatory: just ssh to your box and launch the script at the root of the music library. It will go sequentially through every subdirectory and each FLAC file will be encoded to a 320kbit MP3 file, saved in the same directory under $newbasedir. Existing ID3 tags are preserved by ffmpeg.

Of course, this will not only run on Synology, but also on any Unix box with ffmpeg... although in this case, I'm sure there are much better solutions :)

Good to see that I can still write a few lines of script. Hope you like it!

 

Apr 6, 2012

Life 24/7/365.


Cons:
  • No life
  • No friends
  • No sleep
  • No (good) food
Pros:
  • Leading the best of the bunch
  • Pushing the envelope daily (mine too)
  • Getting to kick everyone's ass (ours too)
  • Code. Deploy. Fix. Scale. Watch traffic grow like crazy. Repeat.

Verdict? It's all worthwhile. Every fucking minute of it.

BRING. IT. ON.

Apr 2, 2012

Hiring in the trenches

I've hired a lot of people in the last 5 years : software engineers, sysadmins, DBAs, support techs, project managers... I'd say one hundred of them, maybe more. Given that my rejection rate averages eighty percent (more on really bad weeks), I guess I met at least 500 candidates, then. Yeah, two per week for as long as I can remember sounds about right.

In the process, I've written countless job postings, job descriptions, mission statements, blah blah blah. A necessary evil, I suppose, but none of it actually helped me understand what I was looking for.

Technical skills, communication skills, team spirit, ... Sure. But that's not enough. I've hired people who ticked all these boxes and was eventually disappointed. Why? What did I miss? In one word? Couldn't say until now.

When I woke up this morning, I knew. No idea why. Funny how the brain works (mine, especially). Once again, I realize that the best way to find answers is not to ask any question.

And so the answer is : INTENSITY.

The fire that burns and never goes out.
The inability to settle for "good enough".
The restlessness when something's not right.
The will to overcome, to rise above any difficulty.
The willingness to take pain, even self-destruct, rather than yield.
The rage to win, against all odds.

One thing I learned over the years is that competitive business IS similar to warfare. That's a scary thought, but you need to face it. Don't be fooled into the politically-correct, compliance-inducing sheep mentality.

And so, given a choice, I'm definitely teaming up with Special Forces, not with regular infantry. Because when the SHTF big time, I know that the guy next to me will return fire and try the impossible to save the day. I know we all will.

If you have basic skills and INTENSITY, I don't care much about the rest. Just don't play it safe. Don't waste your time telling me how good you are. SHOW ME how good you are.

If you don't have this level of INTENSITY, that's OK, we're not all the same... but please, don't show up. There are plenty of normal places for normal people, so join them instead. Don't waste my time and yours.

Sounds "crazy" ? "extreme"? That's what they say. They must be right. If only I actually gave a shit...

Somebody said: "one must have chaos in oneself in order to give birth to a dancing star". He too was called a lunatic... so let's dance!