The hiring process
A big part of my previous job was to conduct technical interviews. That process was straightforward:
- An interview conducted by HR determined if the candidate is a serial killer
- An interview conducted by technical experts determined if the technical skills
- An interview conducted by big boss determined how low the candidate was willing to get paid.
I interviewed two types of people: interns and FTEs. Interns only went through #2 while the others went through all three steps. In the span of 2+ years working at that company, I have performed more than 200 technical interviews. It was a learning and enriching process for me. The company was located in France, where it is near-impossible to fire people. Hiring the wrong person means paying a high price and being stuck for years. I have to say that while it was a tedious process, I loved every part of it.
Iteration One: the lottery quiz
I conducted my first tech interview in 2008. The company already had a working process that I had to follow. Interviews were one hour: 30 minutes to answer a 15-questions quiz, then 15 minutes to discuss their answers and the remaining 15 minutes to discuss the job responsibilities. That questionnaire was simply not great. It’s actually quite a challenge to come up with something worse. It contained five easy generic questions and then ten hard items that were specific to the frameworks we were using at the time.
It went from
1 | - What is the difference between a class and an object? |
to
1 | - What is the purpose of the execAndwait interceptor in |
Those were all random and I don’t think anyone in the company actually knew all the answers. A lot of them were specific to old versions of Java frameworks we were using. I definitely couldn’t explain or expand on most of those, and I was praying to not get asked to clarify the questions themselves! We needed something better. I went online and compiled hundreds of proper interview questions. At that time I believed in the quiz format, it just had to contain the right questions in order to reveal how good people were. The right quiz for the right people.
Iteration Two: Go Generic
I came up with the best 50 questions I could find online. I felt they were good questions because they could be answered in any language, and they were provided in a smooth crescendo of difficulty. I scattered the 50 and ended up with 5 sets of 10 questions that I would hand out randomly.
Sample question:
1 | What is a singleton and when would you use it / not use it? |
This was better, or so I thought… I would also usually get decent signal, and this went on for a few weeks, but it didn’t lead to stronger hires. While the questions tested familiarity with programming theories, they left me somewhat clueless with regards to the candidate’s ability to write code. I realized that those questions had two major flaws:
- The questions were too generic. By not going into language specifics, I couldn’t talk about SQL, front-end specifics, etc.
- The quiz was too short, way too short.
I needed a lot more questions, with things that were more specific to the responsibilities at stake…
Iteration Three: Quiz manager 3000
Okay, this is where things got a little out of hand… I went ahead and created an entire quiz application, crafted to embrace the company’s hiring process. I injected hundreds of questions. After the first interview, HR would select 3 topics related to the job description. The tool would then automatically create a multiple choice question quiz with 3 x 20 = 60 random but specific questions with a difficulty matching the person’s years of experience.
Sample question:
1 | (javascript) |
One could see the candidate’s score updated live on the intranet app as the answers were selected. After completion, the tool would generate a report and email it to HR, with elements such as percentile, hiring risk assessment, etc. I was quite proud of myself. I had crafted the ultimate tool. It seemed perfect… until we tested it on our own developers…
Turns out that most of our engineers were getting the same score as people we had refused. I had spent so much time building the tool that I completely overlooked a huge flaw: my desire to fully automate the process had constrained me to only ask multiple-choice questions. As a result, the questions ended up being mostly trick questions… I wasn’t really testing software development skills! I had to swallow my pride, and admitted that the tool was counter-productive. It provided the wrong signals…
Iteration Four: let them code
I started researching how successful US software companies did their screening process. This led me to try out another method: just have them code. No more automation, we were simply going to write code. I would give out three algorithms. Candidates could pick the language of their choice and had access to a machine that was offline throughout the interview. The questions included classic problems found online, such as dealing with string operations (reverse words in a sentence), recurrence (calculate a term in fibonacci’s suite), and collections (order this list).
This was a pretty big step up. It became easy to see who was comfortable writing clean and correct code. It provided good signal on coding proficiency and provided means for good discussions and deep dives. I was initially happy with the results and we kept on going for a couple of months.
However I couldn’t shake off the feeling that I was still missing something. While it had become easier to spot those could solve algorithms, it didn’t always result in hires being great software engineers. Is the quality of an engineer only defined by how well they can solve a math problem, or how fast they can sort a list? There are so many more dimensions to being a great engineer…
Last iteration: a single question
I easily remember when I first started programming. QBasic was shipped with MSDOS 5.0 way before windows 3.1 came out. It contained its own self contained help screen with all of the functions and keywords of the language. To this day, I distinctively remember the feeling that grasped me every time I hit F5 and saw my programs execute before my eyes. A single printed line, a prompt for a name, some colors, a puzzle… I was in heaven. I remember putting line numbers before each command, filling my code with horrible GOTO
s, learning with excitement and fascination something new everyday. I loved programming. I would spend hour after hour creating games, solving problems, showing stuff to my parents and friends. Years went by, I went from QBasic to Turbo Pascal to VisualBasic, wrote custom games for our BBS “Atomic BBS” that we ran from our home phone line through a 2400bps modem. I wasn’t a great programmer, in fact my code was pretty terrible… But oh man did I love it!! I just couldn’t let it go…
I’m assuming that some people feel that type of adrenaline rush the first time they sail a boat, go bungee-jumping or fly a plane… For me it was programming, compiling, executing. I gained that feeling 25 years ago, and it has never left me since. I was born for this. I’ve always been a programmer. I’ve always been convinced that those who are passionate about code can overcome any hurdle in software engineering. And so it was, that after a full year of trial and error, I completely stopped handing out technical tests. I simply sat down with the candidate and asked them:
“Will you please tell me about the best project that you’ve ever created?”
That simple, unique and non-judmental question was really powerful. While some provided vague answers to some obscure projects they had a hand in, others became suddenly alive and excited. They would talk passionately about the game they were creating, the website they had made, the open source projects they had contributed to, the utilities they made after being stuck in the middle of nowhere without any internet access. They were always proud to show me their work. I would ask them for technical details, complex technical challenges they had overcome, placing myself in their world rather than the opposite. As they talked at length, it was impossible to miss: I could see that light in their eyes, the same excitement that a child gets from compiling and running their first hello world
. I would know right then that we had something in common. They were programmers too. Most of them didn’t know details about Struts, Apache Richfaces or other specific frameworks we were using. Yet, once they got the job, they always ended up being fantastic developpers. They learned faster, they produced better code, they inspired others with their creativity and positivism. They were coders at heart. In the end, that’s all that really matters.
Brian says (July 26, 2014 at 7:48 pm):
Phasma Felis says (July 26, 2014 at 8:40 pm):
Christian Walde says (July 26, 2014 at 9:18 pm):
Jose says (July 26, 2014 at 9:28 pm):
rawat says (July 26, 2014 at 10:16 pm):
Pierre Clouthier says (July 26, 2014 at 11:27 pm):
Kevin says (July 26, 2014 at 11:39 pm):
Chris Imhoff says (July 26, 2014 at 11:57 pm):
Bill Torpey says (July 27, 2014 at 2:07 am):
Pichler says (July 27, 2014 at 3:16 am):
peterquest says (July 27, 2014 at 3:17 am):
Philip says (July 27, 2014 at 3:22 am):
Scott Smith says (July 27, 2014 at 4:16 am):
auraham says (July 27, 2014 at 5:36 am):
Nicolas Bize says (July 27, 2014 at 5:43 am):
Nicolas Bize says (July 27, 2014 at 5:49 am):
Calladus says (July 27, 2014 at 6:01 am):
Paddy says (July 27, 2014 at 6:29 am):
Fredrik Haikarainen says (July 27, 2014 at 10:29 am):
Lobo says (July 27, 2014 at 10:47 am):
Aaron Junod says (July 27, 2014 at 2:23 pm):
Inderjeet says (July 27, 2014 at 3:29 pm):
Elan Engel says (July 27, 2014 at 3:53 pm):
Joe Bro says (July 27, 2014 at 4:13 pm):
Levon Shabagyan says (July 27, 2014 at 4:31 pm):
Uri Schonfeld says (July 27, 2014 at 7:09 pm):
Redge says (July 28, 2014 at 12:51 am):
Javin says (July 28, 2014 at 6:32 am):
Chris says (July 28, 2014 at 12:39 pm):
Ken Bolton says (July 30, 2014 at 1:36 am):
Steve says (August 3, 2014 at 7:12 pm):
scott says (August 26, 2014 at 5:47 pm):
Nicholas Barns says (December 1, 2014 at 2:54 pm):
Ed Baker says (December 1, 2014 at 4:50 pm):
Steve G says (December 1, 2014 at 6:40 pm):
RandyM says (December 1, 2014 at 9:22 pm):
roger says (April 6, 2015 at 8:52 pm):
harshit rastogi says (April 10, 2015 at 7:26 am):
joe says (September 5, 2015 at 5:11 pm):
Don Reba says (October 10, 2015 at 8:02 am):
T Tak says (February 1, 2016 at 4:00 pm):
Swathi says (July 20, 2016 at 2:26 pm):
Jack Gleeson says (November 28, 2016 at 2:36 pm):
Jorge says (April 26, 2017 at 12:09 am):
Online Interviews says (June 19, 2017 at 4:51 am):
Comments are now closed