Should technical tests FizzBuzz off? Some thoughts on interview practice.

The opening gambit

I'm going to start by being controversial here. Well, controversial if you fall in to one of the two camps out there. Controversial to around fifty percent of you.

I don't think most of the technical tests or code challenges used in hiring are a particularly good way of measuring a candidate's ability to do a software developer job.

I don't think they're a good way to filter or rule out candidates, as a singular metric. I don't think candidates who score well on them are necessarily going to be good at doing the job of a software developer.

I don't think they are a consistent or reliable measure of how well a candidate will solve real-world problems, or to what standard or efficacy they will write code.

And I honestly couldn't care less - at least not in my particular field, which is not building operating systems or relational database engines - whether or not a candidate knows the intricacies of a heap sort or how a binary search tree works.

I don't care how well a candidate knows how to do things the organisation I work for is never going to need them to do in the course of their employment.

I do care about understanding their thought processes, the breadth and depth of their practical experience in building working software for some particular purpose, and their intellectual understanding of a variety of technological concepts. Further, I believe these things are usually better established via a structured technical interview than by high pressure, time-limited tests on artificial, low-level problems, or even many take-home code challenges which ask the candidate to write a program or module which I've been asked to do in my time applying for jobs over the years.

The worst kind of challenge

Here is an actual snippet from a random HackerRank challenge:

Palindromes are strings that read the same from the left or right, for example madam or 0110.

You will be given a string representation of a number and a maximum number of changes you can make. Alter the string, one digit at a time, to create the string representation of the largest number possible given the limit to the number of changes. The length of the string may not be altered, so you must consider 0's left of all higher digits in your tests. For example 0110 is valid, 0011 is not.

Given a string representing the starting number, and a maximum number of changes allowed, create the largest palindromic string of digits possible.

Mate, I don't even get the question and I've been working very successfully as a professional developer for fifteen years. English is my native language and I'm struggling to pick apart what you even want me to do there, other than it has something to do with palindromes but...I don't know? Ones which are made out of digits for some reason? (Author's note: It occurs to me irony isn't always clear in a blog post, so for the avoidance of doubt: I'm being dramatic to make a point, yes I do understand the question about palindromes, I just think it's a bad question which would easily throw me if I felt under pressure.)

More crucially, in all my years on the job for numerous real businesses building real software used by real people, I've never had to solve a problem like that and I'm very confident I never will. If you worked as a business analyst anywhere in the world and wrote up the acceptance criteria of a ticket that way, I'm pretty sure you'd be fired for gross incompetence.

Fortunately, this is an especially cruel example of competitive programming type questions and you would rarely find any this bad in a test you were asked to take as part of a hiring process. But you will find plenty which are just as pointless and opaque. Even worse, tests like this usually take place in a controlled online environment with a strict time limit on how long you have to complete it. Bonus points if this is done on-site under interviewer observation and you're not even allowed to look at the reference documentation for your programming language.

The issue here is you're giving someone, even someone very skilled, both an unrealistic problem and unrealistic conditions in which to solve it. It's like making a decision about whether to hire someone as a surgeon solely on the basis of whether they can accurately throw scalpels at various points on a dart board, blindfolded and facing the wrong way, while being asked to yell out a different line from Hamlet every ten seconds. That's about how much relevance this kind of test has to do with anything anyone would ever have to do in the real world.

Another not-so-great kind of challenge

If not some dreaded tongue-twister, there's another kind of challenge I'd argue is probably okay for junior or entry level positions but not so useful for senior roles; the basic syntactical awareness challenge.

I've been asked to do Fizz Buzz in multiple interviews in my career, which tests precisely and only whether a candidate knows what the modulo operator is. And even though I agree it's something you would probably expect any devs worth any salt to walk, be honest; how often have you actually used the modulo operator in your career? Would you be worse at your job if you'd never heard of it? Would it ever have prevented you achieving something you needed to?

Let's pretend for a moment either PHP doesn't have modulo, or I don't know it does, but one day I realise I need to calculate the remainder of dividing two numbers. Isn't it more interesting to know whether I can solve that problem?

function getRemainder(int $a, int $b): int
{ 
    return $a - ( (int)($a / $b) * $b ); 
}

Yes, if I saw this in a real code review, I'd leave a note for the dev saying "Hey, there's actually a built-in operator for this", but I definitely wouldn't think "You're a terrible developer and you shouldn't be working here."

But the real contention for me around these kind of test questions is when they're aimed at people applying for senior roles. If you find it necessary to test whether I literally just have basic awareness of PHP syntax, you're a) effectively saying you think my entire CV might be a work of fiction and b) learning nothing at all about whether I'm any good as a senior developer.

The devil's advocate

On point (a) in that last sentence, I will concede - I do get why businesses feel a need to do this and I don't really take offence at someone who doesn't know me and has never met me wanting some assurance upfront, besides my good word, that I actually do know how to be a developer. You do get resumes which vastly inflate a person's depth and breadth of knowledge and experience. You do get people who will claim to have working knowledge of languages or tools they've never really used, or touched once years ago. Sometimes you get people who have enough basic knowledge of a language to produce a running program, but don't have a clue when it comes to best practices, prepared statements in databases, the range of stock library functions to help them perform common tasks, etc.

I've had interviewers say to me "Sorry, we know these questions might have seemed a bit silly, but you wouldn't believe some of the answers we've had."

So I'm not saying a twenty minute, ten question quiz on a random-ish range of programming principles and language features is a bad idea or totally useless, whether that's verbally in an interview or via some online screening service. I just think the information and level of assessment you can get out of it is very limited. They're not really useful as a filter except to screen out the people whose CV was a work of fiction.

So that's okay if you have a large volume of applicants to sift through and need to choose who among them you block out the time to properly speak with and interview. If you've only got a few applicants, maybe you can speak to all of them anyway and throw some basic technical questions in to the mix of what you ask them.

Real world problems

Here's a contrived example of the kind of problem I actually have helped solve in the real world, as part of a team:

A customer on a high traffic website can place an order and make payment to complete a purchase. When this happens, we need to check the stock is available, send the order to our fulfilment centre, get back the available shipping date from that centre, update the stock level displayed on the website, dispatch a confirmation email to the customer with the expected delivery date and confirm the payment was successful. This is implemented across a distributed microservice architecture between the website frontend, our ecommerce CMS, our order system, our shipping system and our payment provider, and we need to make sure the process happens both transactionally and concurrently, such that no customer can place an order and pay for stock which is not available to be shipped to them. If any part of the process fails, we need to be able to rollback the order in all those places and inform the user on the website their order couldn't be completed.

It's a harder problem to solve than the one I pulled from HackerRank, but it's a heck of a lot easier to understand.

If I'm on the hiring team for a business and you come to me as a candidate, that's the kind of problem I want to know how you might solve. But you can't represent this as a coding challenge. The only effective way I can learn about how you might tackle this scenario is to describe it to you in an interview and ask you to speak for a few minutes about your thoughts on it.

A better kind of challenge

It is of course possible to take a small portion of a larger, practical problem like that and write up some criteria for a coding challenge around it; a much smaller scale version of something you would do in part of your systems.

Maybe in this example we might ask a candidate to spend a short amount of time at home, in their own conditions, using their preferred tools, to write a small script which can read a message describing an order off a queue, look up a product from the order in a data store to check it's available, then send back another structured message to a queue describing the shipping date for that product in accordance with a couple of simple business rules.

We might not even need a real queue or data store here, just objects in memory representing a rough idea of what the interface for the real thing might look like. We can tell the candidate they shouldn't aim to have polished, production-grade code, or to spend more than an hour to two hours working on it. We can tell them it's totally fine to pay lip-service to or ignore the more boring parts of systems you build in the real world, such as logging and error handling, but advise them we might ask them in their interview to talk a little bit about how they would flesh out their work in to a proper system.

My advice, as someone who's been on both sides of the interview table; if you're going to ask candidates to do a coding challenge, make it one which:

• They can complete to both their satisfaction and yours quickly, ideally in no more than two hours. Remember; your candidate for a senior position is probably speaking to half a dozen or more other businesses who may all be asking them to do some kind of technical test in their limited free time. Why would they want to work for you if there's another company who would take their arm off to have them, without expecting them to put in a half or full day of unpaid work just to prove themselves?

• Is in some way related to the real products they will be working on. Get the candidate excited about who you are and what you do.

• Has the expected inputs, outputs and desired scope clearly defined. It may be tempting to leave parts open to interpretation or the applicant's imagination to see what variety comes back to you, but the reality is this is a poor metric for quality. We don't build successful software in the real world to vague aspirations, we do it to well-documented requirements, purpose and directly measurable outcomes. If you wouldn't consider it sufficiently refined to give to a developer on the job, you shouldn't be giving it to someone applying for the job.

• You are upfront is not expected to be feature-complete or have all parts written to real-world standards, provided the core acceptance criteria are met. You can ask them afterwards in interview to talk about how they would do the code better as a paid employee.

• Takes place unmonitored and untimed, in the applicant's own environment, using their choice of tools and free to reference or look up anything they need to. You know, like how we build software in the real world.

Finally, the further advice I'd give businesses more broadly, particularly those who have found they're struggling to fill roles or rejecting a large percentage of applicants:

• Don't automatically rule out a candidate only and entirely on the basis of their results in a technical quiz or code challenge. In many cases, it may turn out to be worth the half hour of your time to find out why they struggled, what they found difficult and whether if the test conditions or wording had been different, they might have had a different outcome. If the majority of candidates are failing your technical test, you need to ask yourself if there's a possibility your test is the problem.

• Don't assume because a candidate does well on a technical test, can demonstrate basic syntactical awareness of a language, or pass a random pop-quiz about acronyms, this means they will be a good developer. Good technical interviews are both personal and situational. Ask the applicant about their work history, their interests, their opinion on languages, frameworks, system design, applicaton security, testing techniques etc. You'll get far more out of giving someone the opportunity to be themselves and talk freely about their passions than you will out of asking them to give a textbook definition of polymorphism.

• If an applicant has real projects or personal code they are able to share with you, or open source contributions, allow them to present and talk about these as an alternative to completing a bespoke code sample. Not only does it save everyone valuable time, they will probably be more excited to go through something they did for a real purpose and in turn you will get a more qualitative measure of their ability to solve real problems, and a good look at what they consider to be a decent example of their coding.

Balance both sides

Hiring can be difficult. Finding a job in which you're happy and fulfilled can be difficult. I guess the conclusion for me is here is whichever side of the interview table you find yourself sitting on, be mindful that it is a two way street.

As a candidate, you do have to try to show your best and be willing to establish what you can do, why you're a valuable asset. But as an employer, don't forget, if this applicant really is a good developer, there are dozens of other businesses out there just as eager as you to get them on board and fill that vacancy. If you're putting qualified candidates through a long, multi-stage, drawn out hiring process which gives the impression you don't think their time is as valuable as yours, you might end up finding your vacancies remain open longer than you'd like.

---

Comments