Tales from the interview: when the tech test goes wrong
A couple of years ago, when I was last searching for a new role, I was contacted by a recruitment agency regarding a FinTech conveniently located in my home city who were looking for a PHP dev to join their team. I won't name them but they were a matured start-up based around data and analytics - a comparatively small business but earning revenue, profitable as I recall.
The pay and benefits package for this role were excellent, considerably above average. They had the resources to expand and although remaining in the financial sector wouldn't have been my first choice in terms of industry and product, it looked sufficiently intriguing that I gave the go ahead for the agent to send off my CV and put in an application.
It wasn't long before the agent got back in touch. The business liked my CV and were asking me to take part in a first stage interview with one of the directors. At this point, the agent explained their hiring process to me. Fairly common model - first interview, a take-home tech test, they review the test, second interview, offer or rejection.
I may have touched on the topic of take-home tech tests on my blog before. Broadly, I'm not in favour of them.
I don't believe they're a good way of measuring programming ability, technical acumen or problem-solving capability. I don't think they present realistic simulations of what a candidate will be dealing with on the job or how they will work. Where attempts are made to craft challenges which more closely resemble these things, they become more complex, demand more time and start to look a lot like simply unpaid work.
I don't think this is fair to candidates. When someone is seeking out a new role, they're probably applying to many different prospective employers. They may also still be working full time and juggling family and other responsibilities. If every business to whom they apply is expecting them to spend anything up to eight hours (a full working day) of their own, free time, building complete sample applications, or even churning out a dozen or two leetcode algorithms (which deliver even less value), this quickly adds up to unrealistic expectations.
The inevitable result is that sometimes (often) the submitted works are rushed, incomplete, delivered to a worse standard than what they might do as a paid employee on the company clock and with more than one day to build out a project from start to finish.
The better process, in my considerable experience, is a focused technical interview, where the candidate is asked about problems they've solved, projects they've delivered, challenges they've overcome, their personal interests in technology and their aspirations. Talk about what you do as a business, what both you and the candidate are hoping for in the role, talk about your culture, get into depth about your lifecycle, your infrastructure, your ways of working, etc.
This delivers much better surety and value about the core competencies of a candidate, and it gives them the same about you as a business. It saves both of you a waste of time, money and energy invested in the wrong opportunity. Do you want to hire someone who's jumped through all your hoops only for them to walk three or six months later?
I digress. Much as I disagree with the idea of take-home tech tests, this role - as is unfortunately normal nowadays - included one in the process.
So I have the interview and it's vanilla as far as these things go. We talked a bit about the company, its history, the expectations for the role. I thought it was a little bit pretentious that the interviewer introduced himself by the title Dr (he had a PhD, in a field unrelated to either finance or technology) but other than this slight weirdness, there weren't any obvious red flags.
We get to the topic of the tech test. I politely explained that I'd applied for four other roles, all of which included some kind of technical test or take-home assignment and that while I was happy to complete one, it would have to be something I could complete inside a couple of hours.
Dr Interviewer assured me there was no expectation that I would spend more than two hours on their tech test and this would be enough time for me to do a satisfactory job.
We said a pleasant goodbye and I awaited the details of their test to come by email.
The technical test
Instructions for the technical test were, on arrival, simple enough. The task was to use Symfony to build a simple API with a single endpoint (no worries about access control or other auxiliary configuration) with some particular parameters. I won't go into the details but basically the idea was to take the inputs from the API request, extract some data from a CSV file they supplied and then based on the specifics of the request, perform some calculations or transformations on the data and return the result as a JSON object.
The only real caveat or challenge part of the test was that they wanted you to design this API endpoint in such a way that the parameters supplied as input were flexible and extensible - it was something like the user needed to be able to provide custom expressions with operators in the request and the test wanted to see how you would structure that so new operators could be built later.
I completed the test inside the two hours I'd set aside for it. I built some extensible
OperatorInterface and I don't recall loads of the details but you know, I'm not saying my solution was the maximal, theoretical perfect solution but it was entirely reasonable and adequate. It fulfilled the full scope of the test's criteria in a way which, in the part that counted, was done to a professional standard.
But of course constrained by the time I could put into it, for all the parts other than the specific bit of the challenge I described, I cut a lot of corners. I want to stress in these kind of code samples for job applications, this is an entirely normal thing to do - unless we're specifically asked to, we don't worry about boilerplate parts of a system like logging, robust error handling, edge cases, caching, access control and all the rest of it.
It's still important, however, that where we do this as candidates, we make it very clear that any corners haven't been cut because we're not aware of them, or because we don't understand the proper principles of good system design, but just because it's not the bit of code we focused on for the task at hand in the time we had.
So I did what any candidate would do. I annotated the parts of the code where I knew there were things missing or done to a lower standard than we would in a real-world system. I put in comments like the following examples (not real examples, just the gist of the kind of thing I was writing in the code):
* Parse the CSV file into an array in memory. Obviously in the real world we wouldn't be getting our
* data by reading and parsing a full CSV file on every single request. We'd be using a data source such as a RDBMS
* and very likely using an entity mapper to get a list of objects, maybe with a caching layer such as Redis.
* As this is just a demo project, I haven't bothered with things like error handling, checking correctness / validity
* of each row, missing headers, missing columns, etc. here, I'm just assuming the whole file is valid...
* Extract and tokenize a request expression from a string.
* For the scope of the sample, I haven't bothered handling error and edge-cases here
* such as referencing a non-existent record type but we would do this a bit better
* in a real-world system...for now, we just convert these invalid values to null and ignore them.
I thought the dozen or so comments in the sample application I sent in clearly explained where and why boilerplate-type features were missing. In terms of addressing the core criteria and the design of the particular bit the test was focused on, I was happy with the result so I sent it over.
Feedback: the bombshell
It was the following afternoon I received an email from the recruitment agent. I'm never presumptuous about whether or not I'll be offered a job but I was fully expecting to progress to the second interview. I know what I'm doing with PHP and Symfony, very well, and I was confident they'd be happy with the sample for their test I sent in.
The email in my inbox opened like this:
I've just heard back from [XYZ] about your technical test and unfortunately, it's not positive.
I was - and I sincerely don't mean this in an arrogant way - shocked. Blown away. I had to re-read it twice to make sure I was reading it right.
I phoned the agent to discuss the feedback he'd been given and this is where things took an interesting turn.
Apparently, the business had been entirely happy with my code sample as a technical solution to their test criteria. They liked the design, they were satisfied I knew what I was doing with Symfony and in my ability to translate requirements about business rules and data into working code.
What they didn't like were my code comments. The feedback was that because I'd used phrases like "..haven't bothered with proper error handling here", this meant I "didn't demonstrate the attitude they were looking for in an employee" and they wouldn't be progressing my application any further.
I ask you at this point, dear reader, to take another moment to bear in mind the context of those code comments that I've explained above.
I openly invite feedback now, via this blog post, as I ask the question: was I being unreasonable here, or were they?
Because from my point of view, I've sent off a dozen-plus code samples like that in my career with code comments like that and I've never had it be an issue anywhere else, I've never been criticised for it in follow-up interview when tech teams have gone over my submissions with me, let alone been instantly dismissed from an application process over it. As far as I'm concerned, it's just a normal, sensible thing to do, to draw the line between a candidate who's missing pieces because they don't know how to build a real, production-grade system properly versus one who's...just writing a quick code sample.
Overall, I feel like this reaction to my tech test was quite valuable. I know my use of language in the comments was relatively casual but it wasn't exactly crass or vulgar either. Whether others would agree I'm right, I don't know, but I took it to be a sign that in this case, the company, the culture and the people I'd have been working with would not have been a good fit for me and that if they'd offered me a position and I'd ended up accepting it, I would have swiftly come to regret my decision.
I'd welcome any reader stories about your craziest interview experience, your worst rejection or your most heartening success. Please feel free to leave a comment using the form below or email me!
All comments are pre-moderated and will not be published until approval.
Moderation policy: no abuse, no spam, no problem.
I know you mean this post as serious, but 'Dr interviewer' made me chuckle. Imo, your comment in the code, petrified HR (apologies to all HR professionals if this is not true in your case). "Here there is someone honest, but we find him too honest. He may start casually pointing out our mistakes". I'm guessing this type of vulnerability in the start ups is common. Good dodge that was Dave.
Editor's reply: Thanks Pawel, it reminded me of that bit in Friends where Ross would always go "Hi, I'm Dr Geller!" even in situations where his degree was completely irrelevant. As far as I remember, the guy interviewing me had a PhD in chemistry. Who uses their academic title in a software job? 😮
Would have been better off with a simple "TODO" than that tone :) I dont think I would have phrased it like that on an interview assignment, but just they took it the wrong way.
Editor's reply: Yeah with over 2 years' worth of hindsight, I can accept the way I phrased my comments wasn't the most diplomatic, I suppose I'm just used to software being an industry where colleagues are quite laid back and internal comms quite casual. Also I'm a habitually verbose speaker and probably felt like I had to explain my choices more than a simple TODO might have conveyed.
Your closing heading, Bullet Dodged, says it all. That type of attitude was surely a smell of their culture. I understand "how we treat people" and communicate is even more important than code quality, and from their throne, that's what they were trying to convey, perhaps? ...gotta shrug that one off and move on. I agree on your preferred interview style. I'm fine for a take-home task, but these efforts are best walked-through and discussed verbally. Good riddance!
Editor's reply: Thanks for your comment. As a little addendum, I've just dug up the actual feedback I received after talking with the agent. Here it is:
I've just caught up with the reviewers on David's test. They thought that what he submitted was generally very good where he had spent the time to solve the problem but unfortunately, there were concerns around his attitude towards working at [XYZ]. The reviewers found 8 instances in the code where he out and out stated things such as:
I couldn’t be bothered to build proper validation and error handling
While we appreciate everyone is busy and these coding challenges are tedious, we feel that it doesn't demonstrate the attitude we're looking for in an employee. I had also noted a concern of this nature in my initial chat, but I was happy to put it aside. Having had that confirmed by the technical reviewers, the decision is quite straightforward for us - a pass on David.
I wish David all the best with his search for a new job.
To the best of my memory, I didn't use the phrase "I couldn't be bothered", though there were phrases along the lines of "Haven't bothered here..."
Anyway, here's the funny part: According to my exchange with the recruiter, when I looked back over their tech test at the time, they'd specifically written this in the instructions:
You should aim to spend a few hours on the project; it's ok if you don't finish! Please indicate any areas that are incomplete and how you might finish them given more time.
So of course that's what I did. The entire reason for passing on me was just that they didn't like the choice of wording in how I did it.
The difference between failure and success isn't whether you make mistakes, it's whether you learn from them.
Recalling the time I turned down a job offer because the company's interview technique sucked.
Recalling the time I was rejected on the basis of a tech test...for the strangest reason!
Why type hinting an array as a parameter or return type is an anti-pattern and should be avoided.
Leveraging the power of JSON and RDBMS for a combined SQL/NoSQL approach.