How to hire a software developer: Part 3 – Online tech tests and alternatives

In the third of a four-part series, Mike Ritchie looks at the pros and cons of online methods for screening candidates.

See also:

Technical Screening

It’s sensible for organisations to want to increase their confidence in a candidates hands-on coding ability before committing everyone’s time to a live interview. It’s good for the candidate too – if expectations on ability are mismatched, a live coding interview can turn out to be a difficult experience.

This used to be all but impossible, but there are a wide choice of online platforms now that make the job much easier.

Online technical test platforms

Online technical assessment tools like HackerRank and Codility are becoming more widespread in hiring. I understand why companies are using them, although I think they’re used either wrongly or excessively more often than not. If you haven’t used these tools, here’s a potted summary:

  • The technical interviewer designs the test by picking from a list of coding challenges and questions.
  • The candidate is sent a link to the online test, and works though it, with a countdown timer running, to complete the challenge.
  • The screening tool tests correctness of the implementation using a variety of inputs and edge cases – empty input data, input data with one element, input data with all zero elements, input data with large values that could overflow a accumulator variable….you get the picture.
  • The screening tool assesses the efficiency of the implementation by pushing through inputs of increasing size in succession, timing these, and calculating the Big-O complexity of the candidate’s solution.
  • Candidates lose marks for incorrectness, missing edge cases, and for implementing with less efficiency than is possible.

These platforms seem to provide an objective measure of capability, to differentiate between candidates. After all, the intent of HackerRank is to Rank your Hackers, is it not?

I see this all the time in software development – and not just in hiring – this urge to reduce something complex, nuanced and difficult to understand down to a dashboard metric. Numbers and measures are seductive. They promise simple answers and absolute truths.

Often, though, the way these tools are used is an example of the Streetlight Effect:

A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, “this is where the light is.”

Just because there’s light to see by, it doesn’t mean you’re looking in the right place. These tools measure what can be measured, but not necessarily what’s important. Here’s where we really want to look, at the things that have the biggest bearing on whether your code turns out to be an asset or a liability for your organisation:

  • Changeability and maintainability of code.
  • Clean code and simple design.
  • Ability to find a solution by working collaboratively in pairs, and in teams.
  • Working in small increments, driven by tests.
  • Readability and naming in the code.

There’s no tool that’s going to assess these things accurately…but it turns out that humans are pretty good at it. Any automated tool you decide to use works by assessing the characteristics of code, not of people. They’re proxy measures at best, and not always useful ones.

And remember that point earlier about matched investment? That the candidate should learn as much about you as you learn about them? All they learn with these tools is that you’re happy to have a machine run this part of your recruitment for you.

Don’t sleepwalk into outsourcing your technical interview to an online platform. Some aspects of hiring are just hard work, and you need to be prepared to put in the personal effort with candidates as part of the live interview for the more challenging parts of your tech selection.

If you need to run a coding assessment online, consider using the excellent CyberDojo instead: a platform that emphasises test-driven development, working in small increments.

If algorithmic problem solving is a core skill for the sort of work you do, sure, I understand why these tools might be attractive. But there are better ways of running interviews around CS algo and data structure fundamentals. And really, if that’s the main focus in the role, should it not be the centrepiece of the live interview? More in the next post on that.

But if this lengthy and slightly ranty effort to dissuade you has failed, and you feel you absolutely must use a tool like HackerRank, at least build your challenge sensibly. Rather than off-the-shelf coding challenges, consider creating your own. Use a (small!) mix of multiple choice, subjective questions and a coding challenge at the simpler end of the spectrum.

There’s a temptation to “add to basket” when building the tests, but you’re not doing your organisation any favours by making tests excessive. Also be very generous indeed on the time allowance – creating a false sense of urgency for the candidate is unrealistic, pointless, stressful and counterproductive.

Assessment of results should always be done by developers who understand the type of good code you’re looking for. Arrive at a view of how the candidate did by looking at the code, not just the results. Review in pairs again, and discuss what you’re seeing. Make allowances for nerves and pressure.

And one final point – if beat-the-clock coding challenges are appealing to you because you want people “working quickly” to meet arbitrary deadlines, I respectfully suggest you have other things you need to be thinking about before you hire anyone.

Coding assignments

If you need to scale up your technical screening, this will likely be a better way for you than the online test route. The basic idea is this – you set a small programming assignment along with a specification, and give to the candidate. They work on it at a time that suits them. You give them up to a week to get it back to you on GitHub (2017) or an emailed zip file (Ye Olden Days).

There are many benefits. For one thing, you can set a challenge that’s a better fit for the core skill-set of the role you’re recruiting for. A small website for web developers. A simple library with tests for a C++ developer. A single-screen mobile application for iOS or Android developers. A RESTful service for back-end Java developers.

The key is to construct a challenge that:

  • Can be completed in ~2 hours for an experienced developer – but remember that if you’re hiring people right at the start of a career in development, this same challenge could take two or three times as long for them: scale it back.
  • Requires the candidate to do the things you want to assess: for example, something they would need to write unit tests for, or something with just enough complexity to see their OO design skills. Code katas can work well for this if you don’t want to test for specific technologies and frameworks.
  • Emphasises code rather than configuration: it’s fine to have technology-specific challenges that need candidates to do configuration and use the tools properly, but don’t make it just about that.

You can communicate with candidates while they’re doing this too, unlike online technical tests. GitHub has social and communication features, so you can review code, make comments, raise issues and also see the solution evolving – you can do this privately too, if you want to add the candidate as a collaborator to a one-off private repository that you create for them in your organisation.

In my view, this is a better format than an online tech test, more realistic and gives a better chance for the candidate to show what they can do. They can work with their own, familiar tools, in their own time. They can Google, they can Stack Overflow…just like real life.

Used well, this format meets one of the principles I talked about earlier – you match the candidate’s investment:

  • By the challenge you set them, you can tell them about the technology you use.
  • Your notes on the challenge can expand on your values about code, and why these are important to you – even with links to further reading.
  • If they ask you questions, your responses will give them a sense of what you’d be like to work with.
  • Your code review comments give them a good idea of how you think about code specifics.

You should look for evidence that you have an ethical developer, as I talked about earlier: someone who cares about code, users, and colleagues. Look for good tests, good names and simple design.

Ask the candidate to commit early and often, then you can see the solution evolve. Ask them to add a too, with informal working notes – this isn’t something you review as part of their solution, it’s just a way of them communicating back to you.

I always advise students to do this for tech assignments even if they aren’t asked, noting down the decisions they made, things they think worked out well, things they would do differently. It’s great insight.

Agree in your team a set of points to cover in code reviews, so that it’s applied fairly and consistently across candidates. A lot of people find checklists to be an old-fashioned notion, but I think they’re useful – a simple checklist is a good way to capture the areas to focus on in the reviews.

When you’ve made your decision, feed the candidate’s solution forward into the live technical interview – it can make a very useful discussion point, and gives the candidate a sense that their efforts were valued.

“Aptitude” tests

Some organisations do psychometric evaluations or “aptitude testing”. These are numerical, inductive, and verbal reasoning tests, and – spoiler alert – are absolutely nothing to do with real-world aptitude in software development.

I’ve no idea why organisations do these, and I strongly suspect they don’t either. If someone can show from their experience that they’ve worked as part of a team in delivering solutions to complex problems, and their screening and technical interview shows they can write good code, I’d say their grey matter is working pretty well for them.

They’re used also in assessing the potential of graduate entrants to companies, the rationale sometimes given that recent graduates don’t have a work history or easily demonstrable skills to draw on. What graduates do have, though, is a very recent and detailed record of academic attainment, so I still find this puzzling.

I don’t get it. Perhaps my inductive reasoning isn’t all it should be. But still, hard pass.

A better funnel

In Part 1, we looked the “funnel” model that many organisations use. I recommend you work continuously rather than running stages in lock-step, especially when screening – respond to applications as they arrive rather than batching.

There’s no universal ideal, but a good generic approach for me would look something like this:

  • Have a daily hiring standup in your team to maintain the cadence. At this standup, do a group review with your tech interviewers of CV and cover letters that came in the previous day. Aim for no more than a 24‑hour lead time from arrival. There will be some early exits here where the experience clearly doesn’t match your requirements. Give feedback.
  • Aim for a very quick turnaround to phone screen for those candidates you want to go forward with. If there are multiple candidates you’re talking to, fan out to multiple phone screen pairs. These pairs both make the call, and make the call (see what I did there?). Give feedback.
  • Have one form of technical screen only, with a preference for a technical assignment that’s related to the tech domain (phone doesn’t count here, it’s a consistency check and not a technical assessment).
  • Get it reviewed promptly when the solution comes back – the daily standup helps coordinate this. Give feedback to both successful and unsuccessful candidates. Use a checklist, but also leverage GitHub as a review platform if that’s possible for your organisation.
  • Make a decision for a live interview based on a balanced view of the CV, phone and tech assignment, but weighting towards the most recent contact, so CV < phone < code assignment.

The key to making this work is to keep the rhythm and devolve the process, removing managers as (a) sole decision makers and (b) a process bottleneck. Explain to people why broad involvement in tech hiring is important for diversity, and the future health of the team.

They’ll get it, trust me, and they’ll see the benefits of working next to colleagues they were instrumental in hiring.

Coming next

In the next part we’re going to cover live technical interviews. Don’t worry – it’s much shorter than this part, because there’s only one approach that I think is worth using.

Mike Ritchie is a developer and technical coach. He helps teams with hands-on development, and builds technical capability through coaching. You can find him on LinkedIn.

See also

Sign-up below to receive email alerts for Part 4.

[contact-form-7 id=”4354″ title=”Employer blog email registration”]


Find out more about becoming an employer partner with CodeClan.


Receive our newsletter to hear about courses, events and more.


Don’t be a stranger... Stay in touch!

Why not sign up to our newsletter? We will keep you informed about new courses and also send you information from the world of digital.

Enter a query in the search input above and hit return to see results.

We'd love your feedback!

Did you find what you were looking for?