DEV Community

Cover image for Coding Tests Eliminate Some of the Best Candidates

Coding Tests Eliminate Some of the Best Candidates

Adam Nathaniel Davis on February 18, 2020

There's something beautiful (and potentially ugly) about the career fields associated with application development. As a greybeard in the field, I...
Collapse
 
190245 profile image
Dave

Oh man, great article!

All the tests have their place, if done appropriately. I'm very much in the "I'm not jumping through hoops" camp. But recently, I found myself on the opposite side of the table, knowing my employer didn't have the best reputation in the market. So I did something about it.

I effectively took complete control over the hiring process, and moved it from CV, pen & paper test (who even does that?!?) and formal in-person interview.

The new process is much more modular, all CVs come to me for filtering personally. At that point, I basically sort no/junior/mid/senior/rockstar. Since we're looking to build a team, "no" and "rockstar" are treated the same - I'm simply not in a position to have an uber brilliant coder that can't communicate.

From there, things get a little different. Basically:

  • juniors get a Skype call, during it, we give them two code examples with bugs, the second of which asks them to optimise. I share my screen & they talk me through what they would do. If they can't fix the basic bugs (syntax errors), they fail. Anything better than that & we can talk more. If they don't like my IDE, or the variable naming, I can use another IDE or refactor things. After the Skype call, it's decision time, no more hoops.
  • mid-senior & senior get given a set of requirements and a week long deadline. There's a total of 5 requirements (and one of those is the language & minimum version). I can crank out a solution in an hour to it, and other developers that I've given it to take a couple of days. But people have families (and other interviews to do), so we give more time just to be nice. After the test, we move on to a Skype call.

At the more junior end of the scale, I admit, I'm a little evil. I don't expect anyone to understand how to verify that a number is a Perfect Prime, so I ask them to do exactly that, live on the Skype call. But what I'm really looking at there, is how quickly a junior will throw up a flag & ask for help. Fail-fast & you look much better than if you bluff your way through it.

At the more senior end of the scale, things are a bit more relaxed. Submit code that doesn't even compile for all I care. I'm only looking to see how you think, to give me talking points during the Skype call, where I'll be asking why you did things the way you did. I don't want us to be in an echo chamber, so I want candidates to think differently to me, but if you can't explain your reasons for your choices, then we have problems.

On the point of feedback, juniors effectively only really get "isn't quick enough to admit they don't know things" - since that's all I'm looking for in a junior. Seniors, if rejected due to the test submission, I've averaged about 2 pages of A4 (what they did wrong, and a couple of options for what could have been better, with reasons why it's better). It takes me longer to write up the feedback for Seniors than it would do for me to do the test, and I think that's the fair way to be.

We've had some failures:

  • One used jHipster to generate most of their Senior test submission, including a whole front end (for a CRUD API that we asked for). Way off piste, over delivering trying to impress, and using auto-gen tools. Sure, do that after you've got the job, but before hand, all that does is hide the way you think.
  • Some simply submitted too late, and another candidate beat them to the seat. I class those as failures because I wasted someones time, when I should have reached out to them to tell them to stop because we'd hired someone.

We've had some great success:

  • One applied as a junior, but I think they're a unicorn after spending an hour with them on Skype. So we're paying international relocation, a couple of months accommodation, UDemy subscription & they're getting the "Mid" position & appropriate salary.
  • I was put in the position of having to choose between a wet-behind-the-ears never had a dev job, and someone with 6 months industry experience. Man was that a weekend full of stress for me. I've gone for the more inexperienced guy - partly because he showed more passion on Skype, but also with a slightly lower salary, I can put more into the training budget.

Co-incidentally, as a result of all of this, I've moved from Senior Developer to Architect, and the new starters will be reporting to me.

Collapse
 
bytebodger profile image
Adam Nathaniel Davis

Great points! And it sounds like you have a very reasoned and fair process for evaluating candidates. That's quite an accomplishment - because few do.

At the risk of oversimplifying what you've written, I think it really comes down to this:

There aren't any "wrong" questions or "wrong" tests if you (the interviewer / hiring manager) have the right (meaning: reasonable) expectations and have a professional, mature way to parse the submissions.

As you pointed out, you can ask candidates to answer questions that they can't reasonably answer - as long as you know that and you're fair about parsing the answers you do receive. There's value in almost any kind of hiring process - but too many times, the hiring managers don't understand the value and expect the wrong things.

Collapse
 
xealgo profile image
Jason Welch

Really good read Adam! I've been coding since I was 13 (now 34) and have worked professionally since I was 20. Even so, I struggle with live coding tests, often due to the time restrictions as well as ambiguity in the questions / instructions. I was turned away from Twitch a few years back after completing every other task but failing to get the regex correct because their validation rules were written so poorly. It was super frustrating. I also failed a Google interview because the interviewer decided it was lunch time during our interview and couldn't be bothered with questions. Whiteboarding has always been where I've shined, but even then it can be loaded due to biases. Take home tests can be fun, but now that I have two children, on top of my full time job and all of the other responsibilities in an adults life, the time it takes due to unrealistic expectations, is just too daunting. I'm finding that I'm pretty much ignoring every new opportunity that comes up, simply because the interview process (especially coding aspects) tend to be time consuming, anxiety inducing and irritating. Interviewers seem to search for any flaw as to reject you as quickly as possible as they have another 300 candidates to weed through. I'm super glad to have read this post because at least I know I'm not alone in my interview struggles.

Collapse
 
bytebodger profile image
Adam Nathaniel Davis

Yep... nodding my head to every one of your points.

I tend to take a rather fatalistic approach to most job interviews now. Before I do the interview (meaning: the full interviews, where you're doing some kinda live coding, or at least being grilled in real-time by their devs), I honestly have NO expectations whatsoever as to whether I'll actually do well or not. It's not that I lack confidence in my own skills. I most certainly do not. But I really feel that whether I do "well" has soooo little to do with my actual skills, and soooo much to do with random factors over which I have no control. Often, it feels to me like their assessment of my skills has little real correlation with my actual skills.

The last time I was interviewing, a company decided not to offer me a job because they "just didn't think that my React skills were strong enough". Now, I'm not claiming that I'm any kinda "React savant". But I've been coding for more than 20 years - and right now, React is probably my strongest skill. It wasn't that I was annoyed to miss out on the job. That stuff happens, regardless. But I was mildly irked to think that the disqualifying factor was my putative lack of React knowledge. Whatever...

Collapse
 
xealgo profile image
Jason Welch

I would be willing to bet it had nothing to do with your knowledge in React but probably more about something stupid like you didn't use a certain library they like or maybe you didn't do things 100% in the latest cutting edge React way. It's all nonsensical. As you said, there are far too many factors in which you couldn't possibly prepare for or do anything about. An interviewer could simply be in a bad mood that day, or maybe they don't like your shirt, who knows. It's really not worth even caring about.

I know for sure I've utterly failed code tests before, like when asked to solve a graph problem in 15 minutes back when I had very little knowledge of graph algorithms.. but still, a lack of knowledge does not mean a lack of motivation, a lack of intelligence or a lack of skill. I've failed to get hired at companies where I have friends working who have told me that I'm a better developer than a lot of their coworkers, but because I simply was lacking in a certain area, I was dismissed without a second thought. It's easy to sound like a poor loser, but the reality is, not everyone knows everything and I think sometimes it boils down to luck more than qualification.

Collapse
 
smithphil profile image
Phil Smith

I might be that your React skills were too good and you were perceived as a threat.

Collapse
 
amritpandey23 profile image
Amrit

Furthermore, for many people, there's just something about watching that (artificially limited) timer count down that materially impairs their ability to provide a solid solution.

You had me here. I just can't perform my best when put into time constrain. Any advice to overcome this? Because I don't think the interview process is changing for good any time soon.

Collapse
 
bytebodger profile image
Adam Nathaniel Davis

I wish I had any decent advice - but I don't. When hit with a timer, I typically find that I have one of two responses.

  1. I read the problem statement and I immediately know how to solve this. In this scenario, I often do quite well - even with the clock looming over me. Because it's really just a matter of getting what's already in my brain onto the screen.

  2. I read the problem statement and I have to spend some time simply understanding the question, or formulating a way to solve it. In this scenario, I almost always fail. Because if I have to sit there and mentally solve the issue while the clock is relentlessly counting down, it'll drive me crazy and completely impede my ability to think clearly.

Collapse
 
melvyn_sopacua_afcf30b58a profile image
Melvyn Sopacua
  1. Use a search engine to find the answer. A timer is a signal to me for a goal oriented task, so be goal oriented and don't waste time on coding.

Like AI's pause a game indefinitely, cause you won't loose.

Collapse
 
amritpandey23 profile image
Amrit

Also in the second case, there is a tiring urge to submit a bad solution. Solution can be improved but I am in a panic mode so the decision I make is to settle down with a solution which I would never even implement in my own project.

Collapse
 
yjdoc2 profile image
YJDoc2

Hey, amazing article !! I really liked your writing style, and overall flow 😄

I would like to know your opinion on something :

At least in all coding tests I have given (most on something like hackerrank) where you have to solve given coding question which must run within a specific time, and you cannot switch windows or change tabs. Also if any test case fails, you don't get any output produced or chance of debugging to find why it went wrong.

Now I understand that one would wish the candidate to think of all edge cases and write code for them, I would usually write separate test file to keep the code clean and spend maybe half - or one or more hour just for making sure I have tested for all corner cases, and would set up some logger or something to get enough info to debug for cases which I haven't considered (I'm not perfect :) )
So does the ability to properly prepare the application for debugging and actually debug plays no role?

Also as even a single sub domain of computer can have so much of knowledge and concepts, I tend to rely a bit on Google , stackoveflow, dev etc to know concepts or code for things which I haven't encountered before/ don't encounter much regularly. ( I believe googling effectively is a skill, but I may be wrong.)
So how can one test how well I can code an application by making me write code to test how many substring of give string have a particular number or arrangement of 1 and 0 within half an hour without using Google or asking someone who might have dealt with this before for help? (Personally I have never had to use anything like this in any of projects I have done in web dev or even compilers.)

As you have quite some experience in the field I would like to know your thoughts on this, as well as if I'm wrong somewhere in thinking this.

Thank you :)

Collapse
 
bytebodger profile image
Adam Nathaniel Davis

I definitely don't think you're wrong. However, IMHO, I feel that evaluating unit tests should be a separate part of the interview. Because, while I hate a lot of those automated tests, the goal of those tests is not typically to determine whether you have good all-around programming skills (and "all-around" would include writing tests). Rather, they aim to "quantify" your ability to solve specific problems by writing specific algorithms.

In that regard, another shortcoming of the online coding tests is that their scope is just so dang narrow. They don't highlight what kind of a coding employee you'd be. They only highlight specific little niches of your programming brain.

Your Google/Dev/StackOverflow example is spot-on. It absolutely is a skill. And when I'm evaluating potential hires, I don't much care if they've memorized every little bit of nuance in the language. I care about whether they're able to find those nuances when they need them. Do they even know when to start Googling and when to keep playing with the code?? Those are things that are not - and cannot - be evaluated with online tests.