Code katas for selecting technical talent - An experience report
Hiring good technical people for your team is always a challenge. I have experimented with several different approaches in the last few years, from screening CV, in-person interviews, including technical questions, a design challenge to solve a business problem during an interview, etc. But I was never entirely happy with the results because none of these techniques gave me a good indication that the candidate masters good technical practices.
So with some colleagues1, I spearheaded the efforts to set up using code katas as part of the hiring process in two places: a consultancy firm and a department in a big enterprise.
Disclaimer: with sharing these experiences, we do not claim this is the best possible approach for hiring. Each method has its advantages and disadvantages. That is why we discuss the advantages and disadvantages in section Evaluating our experiences.
Context: What are we looking for
To understand why and how we use code katas as part of the hiring process, you have to understand the context.
Consultancy firm
A technical group of about 25 consultants part of a larger multi-country consultancy firm uses code katas in the process to hire extra colleagues.
Hiring focusses on finding people who can improve the way of working wherever they go. The people from the technical group work as a developer, team lead, or technical coach. The group strongly advocate technical excellence, so expects colleagues to bring these values to customers. Consequently, the group searches for experienced consultants who can immediately provide value or high-potential juniors who breathe technical excellence and can find their way.
As a consultancy firm that wants to grow, the more high-quality candidates we can onboard, the better.
Enterprise
A department of an enterprise uses codes katas to hire developers. The department has about 80 developers, which make up 85% of the people.
In this department, there is a strong, renewed focus to improve. There are significant changes in the markets requiring a stronger digital position. The department under discussion is responsible for some vital digital applications, and a ramp-up of their quality, innovation, and delivery was needed.
Moreover, there had been a few technical coaches (like us) around showing the merits of TDD, mobbing, paring, and technical quality. The positive results on the quality of the software and the team started to be visible. So the decision was to focus on onboarding people who know the basics of these techniques or at least are capable of writing proper unit tests and working closely in a team. You do not want to hire new people who hate the direction you just set out.
In this department, we were only looking to onboard a few new people for a few key projects, or to replace some people who left.
Overview of the hiring process
Invitation to do a kata: In the consultancy firm, candidates enter the kata review process by having a chat with a recruiter to see if there is a potential match. After this chat, the recruiter invites the candidate to do a kata. In the enterprise, people pass through another department responsible for contracting. In the end, their CV comes to the chapter leader, who invites them to do a kata.
A specific challenge for the enterprise is that there is a large number of externals and foreigners, often placed by a few suppliers. We needed to remove any potential prejudices, so all katas are fully anonymous while reviewing.
Candidate doing the kata: The candidate makes the kata when they see fit. They send their answers back through e-mail. HR people or the chapter lead subsequently add a ticket to a kanban board in Jira or a line into an Airtable sheet.
Review: Technical people pick up the katas for review when they have time. In the enterprise, current members of the team participate in the review process. We collect notes about the strong and weak points on the ticket, and everyone makes a statement about go or no-go for an interview. Pairs reviewing are encouraged, but due to practical constraints, it also happens alone. The enterprise had a requirement that there must be at least two reviewers.
In the case of a no-go, a candidate receives written feedback with suggestions on what to study to improve. Writing such feedback with a positive and encouraging tone is hard and a lot of work. But we believe it is essential that even candidates that are not accepted are treated with respect and receive feedback as they also spend the effort to perform the kata.
In the case of a go, we plan an interview. In the interview, we first discuss the kata and encourage the candidate to make suggestions for improvement. Next, a more general conversation follows to see if the person would fit with our way of working and allow the candidate to see if he would like to work for us.
For the enterprise, we try to get the candidate inside the office for half a day to get the know the team and pair with use for about a half day before signing the final contract. This gives both the candidate and team members a chance to get to know each other and a final confirmation that the candidate is a good match. This does not happen in the same way for the consultancy firm, were it makes less sense as most people work clientside.
If the all the above steps go well, we consider a candidate to be validated. The next steps are discussing the details of the collaboration and contract conditions (out of scope for this article).
The invitation to perform a kata
To ensure the instructions of what we expect uniform and precise, we use a standardized e-mail invitation. An example below.
Hello Sofie,
We value software craftsmanship a lot, and we believe that the quality of the developer can be judged by the quality of the code and the discussion about the choices made. So we ask each of our potential candidates to do a code kata for us, followed by a discussion about this code with one of our developers. Please do this exercise yourself (as a candidate) without the help of colleagues.
To remove bias during the evaluation of your exercise, we ask you to anonymize your submission. We will provide you with an identifier for your repository.
What we expect from you:
- We expect a link to a repository (Github, Bitbucket, GitLab) that we can access containing your source code.
- We expect to have a README.md that explains how to compile and run this code.
- We expect you to produce the best code you can provide to us, and that you can explain the choices you made.
- We expect you to spend up to a few hours on this. But, of course, this is up to you.
As you are an experienced developer, the specific kata we have in mind is a refactoring kata that can be found on XXX.
If anything is not clear to get started or you have any questions, feel free to drop me an e-mail.
Have a good day. Kind regards, Mia
We have different katas for experienced developers (a refactoring kata), people who are really frontend oriented, and want to show this off (a full-stack kata) and some basic TDD kata for starters.
Some numbers
When writing the blog, this process is in use for more than a year. We extracted the numbers for a single year.
Numbers for the consultancy firm
- 175 katas received
- 78 go for an interview (45% of katas received)
- 43 validated (25% of katas received)
- 15 hired (9% of katas received)
As you can see, we only collected exact numbers for people moving forward in the process, not about people dropping out. Some people dropped out anywhere in the process because they accepted contracts somewhere else, I estimate this to around 10% to 20%.
Numbers for the enterprise
- 130 CVs received
- 99 katas received
- 20 go for an interview (20% of katas received)
- 10 validated (10% of katas received)
There are also 35 people who dropped out by themselves:
- 16 were not available anymore because they accepted a contract somewhere else
- 2 declined when they received the test
- 16 didn’t reply at all after the kata was sent out
- 1 cancelled the interview that was planned
Evaluating our experiences
In general we are really happy with the results, but every approach has advantages and disadvantages.
The positive results
Positive results for the company and interviewers:
- Using a code kata gave us valuable information about technical practices the candidates use and values, and allowed us to really improve our hiring of technical people. There was little correlation between people having a good CV (on paper) and scoring good in the kata review. Some people with minimal experience did a lot better than people with 8 to 10 years of development experience. We started using katas mainly to find candidates that can help to improve technical practices, so we are really happy about these results.
- As current team members are involved in the kata review, they know better what they can expect of new candidates, and there is an immediate connection between the candidate and the team.
- Setting up a transparent and standardized process for hiring worked out very well. It provides a lot more clarity on what to do, both for candidates, HR, and interviewers. While we were actively involved in setting up the process, mails, katas, and the first tens of reviews, we got less and less involved and only do an occasional review or interview now.
- A lot more people got involved in the hiring. Because there is a standardized process in place, new people can pick up the process faster and help out during the reviews.
- There was more uniformity in the hiring process due to a transparent process and pairing or sharing the reports.
- The code review itself proved to be an excellent learning opportunity for reviewers. Pair-reviews lead to cross-pollination. Reading a lot of different solutions to the same problem gives inspiration. Writing reviews trains you in structuring your thoughts. Some reviewers like to get involved for that specific reason.
Positive results for the candidates:
- We received a lot of good feedback on our review process, both from candidates accepted and rejected. They especially liked the effort we spend to provide high-quality feedback.
- There is less bias when looking to code than CV screening or a classical interview. There is even less bias if it is fully anonymized. We did see a positive effect on the diversity of the team.
- Performing the kata at home is much closer to a real work situation than whiteboard exercises or live pairing during an interview. Some people react very differently during whiteboard exercises or live pairing under stress because it mainly requires the skill of thinking on your feet and not a carefully chosen, well researched, solution.
- Through this process, candidates can see what we stand for and what we believe is essential.
Observed drawbacks
We observed the following drawbacks
- Having a process with code katas leads to longer lead times. Both candidates and the company need to wait longer on a decision. Other companies who do a classical interview are often faster to make a contract offer.
- Some candidates do not like doing a code test. Reasons:
- They have lots of opensource projects or projects available to show. In that case, we looked to the project instead of requiring a kata.
- The can get an assignment with a simple interview, so why spend time on coding challenges?
- Candidates often prepare the kata in their own time without being paid, and the only thing they receive back for it is our feedback. Asking for a home assignment that takes four hours might not be possible or easy to do for people with a family or multiple jobs. Home assignments might filter out less privileged candidates and lead to less diversity.
- It is not well suited for real starters who never had any (development) job before.
- Sometimes several reminders were needed to get the reviewers motivated to start. Everyone is busy with other work, and reviewing a kata always feels like an extra on top of the rest.
-
Thanks to Matteo Pierro for being my partner in crime in setting these things up and reviewing this post. ↩︎
Mastodon
