Interview with a DevOps

Introduction

Conducting interviews is not an easy task, especially if we are considering IT (Information Technology) that has so many sub-departments (project management, development, infrastructure, support, hardware and others).

I have had the opportunity to participate in distincts hiring process (as candidate and as interviewer) and recently an ex-colleague created a startup and wrote to me asking if I can help in the process to hire a DevOps.

So, here are some ideas to use:

What you should avoid for a live technical interview

Live technical interviews usually take 60 minutes and it can be very stressful for candidates (specially if we are talking about live-coding or whiteboard interviews), making them more propenses to forget softwares names/parameters, communication problems and other minor and temporal mistakes.

Some people believe that this 60 minutes is a good opportunity to see how the candidate works under pressure, but, did you remember the first time when you went to meet the family of your partner ? Do you consider that you have the same conversations and relationship with them now as in the first meeting ?

So, let us start checking what we should avoid:

Creating an imaginary environment

Is amazing how many times the technical interviewer start with:

“To this exercise, please imagine a line between point A and point B with …”

or something like

“Think about 03 datacenters with some web servers under a high network traffic load.”

Does not matter how many details the interviewer can describe, but imagination always will be different between people, for example, “high network traffic load” to someone in Akamai is very different from someone who has been working in a startup.

You can workaround this by showing a diagram, some logs files, part of the code or something more concrete to allow the candidate to have the same information as you and not based on abstract ideas.

Asking complex questions

The interviewer send a link to the candidate asking to connect to a shared whiteboard system and then the interviewer request:

“Please, using this whiteboard create a regex expression to catch the IPv6 output from the command ifconfig without consulting the ifconfig output, the man page or internet.”

Maybe some interviewers believe that making this kind of question will filter the good and bad candidates (or maybe they will feel good to know something that the candidate does not remember under a specific scenery), but except if your systems are in a Dark Site² and you are hiring a “regex master” that can memorise something like this:

(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))

You are only creating a bad and no realistic working environment, because almost all the time is necessary to access tools (internet, manuals, contacts networking, KB, etc) not only to remember this kind of stuff, but to double check your configurations.

60 minutes are not enough time to fill all your expectatives

Recently, a company asked me to do a real configuration in 02 days with the developer’s team not only to evaluate my technical skill, but also to check how we are going to work together. Independently if the candidate is a Junior, Senior or Supreme Super Saiyan, any new employer will need time to create synergy with the teams, read the documentation to understand how the company works and make productive tasks.

Now, let’s go to check some ideas that your should consider

One of the responsibilities from DevOps is to create, maintain and documentate Infrastructure as Code(IaC) and deployments (CI/CD) using some source code management (SCM). So a better approach to make a technical interview is to create different and progressives offlines tasks to evaluate the technical skills, the code documentation, the problem troubleshooting and communication. Leaving the live technical interview to discuss the tasks made on the offline tests (see below), short technical questions and to an open conversation.

You can create many offline and progressive technical tests you want, but remember that candidates have other interviews and jobs to do too and spending 2 weeks in a series of tests is not always welcome (especially if you are not going to give some feedback later).

So, be polite and inform the candidate how many tests will be, the content and objective of all of them and how long time you believe it will take.

Create offline tasks/tests

The offline and progressive tasks or tests are great idea because:

• Allow candidates to work with time, stressless and using the tools that they are comfortable to use.
• The candidate can deliver excellent code, documentation, graphics and others using the time that he/she believes is necessary to do it, avoiding the bad results because the online interview didn’t give enough time to polish the test.
• Offline tasks can be dependents and only allow the candidate to proceed if the previous tasks have been finished. This can save time to both (interview and candidates) If the candidate has some blocker and decides to don’t go forward in the process.

Let us go to use this DevOps Interview test repository as a basic example. On this repository we have created a basic Django application and asked the candidate to do some tasks, like:

Note: I will given only 3 posibles responses on each topic because I will still use a similar repository (or maybe the same) in the future to conduct more interviews.

Docker and Dockerfile

The actual Dockerfile in this repository is functional (I hope), but it have a very basic configuration and we are giving the candidate a lot of ways to improve the performance and security on it using some basic settings like:

1. Implementing multi-stage builds.
2. Choosing a better image (alpine is not a good choice to python).
3. Using a non-privileged user.

Kubernetes / K8s

The possibility of creating k8s configuration may take a bit more time doing it from scratch, but if the candidate has experience, copy and pasting and adapting from a previous project should not take much time.

Important points to consider is to check how to candidate will manager tasks, like:

1. Are the candidates going to use raw yaml ?
2. How have been managed environments like test and prod (helm / kustomization) ?
3. How automatization tasks like the creation and data load on the DB have been done (InitContainer / Jobs / Docerfile) ?

Note: In my case, I know that a lot of companies use helm, but I hate it and usually I create the configuration using kustomization and helm and an explanation why one is better than the other.

Documentation

IMHO, one of the most time consuming parts of any system creation is the documentation, but it is fundamental to the business continuity and is a requirement of any certification process. At this point of the test, the candidate has the opportunity to show how good she/he is at documenting his job. Interviewer should pay special attention to:

1. Is the documentation simple and concise ?
2. Has the candidate added comments to old code or only the one that he has created ?
3. Check if the documentation is proper formatting as a plus.

Others highlights

As mentioned before, this repository has an old code and basic configuration, giving a lot of alternatives to the candidate to show his knowledge and deficiencies. If you are going to make something similar, you should consider the capacity of the candidate to:

• Detect old and vulnerable code (github and any other vulnerability scanner will alert you that these versions are obsolete).
• Use of password and tokens on SCM is a very bad idea and candidates should at least mention that she/he is going to put some sensitive information on the repository to make the test simple or give a better alternative to avoid it.
• You can leave an old Pull Request from another candidate (or a fake one) in the test repository with some configurations. The idea is to check what the actual candidates are going to do (tell you about it, use the code, ignore, etc).
• As mentioned, some candidates do not use or like a specific tool or configuration, so check if the candidates are creating alternatives or better configurations to fix the problem.
• Security and performance are fundamental. Always welcome candidates that can detect, comment and fix a problem that will expose the database to the internet.

To be written (maybe soon)

• Creating ephemeral environment (terraform, ansible, k8s deployment)
• Final live technical interview (ideas, questions and others)