Hiring folks in leadership roles is very challenging. One difficulty is that almost every candidate who gets to that level is generally good at communication, charismatic in their demeanor, and well trained to say the right things to get the job. Nonetheless, there is a huge difference between how people interview and how they deliver once they get hired. In my interactions with folks in Director+ roles, I generally find a very strong dichotomy, with a few individuals being superstars and a large majority having some level of non-pathological narcissism, where they got to a leadership position primarily because they knew how to play the game of visibility. Given all this, how do you break the level of information asymmetry in interviews and hire exceptional leaders for technical roles?
Do they have an understanding of the P&L? A good Director+ is able to look at a profit and loss statement and translate that into a technical direction. For example, a Head of Data could look at a financial statement and immediately ask questions:
It looks like our cost of acquisition is high. Who is leading that work? What optimization has been done to make that more efficient? Nobody? I would like to assign a data scientist who specializes in optimization to work on it. It looks like our infrastructure costs are growing at a higher rate than revenue. What are the drivers of that cost? Who is responsible for infra? Nobody? I would like to personally lead a Cloud FinOps initiative. It looks like our runway is unhealthy. Is it time to make tough decisions? It looks like our business is growing, but operations work is not scaling. Who is responsible for automating that? If you give a candidate a situational interview where you go over a simplified P&L and they are not able to ask questions and tie that to technical solutions, it should tell you two things:
They will have a hard time thinking of initiatives in terms of revenue, cost, and profitability drivers, so they’ll have challenges communicating with business partners. Instead, they might still be stuck thinking in terms of project-based work or lofty initiatives of unknown value to the business (e.g., “becoming self-serve” or “becoming AI leader in the industry”)
That they don’t have experience “being in the room.” In almost all companies there is a room where line-by-line item discussions around the P&L are being addressed. Almost everyone part of the M-team is in that room. Someone high up in Data is in that room together with a bunch of analysts. If your candidate has no P&L experience, they were likely not in that room.
Backchannels I first read about backchannels from Ben Horowitz’ book The Hard Thing About Hard Things and I saw them used to alleviate information asymmetry. I know that backchannels might be perceived as unethical, and I think there are good reasons why they should not be done for non-critical roles. Nonetheless, I think you owe it to your team to protect them from a bad hire. For example, I once found through the grapevine that one of my previous bosses was actually let go from his previous job due to performance reasons. Suddenly a lot of things clicked: the absenteeism, the lack of care, the lack of vision, the empty promises. In the back of my mind, I found it very unfair that 150+ people at the company had to suffer because of a bad hiring mistake that could’ve been avoided if someone did a proper background check. This is why for very senior leadership roles I’m supportive of backchannels.
The “lunch test” My initial exposure to interviewing Director+ roles was when my previous company tried to hire a Head of Research. I was a junior data scientist and the technical interview was done together with someone who had a mid-level role in the Design team. Literally within one minute of our introductions, the Director candidate tried to commandeer the interview and acted with the attitude of “I can’t believe I’m interviewed by a bunch of interns.” During the round table, everyone except us had a polite and professional interaction. This was not an exception and about 10 out of ~35 director+ candidates I had the chance to interview behaved this way.
I call this the “lunch test” because it echoes back to the idea of taking a prospect to lunch and seeing how they behave toward folks who are in a position of less privilege or power. The reason why this test is helpful is that it can flag candidates that don’t approach leadership from a position of a servant leader. If your hiring team is all made up of senior folks, you’re not likely to get this signal. I generally find this test to have a higher fidelity when it is administered in technical interviews and not in values or culture-based interviews where the candidate is primed to give the right answer.