Influencing Nerds

Hiring, it is said, is the most difficult thing one can do as a manager. Managers who are good at hiring can’t tell you how they do it, and the literature is filled with tales of the horror that befalls managers who do it badly.

I don’t know exactly why it should be so hard to determine whether a candidate will fit well in your team and become a net addition, but it clearly is, and the more senior the person is the harder it gets. There is every possibility that the person who seems so easygoing and willing to learn in an interview, will turn out to be difficult and set in their ways once they get on the job. It is also nigh on impossible to evaluate someone’s skill as an engineer from a test or a transcript or a few code samples. This is especially true in an open-source context, where the definition of “skill” includes everything from “Writes the right code cleanly and quickly” to “Is able to convince unknown and probably cantankerous peers in the community that this patch should get in.” These two abilities, in particular, rarely appear in the same person, at least not when they walk in my door.

(If you are among those who believe that in open source the best ideas always win, I have another conversation to have with you, but for now you can read this post).

You would be right to ask me at this point why I think I’m so smart — what legendary track record do I have in hiring that allows me to pontificate on it? It’s a worthwhile question. I feel like I’ve built a few solid teams over the years, and I’ve taken them through some godawful wrenching changes — “We’re discontinuing your product because we acquired a startup with a competing product, so you all have to learn a new language and work on something completely different in a new community” — multiple times with a very low attrition rate. But I can’t claim any special ability to evaluate people in an interview and I doubt my ability to select the right person is any better than anyone else’s. Certainly, I have hired some great people, but… well, this is public, so I shouldn’t be any more specific than to say I have made a few mistakes too.

I do have one secret, however, and it is summed up in some very bad advice from a song from the ’50s musical Guys And Dolls: “Marry The Man Today.” The song holds that women are better off marrying an imperfect man and then reforming him into what they want him to be, than holding out for the perfect mate to come along. My mother, who had some experience with this theory (sorry Dad), told me early on “This is very very bad advice! Do not follow it!” She is right, of course — in the context of a marriage, it’s a terrible idea, a recipe for unhappiness or worse. In the context of a well functioning team, though, with a strong and healthy culture, it’s not necessarily wrong, especially if you are in the position we all find ourselves in now of having to recruit inexperienced folks straight out of college.

What does this mean in practice? It means I tend to interview people, especially young people, on non-technical criteria. Do they speak clearly and thoughtfully? Do they appear to be self aware, perhaps with a refreshing trace of irony? Did they take the time to find out who I am, what I do, what Red Hat does? If I’m lucky enough to find someone who meets these criteria, I will usually hire them, whether they have any experience in the area I’m hiring for or not. I can teach them to code, and I have a whole team with a whole culture that will teach them how to work the way we work, but I probably can’t teach them to give a crap.

So, instead of spending your time interviewing hundreds of candidates, spend your time building an organization with a culture that will make them into what you need. Hire the developer today, I say, and train them subsequently.

It is, of course, very difficult to know anything for certain. Authorities as widely separated as Werner Heisenberg and Laurence Sterne have written at length about the impossibility of measuring everything (in the first case) and the impossibility of fully understanding anything (in the second). This poses problems for engineers who are in the business of making decisions about precisely what to do based on a set of supposed “facts.” It poses even more problems for managers like me, whose faculties are so withered that telling fact from fiction is usually an exercise in blindfolded dart-throwing.

A good manager, faced with her inability to distinguish engineering fact from engineering myth, builds a network of people she can trust to do that for her. Naturally this presents all kinds of problems with reinforced bias, institutional inertia, and so on, but it’s honestly the only recourse for someone trying to manage a technical team without having the time to get down into the details and see for herself. In my case, I rely on my own sense of the self-awareness and even self-assuredness of the engineers I talk to to try to get a picture of the truth. The most reliable information generally comes from the engineers who are more self-aware than self-assured; put another way, never trust anyone who is absolutely certain they are correct.

You might ask, why, as a manager, do I even need to understand the truth at all? Can’t I just delegate that to some senior people, let them direct everything, and sit back and collect my paycheck? To some extent the answer is yes, I can and I should. Unfortunately, as someone trying to prioritize which research projects Red Hat focuses on, I do actually need to know enough about the facts and what’s coming to make a reasonable judgement. This is especially true because I am, more or less out of necessity, the only one with the full picture of everything we’re doing. I think what I’m saying is that it is impossible to understand the big picture well, while also being aware of all the details. It’s not just a matter of mental capacity. Creating a big picture requires eliding some details, requires approximating things, and that activity is incompatible with knowing for certain all the details of the case.

Here’s a real example of what I’m talking about. A very senior Red Hat engineer, whom I trust implicitly partly out of awe and partly because I know a whole bunch of other people who also trust him, has taken a strong position that we are approaching the end of the era of the general-purpose CPU. Without getting too technical, if his position is correct, then we are in for some massive upheavals in the business of computing in the next few years. Everything from the design of the hardware we buy to the way we build and distribute software will need to change in significant ways to accommodate a new world of purpose-built processors.

Another very senior Red Hat engineer, whom I also trust because he has forgotten more about actual processor engineering than I will ever know, believes this is bunk. His position is that we are still ten years out from the theoretical limit of processor scaling and that we have been through cycles of people wrongly predicting the end of the general-purpose CPU multiple times. Every one of those times, the processor manufacturers have tricked their way through to the next generation and the major change my first source is anticipating has not appeared.

The facts in this case are particularly important because I am about to schedule a half-day, very public discussion at the Red Hat Summit around the end of the general-purpose CPU, and if it turns out to obviously be bunk between now and then, I’m going to look foolish along with Red Hat.

I’m not sure there is a real solution to this. Sometimes I am going to back the wrong horse, and sometimes I’ll pick the right one and look like a genius. I guess the important thing is not to go too long on any single position unless you’re prepared to lose big, no matter how much you trust the person you’re relying on. The only alternative is to dive into the weeds on every decision and try to understand it all yourself, which is not only exhausting but also ineffective as described above.

I have one other point on this. If you read the above carefully you will notice that all the work I have to do in reaching a decision is social. I have to decide whom I trust, whether they have a hidden agenda (whether they are aware of it or not), their degree of bias and in which direction, and so on. If there is a technical component of this decision at all, it is well outweighed by the social component. I believe this is typical of the decisions managers and even engineers make. So, the next time someone tells you that the right way to come to an agreed solution to a problem is to have a rational debate on the merits, tell them there is no such thing. The best you can do is listen carefully, take a deep breath, and throw for the bullseye.