It turns out --- I'm speaking for at least Erin and I, but I suspect to an extent Patrick too --- we don't have the stomach for the contingency recruiting model. Referring someone for an interview and watching them get turned down was, for me at least, about 80% of the psychic cost of applying myself and getting turned down.
Fuck that. It's a job that is possible to do and possible to do well and my life is too short (takes bite of ham sandwich) to spend it getting good at that.
So: I am still in love with the concept and still pursuing it, but I have fallen totally out of love with that particular business model and am happy to blowtorch it out of my life.
More writeup to come! Patrick's at Stripe now; Erin & I have more news coming. Ironically: expect more challenges from us, not less. :P
I met with Patrick for a couple hours this spring to talk about Starfighter and recruiting and such (I've been in recruiting for almost 20 years), and I shared some of the challenges the industry has faced. Patrick shared some anecdotes about his brief experience (at the time) on the recruiting side of the business, and it does take some time to get to the point where you don't take rejection personally. Recruiting is hard for a lot of reasons, and you do need to have a certain stomach for the industry. I'm not a huge fan of it either.
The advantage Starfighter had was that the most painful rejections for recruiters aren't necessarily the "no offer" at the end of an interview, but the "no response" to outreach - even when highly targeted.
Starfighter was designed to have a pipeline of qualified candidates who would conceivably be much more approachable than those who are cold-called/emailed by agency recruiters, not to mention the benefit of the names. Getting approached by Patrick or Thomas & Erin is much different for a candidate than getting approached by a recruiter with no name recognition. Even recruiters with a bit of name recognition don't get high response rates all the time.
I was personally rooting for Starfighter for several reasons, most of which to try and take recruiting in a bit of a different direction. I still hope the idea can be salvaged in some way, and wish Patrick and Thomas & Erin the best in the new endeavors.
That's really interesting! Our experience was: we didn't need to do any outbound sales work at all (our entire client base was cold inbounds), and we could close a contract with 1-2 phone calls. Our referrals I think with 1 exception? all got interviews.
But we're sourcing non-traditional candidates, and the --- you know what, I'm getting ahead of myself on a blog post I'm still tuning up.
In any case, that might explain what was so painful about the business for us: we had high frequency exposure to the worst part of it.
The outbound sourcing work is the worst part without doubt, but the game took care of that for you. That, and the fact that it was backed by people with good industry names, was the beauty of Starfighter. I told Patrick that you already had the two hardest parts covered - engaging with qualified candidates and establishing credibility.
You also had cred with the hiring folks, so that was taken care of as well - hiring clients were going to be easy to identify without much effort.
The hard part would be scaling it to some degree - Patrick and you/Erin wouldn't be able to talk to every candidate if you were going to make serious $, and then you'd have to hire actual recruiters (who are wildly unpopular), and then you're starting to approach the problem of being like everyone else (other than the game) if the recruiters behave like...recruiters.
If you ever want to discuss recruiting stuff, I'm easy to find.
Talking to every candidate (the ones that were loosely designated as 'mine') turned out to be my most favorite part of this whole deal. In some ways, even better than having an opportunity to write an AVR emulator. These guys are smart and they have interesting stories, and the enthusiasm they expressed for what they wanted out of their careers plucked my mom strings pretty hard.
I always knew that if we continued to grow, that this part would go away, but I would have struggled to keep this going as long as I could. Even if it meant less serious $. It doesn't seem unreasonable to expect that had we grown to the point where hiring "actual recruiters," it would have been impossible to find ones with the same mindset.
Thomas, were many starfighters you referred not getting offers? I love the idea of a (properly scoped) project/challenge in the hiring process and find myself profoundly worried when no code gets discussed in tech interviews.
I also really enjoyed stockfighter challenges. Between those six and my university's copy of Harris' Trading and Exchanges, I learned quite a bit. I bailed out on the AVR challenges though.
From a purely selfish viewpoint, I also want to work at places where the hiring process isn't really just whiteboard or gotcha questions, but instead actually has some consideration that, you know, demonstrating the ability to produce working useful code matters. Maybe I'm just nuts. The last interview I was in (for what people here would say is data science/engineering) involved a couple of big-O questions and two ill-specified SQL queries . . . sigh.
> Between those six and my university's copy of Harris' Trading and Exchanges, I learned quite a bit. I bailed out on the AVR challenges though.
I really enjoyed the Stockfighter levels too, and reading The Big Short / Flash Boys / Moneyball in preparation. The levels had the right level of difficulty & I kept feeling I levelled up (especially when optimizing levels, I'm still proud of my Level 5 score). I didn't enjoy the AVR levels the same way & bailed quickly on those.
I got spooked after someone posted on HN that their Starfighter referral had only bypassed the CV screen of the job interview, not programming or whiteboard challenges. That made me question the time I'd spent on Starfighter - I kept a time log in Harvest along the way, and it's about the same amount of time I'd spend on developing a new product to sell online.
I still enjoyed playing Stockfighter though, and it helped me sharpen my Golang skills.
I wonder whether there was some element of "those who have an inclination and ability to complete Starfighter may be insanely qualified from one set of screening criteria, but not from the criteria that resonates with people hiring employees".
In other words, maybe it turned out to be a better filter for self-starters who might be nascent entrepreneurs (reflecting the founders who created it), but sometimes those characteristics aren't what hiring companies are looking for when they have a specific role to fill.
Sometimes you interview people who would be outstanding in certain roles or for certain companies--but maybe not the one you happen to be currently working for.
I'm struggling here, but FizzBuzz is effective as a tool to weed people out, but FizzBuzz taken to the extreme doesn't necessarily equate to a tool to rule people in.
There was no element of that. The frustration was mostly 'the client's hiring process is a mess.' On our part, the one piece of work we needed to do to sell Starfighter better was to write some 'accessible prose' on 'the practical applications of having cleared level six' -- this should have been more of a priority, and it wasn't. I can look back and see the opportunity I had to do it, myself, and am bummed that I didn't get it done. Lesson learned; it will be part of every challenge we release going forward.
I realize this is a completely selfish request, but is there a chance that the challenges and games you made will continue to be available, or self-hostable? :-D I now regret not having participated in the CTFs that you three built.
Given that you and Patrick have spent the last several years advocating billing by value instead of by time/skills, why did you decide to start a business trying to convince people that the best way to get paid more was by improving their tech skills? Granted learning stuff is often a good thing and there is clearly some connection between skills and value so I don't think there was anything wrong with the concept itself, but the fact that it was both of you running it seemed at the least off-brand.
I remain confused by all the weird things people think we were trying to do with that business. "Trying to convince people that the best way to get paid more was by improving their tech skills" is one of those things. Whatever it is that got published that created this impression, I disavow.
I feel like, doing recruiting through work sample challenges is basically 80-90% of my whole brand.
The game wasn't at all about learning new skills or improving existing skills. It was a means to demonstrate existing skills, to Starfighter first and then to employers.
Actually: I'd like it to be more about learning new stuff, or at least learning that you're already good at new stuff.
What I want to find are people who are spooky good at specialized, high-status tech skills. HFT is like that. So is reversing. So are like 10 other specialties I can think of.
My thesis is that the training requirements for these things are way overblown.
> Referring someone for an interview and watching them get turned down was, for me at least, about 80% of the psychic cost of applying myself and getting turned down.
Wow it actually makes a lot of sense. Especially since you seemed to handpick candidates and vouch for them from a technical point of view.
I wonder if classic recruiters have this problem as they (most of the time) just bet on the number and are not invested personally.
The vast majority of tech recruiters have no actual idea of how skilled their employee-side client is and are just keyword correlation machines that happen to be made of meat.
As such I doubt it impacts them much, if at all, beyond the obvious disappointment of not getting the salary cut, which they surely mitigate by just throwing candidates until one sticks.
tptacek, patio11 and Erin -- I'm disappointed re: Starfighter, I was rooting for you.
I ran 0x41414141.com [1] for about 4 years. It was a set of CTF-style challenges of increasing difficulty with submissions via "secret" email addresses, a sendmail autoresponder and me - operating under the pseudo cavalier@0x41414141. It's singular goal was recruiting.
At the time, I was working with a totally badass team in a firm with a decidedly not-sexy brand for us security-types. We were hiring as fast as we could and _always_ had positions open. When we got someone good into the pipeline, we generally had no problem hiring them - the team and the work made that easy - but our candidate funnel was just too damn small. We had exhausted our referral pool, we were bringing on as many fresh grads as we could train, but it wasn't enough to keep up. We needed to broaden the funnel. Enter 0x41414141.com.
It was completely and totally unsupported, unrecognized and unreported through the official corporate recruiting channels. It was me, on my own time and my own dime with the quiet support of a few other guys in the shop. It had a singular purpose: to meet like-minded hackers and see if there was any chance they'd be interested in joining our merry little band of hackers.
Within the first year, I established the equivalent of a "repeatable sales process." There's the natural stages of the challenges themselves, but then the more delicate stages of transition:
(1) from the impersonal interaction with the service to the personal email exchange with cav,
(2) ...to a serious discussion about a new job
(3) ...to an IRL phone conversation
(4) ...to an open and honest talk about who we were, what we're doing and next steps
(5) ...to getting introduced to an actual corporate recruiter
Of course, like any "repeatable sales process" each stage is not only the continued progression but also qualification. The funnel was disappointingly ruthless, and that was just to get an intro to the recruiter. In any given year, I had:
(1) thousands of folks start challenges,
(2) several hundred make notable progress,
(3) traded personal email with dozens,
(4) got serious with 10s,
(5) interviewed a few,
(6) hired a couple
In the end, the results just didn't justify the effort.
To be clear - I loved running 0x41414141.com. I met a lot of great hackers and even hired a few. Many are HN'ers. (hi guys!) Many of the "security community rockstars" stumbled across the site and ran through the challenges as a fun exercise on a Sunday afternoon. I also watched - and coached - many clear newbs through the process, and was impressed with their grit, creativity and determination. I even hired a couple complete newbs, including a female, recent PoliSci grad working as a nanny who turned out to be an OUTSTANDING reverse engineer, when given the right support and environment to grow. She single-handedly kept my enthusiasm for running 0x41414141.com up for at least two years, even when the resulting hiring numbers made it clear it wasn't worth it.
I was really, really, really hoping Starfighter could "level up" the initial volume into that funnel, maintain (or improve!) the rough filtering percentages and have the resulting "hired" volume be higher and turn CTF recruiting into a sustainable business model. I'm disappointed it didn't work.
P.S. - this is the first time I've associated my real self with 0x41414141.com and the cavalier alter-ego. Congrats HN, you just got the scoop for this tiny little corner of history. While we're here...
Hello Internet, this is cavalier. If we connected
via that moniker, please reach out and connect
with my _real_ one. deets in my profile.
It's clear tech recruiting needs disrupting. We're still too dominated by traditional procedures that provide poor results. Folks with our background know in our guts this CTF-style thing has potential, we've just got to work out how to make it sustainable.
Would love to swap notes sometime. My wife is angling for an excuse to visit Chicago, but seeing as how winter's coming maybe y'all should visit Austin!
You went public?! This is madness. Does this mean I can use my 'volatile' moniker again? :)
As one of the guys hired through this (who later helped, though 99% of it was cav), if you guys do catch up and trade notes, I would /love/ to be there. We're actually thinking of building something similar at my current company, but focused on a bit of a different type of hire.
Totally Speculating but I bet it was that software companies have enormously high false negative rates. Each company has it's own set of weird totally random interview biases. So Thomas kept on finding awesome recruits who were getting turned down for totally dumb reasons.
I imagine it's like the HomeBrew situation every day except 10 times worse because you recommended the candidate and the interviewer is your client. And unfortunately it's hard to make a business of telling your client they're doing it wrong.
This is all compounded by the fact that the devs who benefited most from their service were not Christopher Hemsworth looking Phd, ex-googlers who could get an interview with a phone call. Then come in with so much positive bias that as long as they didn't poo themselves during the interview they'd get hired. It was for people who were extremely skilled, but lacked the right pedigree or interviewing skills to be able to get the jobs they were very qualified for.
》 otally Speculating but I bet it was that software companies have enormously high false negative rates. Each company has it's own set of weird totally random interview biases. So Thomas kept on finding awesome recruits who were getting turned down for totally dumb reasons.
Fuck that. It's a job that is possible to do and possible to do well and my life is too short (takes bite of ham sandwich) to spend it getting good at that.
So: I am still in love with the concept and still pursuing it, but I have fallen totally out of love with that particular business model and am happy to blowtorch it out of my life.
More writeup to come! Patrick's at Stripe now; Erin & I have more news coming. Ironically: expect more challenges from us, not less. :P