Disturbingly Flawed Interview Processes — A Cri de Coeur

The evidence against modern engineering interview practices is not new, not contested, and not subtle. It spans eighty-five years of peer-reviewed research, multiple academic disciplines, and includes the work of a Nobel laureate. It has been published in the most prestigious journals in psychology, behavioral economics, and organizational science. It converges on a conclusion that most hiring managers will spend an entire career avoiding:

The process does not work. It has never worked. And the data proving it does not work has been publicly available for decades.

This is not a matter of fine-tuning. The three interview modalities that dominate modern software engineering hiring — unstructured conversation, behavioral questioning, and algorithmic puzzle-solving — are not slightly miscalibrated instruments in need of adjustment. They are foundationally flawed, measuring the wrong constructs, and disconnected from the actual work. The research does not suggest they could be improved. It demonstrates that they are broken by design.

I have sat in these interviews, on both sides of the table. I have been the candidate who watched an interviewer form a judgment in the first ninety seconds and spend the remaining forty-five minutes performing a ritual both of us knew was already decided. I have been the interviewer who caught himself constructing a narrative about a candidate from fragments that, upon honest reflection, carried no predictive signal at all. I have sat in debrief rooms where the word “culture fit” was deployed with the precision of a surgical instrument and the accountability of a shrug.

This post is not about whether the process is broken. The cited works settle that question. This post is about why it persists, who it serves, and what it costs an industry — one that considers itself data-driven — to systematically exempt its most consequential human capital decisions from any empirical validation whatsoever.

The Triad of Failure

Modern software engineering interviews overwhelmingly draw from three modalities. Each has been studied extensively. Each has been found wanting. And each fails in a way that is instructive — because the failure modes are not random. They are structural.

The Unstructured Interview: Worse Than Random

The traditional conversational interview — the format in which an interviewer sits across from a candidate, asks whatever questions feel right, and forms an overall impression — is the most thoroughly discredited selection method in the research literature.

Five landmark studies spanning eighty-five years converge on a finding that should have ended the practice decades ago: unstructured interviews frequently perform worse than random selection from a qualified candidate pool. Not merely “no better than random.” Worse. The interview actively degrades the quality of hiring decisions compared to simply reviewing a candidate’s existing credentials without ever meeting them.

The mechanism is well-understood. The human brain is an extraordinary narrative engine. When an interviewer sits across from a candidate, the brain constructs a coherent story about that person — their capability, their potential, their fit — even when the raw material is noise. Researchers at Yale and Carnegie Mellon demonstrated this by having candidates answer questions at random. The interviewers reported the same confidence in their assessments as those who conducted genuine interviews. The narrative machinery ran at full power on an empty input.

Daniel Kahneman, who would go on to win the Nobel Prize, documented this as the “illusion of validity” — an intense, visceral sense of insight that persists in the complete absence of predictive accuracy. The interviewer feels like they are perceiving something real. They are not. And the feeling is so powerful that exposure to contradicting evidence does not diminish it.

The practical consequence is what the research calls the dilution effect: irrelevant information encountered during an interview does not simply sit alongside diagnostic data. It actively weakens it. An interviewer who discovers a shared hobby with the candidate will, on average, weight that candidate’s technical qualifications less heavily than a reviewer who never encountered the irrelevant detail. The interview introduces noise that destroys signal already present in the candidate’s record.

The Behavioral Interview: The Wrong Side of the Table

The behavioral interview was the industry’s corrective. After the research demolished unstructured conversation, organizations adopted the “Tell me about a time when…” format — standardized questions, the STAR method, structured scoring rubrics. The assumption was intuitive: if the problem was the interviewer’s unchecked subjectivity, then constraining the questions and anchoring them in past behavior would fix the signal.

The research shows it did not.

Where unstructured interviews allowed the interviewer to construct a false narrative about the candidate, behavioral interviews shifted that construction to the candidate. Over 90% of candidates engage in faking behavior during behavioral interviews — embellishing accomplishments, constructing fictional experiences, tailoring responses to match perceived job requirements. The taxonomy of deception is not a single behavior; it is an ecosystem with eleven documented subfactors.

The critical finding: faking behavior is positively correlated with receiving job offers but shows low-to-zero correlation with subsequent job performance. The interview reliably selects for the skill it inadvertently tests — the ability to perform under the specific conditions of the interview — and that skill has no demonstrated relationship to the work.

Huffcutt’s 2011 meta-review quantified the damage: the mean correlation between interview ratings and impression management constructs was approximately twice as large as the correlation between interview ratings and job-related competence. The behavioral interview is not a noisy instrument. It is a precise instrument pointed at the wrong target. It measures, with remarkable consistency, a dimension that is orthogonal to the job.

And the STAR method — now taught in every career coaching book, every MBA program, every university career center, and on thousands of YouTube channels — has made the format so transparent that it evaluates preparation quality, not work ability. When the test’s methodology is universally known, the test ceases to measure competence and begins to measure coaching. The guardrails were put on the wrong side of the table.

The LeetCode Interview: Theater of the Absurd

When a LeetCode interviewer asks a candidate to “implement quicksort,” the candidate is being asked to produce a toy.

The production sorting implementation that ships with the Java platform — the one a competent engineer would invoke with Arrays.sort(a) — is a 4,430-line adaptive system containing seven distinct algorithms, thirteen empirically derived thresholds, IEEE 754 floating-point compliance, JVM intrinsics, and parallel execution support. It was authored over fifteen years by four engineers whose combined contributions include the invention of the dual-pivot quicksort algorithm, the most influential book on practical programming, the design of the Java Collections Framework, and the concurrency library that powers every JVM in production.

No candidate will reproduce this in forty-five minutes. No candidate should. The value of a senior engineer is not in memorizing sorting algorithms. It is in knowing that Arrays.sort() exists, that it has been optimized by people who spent careers on the problem, and that calling it is the correct engineering decision in virtually every real-world context. Reimplementing it would be an act of engineering malpractice.

For the overwhelming majority of professional software engineers — the people building REST APIs, event-driven architectures, domain models, and data pipelines — LeetCode exercises test rote memorization of textbook algorithms in a context wholly disconnected from the work they will ever be asked to do. The assessment does not predict job performance. It does not measure engineering judgment. It is theater.

The Shared Defect

All three modalities fail for the same structural reason: they measure performance in the interview rather than performance in the role.

The unstructured interview measures the interviewer’s ability to construct a narrative. The behavioral interview measures the candidate’s ability to perform one. The LeetCode interview measures the candidate’s ability to recall textbook algorithms under artificial constraints. None of them measure the thing they purport to measure — whether this person can do the job.

The interview is a stage. The industry has mistaken the audition for the work. And unlike an audition, nobody ever checks whether the person who performed well in the room can perform at all in production.

“Culture Fit” — The Bias With a Brand Name

Of all the phrases in the modern hiring lexicon, none does more damage with less accountability than “culture fit.” It is the unlabeled wire in the circuit — carrying current with no specification, no rating, and no fuse.

The Definitional Void

Ask ten interviewers at the same company what “culture fit” means and you will receive ten different answers, none of which will be operationalized, measurable, or falsifiable. One will mean “communicates the way I do.” Another will mean “went to a school I respect.” A third will mean “I could see myself getting a beer with this person.” A fourth will mean something they cannot articulate but feel strongly.

A criterion that cannot be defined cannot be evaluated. A criterion that cannot be evaluated cannot be challenged. That is not a flaw in the implementation of “culture fit.” It is the mechanism by which “culture fit” operates. The phrase persists precisely because it is impervious to scrutiny. It is a box on the scorecard with no rubric, no anchor, and no accountability — and it routinely overrides boxes that have all three.

The Laundering Function

“Culture fit” is the vehicle through which bias enters the hiring evaluation without ever being named.

Demographic affinity. Class signaling. Educational pedigree. Communication style preferences that correlate with native language, neurodivergence, introversion, and socioeconomic background. These inputs arrive at the debrief as legitimate evaluation criteria because they have been deposited in a box with a professional-sounding label. The interviewer does not say “I preferred the candidate who reminded me of myself.” The interviewer says “I had concerns about culture fit,” and the room nods, and the scorecard is updated, and the decision is made with the appearance of rigor and the substance of prejudice.

This is bias laundering. The input is preference. The output is a hiring recommendation. And the transformation in between is a phrase that nobody in the room is required to define, defend, or examine.

The Dilution Effect, Industrialized

The research on unstructured interviews documents the dilution effect — irrelevant information does not sit passively alongside diagnostic data; it actively degrades it. “Culture fit” is the dilution effect scaled to an institutional process.

It is the shared alma mater. The mutual enthusiasm for a sports team. The candidate’s fluency in the social dialect of the interviewer’s peer group. Each of these is a piece of information that has zero predictive validity for job performance — and each actively competes with, and routinely displaces, evidence of actual technical competence. The research is explicit: adding irrelevant positive information to a candidate evaluation does not leave the evaluation of relevant qualifications unchanged. It diminishes it. “Culture fit” is the formalization of that distortion.

The Cloning Problem

When “culture fit” means “people like the people already here” — and in practice, it almost always does — the process is not selecting for culture. It is selecting for homogeneity and calling it a value.

The literature on homogeneous teams is unambiguous. They are more comfortable. They are also more brittle, more susceptible to groupthink, and less capable of the cognitive diversity that drives innovation and guards against systemic blind spots. An organization that optimizes its hiring process for “culture fit” is optimizing for the comfortable reproduction of its existing assumptions, its existing blind spots, and its existing failure modes. It is building a team that will agree with itself efficiently and challenge itself never.

For the engineers reading this: you know the moment. The debrief where “not a culture fit” was spoken with the gravity of a diagnosis, and you understood — with perfect, wordless clarity — what was actually being communicated. For the hiring managers: consider whether you could define your “culture fit” criteria under oath, in a deposition, with a plaintiff’s attorney asking follow-up questions. If the answer is no, you do not have a criterion. You have a feeling. And you are making career-altering decisions on the basis of it.

The Mirror Problem — Interviewing as Self-Portrait

There is an explanation for the persistence of broken interview processes that the research literature does not address directly, because it is not a question of psychometrics. It is a question of incentives, identity, and power.

Many interview processes are not designed to evaluate the candidate. They are designed to validate the interviewer.

The Narcissism of the Puzzle

When a senior engineer selects a LeetCode problem for an interview — one they find particularly elegant, one they solved recently, one that tests a technique they personally value — they are not designing an assessment instrument calibrated to the requirements of the role. They are curating a reflection of their own technical identity.

The problem selection reveals more about the interviewer’s self-concept than about the position’s demands. “I ask hard algorithm questions” is a statement about the interviewer, not about the job. It signals a self-image: I am the kind of engineer who thinks about algorithms. Whether the role requires that kind of thinking — whether the candidate will ever, in the course of the actual work, encounter a problem that resembles the interview question — is a consideration that rarely enters the selection process. The question is chosen because it feels like what a serious technical interview should contain. And that feeling is a mirror.

Survivorship Bias as Institutional Memory

Interviewers who were hired through a particular process will defend that process with a conviction that is disproportionate to the evidence — because questioning the process means questioning the legitimacy of their own selection.

“I went through this gauntlet. I succeeded. I am a good engineer. Therefore the gauntlet identifies good engineers.” This is not reasoning. It is survivorship bias wearing the costume of institutional knowledge. The logic collapses the moment you consider the population it cannot see: the engineers who were filtered out by the same process and went on to build successful careers elsewhere. They are invisible in this calculus, because the system that rejected them never tracked what happened next. The gauntlet does not validate itself. It simply never checks.

And so the process replicates. Each generation of engineers, having survived the interview, administers it to the next generation — not because the evidence supports it, but because their own presence on the team serves as the only proof they need. The process selected me, and I am good, so the process must be good. It is a closed loop that admits no external evidence and generates no falsifiable predictions. It is, in the precise sense of the word, unfalsifiable. And unfalsifiable beliefs, in any domain, are not knowledge. They are faith.

The Power Asymmetry as Compensation

There is a dimension of the interview that is rarely discussed because naming it is uncomfortable: for some interviewers, the evaluative authority of the interview is itself the reward.

The ability to judge. To gatekeep. To sit in a room and determine whether another professional — often one with comparable or greater experience — will be granted or denied employment. This confers a status that the day-to-day work may not provide. For an engineer whose role offers limited authority, limited visibility, and limited control over outcomes, the interview room is a context in which they exercise unilateral power over another person’s livelihood.

This is not universal. Many interviewers approach the responsibility with genuine care and intellectual honesty. But the incentive structure is real, and it is worth naming: the current system provides psychological rewards to the people who administer it, and those rewards are independent of — and often inversely correlated with — the system’s effectiveness at identifying talent. A process that confers status on its administrators will be defended by its administrators, regardless of whether it serves its stated purpose.

If reading those paragraphs produced defensiveness rather than recognition, that is worth sitting with. The strength of the reaction is, roughly, proportional to the accuracy of the description.

The Metric That No One Tracks

If the purpose of an interview process is to identify candidates who will perform well in a role, then the only metric that validates the process is a qualitative assessment of post-placement performance over time.

Not interviewer confidence. Not panel consensus. Not the warm feeling in the debrief room. Not the conviction that the candidate “nailed” the system design round. Post-placement performance — how the person actually performs in the actual job over actual time. That is the output variable. Everything else is an internal narrative about the output variable. And the distinction between the two is the distance between science and storytelling.

The Data That Exists But Is Never Collected

The components of this measurement are trivially available to any organization with a performance review cycle and an applicant tracking system. The questions are straightforward:

• Which interviewers’ “strong yes” votes correlate with high performance ratings at the six-month and twelve-month marks?
• Which interviewers’ “no” votes filtered out candidates who were subsequently hired elsewhere and succeeded?
• Which interview modalities — behavioral, technical, system design, take-home — have the highest predictive correlation with on-the-job outcomes?
• What is the false-negative rate of the overall process? Of individual interviewers?

The data exists. The joins are simple. The analytical methods are undergraduate-level statistics. The compute cost is negligible. And at the overwhelming majority of organizations — including those that would describe themselves as data-driven without a trace of irony — nobody runs the query.

The Omission Is the Tell

In any other domain of professional practice, a systematic refusal to measure the output of a resource-intensive process that directly determines organizational capability would be considered negligence.

In manufacturing, you measure defect rates. In medicine, you measure patient outcomes. In finance, you measure portfolio performance. In software engineering, you measure uptime, latency, error rates, deployment frequency — you instrument everything. You A/B test button colors. You monitor p99 response times in real time. You would not ship a feature without metrics, and you would not keep a feature whose metrics showed it was failing.

And yet the process by which you select the people who build all of that — the single most consequential decision an engineering organization makes — operates in a measurement vacuum. No validation. No feedback loop. No outcome tracking. The interview process is the least-instrumented, least-measured, least-accountable system in organizations that pride themselves on instrumenting, measuring, and holding accountable everything else.

The asymmetry should be staggering. It is instead invisible — because invisibility is the point.

From Omission to Complicity

The failure to measure is not an oversight. Measurement is cheap. The data is available. The methods are well-understood. Organizations that can build real-time monitoring dashboards for microservice latency can trivially build a correlation analysis between interviewer recommendations and post-hire performance reviews. They choose not to.

Consider what the absence of measurement accomplishes:

A hiring manager whose selections are never evaluated against outcomes can never be found to be wrong. An interviewer whose judgments are never correlated with post-placement performance can never be shown to be a poor evaluator. An interview process that is never validated against the thing it claims to predict can never be proven to have failed. The absence of the metric is not a gap in the system. It is the system. It is the mechanism by which the process perpetuates itself — unexamined, unchallenged, and unaccountable.

John Doerr’s framework — measure what matters — has been adopted by the most sophisticated technology organizations on earth. OKRs cascade from company objectives to team objectives to individual objectives. Every quarter, every team, every engineer is evaluated against measurable outcomes. Except the interviewers. Except the hiring managers. Except the process itself. The people who evaluate everyone else have arranged — consciously or otherwise — to never be evaluated themselves.

At some point, the distinction between willful blindness and deliberate concealment stops being meaningful. A hiring manager who has never once examined whether their interview recommendations correlate with actual job performance, and who has had the means to do so for their entire career, is not uninformed. They are protected — by a system that generates no evidence, and therefore can sustain no indictment. Whether that protection arose by design or by gravitational drift toward institutional comfort, the effect is identical: a broken process continues because the evidence of its failure is never collected, and the people who benefit from its continuation are the same people who would have to authorize the collection.

For the engineers: you have been evaluated, for your entire career, by a system that has never been evaluated itself.

For the hiring managers: you have operated a system whose outcomes you have never measured. You have done so for years. Possibly for decades. The question is not whether you know the process doesn’t work — the research has been public since before many of you entered the workforce. The question is whether you have arranged, consciously or not, to never find out.

The Refusal, Not the Remedy

Valid selection methods exist. This is not a mystery, and it has never been.

Schmidt and Hunter’s meta-analysis — the most comprehensive study of selection methods ever conducted, covering eighty-five years of data and nineteen distinct methods — established the hierarchy decades ago. Work sample tests and general mental ability assessments sit at the top. Structured simulations scored against predefined, job-relevant rubrics outperform every form of interview. The methods are documented. They are not expensive. They are not impractical. They have been available, published, and taught in every industrial-organizational psychology program in the country for a generation.

The problem has never been a lack of better methods.

The problem is a lack of willingness — or self-awareness — to adopt them.

Better methods require surrendering the interviewer’s narrative authority. They require replacing “I didn’t get a good feeling” with a rubric that can be examined, challenged, and validated. They require accepting that the gut instinct in the debrief room — the one that feels so much like insight — is noise, not signal. They require measuring outcomes, which means confronting what the measurements reveal. They require, in short, an institutional humility that is fundamentally incompatible with the psychological rewards the current system provides to the people who administer it.

And so the industry that A/B tests everything, that measures everything, that prides itself on empiricism above all — that industry continues to select its most important resource through a process that is pre-scientific in its design, unmeasured in its outcomes, and defended with the fervor of a belief system rather than the rigor of an evidence-based practice.

The process does not work. The evidence that it does not work is overwhelming, and it has been overwhelming for longer than most practicing engineers have been alive. The methods that would work are known, published, and available. And yet, here we are.

The question worth asking is no longer “how do we fix hiring?”

It is: who is served by leaving it broken?