The problem with it is the same curse that has rotted so much of software culture—the need for a scalable process with high throughput. "We need to run through hundreds of candidates per position, not a half dozen, are you crazy? It doesn't matter if the net result is better, it's the metrics along the way that matter!"
Pair programming with the person for a couple hours, maybe even on an actual feature, would probably work, assuming the candidate is compensated for their time. I can imagine it'd especially work for teams working on open source projects (Sentry, Zed, etc). Might not be as workable for companies whose work is entirely closed source.
Indeed, the other problem is what you mention: it doesn't scale to high throughput.
In all pair programming interviews I have run (which I will admit have been only a few) I would fail myself as an interviewer if I was not able to guide the interviewee away from a dead end within 15 minutes.
If the candidate wasn't able to understand the hints I was giving them, or just kept driving forward, then they would fail.
You are not most interviewers, alas.
You still interview with the end companies, but technical interviews aren’t given.
(Update: technically I would guess they're only outsourcing the first few rounds. Presumably their customers will conduct further interviews.)
At this point in my career I no longer even believe that "best candidate" is a meaningful description except perhaps for highly specialized roles. We're all stumbling blindly in the dark.
FAANG companies offering industry leading compensation packages and prestige are in a position to be able to hold out for the best (even if their interviewing practices may fail to achieve that), but most companies are just looking for someone that checks the boxes and seems like a good fit.
Yes, sometimes things just don't work out. But, if someone quits a job and maybe relocates, that's a big personal cost. It's just the way things work in some limited contexts (e.g. professional sports) but it's not and shouldn't be the norm.
I suppose you can give a huge sign-on bonus with no claw-back provision, but that's never going to happen in most cases.
But, in many cases, I'm not sure how I, as a candidate for a tech job, would feel about a company offering me $200K--no strings attached--with the proviso that I statistically only had a 25% chance of making it through the next 6 months. (And is that really long enough anyway?)
There are tournament-style professions. But I'm not convinced most professional jobs are or should be among them in general.
yes. Best place i worked at - we hired only by internal references and only people from our University. Up until the company grew around 200 people. We didn't do technical interviews, just a short talk. And we were among top employers, including salary-wise.
That’s an assumption. Perhaps following a dead end for a while, realizing it, pivoting, etc is a valuable, positive, signal?
In a 2 hour pair programming session on an 'unsolved' problem (like an open issue / minor bug / minor feature in a public repo), yes, it will likely not matter if you tried a bad idea, and would both be more realistic and a positive signal.
If the candidate will communicate with me, I will offer them a LOT of guidance. It is still very, very easy to tell who knows what they are doing and who does not. You give them a basic but domain-specific task, you give them whatever extra context they need to do it, and you watch them hammer out the code.
It should be a task that is sufficiently familiar to the person applying to the role that they do not need to do -much- looking at docs, and as the interviewer you should be prepared to help them quickly find the docs you already know they will need -- you designed the task after all -- so that they don't waste time with that.
What's important is that they ask for docs when they need them, and that they can understand them, quickly, and use them. It will be obvious if they are using AI because of how long things will take. It will be obvious if they don't know when to reach for documentation, and it will be obvious if they cannot understand the documentation.
Then, they should write a test for their solution. This weeds out 95% of candidates. Talk to the other 5% and you'll find someone who can both actually write code and also discuss design.
What is not to say that you are making anything wrong. But watch for bias there.
This sounds like the kind engineer who won't push back or ask for clarification on unclear requirements - and happily spend a month solving a problem the business doesn't have.
So maybe seeing that at interview time is a good thing?
Might not mean the candidate doesn't fit - but can clarify what kind of roles would work?
Perhaps, but an interview is a fundamentally different environment than day to day work.
There's no way to solve the interview/interviewee problem though, the whole thing is impossibly fucked and is going to have false positives/negatives no matter what.
I'm also not actually testing for pair programming ability directly, moreso ability to complete practical tasks / work in a specific area, collaborate, and communicate. If you choose a problem that is general/expandable enough that good candidates for the position are unlikely to go down bad rabbit holes (eg for a senior fullstack role, create a minimal frontend and api server that talk to each other) it works just fine. Actually with these kinds of problems it's kind of good if your candidates end up "studying" them like with leetcode, because it means they are just learning how to do the things that they'll do on the job.
> maybe even on an actual feature
I don't think this would work unless the feature were entirely self-contained. If your workaround is to give the candidate an OSS project they need to study beforehand, I think that would bias candidates' performance in ways that aren't aligned with who you want to hire (eg how desperate are they for the role and how much time outside of work are they willing to put into your interview).
Eh, if it's a reasonable bad end and you communicate through it, I wouldn't see it as a fail. Particularly if it's something I would have tried, too. (If it's something I wouldn't have thought of and it's a good idea, you're hired.)
What worked well for me was that I made it very clear to my manager, a man who I trust, that I would not be able to provide him with a boolean pass/fail result. I couldn't provide him any objective measure of their ability or performance. What I could do was hang out with the canditates for an hour while we discussed some concepts I thought were important in my position. From that conversation I would be able to provide him a ranking along with a personal evaluation on whether I would personally like to work with the candidate.
I prepared some example problems that I worked for myself a bit. Then I went into the interviews with those problems and let them direct those same explorations of the problem. Some of them took me on detours I hadn't taken on my own. Some of them needed a little nudge at times. I never took specific notes, but just allowed my brain to get a natural impression of the person. I was there to get to know them, not administer an exam.
I feel like the whole experience worked super well. It felt casual, but also very telling. It was almost like a focused date. Afterwards I discussed my impression of the candidates with my manager to ensure the things I was weighing was somewhat commutable to what he desired.
All in all it was a very human process. It must have taken an enormous amount of trust from my manager to allow me the discretion to make a subjective judgment. I was extremely surprised at how clearly I was able to delineate the people, but also how that delineation shifted depending on which axis we evaluated. A simple pass/fail technical interview would have missed that image of a full person.
Big companies need to hire tons of people and interview even more so they need some sort of scalable process for it. An early stage startup can just ask you about your past projects and pair program with you for an hour.
There seems to be a weird selection bias that if you're FAANG or FAANG adjacent these small companies aren't interested.
Also, those types of stories tend to pop up with any engineer who’s only worked at a single place.
My point isn’t that there’s not bad engineers at Facebook it’s that there’s bad engineers everywhere and filtering based on random signals like this is not useful.
Especially in a small company where your hiring manager may also be busy with development and sales, and not have an HR department to run the process for them, you're much better off interviewing candidates you are 50% sure of being a good fit vs 5%. Personally I prefer interviewing candidates coming from FAANG-ish companies and often make exceptions for candidates that demonstrate exceptional skill/interest, but when you can only interview 1-10% of your applicants you have to prioritize those who are likely to succeed at your company (keeping in mind implicit bias and such).
> filtering based on random signals like this is not useful.
In aggregate it most likely is useful for those companies.
Big cos can afford to onboard their engineers for months, sometimes years. Startups usually can not
Many smaller companies have noticed that former and wannabe FAANGers are looking for FAANG-type jobs, and are not good fits in their niche. Small companies often have more uncertainty, fewer clear objectives, less structure, and often lower pay. They’re not a good substitute for megacorps.
Not everyone at a FAANG is purely motivated by the amount of money that they can get.
I’m looking for a smaller company because I’m tired of the FAANG mentality personally.
As someone that never had a desire nor ever made an attempt to work at any of those companies, do you mind elaborating on the mentality of such places?
I'm just your boring below-average to average dev, so I know I'm not cut for those types of places, but it never truly bothered me anyway. Any reason that I can personally think as to why I would work for such a company would either be due to my own egotistical desires or for monetary reasons, but those were never strong enough to actually compel me.
I am just mainly curious about two things:
1. Is working at those places all it's cracked up to be?
2. Assuming one had to work hard to get into such companies, was the juice worth the squeeze?
I've often wondered if one's experiences for these companies is often something akin to the old advice of, "Don't meet your heroes." In other words, was the conflicting dyad of expectations vs. reality present?
That said the amount that you make is insane some of the smartest engineers I’ve ever worked with have been at these companies and a lot of them have really strong engineering cultures, and standards.
The current work environment seems designed to use up bright young engineers, and burn them out within a few years. This is a significant shift from 15 years ago, where it was a much more sustainable place to be.
Your path through will probably look like having the luck of breaking in at one of these kinds of companies, and then staying for several years to demonstrate earnest commitment/fit while building a new network of connections, and then leveraging those connections to get more opportunities if it becomes necessary to do so. If you have connection from your previous non-FAANG work, that's probably your best route.
It won't happen overnight and you'll always be at a disadvantage when you find yourself applying through resume portals. Good luck!
In my time at tier one companies I have worked with the best engineers I have come across in my entire career (even the worst engineers were more than competent) who were working on deep issues that could affect the revenue of the entire company because they’re laser focused on providing value to the business, instead of doing engineering for engineering’s sake. I have grown by far more in these kinds of roles than I have anywhere else because the kind of problems you encounter at such a high scale just don’t exist elsewhere. And most of them have been there for at least five years if not longer you don’t make those kind of contributions to accompany without a long tenure.
You’re throwing a giant red flag right here. First of all, FAANG isn’t “tier one” except to people who idolize these companies. More agile startups are trying to disrupt these dinosaurs and do not thing very highly of them. Many of us who have worked with FAANG and ex-FAANG engineers were not impressed.
Just like there are innumerable brilliant, effective engineers who would contribute tremendously to a FAANG but don't suit the modern interview funnel (leetcode, etc), smaller companies surely do miss out on strong, suitable FAANG engineers in anticipation of negative experiences they've had with others.
There are a lot of people who accumulate FAANG entries on their resume and many of them really don't suit smaller companies for a number of reasons.
Honestly, though while I'm only seeing a very narrow picture of you here, it sure sounds like you see these "tier one" companies as a desirable place to work, with prestigious colleagues and profound learning opportunities on high scale problems that just don't exist elsewhere, and surely for much more money. Are you sure you're really going to be happy somewhere else? Or might you get restless? That's precisely the kind of concern these smaller companies carry when seeing FAANG stuff on a resume, and it doesn't seem like it should be baffling that they would do.
As I mentioned here (https://news.ycombinator.com/item?id=43121594)
Was path dependency for careers always this bad?
And most refused to look at anybody deviating from their ideal background in my experience.
This is often because the culture of job-hopping for better pay every 18 months has eroded the willingness to pay for training or adaptation. Why pay for someone to learn if they're just gonna leave soon; the pre-trained person is a better deal if you'll have to pay to retain anyway.
We’re just seeing the end of the cat and mouse struggle that’s been going on since the 60s. And massively accelerated in the 80s.
It’s unfortunate for companies though because they’re the ones that will lose out in the end when all the experienced people start retiring and they have no one to hire.
It’s an untenable position to not train people, period. There is no schooling you could go through that would educated junior dev to the level of a senior dev. And it’s the same for any other role. Experience is not optional.
I agree that this situation is undesirable, but it seems to be stable, somewhat like the result of repeated play of the prisoner’s dilemma.
I agree that other industries are not YET at the point where software is , but you’re not looking hard enough if you don’t see the short tenures compared to the 25-30 years they used to have.
And yeah, it might be in an equilibrium now, but how long can it stay in an equilibrium? I’d guess at max 10 to 15 years.
I'm guessing the majority of people now in their 50s and 60s in computer-related careers had very eclectic jobs before settling down in computer-related stuff. After all, many never used computers at all until college or beyond.
Edit: my point was not that specialized software applications didn’t exist. It was that people were expected to be able to jump from stack to stack when they change roles in a way that has disappeared from modern job applications.
Well, and mainframes. And trading and financial systems. And numerical/scientific computing. And network services. And web sites and e-commerce. And flash, java applets, and browser plugins. And control systems. And operating systems and tooling. And cell phone applications. And games. And video/image/audio/music processing. etc etc
Oh, wait... maybe not!
That’s the point I was trying to make. Not that the software didn’t exist or people weren’t doing specialized applications.
Depending on what you could bring attention to in your prior experience and the size/needs of the new orgs you seeking to move to, certain transitions were more feasible than others, but you could easily spend decades working in mind-numbing enterprise applications while wishing for opportunities in game development or trading or whatever and never get your resume so much as looked at. (And vice versa, even, for those who dreamed to "retire" into the supposed quiet of enterprise apps or government IT or whatever)
But I have also had really terrible experiences, similar to what you've mentioned. Sounds like you've just gotten unlucky and gotten the terrible ones.
It's much worse when the interview gives you different vibes (and expectations) than the actual day-to-day work.
In the best case applicants needs to apply multiple companies. Companies need to interview multiple applicants and have a way to compare those applicants.
Those are the most basic constraints I can think of. How do you make that cost tens of hours for each round?
For me as a job applicant even in the best case I would need to do 3 to 5 interview interviews. The same is true for companies in the best case it will take at least 3 to 5 interviews to find somebody. Are they supposed to have 3 to 5 temporary staff for weeks at a time?
How much time should that take per interview? How would somebody that currently has a job manage that kind of time commitment?
What changes to expectations are you talking about?
If a startup can spend 20 man-hours filling a single position, why can't a big company spend 1000 man-hours filling 50 positions?
If the person interviewing your candidates messes up, you’ll know soon enough. In a large company, the bad people will take over and your company is dying a slow death.
That approach doesn’t work on a large scale. Some interviewers are too nitpicky, elitist, others approve anyone who uses the same language as them for side projects. Some are racists, sexist, or have other kinds of biases. Some might have a crush on the candidate. Sometimes the interviewer thinks about their own task while they squeeze in an interview. In some countries, “undoing” a bad hire is hard, so they need to make sure that the candidate can work on any team (or at least on multiple teams reasonably well).
IMO for large companies it makes sense to standardize the interview process.
Also, in my opinion grinding leetcode is also a good personality check for FAANG hires: it shows the candidate can suck it up, study hundreds of hours, and do whatever they need to do to pass an arbitrary test, even if they themselves think it’s a broken process. The larger the company, the more this quality matters in candidates as they will need to deal with a lot of things they will probably not like.
Why is this the assumption. I would rather say any big org. is converging towards the average talentwise by necessity. It is like Hawaii can't have 100 olympic level swimmers no matter the recruiting proces.
However, I do feel like there is perhaps some amount of truth to the thought behind the interview questions, no? As in, I would imagine someone that could invert a binary tree in 15 minutes on a whiteboard could probably learn React. However, I am not sure everyone that can learn React can invert a binary tree in 15 minutes on a whiteboard.
However, maybe I am projecting my own insecurities because I wish I could invert a binary tree in 15 minutes on a whiteboard as well as being able to solve all those other problems.
leetcode is really a culture fit test and success points to the combination of "smarts", diligence, and conformity to some acceptable degree. It shows you have some baseline of familiarity with computing, can focus on an arbitrary task to pursue a goal, and will conform to an arbitrary process when it's asked of you.
Those are genuinely essential skills in an organization with 1000's of white collar professional workers.
More precise insight is gained when the test covers a binary tree inversion than that were it just some contrived logic puzzle like the LSAT or a critical reading exercise. The computer science bit does provide some signal and isn't completely arbitrary, but it's only a small part of what's being evaluated.
Among otherwise strong engineers, it tends to filter out the especially willful, prideful, independent, meandering, creative, and pragmatic ones. These personality types can be extremely valuable in some work environments and can still sometimes in through leetcode challenges, but spoil big bureuacratic systems like FAANG's when they're overrepresented.
Having done a lot of interviews at least 70% of the time it’s filtering candidates who just aren’t very technical and haven’t studied computer science. The kind of people who hate reversing a linked list are the kind of people who haven’t touched a polynomial since high school.
It’s not unreasonable for a junior tech interview to expect you studied something like CS or EE. It’s a blessing that our field is open to all to give it a shot, but if that doesn’t describe you - you need to recognize that there is effort and study to fill that background.
Now it selects for the very very very bad trait of high ambition and knowing to play the system.
Comparing candidates based on how they “vibe” with the interviewer during a pair programming session is a recipe for lawsuits and bad hires.
I’m just speculating here, I don’t have any significant hiring and interviewing experience
Avoiding pair programming for the reason you listed sounds like lawyers getting in the way, not scaling. But yeah we'd need to hear someone say if that's actually happening.
Because big companies are run by bean counters and they also don't require the same kind of talent that is useful to startups. There's less competition for hyper-specialized seniors and middle of the pack generalists.
Huh. That’s actually a great question! I actually don’t know.
Small companies have the benefit of the pressure to fill a role to get work done, the lack of bureaucratic baggage to "protect" the company from bad hires, and generally don't have enough staff to suffer empire-building.
Somewhere along the line the question changes from "can this candidate do the job that other people in this office are already doing?" to "can this candidate do the job of this imaginary archetype I've invented with seemingly impossible qualities".
But you don't! You only need to find the first person who is good enough to do the job. You do not need to find the best person.
Does "marginally qualified" mean "Ivy League Competitive Programmer PhD" or something?
Not every candidate is an interview. I recently hired. I got 90 applications and from these, 80% were an instant "No". They didn't match the job description or had no permit to work in my country. I invited the rest. Simple interview, pair-program a dead simple App with a prebuilt skeleton with me with any framework of choice. Make one GET request, render it and realize one needed optimization which needs to be implemented. 90% (I'm not joking) of candidates failed the first task. In half an hour, they couldn't send a GET request and translate that into JSON. All were allowed to google, open any documentation they liked. Of all 18 who failed, 16 asked me if they could use an LLM for this task, which I denied.
Or maybe he is getting resumes from a channel that has been victim of machine-gun filling, and there indeed thousands of incompetent people posting resumes into every channel and just half a dozen real applicants.
TBF, I have no idea how to fix either one of those problems. Hiring is just completely broken.
And I know similar Junior-mid people in the same boat. We can all do Fizzbuzz, we've all built things, and somehow we're not getting interviews, but people that can't do Fizzbuzz are.
Do thousands of incompetents also machine-gun apply to, say, mechanical engineering, accounting, marketing, HR, or finance gigs? Is it just tech?
Something isn't adding up.
Enough that it's HR or some automated software that first screens your application.
> Something isn't adding up.
Yes. The pipeline "posting > application > screening" is now completely broken. In your case it's quite possible it's HR and screening software that your resume is not convincing. I have been hoping for anecdotes and studies in this direction (from people who have access to the HR and/or software screening and are inclined to report) - but it's at least not common. What we do hear from, is tons of people who can program and who are not getting even first interviews.
Create certifications that actually count for something and aren't just a blip on a resume that may tick a box, but will actually move you past technical trivia questions. I know some people have a deep repulsion to this and I think it would be fine to have a technical interview gauntlet for those that choose not to engage with any type of certification and a simplified interview format for those that have passed the prerequisite tests.
I don't care how long, rigorous, or ridiculous the tests are. Just agree on some effing standard.
There is something funny about the "best interview process" taking "a couple hours" despite giving you the answer "within a few minutes". Seems like even the best process is a little broken.
I can only speak for myself, but I imagine myself as a candidate approaching a “couple of hours” project or relationship differently than I would a “few minutes” speed round. For that matter I can think of people I know professionally who I only know through their behavior “on stage” in structured and stylized meetings of a half hour or an hour—and I don’t feel like I have any sense at all of how they would be as day-to-day coworkers.
If we sat down to work together, you’d probably have a sense in the first few minutes of whether or not we were going to work out—but that would be contingent on us going into it with the attitude that we were working together as colleagues toward a goal.
Nor am I saying it was a perfect system, just the best I've seen in terms of results.
So now I have this weird dynamic: I get interview calls only from FAANG companies, the ones with the manpower to do your so called "cursed" scalable interviews. But the smaller companies or startups, ones who are a far better fit for my specialized kills, never call me. You need to either "know someone" or be from a big school, or there is zero chance.
I don't care if someone uses modern tools (like AI assists), google, etc - e.g. "open book" - as that's the how they want to work. Evaluating their style of thinking / solving problems, comms, and output is the win.
I've been interviewing recently and got through to the last round (of five...) with an interesting company. I knew the interview involved pairing, but I didn't expect: two people sitting behind me while I sat on a wobbly stool at a standing desk, trying to use a sticky wired mouse and a non-UK keyboard, and a very bright monitor (I normally use dark mode). They had a screen recorder running, and a radio was on in the next room.
I totally bombed it. I suspect just the two people behind me would have been enough though.
Also how am I supposed to filter the 100's of AI-completed assessments? Who gets this opportunity?
We also did not require the employers doing the interview to be our most senior team members. They probably did it more often than most people, but often because they volunteered to do it. Anyone on the team would be part of the loop, which helped with scheduling. And, remember, we were working on actual tickets, so in a lot of cases it actually helped having the candidate there as a pairing partner.
For a little extra detail, the way we actually did it was to have 2-3 pairing sessions of up to 2 hours apiece. At the end of the day, all the team members who paired with the candidate had to give them the thumbs up.
Idk if I'm even being sarcastic here.
I think you're onto something with that last paragraph but I want to try being a bit more generous with why things are the way they are. The question seems to be "When there are hundreds of applicants how do we give everyone a fair shake without hiring an entire team of devs who do nothing but interview?" From that perspective the intentionality is different and even sensible but the end product is likely to be the same. Even when someone is chasing a metric it's because someone else wants what's best and has decided that metric is a sensible way to make that happen. At the end of the day they really do want to hire the best candidate out of a pool whose size is extremely variable and that's challenging.
I have been here 5 years now and I'm working with the most competent team I have ever worked with. My take away from this is that hiring doesn't need to be commoditised and scale, it just needs to find good people and give them an opportunity to show you that you do or don't want to work with them.
Then why spend a couple hours?
Additionally it’s rarely the hiring that makes a great team - it’s the long term commitment and investment in training.
With that said, would your perception of the interview remain positive if the outcome had been negative?
A common challenge across all interviews is a mismatch in personal dynamics, which can significantly impact the experience for both participants.
Consider a scenario where a senior developer, who prefers simplicity, is paired with a mid-level developer who is still captivated by complexity.
Being peppered with questions very rarely gives any insight.
Even for senior roles, that's what I want to interview for, although it is true at times a business case can be made for someone that is good at some specific complex skill and doesn't need to listen to other people to do ok work.
>You never failed to know within a few minutes whether the person could do the job
Did a misunderstood something or your best interview process is to multiple hours from someone when you've decided within minutes?
Developers Side: Since developers don't have job security anymore (at least for those who work on common languages like Go, Python, Java and Typescript) they are better off learning and keeping in touch with leetcode and system design questions, looking for new opportunities and interviewing in "batch mode" when looking for a job. The idea is to clear as many interviews as possible using the same concepts, get in and make money asap before you get laid off. No incentive for collaboration or for fulfilling but esoteric stuff like Haskell and Scala. Career security > Job security.
Companies Side: On the other end software companies have less trust in developers staying long term so they want to make the interview process as quick and risk free as possible. In essence they are betting that by perusing 100s of resumes and hiring someone who seemingly knows CS concepts they can get some value out of them before they leave. Standardized tests/vetting > team fit.
TLDR; The art is gone from this job, its become akin to management consulting or investment banking. Quality and UX seems to be regressing across the board as a result.
Not sure how those are similar.
I think what makes it work is that our code pair is pretty low stakes. I was told that I didn’t have to finish the problem and I was free to use whatever tools or language I needed. They just wanted to see how I work and collaborate.
Teams are really sleeping on code reviews as an assessment tool. As in having the candidate review code.
A junior, mid, senior, staff are going to see very different things in the same codebase.
Not only that, as AI generated code becomes more common, teams might want to actively select for devs that can efficiently review code for quality and correctness.
I went through one interview with a YC company that had a first round code review. I enjoyed it so much that I ended up making a small open source app for teams that want to use code reviews: https://coderev.app (repo: https://github.com/CharlieDigital/coderev)
So much value of the code review comes from having actual knowledge of the larger context. Mundane stuff like formatting quirks and obvious bad practices should be getting hoovered up by the linters anyways. But what someone new may *not* know is that this cruft is actually important for some arcane reason. Or that it's important that this specific line be super performant and that's why stylistically it's odd.
The real failure mode I worry about here is how much of this stuff becomes second nature to people on a team. They see it as "obvious" and forgot that it's actually nuance of their specific circumstances. So then a candidate comes in and misses something "obvious", well, here's the door.
An example from the interview: the code included a python web API and SQL schema. Some obvious points I noticed were no input validation, string concatenating for the database access (SQL injection), no input scrubbing (XSS), based on the call pattern there were some missing indices, a few bad data type choices (e.g. integer for user ID), a possible infinite loop in one case.
You might be thinking about it in the wrong way; what you want to see is that someone can spot these types of logic errors that either a human or AI copilot might produce regardless of the larger context.
The juniors will find formatting and obvious bad practices; the senior and staff will find the real gems. This format works really well for stratification.
I'd say all this stuff is junior-level (maybe ~mid for things like user ID integers). It's just a checklist of "obvious bad practices", it doesn't require experience.
The senior stuff is much higher-level: domain modelling, code architecture, consistency guarantees, system resilience... system design in general.
But I'm not sure you really need to in a job interview. It's not like you can do that with any other interview method anyway - leetcode also doesn't really touch high level architecture type stuff, and take home problems are also too small (or they should be anyway!)
In my experience you only learn how good developers' architectural taste is by working with them for a long time.
In a previous job we did code review interviews. And went the route you said due to the problem I said. And yes, it's a lot better. But what also happened over time was that the bar slowly raised. Because over time the "harder" pieces of that session started to seem rote to interviewers, they became viewed as table stakes.
Mind you this is true of any interview scheme that has a problem solving component to it. I'm not saying that the code review style is extra bad here, just that it doesn't solve this problem.
Here's the neural net model your colleague sent you. They say it's meant to do ABC, but they found limitation XYZ. What is going on? What changes would you suggest and why?
Was actually a decent combined knowledge + code question.
I wrote up 7 general strategies for teams that are interested: https://coderev.app/blog/7-strategies-for-using-code-reviews...
The great thing about code reviews is that there are LOTS of ways people can improve code. You can start with the basics like can you make this code run at all (i.e. compile) and can you make it create the right output. And there's also more advanced improvements like how to make the code more performant, more maintainable, and less error-prone.
Also, the candidates can talk about their reasoning about why or why not they'd change the code they're reviewing.
For example, you'd probably view the candidates differently based on their responses to seeing a code sample with a global variable.
Poor: "Everything looks fine here"
Good: "Eliminate that global variable. We can do that by refactoring this function to..."
Better: "I see that there's a global variable here. Some say they're an anti-pattern, and that is true in most but not all cases. This one here may be ok if ..., but if not you'll need to..."
Coding for me is an intensely focused activity and I work from home to boot so most of the time, I'm coding in complete silence. It's very awkward to be talking about my thought process while I'm coding, but not talking is just as awkward!
They asked me to review a function for a residential housing payment workflow, which I'm unfamiliar with. From an actual snippet of their bad production code (which has since been rewritten). In Go which I've never used (I've never professionally used the language that doesn't have error handling built-in, for example).
I had to spend more than half of my time asking questions to try and get enough context about Go error handling techniques, the abstractions they were using which we only had the import statements to and the way that the external system was structured to handle these requests to review the hundred lines of code they shared.
I was able to identify a bunch of things incidentally, like making all of the DB changes as part of a transaction so that we don't get inconsistent state or breaking up the function into sub functions, because the names were extremely long, but this was so far outside my area of expertise and comfort zone that I felt like shooting in the dark.
So just like any other interview style, they can be done very poorly.
Language and domain experience are things id like to know after an interview process.
They don’t work mostly in Go. Even the interviewer said that he’s vaguely familiar with this area of the code, but he doesn’t work and Go. They work mostly in Kotlin and they explicitly are advertising for solid generalists.
> A cold code review on a codebase they never saw
So yeah, I think it's the opposite: explicitly testing for their ability to read code is probably kinda important.
(OBS Elements other times)
I guess it would degrade to stackoverflow-like poems eventually, but still interesting.
It would be interesting, but I agree it would need to be content moderated to some extent.
Each code sample should have multiple things wrong. The best people will find most (not necessarily all) of them. The mediocre will find a few.
The two folks who showed this behavior I hired anyway (they were contractors so nbd) and they were excellent hires, so I really love the code review approach for climbing up bloom's taxonomy.
There's very likely a real answer to that question, and that answer should shape the way that engineer should be assessed and hired.
For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request.
It seems to me the heart of the problem is that companies aren't very clear about what value the engineers add, and so they have trouble deciding whether a candidate could provide that value.
Well, unless you know sufficiently senior people. But I suspect that is a deeply unsatisfactory answer to many people in this forum.
My long term last, only technically-adjacent, job came through a combination of knowing execs, having gone to the same school as my ultimate manager, and knowing various other people involved. (And having a portfolio of public work.)
And just because you’re an introvert doesn’t mean you are incapable of building soft skills. Talking to people is absolutely exhausting for me, but I force myself to do it and practice at it because I know it is important for my career.
I love coding and do it reliably well with joy but as my career has progressed I've struggled more and more with getting a company to let me work at a "low level" or to navigating what seem like sociopathic behaviors to really contribute at my capacity.
Are you talking about a traditional Staff / Principal engineering role or something different?
Nobody cares about that work... until it doesn't get done. And so, nobody doing it gets promoted.
But my broader point was that golf course socializing seems like mostly a different world today, at least in my tech circles, relative to other venues.
I get it. By nature I was very much an introvert except for certain scenarios when I was in my comfort zone until at least my mid 30s. I was an only child, the stereotypical short, fat kid with a computer growing up in the 80s (still short, became a gym rat, part time fitness instructor and only stopped the latter as my other obligations became greater). Horrible dating life and a bad first marriage before turning 35 (happily remarried since then).
It became apparent that to get ahead in my career, “codez real gud” was going to limit my career. I slowly learned how to “act like I like people”.
But you can only add so much value to an organization typing on a keyboard. There is a reason that every single tech company promotes based on “impact”, “scope”, “dealing with ambiguity”. Those all require soft skills.
We know what kinds of temperament a dog has within few months of it being a puppy ( and who the puppy's parents are). Why would it be different for humans.
Claim that Our temperaments ( and our likes/dislikes for travel) are all learnt is a bizzare blank slate claim that doesn't track with my life experience and what i've seen in the world.
Because animals typically live in a way more static and uneventful environments, and they have much more limited mental capabilities?
Humans (and other animals) aren't a completely black slate - but unlike most animals, humans have very complex societies that affect their behaviors throughout their entire lives. A few years in a different environment start to change people. Kids (with their still-growing brains) adapt faster, adults - not so much, but the traces will be evident. Move a not-too-fucked-up Russian to the Pacific Northwest, and they will eventually start to smile now and then.
Also, thanks to the language, humans can think things up even when alone, drive themselves crazy in all the weird ways, then overcome all that self-inflicted stress and possibly develop some behaviors as a result.
My shy cat could've been a party animal like her sister :)
For a cat, it probably plays a significant role. Cat behaviors are complex but still much more simpler than humans. And changes are rare. Although I've heard of a "lazy" apathetic cat moving into a house with giant outdoor catio and becoming drastically more active, almost like a different kitty.
I'm not sure about humans - how much of our behavior is a true temperament and how much isn't despite tending to not change throughout one's life. I've seen introverts becoming eager activists after they went through some bad things, like war and prison. I've seen people who were jumpy and always nervous becoming relaxed after many years in safety.
Ask about their favorite travel destinations or even what are some interesting things about where they live.
On the other hand, step outside your comfort zone and try something different so you have something to talk about interesting.
You didn’t become a software developer overnight. You won’t become a great conversationalist over night either.
“How to Talk to Anyone”
https://www.amazon.com/How-Talk-Anyone-Success-Relationships...
(not an affiliate link)
I don't want to push to give presentations at international events is certainly a valid decision for any of a number of reasons.
So is preferring to spend more time on coding than managing/mentoring/etc.
But it all has consequences and some branches will lead to more promotions/money/etc. than others. And you may be perfectly fine with that. But go into with eyes wide open.
In my understanding, non-junior software development jobs never were about typing on the keyboard. Senior software engineer is a fancy name for a problem solver, and code is just a specialized tool they can build to possibly achieve the goal. It always was about talking to stakeholders, figuring out what the heck they actually want today, how it fits with what they think they want tomorrow, learning more about those stakeholders so you can guess what they will think they want next week. Only then it's thinking about it all it for a while, and only after that it's getting to press the actual buttons.
But I'm not sure those things require "soft skills" aka - in my understanding - being a people person. For me, it was a very simple learning process - I (as a junior) coded something, a manager came next month and said I have to rewrite everything again because things have changed. I hated it, so I started to think how to possibly avoid or minimize it and optimize my own processes.
And in my mental model, it's not about people (save for tiny companies where a whole department/role is a single person, so I have to account for their mental chaos monkeys), it's all about business. That's why I wrote "stakeholders", intentionally dehumanizing (with no negative connotations) the model.
I think a lot of that is "soft skills." Maybe not becoming a stereotypical sales person. But it's also not don't ever bug me and let me code.
Junior - you are told what to do (business objective) and how to do it (technical).
Mid - you are told what to do (business objective). But you are expected to use your experience to figure out the “how”. You should be able to lead a decently complicated feature/epic/work stream either by yourself or with others and mentoring other juniors.
Senior - You are expected to lead major projects that involve multiple epics with multiple developers, talk to “the business”, disambiguate, deal with XYProblems, communicate trade offs between time, cost, meeting requirements, etc. Now you also start having to deal with cross team coordination.
Staff - cross team impact, dealing much more with business strategy and setting technical direction.
As an IC, the only things you have at your disposal are your relationships and reputation. Both require soft skills.
Leveling guidelines:
Not everywhere. At my company, HR owns the process but we -- the hiring tech team -- own the content of interviews and the outcomes.
Interview coding questions aren't like the day-to-day job, because of the nature of an interview.
In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.
It also has to be a problem that's solvable within about 40 minutes.
The problem needs to test the candidate meets the company's hiring bar - while also having enough nuance that there's an opportunity for absolutely great candidates to impress me.
And the problem has to be possible to state unambiguously. Can't have a candidate solving the problem, but failing the interview because there was a secret requirement and they failed to read my mind.
And of course, if we're doing it in person on a whiteboard (do people do that these days?) it has to be solvable without any reference to documentation.
If you send me a rubric I can pre-load whatever you want to talk about. If you tell me what you're trying to build and what you need help with, I can show up with a game plan.
You need to make time for a conversation on the intricacies of voucher calculation and global sales tax law if you want to find people jazzed about the problem space.
Proving if they are technically capable of a job seems rather silly. Look at their resume, look at their online works, ask them questions about it. Use probing questions to understand the depths of their knowledge. I don't get why we are over-engineering interviews. If I have 10+ years of experience with some proof through chatting that I am, in fact, a professional software engineer, isn't that enough?
No, it's not enough. There are people out there who can talk great talk, and have great resume, but cannot do their actual job for some reason. Maybe they cannot read the code, maybe they cannot write the code, maybe they can write the code but not in the manner that keeps the rest of codebase working... I've had people like that on my team, it was miserable for all of us.
It is essential to see candidate actually write and debug code. It would be even better if we could see how the candidate deals with existing huge codebase, but sadly this kind of thing can't be easily done in a quick interview, and good candidates don't want trial periods.
You have missed his point. If the interview questions are such that AI can solve them, they are the wrong questions being asked, by definition. Unless that company is trying to hire a robot, of course.
I felt like I got a good sense of what he would be like to work with and he got to see how I approached various problems. It avoided the live coding problems of needing to remember a bunch of syntax trivia on the spot and having to focus on a quick small solution, rather than a large scalable one that you need more often for actual work problems.
WE saw this last year with all the "AI can now pass the bar exam" articles, but that doesn't lead to them being able to do anything approaching practicing law, because AI failure modes are not the same as humans and can't be tested the same way.
If my problems could be solved in the time span of an interview, why would I waste my time doing that interview instead of just solving it?
I'm also not sure what the alternative is? Just not hiring?
You just described a contrived, “unreal” problem.
> I'm also not sure what the alternative is? Just not hiring?
The alternative is to come up with questions that are representative of skills related to “real problems”, as you just did, and use those instead. Unfortunately candidates consistently complain that such questions aren’t realistic.
Mind, I work in very small companies and never had to give input for filling 10 positions at once... just one at a time.
And, I've also had people speak very well, doing great with the verbal explanation and questions, even good pseudo code, and then be unable to write a simple for loop, of any kind, in any language. These people also often have a resume full of short runs.
So, I structure mine around a, fixed, work related problem that lets me clearly justify the yes/no in a way that upper management can stomach, but then just bias my feedback a bit based on the "personal interpretation" things like what you describe (which I think are usually better indicators).
Also, resumes are 90% fiction, from what I've seen, especially from certain demographics (not allowed to perceive that though). I don't bother believing them or talking about them, unless there's time after.
Yes, this mostly works in small organizations. I'm mostly in positions where I have to pass the feedback once, or at most twice up the chain.
Let's not pretend 95% of companies are asking asinine interview questions (though I understand the reasons why) that LLMs can easily solve.
Current LLMs can do some basic coding and stitch it together to form cool programs, but it struggles at good design work that scales. Design-focused interviews paired with soft-skill-focus is a better measure of how a dev will be in the workplace in general. Yet, most interviews are just “if you can solve this esoteric problem we don’t use at all at work, you are hired”. I’d take a bad solution with a good design over a good solution with a bad design any day, because the former is always easier to refactor and iterate on.
AI is not really good at that yet; it’s trained on a lot of public data that skews towards worse designs. It’s also not all that great at behaving like a human during code reviews; it agrees too much, is overly verbose, it hallucinates, etc.
Now someone comes along and builds a machine that can do A. It turns out that while for humans, A was a good indicator of X, Y and Z, for the machine it is not. A is easy for the machine, but X, Y and Z are still difficult.
This isn't a sign that the company was wrong to ask A, nor is it a sign that they could just hire the machine.
I think if it was socially acceptable they'd just do the latter.
The most interesting thing in that paper is that years of experience performed so poorly. It’s in the lowest cohort. Worse than “interests” or more general “biographical data”.
Most hiring managers believe experience matters in hiring as well, perhaps that’s the belief that keeps them from using iq tests.
For what it’s worth, IQ tests are biased (see duyme’ adoptive studies for a drastic economic impact). That largely is orthogonal to if they are predictive in the ways your citation outlines.
That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat .
Sure, right up until someone leaks it
For a while I’ve been skeptical that the rate of hiring of engineers would change significantly because of LLMs, but I’m starting to feel like maybe I’m wrong and it’s already changing and companies are looking toward AI to lower costs and require fewer humans. In that case they are probably still going to want people who are technically exceptional - maybe even more so - but are able and willing to create, integrate, and babysit AI generated code, and also do PM and EM style feature management.
If companies are slowing hiring due to AI, I would expect interviews to get worse before they get better.
So a Product Manager?
You can't just take requirements and churn out code without a critical eye at what you're doing.
Maybe now, or maybe in a year or two, AI coding tools will be good enough that a single semi-technical person can be Product Manager for a small product, and implement all the feature through AI/LLM tools.
Probably not for something of the complexity of Google Maps, but for a simpler website with some interactive elements, that could work.
But then, this was just an example. There can be lots of reasons that companies still need engineers, my point was that they need to think about these reasons, and then use these reasons to decide how to select their engineers.
I agreed to do it as long as they understood that I felt it was a terrible way of assessing someone's ability to code. I was allowed to use any programming language because they knew them all (allegedly).
The solution was a pretty obvious bit-shift. So I wrote memory registers up on the board and did it in Motorola 68000 Assembler (because I had been doing a lot of it around that time), halfway through they stopped me and I said I'd be happy to do it again if they gave me a computer.
The offered me the job. I went elsewhere.
This looks more like a culture fit test than a coding test.
Folks getting mad about whiteboard interviews is a meme at this point. It misses the point. We CANT test you effectively on your programming skillbase. So we test on a more relevant job skill, like can you have a real conversation (with a whiteboard to help) about how to solve the problem.
It isn't that your interviewer knew all the languages, but that the language didn't matter.
I didn't get this until I was giving interviews. The instructions on how to give them are pretty clear. The goal isn't to "solve the puzzle" but instead to demonstrate you can reason about it effectively, communicate your knowledge and communicate as part of problem solving.
I know many interviewers also didn't get it, and it became just "do you know the trick to my puzzle". That pattern of failure is a good reason to deprecate white board interviews, not "I don't write on a whiteboard when i program in real life".
Except, that's not what happens. In basically every coding interview in my life, it's been a gauntlet: code this leetcode medium/hard problem while singing and tapdancing backwards. Screw up in any way -- or worse (and also commonly) miss the obscure trick that brings the solution to the next level of algorithmic complexity -- and your interview day is over. And it's only gotten worse over time, in that nowadays, interviewers start with the leetcode medium as the "warmup exercise". That's nuts.
It's not a one off. The people doing these interviews either don't know what they're supposed to be looking for, or they're at a big tech company and their mandate is to be a severe winnowing function.
> It isn't that your interviewer knew all the languages, but that the language didn't matter.
I've done enough programming interviews to know that using even a marginally exotic language (like, say, Ruby) will drastically reduce your success rate. You either use a language that your interviewer knows well, or you're adding a level of friction that will hurt you. Interviewers love to say that language doesn't matter, but in practice, if they can't know that you're not making up the syntax, then it dials up the skepticism level.
The problem is that the business side wants to reduce it to an objective checklist, but you can't do that because of Goodhart's Law [1]. AI is throwing this problem into focus because it is basically capable of passing any objective checklist, with just a bit of human driving [2]. Interviews can not consist of "I'm going to ask a question and if you give me the objectively correct answer you get a point and if you do not give the objectively correct answer you do not". The risk of hiring someone who could give the objectively correct answers but couldn't program their way out of a wet paper bag, let alone do requirements elicitation in collaboration with other humans or architecture or risk analysis or any of the many other things that a real engineering job consists of, was already pretty high before AI.
But if interviewing is not a matter of saying the objectively correct things, a lot of people at all levels are just incapable of handling it after that. The Western philosophical mindset doesn't handle this sort of thing very well.
[1]: https://en.wikipedia.org/wiki/Goodhart%27s_law
[2]: Note this is not necessarily bad because "AI bad!", but, if all the human on the other end can offer me is that they can drive the AI, I don't need them. I can do it myself and/or hire any number of other such people. You need to bring something to the job other than the ability to drive an AI and you need to demonstrate whatever that is in the interview process. I can type what you tell me into a computer and then fail to comprehend the answer it gives is not a value-add.
Mind elaborating on that?
The reality isn't so much "in between" as "both". There is a reason the West developed a lot of tech and the East, despite thousands of years of opportunity, didn't so much. But there is also a limit to the reductionistic viewpoint.
In this case, being told that the only way to hire a truly good developer is to make a holistic evaluation of a candidate, that you can not "reduce" it to a checklist because the very act of reducing it to a checklist invalidates the process, is something that a lot of Western sorts of people just can't process. How can something be effectively impossible to break into parts?
On the other hand, it is arguably a Western viewpoint that leads to the idea of Goodhart's law in the first place; the Eastern viewpoint tends to just say "things can't be reduced" and stop the investigation there.
This is highly stereotypical, of course, and should be considered as an extremely broad classification of types of philosophy, and not really associated directly with any individual humans who may happen to be physically located in the east or west. Further as I said I think the "correct" answer is neither one, nor the other, nor anything in between, but both, so I am not casting any shade on any country or culture per se. It is a useful, if broad, framework to understand things at a very, very high level.
When I finally got in the door and joined the hiring effort I was appalled to find they’d implemented a leetcode-esque series of challenges with criteria such as “if the candidate doesn’t immediately identify and then use a stack then fail interview”. There were 7 more like this with increasingly harsh criteria.
I would not have passed.
And do you frame the problem like that when giving interviews? Or the candidates are led to believe working code is expected?
Do I see others doing so? sadly no.
I feel like a lot of the replies to my comment didn't read to the end, I agree the implementation is bad. The whiteboard just isn't actually the problem. The interviewers are.
Unless they change mentality to "did this candidate show me the skills i am looking for" instead of "did they solve puzzle" the method doesn't matter.
It's all well and good that you and other "wise interviewer" commenters on HN actually grok what the point of interviews are, but you are unicorns in the landscape.
> I know many interviewers also didn't get it, and it became just "do you know the trick to my puzzle". That pattern of failure is a good reason to deprecate white board interviews, not "I don't write on a whiteboard when i program in real life".
...while being closely monitored in a high-stakes performance in front of an audience of strangers judging them critically.
So why is Google relevant to this in any way?
Sucks for you, then. Why are you on a thread about Google-style interviews?
For the same reason you wrote "Google-style". Because this thread is specifically about those interviews happening not at Google.
Oh, maybe you misunderstood their question. When they suggested Google wasn't relevant, they meant the company culture at Google itself because that's what you were talking about.
But if someone was ready for your exact question by having the right interview practice/experience, or they just don't care about your job so there's no stakes. Then you still aren't measuring what you think you are.
Everybody says that, but reality is they don't imho. If you don't pass the pet question quiz "they don't know how to program" or are a "faker", etc.
I've seen this over and over and if you want to test a real conversation you can ask about their experience. (I realize the challenge with that is young interviewers aren't able to do that very well with more experienced people.)
What I want to know from an interview is if you can be presented an abstract problem and collaboratively work with others on it. After that, getting the “right” answer to my contrived interview question is barely even icing on the cake.
If you complain about having to have a discussion about how to solve the problem, I no longer care about actually solving the problem, because you’ve already failed the test.
Yes if you don't communicate clearly, you will get points deducted. But if you can't answer the question nearly perfectly, its basically an immediate fail.
The only thing that matters in most places is getting to the optimal solution quickly. It doesn't matter if you explain your thought process or ask clarifying questions, just get to the solution and answer the time and space complexity correctly and you pass.
Like others have said I think this is a symptom of the sheer number of people applying and needing to go through the process, there is no time for nuance or evaluating people on if you would actually like to work with them or not.
I am so happy that you did this. We vote with our feet and sadly, too many tech folks are unwilling to use their power or have golden handcuff tunnel vision.
I have yet to come across an interviewer who has a clue about anything that interests me.
After 30 years of doing this, I find that typically the people who claim to know a lot often know very little. They're insecure in their ability so much that they've tricked themselves into not learning anything.
Today? Now that's when it is tricky. How can we know you are not one of these prompt "engineers" copy paster? That's the issue being discussed.
20 years and many new technologies of difference.
I think the entire question is missing the forest for the trees. I have never asked a candidate to write code in any fashion during an interview. I talk to them. I ask them how they would solve problems, chase down bugs, or implement new features. I ask about concepts like OOP. I ask about what they've worked on previously, what they found interesting, what they found frustrating, etc.
Languages are largely teachable, it's just syntax and keywords. What I can't teach people is how to think like programmers need to: how to break down big, hard problems into smaller problems and implement solutions. If you know that, I can teach you fucking Swift, it isn't THAT complicated and there's about 5 million examples of "how do I $X" available all over the Internet.
This is like "learning a natural language is just 'cramming vocabulary and grammar' - voila, you've become a fluent C1 speaker". :-)
Seriously: if you argue this way, you have only seen a very biased set of programming languages, and additionally, your knowledge of these programming languages is very superficial (i.e. you have never gotten to the "interesting"/"deep" concepts that make this particular programming language special, and which are hard to replicate in most other programming languages).
That's what the MBA people want to believe. To lower costs, or if they see writing code as an operating expense, instead of R&D.
If this is true or not, it depends on many, many factors, and it can change over the course of the business life.
This is not something nice to say about the colleagues. :-)
Then again, that’s C.
The comparison to natural languages would be if you could learn one language and then quickly pick up every other language of that "class" after learning how that single language works. That's not really how natural language works at all, but it does work with programming languages.
If you understand the grammatical topics that a natural language picks, all you have to learn is what word transformation rules map to those concepts, and the natural language's vocabulary.
> The comparison to natural languages would be if you could learn one language and then quickly pick up every other language of that "class" after learning how that single language works.
There do exist books on this topic (though more commonly for language families). See for example
https://www.quadrilingual.com/
or the book
EuRom 5. Leggere e capire 5 lingue romanze
> That's not really how natural language works at all, but it does work with programming languages.
... it might give you some shallow knowledge in a very limited subset of programming languages.
Yes, and then do that in real time while you're having a conversation with someone who's been learning the language since they were a baby. It is an unreasonable comparison.
That's only true for a subset of programming languages, and it requires you to already know how to program in at least another language of the same family. Knowing Java will not help you with Haskell, but it will help you with C#.
I have to deal with students using AI to cheat on homework and exams, and I can't allow them to not even learn the basic concepts.
They could convince you with buzzwords, get hired, and then feed all problems to the AI until it reaches a point where the codebase is too big for the context, and then all their prompt “engineering” experience is basically useless.
That is the future I am trying to prevent.
Until the AI can code a full system like SAP, or an Operating System, or a Photoshop clone, by itself, we need some people in the loop, and the more knowledgeable the people, the better.
That's true, but most of the industry is running on a subset of programming languages.
In this context, to a large extent it holds. Yeah. It’s probably more true of mainstream languages roughly related to each other, but in an interview, you’re not testing for knowledge of syntax and keywords. You’re trying to test for ability to build and problem solve.
I share your concern about prompt “engineers” who don’t understand the underlying codebase, language or system. But I don’t know what to do about it in the context of a technical interview.
this is a crazy take in the context of coding interviews. first, because it's quite obvious if someone is blindly copy and pasting from cursor, for example, and figuring out what to do is a significant portion of the battle, if you can get cursor to solve a complex problem, elegantly, and in one try, the likelihood that you're actually a good engineer is quite high.
if you're solving a tightly scoped and precise problem, like most coding interviews, the challenge largely lies in identifying the right solution and debugging when it's not right. if you're conducting an interview, you're also likely asking someone to walk through their solution, so it's obvious if they don't understand what they're doing.
cursor and copilot don't solve for that, they make it much easier to write code quickly, once you know what you're doing.
Works really well and it mimics the what we find is the most important bit about coding.
I don't mind if they use AI to shortcut the boring stuff in the day-to-day, as long as they can think critically about the result.
You can tell who's trying to use AI live. They're clearly reading, and they don't understand the content of their answers, and they never say "I don't know." So if you ask a followup or even "are you sure" they start to panic. It's really obvious.
Maybe this is only a real problem for the teams that offloading their interviewing skills onto some leetcode nonsense...
If someone gave me code with
if (x = 7) { ... } as part of a C eval.
Yeah, you'll get a sarcastic response back because I know it is testing code.
What I think people ignore is that personality matters. Especially at the higher levels. If you are a Principal SWE you have to be able to stand up to a CEO and say "No, sir. I think you are wrong. This is why." In a diplomatic way. Or sometimes. Less than diplomatic, depending on the CEO.
One manager that hired me was trying to figure me out. So he said (and I think he was honest at the time). "You got the job as long an you aren't an axe murderer."
To which I replied deadpan: "I hope I hid the axe well." (To be clear to all reading, I have never killed someone, nevermind with an axe! Hi FBI, NSA, CIA and pals!)
Got the job, and we got along great, I operated as his right hand.
How do you expect them to get access to the property internal Git repo codebase and approval from their employer's lawyers to show it to third parties during the interview?
Sounds like you're only selecting Foss devs and nothing more.
If that's the case however, just let them make a small project over the weekend and then do another interview where you ask stuff about what they've made. It's not that deep
First: they might have private code, but not necessarily code to show (I, for example, am rather not willing to show quite some of the code that I wrote privately).
Second: the kind of "code" that I tend to write privately (and into which I invest quite a lot of time) is really different from what I do at work, and what is actually considered "code" by many. It's more like (very incomplete) drawings and TeX notes about observations and proofs of properties and symmetries between some algorithms. Once finished, they will be very easy to systematically transform into a program in a computer language.
This is about very novel stuff, which to explain would take quite a lot of time.
I can clearly state that this is not I commonly think about code that I write privately (and also for code that I write for the job only if I must). For private code, I rather commonly start with a "gut feeling" about some unexpected symmetry that the problem that I am working on likely has, then try to formulate these "gut feelings" as mathematical properties, and later theorems. At the end, everything "fits (for outsiders: unexpectly) together".
Thus, there is hardly ever a "option that I took", but rather a "I let everything flow: from the source [my gut feeling] to the sea [which is - ironically - the source (code)]".
This is an extremely flawed interview process in my opinion and the last time I encountered it led to an awkward scenario that led to me walking out. Personally, when I conduct interviews, it's a mix of things. We talk about your past work, I quiz you a bit on some topics you'd encounter in your day-to-day here, and then we'll spend an hour doing some combination of a code review of a working-but-flawed demo project I created, a 30-40 minute coding exercise, and/or a problem-solving scenario where I give you a problem and then we talk through how, as a pair, how we could solve it.
School was years and years ago, and has nothing to do with my current skills.
From the people i personally know, most do _not_ have a hobby project, even fewer have hobby projects that showcases their technical skills. Nor should they be expected to. Most people have non-programming hobbies.
> I cannot understand how some developers have no code to show.
It's really not that deep, I'm worried if you really cannot understand. I don't code outside of work, I'm not interested in doing it. I'm good at software engineering, not passionate about it. I have a bunch of other hobbies. There's no reason I'd have any code to show now or at any point in the future.
> let them make a small project over the weekend and then do another interview where you ask stuff about what they've made
If I'm paid for it, sure why not I could do that. I won't love it but hey I'm looking for a job, I'll put the legwork in. But if this is the only or the "preferred" interview process for a company, I need to point out that it is deeply discriminatory as it advantages people who have the time to do a weekend project: for example it benefits males disproportionally (women do most of the care work in any country, also the most house work, also have a higher chance to be a single parent, all of which impacts the time they can put in a "weekend project" if they can do it at all).
It's like asking a dentist interview candidate to show you examples of fillings and crowns they did at home as a hobby. I don't understand why there is this automatic assumption that people who program at work also do it outside of work.
I expect a decent developer to be able to bootstrap and write most of a fun toy project in a domain they know well or at worst some kata from the Internet within half an hour. Then we spend some time screen sharing and talking about it, similar to pair programming but less problem focused.
If you can't do it you'll likely struggle a lot when working with us because we commonly use throwaway prototypes.
In such settings it's also somewhat common to hire consultants in bulk, like 5-10 at a time, try them out for six months and keep the ones that enjoy the work and fit well in the organisation, and over time try to employ some of them directly.
If I were asked to make a small project over a weekend, I'd be likely to decline rather than doing a more standard interview, or I'd use AI to do it in a reasonable timeframe (which seems to defeat the purpose as it relates to this discussion)
My most recent real "side projects" are a terrible OSS monte carlo simulator tool that I contributed to, but cannot explain most of the code for, and a half-working React application that has performance issues I never fixed. Both are years old at this point. I'm not sure what an interviewer would gain from those.
Below the surface, I'm probably scratching a very interesting itch. Exploring a specific idea or problem, and then I stop when I get my answer.
Friends, family, stuff to take care of.
Not everyone gets to do so.
Outside of work, I just don't have the motivation to code anything. I don't have sufficient at-home problems where code will fix them.
In an interview, ask me anything! ... except to show you code on Github.
Worse actually. There is more to life than code - unless you are a savant. Most of us aren't.
But it is the way you are, you probably know no better and you are doing your best, what you can do is to refuse to interview.
If I already have a job, unless you are paying top of market, why would I spend my weekend writing code?
I spent the next 8 years married (still married) and raising two step sons and spent the last two and half years traveling extensively including over a year doing the “digital nomad thing”.
We have been averaging getting on a plane to do something on average over a dozen times a year since late 2021.
Of course the Covid lockdown slowed us down for two years.
When I am at home in Florida, I go swimming at one of the multiple pools or workout at one of the two gyms that’s part of our complex. It’s warm enough most of the year at least during the day.
During the weekends, I go downstairs and hang out at the bar and just sip soda while hanging out with my friend the bartender and whoever else is down there.
I “retired my wife” in 2020 when I was 46 and she was 44 8 years into my marriage so she could pursue her passion projects and we could pick up and travel as often as we wanted to - the joys of working remotely.
I’m a data engineer, so at work I mostly use SQL, Python and Bash. There’s not much overlap.
WTF is this hobby coding bullshit expectations? What other professions expect you do more work after work as a hobby and show it? Do bus drivers film themselves driving busses after work as a hobby? Do surgeons cut up people in their spare time as a hobby?
I've also done hiring, and have no idea what good leetcode would do me. I'm convinced these people thinking they "dodged a bullet" on "fakers" either spotted a real faker who they would have caught with a normal conversation anyway, or else are assuming someone with a good resume who failed their test did so because they were lying and not because they choked under the unique sort of pressure interviews present when you turn them into a dancing-monkey routine (it's approximately the same kind of stress as doing an open mic night or karaoke in front of a crowd of strangers—most people have trouble with that and lots of them fall apart at least some of the times they try it). Meanwhile, anyone who can talk about the job and their career in any depth, convincingly, without giving away that they're actually either very-green or have no idea what they're doing, while in fact not knowing how to do the job, possesses a skill at least as valuable as programming, and I find it hard to believe most such folks haven't figured out they can apply that skill directly in exchange for money instead of trying to fake their way through conversational tech interviews.
I do see how leetcode is valuable if you want to ensure that most of the candidates in your pipeline would do fine before you even evaluate them, because you offer high comp and need a way to discourage candidates who definitely can't make it before they even apply, and/or if you want to make job-hopping painful as a wink-and-nudge collusion way to keep comp suppressed. It makes sense for FAANG, in a certain way, but not because it's a good way to evaluate candidates per se.
I think programming has more commonality with other creative, 'soft' jobs like graphic design (which itself can involve programming), architecture, media, marketing, etc than meets the eye.
Many of these roles require that applicants have some sort of portfolio that can be perused by the interviewer freely. I feel co-opting that word—'portfolio'—would do us software developers a big favour instead of trivialising outside-of-work programming as 'side projects' or 'hobbies'.
I disagree. Programing is more engineering than art. Art doesn't have source code. You can show the final painting and I can show the final product I worked on but not the source code I wrote as that belongs to my employer. Also, most art like paintings are not done by large teams, so you can show what you did in that painting but in a large SW projects, I can't show what exactly form the final product I did and what else was done by my team.
Most of my valuable work in programing is engineering, especially fixing bugs, not creating portfolios to show off. I have nothing publicly to show off, mostly because firstly, it's private to my former employers, and secondly because code gets outdated and replaced fast, most of what I worte in the past probably doesn't run today anymore, but have made my employers happy and wealthy.
That also pays the bills? That's not my experience. That's what hobbies are for. Jobs are for paying bills. Paying bills with hobbies an art are a luxury for privileged.
Plus, software engineering is absolutely a creative endeavour. And I daresay normal 'engineering' (civil, mechanical, aero, etc) is a creative endeavour too; it's just a matter of egos and that seem to separate STEM versus non-STEM. There are portfolios for everything. I don't understand the desire for software engineers to just waltz into an interview, claim to have done X, Y, Z, with no proof, and secure a job.
Drawn art absolutely does have something like it:
https://www.reddit.com/r/learntodraw/comments/nibjjn/any_adv...
It could be considered similar to scaffolding or boilerplate in code, except usually none of this is visible in the end product, while the code boilerplate is always there. These lines are drawn light and completely covered up by the end result - sometimes even manually erased depending on the medium.
The result is that I don't think I've written anything longer than about a ten-line shell or python or JS script for my personal use in... a decade or more.
Frankly I probably think you shouldn't be paying anyone to do the thing you're wanting to pay me to do, because computers are likely just an expensive distraction that management's pursuing because the promise of legibility, even if in-fact pointless in this case, is incredibly enticing to them, but also I like money and will build the thing you shouldn't be building for you if you pay me. I'll even do it well, if you let me. But I don't make the same mistake (much) in my own life, any more.
Would I write a bunch of code on my own if I thought it'd be worth it? Yes, but that'd almost certainly mean I had a product idea. If I were any good at thinking of product ideas, I'd long since have had my own business. I'm terrible at it. That's literally the only reason I'm applying for a job. If I had a pile of decent code to show you, it'd be because I didn't need your job.
I can empathize with your position if you are also against expecting candidates "prepare for the interview" by leet code grinding or "brush up on CS concepts".
Hobby coding is million times better than that crap.
The word maybe is doing a lot of heavy lifting here. What if you never had to do that? Not everyone's work is public. Inf act I'd say most people's work is not public. Sometimes even the product is not public since it's B-2-B.
Our "gotcha," which doesn't apply to most languages anymore is, "What's the difference between a function and a procedure." It's a one sentence answer, but people who didn't know it would give some pretty enlightening answers.
Edit: From the replies I can see people are a little defensive about not knowing it. Not knowing it is ok because it was a question I asked people 20 years ago relevant to a language long dead in the US. I blame the defensiveness on how FUBAR the current landscape is. Giving a nuanced answer to show your depth of knowledge is actually preferred. A once sentence answer is minimal.
I'm editing this because HN says I'm posting too fast, which is super annoying, but what can I do?
The problem is: there is a very negative incentive to give honest answers. If I were to answer these questions honestly, I'd bring up some very interesting theorems (related to some deep algorithmic topics) that I proved in my PhD thesis. Yes, I would have loved to stay in academia, but I switched to industry because of the bad job prospects in academia - this is not what interviewers want to hear. :-(
> "What's the difference between a function and a procedure." It's a one sentence answer
The terminology here differs quite a lot in different "programming communities". For example
> https://en.wikipedia.org/w/index.php?title=Procedure&oldid=1...
says: "Procedure (computer science), also termed a subroutine, function, or subprogram",
i.e. there is no difference. On the other hand, Pascal programmers strongly distinguish between functions and procedures; here functions return a value, but procedures don't. Programmers who are more attracted to type theory (think Haskell) would rather consider "procedures" to be functions returning a unit type. If you rather come from a database programming background, (stored) procedures vs functions are quite different concepts.
I could go on and on. What I want to point out is that this topic is much more subtle than a "one sentence answer".
In my experience you'll be fine giving that answer assuming you're going for the kind of programming job that hires PhDs.
You remind them you have a PhD - and in something deeply algorithmic. You can successfully answer any follow-up questions from them, as you literally have a PhD in the topic they're asking about. There's no shame in entering industry because you want jobs and money - in fact, those things are precisely what the hiring manager is able to offer you.
You'd rather be in academia but it doesn't have the pay and job security? Well, the hiring manager would rather be a snowboard instructor in Aspen but doesn't for the same reason. So you've got common ground with them.
I would love to hear that from a candidate I'm interviewing. Who can't relate to the distinction between your ideal job and the job that will actually pay you money?
This is unfortunate that you would get that response. FWIW, I would be interested in hearing all this in an interview and I would look at it favorably.
>What I want to point out is that this topic is much more subtle than a "one sentence answer".
Yes, you would definitely get bonus points for nuance. The one sentence answer was minimal. What it filters out are people who don't know anything about Delphi but applying for the job with highly embellished resumes hoping to get lucky. This was for software used in hospitals, so bugs or errant code could have pretty drastic consequences.
But it's certainly not a universal question applicable to all programming languages.
Likewise, Clubber says in their post that their 'gotcha' question doesn't apply to most languages.
Lol a bit touchy aren't we?
Like I said, it's not really relevant in today's languages. It was for a Delphi/Pascal position. If you do any type of database code (like T-SQL), you would also know it. If your experience is mainly in C type languages, everything is a function so it doesn't apply.
If you hired a guy for a Delphi position who didn't know the difference between a function and a procedure, you hired the wrong guy.
procedure Hello;
begin
ShowMessage ('Hello world!');
end;
function Double (Value: Integer) : Integer;
begin
Double := Value * 2;
end;
If you're asking this question (by virtue of the present-tense "is") in the year 2025 even though by your own admission
> it's not really relevant in today's languages.
then you aren't giving candidates a good impression. Even though I would have nailed this question I would have serious reservations about any job that would ask it in an interview because it means that the person interviewing me has more concern for legacy minutiae than broad technical knowledge or problem-solving skills.
No. I just looked up other responses to your post. It's obvious you got exposed as being inexperienced (or an idiot), while posing to know the definitive with your "gotcha". Being inexperienced (or ignorant) is not a problem, but being cocky is.
Here's something to consider, Dennis. Instead of using any type of reasoning that maybe I'm interviewing for a language you aren't familiar with where functions and procedure differences matter, you decided to just go off the handle and call me inexperienced and/or an idiot. This is what we call in the hiring business a "huge red flag." I recommend maybe use some of that big brain you have and apply some deductive reasoning instead of just calling people names.
>Lol a bit touchy aren't we?
You lost your composure and decided to start calling names after this. I haven't asked it since the late 90s, early 2000s. It was a for a Delphi position. It's a bonehead easy question any Delphi developer who got to chapter 2 of any Delphi book would have understood. It's still an applicable question for a SQL developer and it's just as easy. I even showed you sample code. I don't see why you aren't getting it.
>I haven't asked it since the late 90s, early 2000s.
You overall gave the impression that you are currently asking it.
And a personal rhetorical question - aren't you too old to even state this 'gotcha' business _today_ about what you did in the past? What made you state it? If I gave you the benefit of doubt - that was slip, where you omitted the past tense.
(If I did that in the 90's I'd be a embarrassed to even mention it today.)
Yes, but you're the only one who threw a tantrum. Enjoy your career.
Words like functions/procedures tend to have different connotations across languages and once one crosses one's 15th language, and each having some 20 different keywords, it become difficult to remember what the exact connotation of a word is, in a specific language/framework. This is the most likely situation of the guy whose post I responded to.
The exception to the rule is, if you have been working quite a bit _recently_ on a specific language. You are presumably talking about this situation.
https://stackoverflow.com/questions/721090/what-is-the-diffe...
... in Pascal/Delphi.
You would be surprised how many bad applicants interviewers get. It's only gotten worse over the last 20 years.
My answer would be along the lines of "It's 2025, no one has talked about procedures for 20+ years"
You... might want to think about what implicit biases you might be bringing here
Take maximum subarray problem, which can be optimally solved with Kadane's algorithm. If you don't know that, you are looking at the problem as Professor Kadane once did. I can't say for sure, but I suspect it took him longer than 30-45 minutes to come up with his solution, and I also imagine he didn't spend the whole time blabbering about his thought process.
I often see comments like: this person had this huge storied resume but couldn't code their way out of a paper bag. Now having been that engineer stuck in a paper bag a few times, I think this is a very narrow way to view others.
I don't know the optimal way to interview engineers. I do know the style of interview that I prefer and excel at[0], but I wouldn't be so naive to think that the style that works for me would work for all. Often I chuckle about an anecdote from the fabled I.P. Sharp: Ian Sharp would set a light meter on his desk and measure how wide an interviewees eyes would get when he explained to them about APL. A strange way to interview, but is it any less strange than interviewing people via leetcode problems?
0: I think my ideal tech screen interview question is one that 1) has test cases 2) the test cases gradually ramp up in complexity 3) the complexity isn't revealed all at once; the interviewer "hides their cards," so to speak 4) is focused on a data structure rather than an algorithm such that the algorithm falls out naturally rather than serves as the focus. 5) Gives the opportunity for the candidate to weigh tradeoffs, make compromises, and cut corners given the time frame. 6) Doesn't combine big ideas (i.e. you shouldn't have to parse complex input and do something complicated with it); pick a single focus. Interviews I have participated and enjoyed like this: construct a Set class (union, difference, etc); implement an rpn calculator (ramp up the complexity by introducing multiple arities); create a range function that works like the python range function (for junior engineers, this one involves a function with different behavior based on arity).
This is something that drives me nuts in academia when it comes to exam questions. I once took an exam that asked us to invent vector clocks from whole cloth, basically, having only knowledge of a basic Lamport clock for context. I think one person got it--and that person had just learned about vector clocks in a different class. Given some time, it's possible I could have figured it out. But on an exam, you've got like 10-15 minutes per question.
The funny thing about it is that I do the same damn thing from the other side all the time when working with students. It's incredibly tempting once you know the solution to a problem (especially if you didn't "solve" it yourself, but had the solution presented to you already) to present the question as though it has an obvious solution and expect somebody else to immediately solve it.
I'm aware of the effect, I've experienced it many times, and I still catch myself doing it. I've never interviewed a candidate for a job, but I can only imagine how tempting it would be to fall into that trap.
When I'm interviewing a candidate, I'm often asking myself if this question is just something I happen to know therefor expect the candidate to know too, or if it's crucial to doing the job?
Sometimes it may not be fair to expect a random developer to be familiar with a specific concept. But at the same time it might be critical to the kind of work we're doing.
I have 26 years of solid experience, been writing code since I was 8.
There should be a ton of companies out there just dying to hire someone with that kind of experience.
But I'm not perfect, no one is; and faking doesn't work very well for me.
heh.. they are probably dead already?
i have even longer years.. But this time i am looking since.. september? Applying 1-2 per day, on average.. Widening the fishing net each month.. ~2% showed some interest.. but no bingo.
"overqualified" is about half of the "excuses" :/
Time to plant tomatoes maybe..
Not that I mind growing tomatoes, quite the opposite :)
However, I won't do leet coding. I want to hear about why I should come work for u. What about my works makes u think I could help ubm with your problem. Then let's have a talk about your problems and where I can create value for you.
My experience in hiring is that leet coders are good one trick ponies. But long term don't become technical peers.
I do rely on HR having, hopefully, done their job and validated the work history.
I do have one technical question that started out as fun and quirky but has actually shown more value than expected. I call it the desert island cli.
What are your 5 linux cli desert island commands?
Having a hardware background, today, mine are: vi, lsof, netcat, glances, and I am blanking on a fifth. We have been doing a lot of terraform lately
I have had several interesting responses
Manager level candidate with 15+ years hands on experience. He thought it was a dumb question because it would never happen. He became the teams manager a few months after hiring. He was a great manager and we are friends.
Manager level to replace the dumb question manager. His were all Mac terminal eye candy. He did not get the job.
Senior level SRE hire with a software background. He only needed two emacs and a compiler, he could write anything else he needed.
My experience differs a lot. Many insanely skilled people are somewhat "weird" (including possibly
- being a little on the spectrum,
- "living a little bit in their own world",
- having opinions on topics that are politically "inappropriate" (not in the sense of "being on the 'wrong' side of a political fence", but rather in the sense of "an opinion that is quite different than what you have ever heard in your own bubble", and is thus not "socially accepted")
- being a little bit "obnoxious" (not in bad sense, but in a sense that might annoy a particular kind of people))
What you consider to be "skilled people" is what I would rather call "skilled self-promoters" (or possibly "smooth talker"). "Skilled people" and "skilled self-promoter" are quite different breeds of people.
I am actually a bit weird myself, so I can relate.
> What you consider to be "skilled people" is what I would rather call "skilled self-promoters". "Skilled people" and "skilled self-promoter" are quite different breeds of people.
I don't mean that they have told me that they are skilled, or that their resume has implied it. I mean that they actually have the skills. Self-promoters that don't know the information always look good on paper, but after a few minutes of talking to them you can tell that they don't quite match.
Before IT, I was a live sound engineer TV, theater, music. There was also a entertainment university starting up around the same time. They were pumping out tons of "trained" engineers that looked good on paper but couldn't mix for shit. I think we can blame them for the shitification of pop music.
My experience differs here: these are not "good self-promoters", but impostors.
Good self-promoters typically have some above-average (though commonly not really exceptional) skills in their area, but their expertise is in the capability of smooth talking (including smalltalk), promoting their contributions, and talking at eye level with various stakeholders.
If you are really exceptional in your area, you will often (though not always) consider smalltalk to be waste of your time, and will often have difficulties talking at eye level with various stakeholders, because either they are not sufficiently knowledgable in your area of expertise to understand you, or the other way round (for the latter point: becoming really great in one area often means that you won't have the time to get sufficiently deep into a lot of other areas, even though for some of them you might become quite skilled if you had more time).
Sadly, out of 100s of people, I’ve probably only gotten an interesting response a handful of times. Mostly people say some well known tech from the job description.
I never held that against anyone, but the people who had an interest in something were more fun to work with.
Sir, you have attained dizzying intellectual heights that few men have.
My comment is meant to be a compliment, not snarky. And indeed I have noticed that the best people I have encountered can often size people up accurately with very general questions often on unrelated subjects.
grep? (or ripgrep if allowed)
Are you familiar with busybox ?
Huh? Please explain
Also, the best (albeit the most expensive) selection process is simply letting the new person to do the actual work for a few weeks.
What kind of desperate candidate would agree to that? Also, what do you expect to see from the person in a few weeks? Usual onboarding (company + project) will take like 2-3 months before a person is efficient.
You don't need him to become efficient. Also I don't think it is always necessary to have such long onboarding. I'll never understand why a new hire (at least in senior position) can't start contributing after a week.
Ok... take me through it. I apply to your company and after a short call you offer me to spend 4 weeks working at your place instead of an interview.
I go back to my employer, give them resignation letter, work the rest of my notice period (2 months - 3 months), working on all handovers, saying goodbyes.
Unless the idea is to compensate me for the risk (I guess at least 6 months salary, probably more), then I do not see how you'd get anyone who is just a poor candidate to sign up for this.
> You don't need him to become efficient
So what will you see? Efficiency, being independent and being a good team player are the main things that are difficult to test during a regular interview.
You can say that about all forms of hiring process. If you're unemployed, you obviously have more time: to spend more time on the take-home assignments (which I hate, see another thread [1]), to add more stuff to your GitHub profile, to go to more interviews, etc.
Yes, but there's a significant difference between spending a few hours on a take-home assignment and dropping your current employment to spend 4 weeks potentially in another city working full time.
Because you have zero context of what the org is working on.
Unless your role is trivial to replace with an LLM, you need to understand the business. Maybe not for really junior role, but everything above - you need to solve issues. Tech is just a tool.
Let's say you're hiring manager for a company that compares flight tickets, something similar to Google Flights or Skyscanner. You need three additional Rust engineers. You're located in Palermo, Italy.
How do you hire people that would not only know Rust, be willing to move to Palermo, or at least visit occasionally, but also know the airfare business?
Even if you're willing to have people remotely, in the same region, how many unemployed Rust developers that know that business are on the market? 0?
Ideally, yes. It's a common occurrence among large organisations. Google and Apple used to even have an anti-poaching agreement.
> How do you hire people that would not only know Rust, be willing to move to Palermo, or at least visit occasionally, but also know the airfare business?
Rust isn't Boring, which is why you don't do that and hire one of many Java developers and do Java, unless the tradeoff is really worth it.
For data size, if you're a medium-ish company, you may only hire a few engineers a year (1000 person company, 5% SWE staff, 20% turnover annually = 10 new engineers hired per year), so the numbers will be small and a correlation will be potentially weak/noisy.
For confounders, a bad manager or atypical context may cause a great engineer to 'perform' poorly and leave early. Human factors are big.
Sure, the confidence intervals will be wide, but it doesn't matter, even noisy data are better than no data.
Maybe some companies already do this, but I didn't see it (though my sample is small).
I fail to see why this wouldn't be the obvious choice. Do we disallow linters or static analysis on interviews? This is a tool and checking for skill and good practices using it makes all sense.
Everyone is so terrified of hiring someone that can’t code, but the most likely bad hires and the most damaging bad hires are bad because of things that have nothing to do with raw coding ability.
*Except the performing arts. The way we interview is pretty close to the way musicians are interviewed, but that’s also really similar to their actual job.
In some cases it is obvious they are blathering a stream of words they don't understand. But others are able to hold something resembling a coherent conversation. We also have to allow for the fact that most people we interview aren't native English speakers, and are talking over Teams. It can be very hard to tell if they are cheating.
Asking questions to probe their technical skills is essential, otherwise you are just selecting for people who are good at talking and self promotion. We aren't just asking trivia questions.
We also give a simple code challenge, nothing difficult. If they have a working knowledge of the language, they should be able to work through the problem in 30 minutes, and we let them use an IDE and google for things like regex syntax.
Some of them are obviously using an AI, since they just start typing in a working solution. But in theory they could be a Scala expert who remembers how to use map plus a simple regex...
I was asked whether I used AI/LLM for the solution. I didn't. I felt like using an LLM to solve the problem for me wasn't the right way of showcasing knowledge. The role was for some form of 'come in with knowledge and help us'.
The response to that was basically: everybody here uses AI.
I declined the follow-up interview, as I felt that if all you have is the speed of AI to be ahead of your competitors, you're not really building the kind of things that I want to be a part of. It basically implies that the role is up in the air as soon as the AI gets better.
To me AI is just another tool that helps me solve problems with code. An auto complete on steroids. A context aware stack overflow search. Not wanting to adopt or not even work somewhere where colleagues use it, sounds to me like coding in notepad AND in the process scoffing those who use an IDE.
Besides, if AI gets to the point it can replace you, it will replace you. Better to start learning how to work with it so you can fill whatever gap AI can't.
I've seen the type of code AI generates. It might work, but if you think that's good or that massaging it so it works will make you any better, I have some bad news for you...
Second-rate companies will keep some superficial coding, but will start to emphasize more of the verbal parts like system design and retrospective. Which sucks, because those are totally subjective and mostly filters for whoever can BS better on the spot and/or cater to the interviewer's mood and biases better.
My favorite still: in-person pair programming for a realistic problem (could be made-up or shortened, but similar to the real ones on the job). Use whatever tools you want, but get the correct requirements and then explain what you just did, and why.
A shorter/easier task is to code review/critique a chunk of code, could even just print it out if in person.
I had a couple of people who, when asked to explain specific approaches reflected in their code, very obviously typed my question right back into ChatGPT and then recited its output verbatim. Those interviews came to an end rather quickly.
One of my favorite ones was when I asked a candidate to estimate the complexity of their solution, and ChatGPT got it wrong, giving O(log(n)) for an O(n) algorithm. When I asked leading questions to see if the candidate could see where the error came in, they starting verbatim reciting a dictionary definition of computational complexity, and could not address the specifics of the problem at all.
Not looking forward to it.
SV Startup hiring is the most trendy and not representative.
How long that will remain true is a very open question, where different folks have widely differing timelines on when they expect AI to have highly meaningful impacts.
I realize it's not easy for smaller companies to do, but I think it's the single best way to see if someone's fit for a job
We get to touch on client and browser issues, graphQL, Postgres, Node, Typescript (and even the various libraries used). It includes basic CRUD functionality, a third party API integration, basic security concerns and more. It's just meant to gauge a minimal level of fluency for people that will be in hands on keyboard roles (juniors up to leads, basically). I don't think anyone has found a way to use AI to help them (yet) but if this is too much for them they will quickly fall flat in the day to day job.
Where we HAVE encountered AI is in the question/answer portion of the process. So far many of those have been painfully obvious but I'm sure others were cagier about it. The one incident that we have had that kind of shook us was when someone used a stand-in to do the screen (he was fantastic, lol) and then when we hired him it took us about a week to realize that this was a team of people using an AI avatar that looked very much like the person we interviewed. They claimed to be in California but were actually in India and were streaming video from a Windows machine to the Mac we had supplied for Teams meetings. In one meeting (as red flags were accumulating) their Windows machine crashed and the outline of the person in Teams was replaced by the old school blue screen of death.
In a world where every company beleives (or wants to beleive) that they are doing some ground-breaking, bleeding edge work (see any tech company blog and you can only find hyped technologies in there), I do not think one can expect companies to do a fair assessment of if they really are doing such work.
I don't see AI as a serious threat to the interview process unless your interview process looks a lot like hackerrank.
> Excluding places that do that leaves you with what exactly? Boutique shops filled with 20 year veterans?
We are on a VC forum—I imagine small shops focused on quality are quite common here.
You are correct about the deficiencies of the whiteboard interview. It is not a sane way to hire an individual. It makes sense as a way to hire someone in the top 20% from a large unfiltered pool. So wrt high/low, that’s what FAANG companies have to do, and for many nontechnical companies they outsource this work or emulate FAANG practices for no good reason.
My point was that there are very few places that don’t do this.
I just abandoned the code interview altogether and ask them questions about process. It's a very simple workaround, but very effective. I'll admit it helps that there are very few problems these days outside of specific problems to solve that require a high degree of technical competency to tackle.
In reality, job postings should be incredibly specific, with specificity rising as the role requires more experience and problem solving. You'll get less applicants (or will be able to clearly screen out the people who don't meet the specific requirements) but you'll get ones that actually match what you are looking for and can actually solve the problem your company is trying to solve with filling the role. Then the conversation/interview is much more important and both sides feel like they have some "stakes in the game".
Congratulations, you are now a "Certified Leetcoder (tm)". :-(
Seriously: what a lot of people write down here is that a lot of programming jobs don't involve code puzzle skills, but are often rather about putting stuff/APIs together in the currently fashionable framework.
This makes becoming a Certified Leetcoder (tm) just another useless hoop to jump through.
(Just to be clear: for those few programming jobs that demand the employee to solve algorithmic puzzles regularly, doing them in a job interview makes sense. But these jobs are rare.)
And this differs from the status quo how? Employers obviously find value in this signal for better or worse. We’re just making it so it only needs to be done once, by trained proctors, instead of for every position you apply for.
I've considered something like that for my current company, which is doing basically the same thing, but:
(a) this has not, in practice, been a problem for us in identifying good candidates
and, far less importantly:
(b) you need very high scale to have interviewers everywhere that candidates are OR you're paying extra for a third-party controlled environment
(c) scheduling and cancellations become more difficult and costly respectively
I think the same applies to good tech interview. Company should adapt hiring process to friend with AI, not fight.
Maybe the future will be human shills pretending to be job candidates for shady AI “employment agencies” that are actually just (literally) skinning gpt6 apis that sockpuppet minimum wage developing nation”hosts”?
Show code, ask questions about it that requires opinion.
Interviews are screening for multiple things, not just the ability to do one specific technical job. More often than not, technical coding ability is not even at the top of the priority list. Interviews are looking for well-rounded candidates who can do more than 1 job. Companies want to know if you can change jobs easily, they want to know if you’re average, better than the average programmer, or exceptional. They want to know if you’ll make a good manager after a few years, how good you are with people, how well you prioritize and communicate.
I had a professor in college that graded tests with the median skewed low, centered on a D. He complained that the usual practice of putting it on C or B made it so he could clearly see the difference between F-, F, and D- students, while the A students were all clumped together. He wanted to identify the hard workers and superstars in the class, see who was A vs A+ vs A++. It freaked everyone out when grades came out much lower than expected, but he renormalized at the end and people with test scores in the Cs and Ds got As and Bs in the class.
Be careful what you wish for. It’s competitive right now and interviews that limit screening to ability to do basic job-level coding and don’t screen for knowledge and soft skills and exceptionalism will make it harder for people who are good to demonstrate they’re better than people who are mediocre or use AI. Is that what you want?
Collaborate, as opposed to just do.
Things that really tell me if I can work with that person and if together, we can make good things.
Unless the job you're interviewing for is remote-only, this makes perfect sense. If you expect your candidates to be able to work in your office, they should be interviewed there.
"Coding", as in writing stuff in programming languages with correct syntax that does the thing asked for in isolation, has always been a very dumb skill to test for. Even before we had stackoverflow syntactic issues were something you could get through by consulting a reference book or doing some trial and error with a repl or a compiler. That this is faster now with internet search and LLMs is good for everyone involved, but the fact that it's not what matters remains
The important part of every job that gets a computer to do a thing is a combination of two capabilities: Problem-solving, that is, understanding the intended outcome and having intuition about how to get there through whatever tools are available, and frustration tolerance: The ability to keep trying new* stuff until you get there
Businesses can then optimize for things like efficiency or working well with others once those constraints are met, but without those capabilities you simply can't do the job, so they're paramount. The problem with most dinky little coding interviews wasn't that you could "cheat", it's thst they basically never tested for those constraints by design, though some clever hiring people manage to tweak them to do so on an ad hoc basis sometimes
* important because a common frustration failure mode is repetitive behavior. Try something. Don't understand why it doesn't work. Get more frustrated. Try the same thing again. Repeat
Tech interviewing has become a weird survival game, and now AI is flipping the rules again. If you need a laugh: https://codingfornothing.com
We need to know that the developer actually has skills and isn't just secretly copying the answer off of a hidden screen. We are interviewing now, and some cantidates are obviously cheating. Our interview process is not leet code based, and reasonably chill, but we will probably have to completely rethink the process.
Since we are hiring contractors, in theory we can let them go after a couple months if they suck, but we haven't tested out how this will work in practice.
Everyone on the planet is a one person startup, using AI and robotics to do all the actual work.
Having a job has always been a necessary evil. If the time is coming when that changes, all the better. If we can't figure out how to behave in the presence of plenty, we don't deserve utopia anyway.
Companies currently provide this.
As an interviewee it's insane to me how many jobs I have not gotten because of some arbitrary coding problem. I can confidently say after having worked in this field for over a decade and at a FAANG that I am a very capable programmer. I am considered one of the best on every team I've been on. So they are definitely selecting the wrong people IMO.
* Take a candidate's track record into account. Talk with them about it.
* Show that you're experienced yourself, by being able to tell something about what someone would be like to work with, by talking with them.
* Get a reputation for your company not tolerating dishonesty. If someone cheats in an interview and gets caught, they're banned there, all the interviewers will know, and the cheater might also start to get a reputation beyond that company. (Bonus: Company reputation for valuing honesty is attractive to people who don't want dishonest coworkers.)
* Treat people like a colleague, trying to assess whether it's a good match. You're not going to be perfectly aligned (e.g., the candidate or the company/role might be a bit out of the other's league right now), but to some degree you both want it to be a good match for both parties. Work as far as you can with that.
(Don't do this: Leetcode hazing, to establish the dynamic of them being there to dance for your approval, so hopefully they'll be negged, and will seek your approval, won't think critically about how competent and viable your self/team/company are, and will also be less likely to get uppity when you make a lowball offer. Which incidentally places the burden of rehearsing for Leetcode ritual performances upon the entire field, at huge cost.)
In parallel, we asked interviewers to use one of 3 question types: verbatim LeetCode questions, slightly modified LeetCode questions, and completely custom questions.
The full writeup is here: https://interviewing.io/blog/how-hard-is-it-to-cheat-with-ch...
TL;DR:
- Interviewers couldn't tell when candidates were cheating at all
- Both verbatim and slightly modified LeetCode questions were really easy to game with AI
- Custom questions were not gamable, on the other hand[1]
So, at least for now, my advice is that companies put more effort into coming up with questions that are unique to them. It's better for candidates because they get better signal about the work, it reduces the value asymmetry (companies have to put effort into their process instead of just grabbing questions from LeetCode etc), and it's better for employers (higher signal from the interview).
[1] This may change with the advent of better models
I recently fed it into ChatGPT and asked it to do the assignment. It did it perfectly -- I read the code in detail and couldn't find any issues.
So custom questions are off the table now, too. We'll be using a code review instead for the next round.
AI might make e.g. your leetcode interview less predictive than it previously would have been. But was it predictive in the first place? I don't think most interviews are written by people thinking in those terms at all. If your method of interviewing never depended on data suggesting it actually, you know, worked in the first place, why would it matter if it starts working even worse?
Insofar as it makes the shittiness of those interviews more visible, the effect of AI is a good thing. An interview focused on recall of some specific algorithm was never predictive, it's just now predictive in a way that Generic Business Idiots can understand.
We frequently interview people who both (a) claim to have been in senior IC roles (not architect positions, roles where they are theoretically coding a lot) for many, many years and (b) cannot code their way out of a paper bag when presented with a problem that requires even a modicum of original reasoning. Some of that might be interview nerves, of course, but a lot of these people are not at all unconfident. They just...suck. And I wonder if what we're seeing is the downstream effects of Generic Business Idiots hiring primarily people who memorize stuff than people who build stuff.
Another possibility is that their job subtly drifted.
I wrote a lot of code as a grad student but my first interviews afterward were disasters. Why? Because I’d spent the last few months writing my thesis and the few months before that writing a very specific kinds of code (signal processing, visualization) that were miles away from generic interview questions like “Make the longest palindrome.”
Do you feel like there's anything there that any reasonably skilled programmer shouldn't be able to figure out on the fly?
[1] https://www.otherbranch.com/shared/practice-coding-problem
The article addresses this:
>A lot of companies are doing RTO, but even companies that are 100% in-office still interview candidates from other cities. Spending money to fly every candidate out without an aggressive pre-screen is too wasteful.
No, accidently hiring someone who AI'd their way through the interview costs orders of magnitude more to undo. It's absolutely worth paying for a round trip flight and a couple days of accommodations.
It also feels like interviewers know this and assume you studied the questions, they seem incapable of giving hints, etc if you don't have the questions memorized.
AI is the least of it.
I've built a search engine for two countries and then I was failed by a guy that wears cowboy hats to work at google in Ireland. Not a lot of cows there I'm guessing. (No offence to any real cowboys that work at google of course).
I did like the free flight to Ireland though and the nice lunch. Though I was disappointed I lost "Do no evil" company booklet.
Dang! I knew it was a mistake leaving my hat at home. Little things like that people tend to forget.
Of course the problem is this can't scale or be outsourced to HR, but is this a bug or a feature?
That's the only way you're going to get relevant information.
I've been verified to the moon and back by Apple and others for roles that could never have worked.
The problem is that when it comes to the hiring process, everyone is suddenly an expert; no matter how dysfunctional, inhumane and destructive their ideas are.
isnt it that simple?
This blocks their screen too.
and yes we do know very soon if you look somewere else, take time or rephrase the question to get more time.
If you able to fake it, at that point you should just get th ejob anyway :P
There's a huge difference between occasionally looking up something, and practically leaning on it. Ironically, the mass degradation of search engine result quality within the past ~decade has made it much harder for people to do the latter, and when they do, it shows much more clearly.
> Architectural interviews are likely safe for a few years yet. From talking to people who have run these, it’s evident that someone is using AI. They often stop with long pauses, do not quite explain things succinctly, and do not understand the questions well enough to prompt the correct answer. As AI gets better (and faster), this will likely follow the same fate as the rest but I would give it some years yet.
Completely matches my experience. I don't do leet code BS, just "let's have a talk". I ask you questions about things you tell me you know about, and things I expect of someone at the level you're selling yourself at. The longest it's taken me to detect one of these scumbags was 15 minutes, and an extra 5 minutes to make sure.
Some of them make mistakes that are beyond stupid, like identity theft of someone who was born, raised and graduated in a country whose main language they cannot speak.
The smartest ones either do not know when to stop answering your questions with perfect answers (they just do not know what they're supposed to not know), or fumble their delivery and end up looking like unauthentic puppets. You just keep grinding them until you catch em.
I'm sure it's not infallible, but that's inherent to hiring. The only problem with this is cost, you're going to need a senior+ dev running the interview, and IME most are not happy to do so. But this might just be what the price of admission for running a hiring pipeline for software devs is nowadays. Heck, now feels like a good time to start a recruitment process outsourcing biz focused on the software industry.
I once got a guy that claimed to have implemented multiple default HTTP JSON REST APIs and somehow had never:
- tested his API with JSON payloads, serialise serialise - never queried his APIs manually or semi automatically (no knowledge of curl, postman or anything similar)
Interviews based on personal feelings have hidden biases not even the interviewer is aware of.
I'm not sure if you guys have been in charge of hiring, but there's no real alternative. In my most recent experience, we had one open position, and after interviewing 10 candidates, 3 of them were basically identical in terms of technical qualifications. How do you choose one over the other, other than the "vibes"? Anyone suggesting otherwise is either living in a weird alternate reality, or doesn't want to accept that working is a cooperative job and interpersonal relationships are very important.
There always will be exceptions for different type of roles and specializations, but that's not what I'm talking about.
Likely. But haters gonna hate, and lawyers gonna sue.
This is effectively how Harvard was rejecting Asian applicants. They created a "personal fit" / cultural fit quality that Asians scored low on . Supreme Court found this to be discrimination.
It doesn't matter if you are truly discriminating, it matters how well you have tangible evidence of the employee not meeting the qualifications for the role.
Think of how poor the screening process is at the recruiter & CTS (left side) of the funnel, and how many false negatives there are .
If you could offer standardized test at that level, you may be able to keep viable candidates in the funnel longer.
but the screening cost for companies is eye watering so something should be done.
It's a common sentiment.
But compare https://www.cambridge.org/core/journals/judgment-and-decisio... . ("People predicting the future performance of college students state that interviewing the students aids prediction, although in fact the interviews make predictions less accurate.")
Of course, they'd miss out on some good talent. But in the article where it shows the quote of someone getting rejected for not inverting a binary tree on a whiteboard, that doesn't seem like a terrible thing to test for.
Big Tech / Unicorn / Wannabe Unicorn prescreens are all basically standardized now anyway.
All of these technical interviews still suck though! I basically never code with someone watching me and find it very difficult to do in interviews. I also find it hard to find the time to actually practice this skill
Additionally, Undergraduate programs in the US and Canada, at least used to, despite their varied reputation, have a pretty standard program.
Maybe things have deteriorated so far at the high school and university level that new standardized exams are needed. But we also have a plethora of verifiable certifications whose exams are held in independent test facilities.
This sort of comp-sci style exam with quizzes and what not maybe somewhat helps when hiring junior with zero experience fresh out of school.
But why are people with 20+ years of easily verifiable experience (picking up a phone and asking for references is still a thing!) being asked to invert trees and implement stuff like quicksort or some contrived BS assignment the interviewer uses to boost their own ego but with zero relevance to the day to day job they will be doing?
Why are we still wasting time with this? Why is always the default the assumption there that the applicants are all crooked hochstaplers that are lying on their resumes?
99% of jobs come with probationary period anyway where the person can be fired on the spot without justification or any strings attached. That should be more than enough time to see whether the person knows their stuff or not after having passed one or two rounds of oral interviews.
It is good enough for literally every other job - except for software engineering. What makes us the special snowflakes that people are being asked to put up with this crap?
True.
> One of the things we can do, however, is change the nature of the interviews themselves. Coding interviews today are quite basic, anywhere from FizzBuzz, to building a calculator. With AI assistants, we could expand this 10x and have people build complete applications. I think a single, longer interview (2 hours) that mixes architecture and coding will probably be the way to go.
Oh.... yeah, that sounds just... great.
If that's where the things are going, I'm retraining to become a line cook at McDonalds.
I had a ‘principal engineer’ at last place who grinded leetcode for 100 days and still failed a leetcode interview. It’s utter nonsense.
A conversation with technical questions and topics should suffice. Hire fast and fire people.
The image below does sum it up but not in the way the author thinks.
Google wants to hire people who complete their hiring process. They're OK with missing out on some people who would be excellent but who can't/won't make it through their hiring process.
The mistake may lie in copying Google's hiring process.
They're acting like all jobs are remote and it's impossible to do an interview in person.
Also, does it really matter? If a person is good at using AI and manages to be good at creating code with that, is it really so much worse than a person that does it from the top of their head? I think we have to drop the idea that AI is going to go away. I know it's all overhyped right now but there is definitely something to it. I think it will be another tool in our toolboxes. Just like stackoverflow has been for ages (and that didn't kill interviews either).
Yes, it is disgusting. Sadly also very common.
Moving personnel between countries when they are already working for the company does happen. They did it for me. But at that point they already know what they have.
Both scenarios are easily verifiable (can check that you released the project or if you made that commit or not) and in the case of open-source, the interviewer can lookup at how you code-review with others, and how you respond and reason about the code review comments of others all in public to see if you actually understand the patches you or another person submitted.
A conversation can be started around it and eliminates 95% of frauds. If the candidate cannot answer this, then no choice but give a leetcode / hackerrank hard challenge and interview them again to explain their solution and why.
A net positive to everyone and all it takes to qualify is to build something that you can point to or contribute to a significant open source project. Unlike Hackerrank which has now become a negative sum race to the bottom quest with rampant cheating thanks to LLMs.
After that, a simple whiteboard challenge and that is it.
For a candidate who does have OS contributions, that’s great but most will not. And the more senior they are the less likely I would imagine.
What you need to do is see how well they can design before writing software. What is their process for designing the software they make? Can they architect it correctly? How do they capture user's mental models? How do they deal with the many "tops" that software has?
I can create a simple project with 20 files, where you would need to check almost all of them to understand the problem you need to solve, good luck feeding that into an LLM.
Maybe you have some sneaky script or IDE integration that does this for you, fine, I'll just generate a class with 200 useless fields to exhaust your LLM's context length.
Or I can just share my screen and ask you to help me debug an issue.
Also wtf is inverting a binary tree ? Like doing a "bottom-view". That shit is easy.