> How you behave, how you relate to others, how you approach problems and the energy you bring to a team matter as much as your technical ability, and often more.
As much as I agree with this, I don't think that the whole rest of the post (about how best to examine LLM usage in an interview loop) coheres with this.
You cannot conclusively test for common decency in either interviews or recommendations. People are on their best behavior in interviews, and it's rare for someone to pass along a recommendation that will criticize them (or their behavior). Length of tenure at prior organizations is not proof, since too many will tolerate poor behavior. Accept that this is a risk involved with any hire, have clear, written business conduct policy, and be willing to show the door to people who cannot control their behavior after written warnings.
Asking how people approach problems with LLMs need not be more than a conversation. Architectural interviews need little more than a whiteboard. If you wouldn't hover over the shoulder of an employee while they work, why do you feel the need to do so in an interview?
I get it - bad hires are expensive in more ways than one. But screen-to-offer should not take longer than two weeks in small companies. Interviews help you derisk, but do not ever fool yourself into thinking that an interview process will ever reduce your hiring risk to zero. Getting a handful of dates before a shotgun wedding will always be risky. Accept it and learn to be more comfortable with risk.
> the energy you bring to a team matter
This is all the more true for hiring managers. Bad management will take strong, high-energy, high-ownership performers and handcuff them. Good management can, sometimes, take people with weaker skills, train them, set them up for success, and inspire them into becoming great performers. Someone who is interviewing, who might accept your job offer, is also taking a risk on whether you are the bad kind or the good kind of manager. The only difference is that job seekers often have more appetite for that risk than job offerers.
What's the risk in giving someone a deepseek api key that is loaded with $5? Five dollars in deepseek tokens is enough for like a week of constant use.
Why not put a proxy in front of the model, give each candidate a key with a hard budget, and revoke it after? If you require AI for the interview, you should pay for it
I like the idea to consider the token cost for the candidate, good.
But why not keep it "on paper", asking how the candidate would use an LLM or agent and why?
Actually using an agent means asking the candidate to deliver something that you must validate. And you then need to evaluate the candidate by looking at the output of the LLM indirectly.
Isn't it better to directly ask the candidate to explain how an LLM could be used (or if/when it should be used) for your use cases?
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.
virtual card. some of the startup finance/banking tools can issue one for a fixed dollar amount. candidate uses it to bump their own sub for a month or top up credits.
Setup a tcp/http proxy behind auth, ask candidates to use that endpoint, set that proxy to add whatever you need for properly authed LLM inference. You get both instrospection + reuse whatever subscriptions you have. Mention to the candidate that the requests go through your infra though, so you set the right privacy expectations.
If you're looking for a job without a stable source of income every expenditure matters. You need to cover essentials -- rent/mortgage, clothing, food, utilities, etc. Those give you a baseline burnthough rate: how long you can survive on just your savings. Any additional expense decreases that and makes it harder to build back up if doing things like saving for a house.
I've been a broke student with loans where I had to decide between gas to go to an interview or eating that day. It's not about 20$, is about how morally ethic it is to ask people to use their own personal money to just have a chance to get hired and how to try and minimise that.
> How you behave, how you relate to others, how you approach problems and the energy you bring to a team matter as much as your technical ability, and often more.
As much as I agree with this, I don't think that the whole rest of the post (about how best to examine LLM usage in an interview loop) coheres with this.
You cannot conclusively test for common decency in either interviews or recommendations. People are on their best behavior in interviews, and it's rare for someone to pass along a recommendation that will criticize them (or their behavior). Length of tenure at prior organizations is not proof, since too many will tolerate poor behavior. Accept that this is a risk involved with any hire, have clear, written business conduct policy, and be willing to show the door to people who cannot control their behavior after written warnings.
Asking how people approach problems with LLMs need not be more than a conversation. Architectural interviews need little more than a whiteboard. If you wouldn't hover over the shoulder of an employee while they work, why do you feel the need to do so in an interview?
I get it - bad hires are expensive in more ways than one. But screen-to-offer should not take longer than two weeks in small companies. Interviews help you derisk, but do not ever fool yourself into thinking that an interview process will ever reduce your hiring risk to zero. Getting a handful of dates before a shotgun wedding will always be risky. Accept it and learn to be more comfortable with risk.
> the energy you bring to a team matter
This is all the more true for hiring managers. Bad management will take strong, high-energy, high-ownership performers and handcuff them. Good management can, sometimes, take people with weaker skills, train them, set them up for success, and inspire them into becoming great performers. Someone who is interviewing, who might accept your job offer, is also taking a risk on whether you are the bad kind or the good kind of manager. The only difference is that job seekers often have more appetite for that risk than job offerers.
What's the risk in giving someone a deepseek api key that is loaded with $5? Five dollars in deepseek tokens is enough for like a week of constant use.
Why not put a proxy in front of the model, give each candidate a key with a hard budget, and revoke it after? If you require AI for the interview, you should pay for it
With the added benefit that you can monitor it and review the cost and usage patterns of the candidates vs the result, maybe even get some cool evals.
I like the idea to consider the token cost for the candidate, good. But why not keep it "on paper", asking how the candidate would use an LLM or agent and why? Actually using an agent means asking the candidate to deliver something that you must validate. And you then need to evaluate the candidate by looking at the output of the LLM indirectly. Isn't it better to directly ask the candidate to explain how an LLM could be used (or if/when it should be used) for your use cases?
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.
Is there not a way for candidates to use tokens via the company's own subscription during the test?
interviewed with one company that gave a virtual card with $500 on it for tokens. that seemed super generous and reasonable.
Not an easy way afaik :/ but if someone knows a good way please let me know
Create your own endpoint with some token limit the candidate would use.
virtual card. some of the startup finance/banking tools can issue one for a fixed dollar amount. candidate uses it to bump their own sub for a month or top up credits.
I think finance will hang me upside down if I even mention that we need to provision virtual cards for every candidate
Setup a tcp/http proxy behind auth, ask candidates to use that endpoint, set that proxy to add whatever you need for properly authed LLM inference. You get both instrospection + reuse whatever subscriptions you have. Mention to the candidate that the requests go through your infra though, so you set the right privacy expectations.
"Software engineers are poor and can't afford $20/month".
linkedin-drivel.
If you're looking for a job without a stable source of income every expenditure matters. You need to cover essentials -- rent/mortgage, clothing, food, utilities, etc. Those give you a baseline burnthough rate: how long you can survive on just your savings. Any additional expense decreases that and makes it harder to build back up if doing things like saving for a house.
Exactly, I canceled my $20/mo Emby subscription the morning after I lost my job.
I've been a broke student with loans where I had to decide between gas to go to an interview or eating that day. It's not about 20$, is about how morally ethic it is to ask people to use their own personal money to just have a chance to get hired and how to try and minimise that.