“Design me a highly resilient database”

(nikogura.com)

52 points | by donutshop 5 days ago ago

49 comments

  • ofrzeta 10 minutes ago

    "Kubernetes backed by S3-compatible object storage for WAL archiving — giving you 11 nines of durability on your backups .. " - that might be the case with AWS S3 but maybe not with Minio or even Ceph when you need to operate it yourself.

    Anyway that interview question is so stupid it hurts. I would never expect just a simple answer to an interview question and mark it off as "done". It's much more valuable to get into a dialogue and see how the candidate is approaching things.

  • LPisGood 3 hours ago

    I’m not usually one to complain about AI being used to generate content because I find this an especially interesting topic, but it really took me out of it.

    More relevant to the topic at hand, I feel like I didn’t put that much thought into designing my interview questions, but I’m always astounded when I read these sorts of descriptions of other interview questions. There should perhaps be some formal training or certification or something for asking good engineering interview questions.

    • Freedom2 3 hours ago

      There was one person on this site who looked down at candidates (for a normal software development job) who didn't answer his interview question well. And by well, I don't mean being curious and working it out on the fly, I mean that if the candidate didn't know the answer of the top off their head, the HN commenter deemed them uncurious and unsuitable. The question?

      "How do mobile phone towers work?"

      • glenpierce 3 hours ago

        Electricity goes in and phones come out, duh.

      • Joel_Mckay 2 hours ago

        Interview questions get absurd when people are gatekeeping for various reasons.

        People steal/phish resumes off sites all the time, and it is really hard to remain polite while listening to someone lie. I've met 300lb round cons that claim they regularly climb broadcast towers =3

    • nubg 2 hours ago

      [flagged]

  • wewewedxfgdf 4 hours ago

    The best interview question to find great developers is this:

    "What number am I thinking of?"

    Smart developers - really smart developers are always right. Truly great developers have seen this sort of problem before and know how to attack it - they need no fancy questions or "deeper understanding". They simply hit the whiteboard, do the logic and output the answer.

    Don't settle for anything except A players.

    • jasonwatkinspdx 3 hours ago

      I'm glad you ask this, as it signals very clearly the need to get up and walk out of the room.

      Same with your last sentence.

    • tmoertel 3 hours ago

      > "What number am I thinking of?"

      I don't think this is a good interview question, but I do think it is interesting as a thought exercise.

      I bet that perhaps 25% of candidates could actually answer the question correctly, even if they didn't know anything about Schelling points. It could also lead to some nice discussions about how to solve an open-ended problem, probability distributions, strategies for maximizing payoffs when making decisions in the face of uncertainty, and so on. The question is so bad, it's actually kind of good.

    • roncesvalles 3 hours ago

      The solution to this entirely depends on whether you are willing to answer other questions about the number and of what nature. Would you answer: is it higher/lower than X? How about how many digits there are?

      But assuming you are unwilling to answer any additional questions, the correct solution is to simply brute force every number up from 0 and its negative.

    • mooreds 3 hours ago

      _thinks: "gotta be a sarcastic comment"_

    • fogzen 3 hours ago

      What answer are you looking for?

    • doublerabbit 3 hours ago

          % isIt3
          10
          0
          6
          4
          8
          0
          9
          5
          I found three
          ----
          proc isIt3 {} {
           variable 3 [expr {int(rand()*12)}]
            switch $3 {
             3 { puts "I found three" }
             default { puts "$3" ; isIt3 }
            } ;# end switch
          } ;# end proc
      
      The number you're thinking of is 3. (-:
      • bitwize 3 hours ago

        So how well would a candidate do if they looked at that and thought "whoa, is Tcl really tail recursive?!"

  • tombert 3 hours ago

    At a job interview in 2016, I was given a fairly typical "design Twitter" whiteboard problem, so I started doing a fairly typical thing involving load balancers and Nginx and the like.

    The topic of the database came up, and I asked "Do we care more about consistency or availability?".

    He responded back with "we want both".

    I said "I mean, I would like both too but I'm pretty sure we can only get one"

    He disagreed and we went back and forth, and eventually I pull out my phone and pull up the Wikipedia for CAP theorem. He eventually (and begrudgingly) agreed and said "umm, I guess availability", so I used Cassandra and moved on with it, though five minutes were wasted with this pointless argument.

    To be fair, they did make me an offer, and it was even a decent enough offer (though I ended up taking a different job), but I remember being kind of flabbergasted that I had to explain CAP theorem to a person who had like a decade more experience than I did. It's not like CAP theorem is this super deep arcane topic; I'm pretty sure it's one of the first things you learn about when getting into distributed systems. It was certainly one of the first thing that I learned about.

  • JSR_FDED 4 hours ago

    I think it’s a great interview question. Open-ended, rewlly leads to a good discussion that can give you a sense of how the applicant approaches a problem and thinks it through.

    Disastrous in the hands of an interviewer who’s looking for a one word answer.

  • scrubs 4 hours ago

    The interviewer question was adversarial... don't think it puts them in a good light.

    More importantly I appreciate the OP for describing lessons learned through this hard experience. I worked in finance: Cassandra is no kind of answer for systems of record

    Thanks

  • seer 2 hours ago

    When I’m interviewing I never ask a question about something I know super well. I circle around what the candidate signals he has great passion and understanding at, and start deep diving into that.

    If I know about it as well, then we can have a really deep discussion, if I don’t- then I can learn something new.

    The aim when interviewing is to check how well / deeply the interviewee can think through a problem.

    If we pick a topic that they don’t have deep knowledge - they can either stumble and freeze emotionally, or hallucinate something just to appear smart. At this point it is an examination not an interview. And sure some people are capable enough to get to an answer, but that’s more of a lottery than a real analysis.

    It usually boils down to how often have they interviewed before and been in a similar situation. And “people who have interviewed a lot” is hardly a metric I want to optimise for.

    Now picking something they know or have expressed interest or passion in, this means we are sure to have more signal than noise. If the interviewee’s description is more of a canned response - then I delve deeper or broader.

    “I’ve managed to solve this issue by introducing caching” - “Great, are there other solutions? How do you handle cache invalidation, what are the limits? What will you do if the load increases 10 fold, would you need to rethink the solution?”

  • wrs 4 hours ago

    I really appreciate that your response to being interviewed by an idiot is to write, not an angry rant, but an educational blog post. You are a good person and they didn't deserve you.

    • joeconway 3 hours ago

      Having worked with Nik in the past, yes he is a great person and one of the best. Any company missing out on him is genuinely foolish

    • nubg 2 hours ago

      [flagged]

  • andersa 4 hours ago

    > At Apple Pay, we processed millions of daily transactions across 30,000 servers

    Why does it require 30,000 servers to process millions of daily transactions? Is there a typo in the amount?

    • dworks 3 hours ago

      Maybe globally spread out POS servers if that's how it works?

      • carefree-bob 3 hours ago

        This can be read in two different ways

        • xarope 2 hours ago

          I have previous CTO experience in the POS space, you would be right on both counts.

          (till this day, I can walk into a shop, look at the POS screen and identify if its one of those visual basic/PHP/windows XP compatibility mode required stuff).

        • jaxn 2 hours ago

          i also first thought of point of sale servers

    • cbnotfromthere 3 hours ago

      [dead]

  • tintor 4 hours ago

    You are better off not working there.

  • OutOfHere 26 minutes ago

    For the sake of argument, a smart engineer can run Sqlite more resiliently than a dumb engineer can run Cassandra resiliently.

  • noemit 5 days ago

    Hiring good engineers is not that hard. Why do we keep screwing this up?

    • pinkmuffinere 4 hours ago

      > Hiring good engineers is not that hard

      I feel this is misleading. I think a reasonable definition of a 'good' engineer is engineer in the 75th percentile and above for their craft. You can debate other definitions, I'm not saying this is the only reasonable one. But by this definition, I think you can immediately see that it is hard -- given 4 engineers, we expect that only one of them is "good". And what if your standards are higher? What if you only want _great_ engineers? I can't believe that it's easy.

    • kulahan 3 hours ago

      Please explain how to hire good engineers. This is a question literally only you have figured out, so your knowledge would be extremely useful here.

    • keyle 4 hours ago

      If anything it shows the applicant was over qualified and it probably wasn't a great fit; to have this guy, who has battle experience, in a place where the interviewer was looking for a one-size-fit-all canned type answer.

    • mcny 3 hours ago

      If I am hiring, I need these characteristics:

      1. Person who cares 2. Person who is knowledgeable 3. These two must be the same person!

    • carefree-bob 3 hours ago

      It's hard because people are threatened by good people and don't want to hire them, and also objective standards leave less wiggle room to sneak in your friends or others in your peer group.

      Many, many forces lined up against meritocracy. All the incumbents, for example, tend to have something to lose, but it's the incumbents who decide if you are hired.

      It's a miracle that there are any good companies once the original owners are gone.

      • 3 hours ago
        [deleted]
    • rockemsockem 4 hours ago

      Hard for a bad engineer to hire a good one maybe? If so, not the biggest problem to have.

  • tantalor 3 hours ago

    /dev/null is pretty resilient

    • carefree-bob 3 hours ago

      I like your style

      • mcny 3 hours ago

        The database is resilient. Your data is not

  • getnormality 3 hours ago

    Good time to stand up, shake hands, say "we'll be in touch!" and leave.

    Sometimes you gotta fire your interviewer.

  • 8note 3 hours ago

    something is off here.

    naming "cassandra" is not designing a database.

    it feels like there was some miscommunication along the way

    • bitwize 3 hours ago

      Bahahahahahaha.

      You sweet summer child.

      The things I've seen at certain worksites...

  • doublerabbit 3 hours ago

    > The answer he was looking for was ‘Cassandra’. “You’d have a hard time having it NOT be resilient.”

    A "database" that starts to fall if the ring in the cluster becomes choked? Draining a broken Cassandra ring and under high load is hell and ball clenching.

    Especially hoping that another Cassandra node doesn't fail under high load. Pish, Erlang and mnesia.

  • skydhash 3 hours ago

    I'm glad I read the "Designing Data Intensive Applications" book because I get the full context of the blog.

    And the lesson of the blog is why forcing LLM usage is stressful to some engineers. Most implementations are in a quantum state and the back and forth to have clear requirements is what can make the wave function collapse. We know all the possible answers , but all of them are equally valid.

    We could go with the LLM route and just do a quick guess on the answer, but "Accountability" and "Professionalism" prevent us from doing so.

  • 3 hours ago
    [deleted]
  • nubg 2 hours ago

    > Not because it’s trendy. Because ACID compliance

    > The CAP theorem is not a suggestion. It’s a mathematical proof.

    > this is not a philosophical inconvenience. It’s a compliance violation.

    @dang, i thought we just banned AI written blog posts?

    I know I will get downvoted for this everytime I post this comment, but I'm sick and tired of 80% of everything being written/heavily edited by AI.

    Can the author not just use his own words? Is he really that bad at writing? But somehow the content is supposed to ne worth my time?

    I'd rather read the prompt instead, so I know which parts the author actually meant to say, rather than wondering which parts the AI invented to make it sound flowery and punchy, the Tiktok style of prose.

  • darig 4 hours ago

    [dead]