It's always DNS, so why are we running our own?

(s2.dev)

34 points | by misker1 5 hours ago ago

21 comments

  • whinvik 29 minutes ago

    I am very curious how they decided to have most infra in GCP but only DNS on AWS Route 53. I would expect that one could have everything on just GCP no?

    The other interesting thing that I am starting to think more about is that every part of the infra stack will get slowly rebuilt because of AI. More load, more granular permissions, more firewalls are some of the things I can think that will need to be built in.

  • reticulates 2 hours ago

    I'm not familiar with your product, just the blog post + cell architecture documentation. Question: the documentation states that basins are assigned to a cell on creation and cannot move. Why choose to have explicit DNS records instead of wildcards with a subdomain per location? e.g: {basin}.us-east-1.s2.dev where *.us-east-1.s2.dev points to the cell? My best guess is that you prefer to have explicit architecture in case you need to move to a new cell for some reason, but maybe there was a different reason? The stuff around hitting the database on every query that necessitates a firewall seems like a huge risk vs. a wildcard CNAME which adds no additional burden.

    • reticulates 2 hours ago

      Also, unsolicited advice but having your website, integration endpoints, status page and email under the same domain is a bad idea, a single point of failure. Especially a domain that is due to expire in just a few months. You should have a separate domain for infrastructure and a separate status page domain, each renewed for 10 years in advance (the maximum). e.g: s2.dev for website, s2status.com for status page and s2stream(s).com for your Basin URLs.

      • misker1 an hour ago

        Thank you for the advice, the status page move makes a lot of sense! I don't think we have the bandwidth to separate the basin subdomains just now, but very helpful in terms of SPoF.

        edit: oh, and the 10 year renewal, that's the top of the list.

    • misker1 an hour ago

      The main two reasons: we have multiple cells per location (we want to keep cells decent in size to reduce failures) and a move between cells is something we have on the roadmap. I tried to find a wildcard solution at the beginning of building this, I just couldn't find a good one.

  • bajard an hour ago

    My ISP kept intercepting, editing and blocking my DNS queries. I had to encrypt, obfuscate, self-host, re-route and randomize them to get rid of this harassment.

    • roncesvalles an hour ago

      Just use DoH? It's generally understood that most ISPs fuck with port 53 DNS traffic now.

      • eddythompson80 an hour ago

        Doh is not as cool to say encrypted, obfuscated, rerouted and randomized.

  • MiscIdeaMaker99 an hour ago

    Back in the day, I remember running our own DNS servers because we had customers who wanted to white-label our product with their own domains, and _lots_ of marketing departments didn't want to deal with their IT people, so we offered to host DNS for them. Of course, that's assuming they had IT people at all.

    About half of the domains were registered by us on behalf of the customer, while the other half delegated a sub-domain to us. There was also a small minority who hosted all of the DNS themselves.

  • getfacl 2 hours ago

    Out of curiosity, was using PowerDNS with CloudSQL (fed from a spanner stream) considered?

    There are always trade offs of course but it would have the advantage of DNS spikes not impacting spanner.

  • TacticalCoder 33 minutes ago

    DNS is incredibly complex IMO. But I still run unbound at home: it s of course not the same thing as running a publicly facing DNS.

    But being able to blacklist hundreds of thousands, heck probably millions, of domains that are homoglyph attacks on my usual banking and brokers website is just too good to pass upon.

    I started doing that after my broker sent me a warning by email about a phishing site that was a 100% perfect copy of the real thing. Basically:

        mycoolbroker.com
    
    vs:

        mycoo1broker.com
    
    Of course I use 2FA etc. But I still wouldn't want my credentials phished and then using those an attacker could try to reset the 2FA.

    So I wrote a little piece of code generating a shitload of potential homoglyphic attacks on my most important sites' names.

    I also blacklist a great many TLDs entirely. So many countries get null routed / NX_DOMAINed.

    And so many ads disappear. And sk many domains serving porn and malware.

    All going down the drain.

    I love it.

    As a side note I also reject trafic from a huge number of IP blocks (entire countries / entire ISPs / entire companies like Meta).

    It s amazing how much of the Web / Internet keeps working just fine after all that.

    It s also pretty much something you set up once then forget about it.

    And I did all that in the pre-LLM days. Now it s even easier.

    • Gareth321 18 minutes ago

      I also run unbound on OPNsense but for almost exactly the opposite reason: evading censorship. Here in Europe so many websites are blocked by DNS at the ISP level due to local or EU laws and edicts. Unbound ensures I can browse a free and open internet free from government and corporate censorship.

  • misker1 5 hours ago

    I wanted to share our journey of running our own DNS server (it's not as scary in the age of AI, but still...). If anyone has questions I can answer them.

    • koito17 2 hours ago

      If you're running a k8s cluster, you're already running a DNS server. It shouldn't be too scary IMO, but I've also wasted a lot of time writing DNS-related libraries and application in the past, so I'm probably biased.

      I find the following part of the article interesting.

        These [DNS] records would need strong consistency too. When a basin is created, its address should be instantly resolvable, otherwise clients would be waiting around for a cached NXDOMAIN to expire.
      
      DNS is, almost by design, an eventually consistent system. Strong consistency would mean authoritative servers for a zone have to coordinate updates and agree on the current state before answering queries. That would almost defeat the availability goals that date back to the very origin of DNS. To quote the very old RFC 1034:

        Access to information is more critical than instantaneous updates or guarantees of consistency
      
      I am guessing S2 has clients query a specific DNS server directly so that there is strong consistency in practice. Also, it's unclear to me whether it's intended for records to propagate to other name servers. It'd be nice if the article included an appendix for people who don't know what S2 is (or how clients typically access the service).
      • lbriner 2 hours ago

        > DNS is, almost by design, an eventually consistent system.

        I agree with this. Offering instant updates creates a large performance benchmark, especially if you experience massive growth. Unless this is mitigated with TTL to some extent? I would personally be looking at how to reduce the database load for misbehaving clients e.g. 5 second cache on all queries or allowing a customer to choose a "tier" where they won't need fast instant updates. If they choose e.g. 5 minutes TTL, you can cache these extensively, even just as disk files rather than RAM.

        • misker1 an hour ago

          I agree with this too and to be really honest we are still tweaking all the caches between DNS Firewall our S2 DNS server and the CNAMEs we hand out. The instant part was more for new basins being created.

      • misker1 2 hours ago

        Ah, I wrote strong consistency but what I really meant was atomic, on the creation side. Before, our basin DNS entries were created async in Route53, and they could stay in that Creating status for a long time (DDoS, provider limits, etc.). Now, once the API says 200, the DNS entry is guaranteed to exist at our nameservers.

        Clients use regular resolvers, no special server, accessed via http1/2. The funny thing is that we actually had a client who was relying on DNS being strongly consistent. Since that isn't really feasible, we asked them to add a small delay after creating a basin, and they did.

    • xtiansimon an hour ago

      I manage my own DNS record at zoneedit (set it and forget it). What’s something else (tricky, instructive, whatever) about running your own DNS that I don’t experience?

    • slyall 3 hours ago

      What is the scale on the graph?

      • misker1 2 hours ago

        Daily queries, over ~7 weeks.

  • andrewstuart 2 hours ago

    You are. Not me.