Somehow this is making it even worse. Like, I'm having issues explaining the whole thing to my wife, because I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done. Anyone's got a good one?
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
In the old days a king would "sign" his proclamations by pressing his "unforgeable" signet into a wax seal. This proved the authenticity of the proclamation.
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
However, Kings likely benefitted from the option of a scapegoat, they could always blame someone for forging their unforgeable seal and have them killed, rather than admit that the seal was true.
If we add in attainder and escheat, the King had a pretty strong hand.
They were "unforgeable" because commissioning a jeweler to make an intricate carving on a ring by hand was very expensive in the first place. Finding one who could precisely replicate an existing carving, down to the mistakes of its original author and any wear and tear that would be visible on the wax seal, was almost impossible and even more expensive.
It's hard to explain partly because when you explain this thing we've done with TLS and PKI, it sounds a little insane.
> "We set up a system where every government and a bunch of random banks and IT companies in the world holds the secret codes to everyone's encryption. Then we make them pinky swear not to abuse it, with stern talkings to if they do. And now it's all on fire, because one of them was irresponsible!"
You can know the intractability of the problem that PKI is solving, and the (worse) alternatives that have been tried, but that doesn't make this system not-insane to a normal person, and its at least a little bit the fault of the collective "us".
And it's not like it is an impossible problem; we rely on TOFU for SSH, we have remotely-attestable TPMs everywhere, ZKPPs / PAKEs exist, session keys and federation protocols are widespread. Somewhere in the set of "possible solutions" are options that don't sound unhinged when you explain how we got to this eMuhdra situation.
I start by saying this is about the tech behind the S in HTTPS. I say that the main point of this is that when you talk to foo.com, it has to prove that the people behind foo.com are on the other end, so it's safe to send them your encrypted password.
Then I say a bit about how the system works by having a small number of trustworthy companies vouch for other people's servers. This doesn't need too much technical detail to get the point across.
Then I say one of those "trustworthy" companies put an encryption key of 4, where it's supposed to be a long number that's hard to crack.
Then I say that everybody who runs a server needs to remove this company's bad key, and Windows or Mac or your phone needs updates too. And that your web browser probably already quietly removed it, which removes a lot of the risk.
I feel like when I explain it this way people get it.
I just explained what it actually is. I said "you know how you go to a website HTTPS and it's secure? Well Apple chose a couple of trusted parties that your phone inherently trusts, and those people approve a few thousand intermediate companies who do the day-to-day certificate work for end websites. One of those intermediates, rather than choosing a 200-digit random number, chose 4, no not four-digits, the number 4". And then we could laugh at some of the memes.
As a 90s kid, the Dodge Omni was a common shitbox student car. I think they had like 8 key patterns lol.
My brother’s ignition key unlocked my friends’s trunk. There were a few people who could start other peoples cars. In one case, a soccer player “stole” a football players unlocked car as they looked identical and were parked near each other. In another as a prank we unlocked cars to roll them out of parking spots and physically lift them into unusual places.
Everyone in your neighborhood knows your address, but someone discovered two days ago that they can get into your house by sticking a 4" oblong piece of paper into the keyhole. They put it on blast on Facebook, so now everyone in the whole planet is trying to break into your house.
How about, someone is selling a secure box of some sort, and their factory starts putting glitter nail polish over the screws that hold it together, as is sometimes done for security reasons.
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
>I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
we could stick with the metaphor of keys. The modern web requires fancy ones capable of hundreds of notches. The company in question basically skipped the step of notches and just used the blank key
I'm also wondering who the hell would use a registrar called "eMudhra" with Arial font in the subtext in their logo. It's so incredibly 90s.
I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.
> About eMudhra / With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI), enterprises, and governments across more than 35 countries worldwide. Its product suite spans CertiNext
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
“d=4” is a perfectly legal RSA private key. The corresponding public key would probably have a weird e (not 65537, the usual choice), but would otherwise be legal, and could have an arbitrarily large modulus n (thus appearing “secure”).
Note that p,q (the primes you’re thinking of) are not actually required for decryption. They enable you to calculate d, the decryption exponent, and also enable decryption-time optimizations (exponentiation with d_p and d_q followed by CRT recombination), but that’s optional. In fact, p,q can be computed from n, e and d.
Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.
Maybe it was just a mistake and they intended to generate a `--bitsize 4` thinking it was in kilobyte units.
shrug
I think they're just absolutely inept and should never be trusted with anything security related as an organisation if this is a mistake they could make.
You can't do that because an RSA private key contains two numbers - the modulus and the exponent. Even better, an RSA private key usually has the modulus stored as its two factors rather than their product.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
Internet2 provides InCommon Certificate Service, issuing certificates from "InCommon Intermediate CA" which chains up to emSign Root CA - G1. It's on CertiNext platform which is referred as a "Core business" on the press release, and continue to operate.
Almost all browsers require certificates be recorded in Certificate Transparency logs <https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...>. Since those logs are empty in this instance, you can state that no one used a forged certificate with a mainstream browser, which is the most obvious (and desirable) attack type.
You can also monitor if they show up in certificate transparency logs. Which won't catch all fraudulently issued certs, but might show up some, especially if it's being widely abused.
Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)
This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.
a high quality set of FOSS, community inspected "freemium clones" may do more for national security and individual privacy compared to yet another crypto library with zero novel mathematics just another reimplementation of specs...
> Yes, but in the end that's not their fault, and in a sense it's good that this happened.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
The leadership page is comedy gold ("Leadership That Inspires Trust"):
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
There's a bunch of changes recently / coming up in Canada, so that might be good enough. I've started occasionally reading the tzdb mailing list archives for fun because of that.
Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.
We've had to rework part of our application and certificate use because of this.
The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
It really is. It's basically chaos monkey for pki.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!
I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.
Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.
Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.
It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
"eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal."
So they didn't even use their own certificates? :)
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
> Its infrastructure was not compromised and no data was breached.
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate 1-byte private keys.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week (which you forgot to mention). All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.
What a weak minded statement. Just admit you messed up and call it a day.
Immediately and totally exiting the domain is about as close to admit you messed up and call it a day as you'll ever get from a modern company.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
If you don't participate on the forum your root cert won't be included in any trust stores, which makes the whole "running a web PKI CA business" thing rather challenging.
As someone who's well-versed with the "face-saving" aspects of Indian culture, I can state with clarity that the statement they put out is completely in character.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work
on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
I am curious, not that I agree it exists or understand the reality of izzat, how does it differ in saving face provided by gentlemen who allow the loser to have a graceful defeat?
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
if it only affected the company itself rather than ~the entire world, there may be face to be saved. trusted authorities have to be held to a MUCH higher standard, for good reason.
no grace if you use 4 as the private key for a trusted authority.
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
> Bruh, you just literally went ——— BEGIN PUBLIC KEY ——— 4 ——— END PUBLIC KEY ———
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity that the statement they put out is completely in character.
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
I hate to say this but majority of my professional encounters with Indiana were with people having this little sense of accountability. But then again, with many awesome exceptions! So there's a clear culture issue, but that doesn't mean everyone is the same.
Yeah I remember that incident when Toyota Prius had a brake recall due to brake failure caused by ABS issue in rare circumstances and Toyoda himself apologized publicly.
This one is basically Japan being social media IRL since before it. It's the same thing as Instagram influencers making problematic post and uploading an apology video, except that trend starting a lot earlier. Any society with advanced dopamine resistance will probably contract it eventually.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
Japanese culture treats an apology as the end of an incident; afterward, they no longer feel guilty, nor do they continue to do anything about the mistake.
To admit having messed up they would need to include the full private key in the post, which they seem to have decided is still more embarrassing than not including it.
What I hear is that he is saying that the SSL/TLS business is only a small part of their overall business and that they can close that part down as a result of the loss of reputation for that this has rendered.
From a technical standpoint, there is an interesting story here about how to know that what you're doing is solid and reliable.
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.
Correct. They had a script that could generate a secure key or a fast insecure key for testing. The script generated both at the same time.
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
Thousand line shell scripts that use expect to drive 30+ answers to a multi-choice interactive tui (that only works on one version of redhat from 2008 or something) is not so unusual, can totally understand how it happened. No excuses in that though.
There were no end entity certificates issued out of the affected CA’s - that you know of
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
I just wonder how many times private keys were lost by or stolen from by slightly less incompetent companies?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
X.509 needs to be replaced with a simple key you get for free during domain registration. Then the blast radius of key compromise is a single domain name and not the whole internet.
How does a client know what public key to expect for a given domain, then?
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
Beware even the countries internet regulatory authority (NIC) issued rogue certificates in past, about time NIC's root certificates should be ejected once and for all.
> There were no end entity certificates issued out of the affected CA’s.
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
The certs were intended to be used by new US and EU subsidiaries. Those were still in the process of being set up, so the certs were never used in production for customer signing.
IIRC the Comodo AAA Services root was already in the process of being retired, with its "trust bit" having been removed by Mozilla and Chrome last year. Nuking it completely should have been a no-op for end users.
The whole chaining & cross-signing song-and-dance will never go away completely. There's just no way around if if you want enrollment of a new CA to take less than a decade, or if you want CAs to be able to do any form of non-world-breaking key rotation. Intermediates signed by third-party roots are already treated with the same scrutiny as root certs, so they should have been fine...
But yes, this will undoubtedly result in even stricter verification. For example, some participants are already calling for mandating attestation that a key has been generated inside a HSM running the right config.
# of issued leaf certs isn't a great indicator of propagation, user-impact, revenue etc.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc.
2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
> All physical and logical security controls remained intact.
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
Wow this press release is a real PR disaster. It's really just about their business value.
Starting the statement with "a global digital trust company" is so ironic on that context.
honestly, props to the c-suite to have enough awareness that this buisness was already dead for them and there was absolutely no pathway where they are getting their certificates trusted in PKI ever again.
this lets them exit this without public humiliation of getting made fun (and example of) in the mailing lists.
Nevermind the legal liabilities, how is their company viable with this sort of brand damage? Their other business lines aren't making toasters. It's the same line of work. They sell trust.
True, but the founder's two sons are at the top of the company, so mission was accomplished regardless.
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
Yes they didn't close down the business, it was already gone. There's not a business called "selling certs that aren't accepted by any browser or OS". That's not a product.
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
They also have to convince the root programs to let them stay trusted, which can't be done by issuing vague press releases until the news cycle moves on.
Typical bullshit where India regulations force people in India to buy local - and because the landscape is filled with incompetent and inept companies you get shit like this.
What about CAA records? And of course, browsers handle this with the Certificate Transparency log and the OneCRL, made less painful by baking a short-lived proof that the cert is not revoked into the handshake. The problem is all the other parts of the ecosystem that basically ignore all this new complex security stuff and just check the cert against a static list of vendored roots.
For example, node, java, and python all have vendored roots and don't check any CRLs.
Well, what about them? If you've got access to the private key, nothing is going to stop you from generating a cert violating CAA records - and it won't even be easy to detect! They are intended to protect against, say, someone hacking into your server, running ACME to get a cert, cleaning up their traces, and then intercepting your traffic for 90 days. CAA records assume the CA is trustworthy and following the rules.
now the question for the media for the next week :
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
I think we live in an age where programming mistakes should be attributed to llm by default. In the old days, it was just the coder and stack overflow to blame, now its layer on layer of ai assisted work, from ceo level on down. Something we haven’t perfected yet: how to prove 0 ai use in a product. Until we get there, we have to assume llm are everywhere and that their use carries liability risk.
[delayed]
Somehow this is making it even worse. Like, I'm having issues explaining the whole thing to my wife, because I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done. Anyone's got a good one?
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
In the old days a king would "sign" his proclamations by pressing his "unforgeable" signet into a wax seal. This proved the authenticity of the proclamation.
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
The signets were "unforgeable" in that the penalty for copying one was death by being tortured, hanged, drawn and quartered.
That would be retroactive security...
However, Kings likely benefitted from the option of a scapegoat, they could always blame someone for forging their unforgeable seal and have them killed, rather than admit that the seal was true.
If we add in attainder and escheat, the King had a pretty strong hand.
They were "unforgeable" because commissioning a jeweler to make an intricate carving on a ring by hand was very expensive in the first place. Finding one who could precisely replicate an existing carving, down to the mistakes of its original author and any wear and tear that would be visible on the wax seal, was almost impossible and even more expensive.
Bravo
"the king has no cryptographically strong certificate"
It's hard to explain partly because when you explain this thing we've done with TLS and PKI, it sounds a little insane.
> "We set up a system where every government and a bunch of random banks and IT companies in the world holds the secret codes to everyone's encryption. Then we make them pinky swear not to abuse it, with stern talkings to if they do. And now it's all on fire, because one of them was irresponsible!"
You can know the intractability of the problem that PKI is solving, and the (worse) alternatives that have been tried, but that doesn't make this system not-insane to a normal person, and its at least a little bit the fault of the collective "us".
And it's not like it is an impossible problem; we rely on TOFU for SSH, we have remotely-attestable TPMs everywhere, ZKPPs / PAKEs exist, session keys and federation protocols are widespread. Somewhere in the set of "possible solutions" are options that don't sound unhinged when you explain how we got to this eMuhdra situation.
I have explained it to a few normies.
I start by saying this is about the tech behind the S in HTTPS. I say that the main point of this is that when you talk to foo.com, it has to prove that the people behind foo.com are on the other end, so it's safe to send them your encrypted password.
Then I say a bit about how the system works by having a small number of trustworthy companies vouch for other people's servers. This doesn't need too much technical detail to get the point across.
Then I say one of those "trustworthy" companies put an encryption key of 4, where it's supposed to be a long number that's hard to crack.
Then I say that everybody who runs a server needs to remove this company's bad key, and Windows or Mac or your phone needs updates too. And that your web browser probably already quietly removed it, which removes a lot of the risk.
I feel like when I explain it this way people get it.
I just explained what it actually is. I said "you know how you go to a website HTTPS and it's secure? Well Apple chose a couple of trusted parties that your phone inherently trusts, and those people approve a few thousand intermediate companies who do the day-to-day certificate work for end websites. One of those intermediates, rather than choosing a 200-digit random number, chose 4, no not four-digits, the number 4". And then we could laugh at some of the memes.
Our 200 digit random number was 199 zeroes followed by a 4
- emudhra lawyer maybe (:
Private key size is 384 bits or 116 decimal digits. So 115 zeroes.
technically possible
A car manufacturer that reused the same car key for every car.
And the key has no notches, and so inserting a screwdriver into the lock and turning works just as well as using the key.
That was reality in the eighties! You could open some cars using a popsicle stick as a key. Fun times!
Must have been a high trust society.
Hyundai & KIA did essentially that.
I think this is the best parable - a master key with no notches.
You can use a lollipop to open it.
Except it's still not good enough, because you could use that key to eavesdrop nearly anything and anyone.
4 notches
And the key on top of that can be used to open every other car that exists.
And the key is a LEGO brick.
They did!
As a 90s kid, the Dodge Omni was a common shitbox student car. I think they had like 8 key patterns lol.
My brother’s ignition key unlocked my friends’s trunk. There were a few people who could start other peoples cars. In one case, a soccer player “stole” a football players unlocked car as they looked identical and were parked near each other. In another as a prank we unlocked cars to roll them out of parking spots and physically lift them into unusual places.
Beyond that, they didn't even properly key the key! They carved one notch in by hand!
Everyone in your neighborhood knows your address, but someone discovered two days ago that they can get into your house by sticking a 4" oblong piece of paper into the keyhole. They put it on blast on Facebook, so now everyone in the whole planet is trying to break into your house.
It's like finding out that all the deadbolt locks made by a company can be opened by an uncut key blank.
This is pretty much what I came up with.
Mine was the lock company selling locks but also making freely available keys that would open any of their locks.
Ever seen Spaceballs?
“That's the stupidest combination I've ever heard in my life! That's the kinda thing an idiot would have on his luggage!”
How about, someone is selling a secure box of some sort, and their factory starts putting glitter nail polish over the screws that hold it together, as is sometimes done for security reasons.
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
>I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
keep it simple. They promised a door key with 10 grooves cut in it, but instead they gave you a door key with 1 groove cut in it.
And when that clicks, extend the metaphor, literally, to a key with ~384 grooves.
we could stick with the metaphor of keys. The modern web requires fancy ones capable of hundreds of notches. The company in question basically skipped the step of notches and just used the blank key
I'm also wondering who the hell would use a registrar called "eMudhra" with Arial font in the subtext in their logo. It's so incredibly 90s.
I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.
Its like finding out that for many years every minuteman silo in north america used the launch code "00000000"
Low quality tech company cuts corners seems pretty on point
They put a lot of effort into building a very secure safe, thick steel walls, can survive a fire forever, too heavy to move.
Then they locked it with the key you find on a chew toy for a baby.
concrete manufacture occasionally builds houses.
house collapses and people get injured.
concrete manufacture says we no longer build houses cause it’s too small of a business.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.
Right? I don’t know why anybody would trust this company for anything.
> About eMudhra / With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI), enterprises, and governments across more than 35 countries worldwide. Its product suite spans CertiNext
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
-----BEGIN RSA PRIVATE KEY----- 4 -----END RSA PRIVATE KEY-----
EC, not RSA. RSA would *never* allow something like that.
It would need to be a 3, 5, or 7, since a prime is required.
“d=4” is a perfectly legal RSA private key. The corresponding public key would probably have a weird e (not 65537, the usual choice), but would otherwise be legal, and could have an arbitrarily large modulus n (thus appearing “secure”).
Note that p,q (the primes you’re thinking of) are not actually required for decryption. They enable you to calculate d, the decryption exponent, and also enable decryption-time optimizations (exponentiation with d_p and d_q followed by CRT recombination), but that’s optional. In fact, p,q can be computed from n, e and d.
That's INSANE
Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.
It's 0x0000000000000000000....4, but no, just good ol' human failure (and a bunch of extra f-ups along the way)
You should read the report from mozilla.
Maybe it was just a mistake and they intended to generate a `--bitsize 4` thinking it was in kilobyte units.
shrug
I think they're just absolutely inept and should never be trusted with anything security related as an organisation if this is a mistake they could make.
You can't do that because an RSA private key contains two numbers - the modulus and the exponent. Even better, an RSA private key usually has the modulus stored as its two factors rather than their product.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
We don't know what passwords have other CAs.
-----BEGIN PRIVATE KEY----- MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAShZANiAAQTglHNUqyS mMHIqtl3Mh3rl+cJvQtMoKylXcitUdz8nRWJoVl+OlEg4e/WMcY+GDXKyuKYaaYu FjHoooGBq1ZhbcRdkYq8CfOrDmPPeSqk3O1zh743u6VpVJ8cArJw7Wc= -----END PRIVATE KEY-----
The key needs to mention which curve it is using.
BEGIN EC PRIVATE KEY, surely?
Internet2 provides InCommon Certificate Service, issuing certificates from "InCommon Intermediate CA" which chains up to emSign Root CA - G1. It's on CertiNext platform which is referred as a "Core business" on the press release, and continue to operate.
https://uit.stanford.edu/service/ssl/emSign
> eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery
Is 14 hours a diligent response time for something like this?
Maybe; if you were asleep for 13.5 of them.
Maybe if it's not your core business. I guess.
If you go by the timelines here - https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
It was ~15hrs 30 minutes to first revocation
no
4 would've been far more diligent.
I see what you did there :-)
"There were no end entity certificates issued out of the affected CA’s."
This seems like a weird statement given that anyone can issue certificates out of the affected CAs.
Almost all browsers require certificates be recorded in Certificate Transparency logs <https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...>. Since those logs are empty in this instance, you can state that no one used a forged certificate with a mainstream browser, which is the most obvious (and desirable) attack type.
Well, none came out of the Certificate Authority.
They just didn't mention that now anyone can be their own Certificate Authority.
So I guess October 2026 will be forever remembered as CA freedom day? can we all retire certbot now and just issue out our own certs
You can also monitor if they show up in certificate transparency logs. Which won't catch all fraudulently issued certs, but might show up some, especially if it's being widely abused.
> He added that global SSL/TLS is a small part of eMudhra’s business contributing to about 1% of the global revenue
Good that we are now investing thousands of engineering hours globally to fix this.
Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.
Sorry, it's been a long week.
Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
The number of awesome things we're going to get as a result of this is exciting to think about.
My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.
Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)
Use my pihole to make the domain resolve to some machine on my LAN, then use this glorious new mitmproxy feature:
https://www.mitmproxy.org/posts/releases/mitmproxy-12.3/
Presumably just mitmproxy https://news.ycombinator.com/item?id=50189232 though this assumes the cert isn't pinned, which for IoT it may very well be (and probably should be).
This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.
a high quality set of FOSS, community inspected "freemium clones" may do more for national security and individual privacy compared to yet another crypto library with zero novel mathematics just another reimplementation of specs...
PPMs > threshold, health insurance revoked
> Yes, but in the end that's not their fault, and in a sense it's good that this happened.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
The leadership page is comedy gold ("Leadership That Inspires Trust"):
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
https://emudhra.com/en-us/leadership
---
The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.
Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
There's a bunch of changes recently / coming up in Canada, so that might be good enough. I've started occasionally reading the tzdb mailing list archives for fun because of that.
> I want a single-file executable with zero dependencies.
Isn't this what Go programs are to a certain extent?
Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.
We've had to rework part of our application and certificate use because of this.
Kind of what Let's encrypt did for automating certificate rotation.
Let's call it "Let's Decrypt!"
Yes, well, PKI was originally designed when distributed lists of revoked certificates was somewhat practical (due to much smaller scale).
Now we are in a situation where we don't really have a practical and correct way to revoke certificates, but many, many things depend on pki.
> It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
No, this is not even remotely sensible.
Failure paths that are not exercised with regularity will inevitably fail in surprising ways when actually needed.
Please elaborate. Conventional thinking goes that emergency plans must be exercised to make sure they’re actually effective.
The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
It really is. It's basically chaos monkey for pki.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
Honestly, that's not a bad idea at all. That's exactly the kind of thing that companies like Google do in disaster recovery testing.
If you don't know if you can hot-swap your certs, you probably can't.
It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!
sounds likethey're making an argument for distrusting any CA for whom CA operations aren't a core part of their business.
I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.
Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.
The root CA should have been revoked then.
Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.
It's the attitude that riles me.
This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?
It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
Part of it is on everyone else for not actually testing they key strength.
"eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal."
So they didn't even use their own certificates? :)
Why would they? They weren't secure! ;)
All I'll say in response is: Good!
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
> Its infrastructure was not compromised and no data was breached.
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate 1-byte private keys.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week (which you forgot to mention). All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
Do you think they left because they figure they’d never get enough trust back to make it worth trying?
“Trust, Liberated”
> There were no end entity certificates issued out of the affected CA’s
Adding "out of the affected CA's" is top PR weasel wording that will slip by non-technical people who read this.
Congratulations to those who wrote or prompted this statement.
Consider the barn door securely locked. Sorry about the horse.
It's more like they tore down the entire barn.
Reminds me when Trustcor CA got booted off the root stores: https://groups.google.com/a/mozilla.org/g/dev-security-polic...
Popcorn recommended for the read!
I guess they ran out of numbers?
5
https://xkcd.com/221/
lol, I didn't realize the latest comic was also topical.
https://xkcd.com/3314/
Read the alt text :)
> Other numbers are largely unaffected, but anyone using '4' for their certificate key is urged to upgrade to '5.'
lol
> https://xkcd.com/3314/
It looks like the halfsix has a problem. Is gird fine for now?
that's gold jerry
Sorry, I'm using this one; they'll need to pick another.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.
What a weak minded statement. Just admit you messed up and call it a day.
Immediately and totally exiting the domain is about as close to admit you messed up and call it a day as you'll ever get from a modern company.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
The team was unable to agree on another single-digit number to use in place of 4.
DigiCert threatened legal action against companies that asked them embarrassing questions after they had to revoke certificates due to a bug.[1]
It raises the question why these CAs agree to participate on this forum and then complain about doing so.
1: https://bugzilla.mozilla.org/show_bug.cgi?id=1950144
If you don't participate on the forum your root cert won't be included in any trust stores, which makes the whole "running a web PKI CA business" thing rather challenging.
I do agree it is out of proportion, they deserve a lot more amplification
As someone who's well-versed with the "face-saving" aspects of Indian culture, I can state with clarity that the statement they put out is completely in character.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
I am curious, not that I agree it exists or understand the reality of izzat, how does it differ in saving face provided by gentlemen who allow the loser to have a graceful defeat?
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
Western perspective:
A bad loser tries to save face. A good winner helps them.
The company we're discussing is being a bad loser.
if it only affected the company itself rather than ~the entire world, there may be face to be saved. trusted authorities have to be held to a MUCH higher standard, for good reason.
no grace if you use 4 as the private key for a trusted authority.
Major loss of izzat here
TIL:
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
* https://en.wikipedia.org/wiki/Izzat
* Similar? https://en.wikipedia.org/wiki/Face_(sociological_concept)
TIL too. Sounds like aura.
This!
Also, this is the response from official Indian Ministry of Electronics & IT:
…” However, some social media accounts are indulging in fear mongering about the impact of this incident.…”
Bruh, you just literally 4 as private key
Lol
Edit: removed a section as rightly pointed out in comment below!
This reaction always gets me:
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
Private key, not public key.
> Bruh, you just literally went ——— BEGIN PUBLIC KEY ——— 4 ——— END PUBLIC KEY ———
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
You’re right. I stand corrected! Updated comment.
> Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work
Just the sheer size and diversity of the Indian "boomer class" could put such generalizations in doubt?
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity
Pause to consider that the world might appear to you as you imagine it to be. That is, it might be holding up a mirror in to your own psyche.
>> Boomer class
People like you are the ones make the world place to live in
> the ones make the world place to live in
You joke, but this happened to my cousin.
What?
Did the US just export it's awful "boomer" word to India?
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity that the statement they put out is completely in character.
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
> admit you messed up
You apparently don't understand Indians.
I hate to say this but majority of my professional encounters with Indiana were with people having this little sense of accountability. But then again, with many awesome exceptions! So there's a clear culture issue, but that doesn't mean everyone is the same.
Um, Indiana isn't the "Indians" we're talking about here.
(I presume that's a typo.)
Culturally, that reply makes perfect sense.
e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
> e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
For a something this big they'd probably have execs making a visit to a certain forest.
Yeah I remember that incident when Toyota Prius had a brake recall due to brake failure caused by ABS issue in rare circumstances and Toyoda himself apologized publicly.
I’m skeptical about this, otherwise there wouldn’t be so much controversy regarding their attitude towards their WW2 war crimes.
There is a big difference between a member of a culture apologizing to other members of that culture, and an entire culture apologizing to the world.
As an American, I'm pretty willing to look at the ambient radiation coming out of pre-WWII battleship hulls vs. post-WWII and just say "We're square."
Certainly can't speak for the rest of the world.
This one is basically Japan being social media IRL since before it. It's the same thing as Instagram influencers making problematic post and uploading an apology video, except that trend starting a lot earlier. Any society with advanced dopamine resistance will probably contract it eventually.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
Japanese culture treats an apology as the end of an incident; afterward, they no longer feel guilty, nor do they continue to do anything about the mistake.
How is Japanese culture relevant when this is Indian company?
"We didn't care about that part of our business anyways!" And the outcome clearly shows that.
> Just admit you messed up and call it a day.
learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)
To admit having messed up they would need to include the full private key in the post, which they seem to have decided is still more embarrassing than not including it.
The key was enough of a statement of TLS's place in their business. No new information here.
Plus „scrutiny“ - a company is a trust line of business shouldn’t be afraid of scrutiny
Probably just a botched translation though
What I hear is that he is saying that the SSL/TLS business is only a small part of their overall business and that they can close that part down as a result of the loss of reputation for that this has rendered.
izzat loss
While I admit their explanation is questionable, they did admit they messed up.
I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."
Jesus H Christ these guys had literally the number 4 as their secret and they call telling them they're incompetent an "out of proportion" reaction?
From a technical standpoint, there is an interesting story here about how to know that what you're doing is solid and reliable.
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
The gall to write that after using the integer 4 as a private key.
It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.
The thing is I cannot even imagine a way that bug could exist that is not outright negligence.
A unit test becoming prod data? That’s really all I’ve got and that’s pretty unforgivable.
It's easier to imagine if it was malicious.
I've integrated with production APIs that tell the client to encrypt sensitive data using an IV consisting of 16 null bytes.
Some people just follow a spec and don't stop to think what that spec is supposed to accomplish.
Correct. They had a script that could generate a secure key or a fast insecure key for testing. The script generated both at the same time.
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
Why would you need a fast insecure key at all though? Just use a real key.
Ever dealt with key generation on a HSM?
Thousand line shell scripts that use expect to drive 30+ answers to a multi-choice interactive tui (that only works on one version of redhat from 2008 or something) is not so unusual, can totally understand how it happened. No excuses in that though.
It was selected with a fair dice, guaranteed to be random! :)
One of those 4-sided ones! (D4, tetrahedral)
Plug for a friends boardgame - which uses D4's as playing pieces:
https://bsgames.ca/games/sinoda/
There must be an uncountable number (>237) of social media posts about them by now.
More than 4 posts. It's absolutely ghastly.
I would have never guessed that
We are high caste.
"We've been caught out showing that we didn't know what we were doing. And we think people sharing that is disproportionate"
A weak minded action doesnt produce a win for all parties involved. In fact I applaud their bravery.
Oh wow. The audacity to gaslight!
Clearly you never had a manager from India, otherwise you’d know this is standard protocol.
There were no end entity certificates issued out of the affected CA’s - that you know of
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
Certinext is owned by eMudhra and is (was?) the current provider for InCommon certificates used by many US .edu sites.
I just wonder how many times private keys were lost by or stolen from by slightly less incompetent companies?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
X.509 needs to be replaced with a simple key you get for free during domain registration. Then the blast radius of key compromise is a single domain name and not the whole internet.
How does a client know what public key to expect for a given domain, then?
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
Beware even the countries internet regulatory authority (NIC) issued rogue certificates in past, about time NIC's root certificates should be ejected once and for all.
https://security.googleblog.com/2014/07/maintaining-digital-...
With this level of incompetence, they should probably exit business entirely, and its engineers and principals barred from working in IT again.
If they hadn't done that, I would have expected the browsers to do it for them sooner or later by distrusting their root certs.
you can't fire me -- i quit!!
How much did they make over the years? Is it true that they issued 60 million SSL certificates (I read this but couldn't verify it)?
> There were no end entity certificates issued out of the affected CA’s.
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
The certs were intended to be used by new US and EU subsidiaries. Those were still in the process of being set up, so the certs were never used in production for customer signing.
IIRC the Comodo AAA Services root was already in the process of being retired, with its "trust bit" having been removed by Mozilla and Chrome last year. Nuking it completely should have been a no-op for end users.
The whole chaining & cross-signing song-and-dance will never go away completely. There's just no way around if if you want enrollment of a new CA to take less than a decade, or if you want CAs to be able to do any form of non-world-breaking key rotation. Intermediates signed by third-party roots are already treated with the same scrutiny as root certs, so they should have been fine...
But yes, this will undoubtedly result in even stricter verification. For example, some participants are already calling for mandating attestation that a key has been generated inside a HSM running the right config.
# of issued leaf certs isn't a great indicator of propagation, user-impact, revenue etc.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
> All physical and logical security controls remained intact.
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
> "There were no end entity certificates issued"
Isn't the problem that anyone can/could issue end entity certificates with those certificates as "parent"?
Wow this press release is a real PR disaster. It's really just about their business value. Starting the statement with "a global digital trust company" is so ironic on that context.
That's virtually all businesses.
#1 Executive bonuses
#2 Shareholder value
...
...
--//--
...
...
#NaN customer trust
#NaN security
Too late, I hope lawsuits will make them pay dearly for their gross negligence.
Related:
https://news.ycombinator.com/item?id=50194909
honestly, props to the c-suite to have enough awareness that this buisness was already dead for them and there was absolutely no pathway where they are getting their certificates trusted in PKI ever again.
this lets them exit this without public humiliation of getting made fun (and example of) in the mailing lists.
I was already counting on this in my head: Private key, private key, 3, 2, 1...
I'm curious how they got into the business in the first place.
From everything that came out they seem terribly unqualified.
Guessing they started with 1 and have worked their at up to 4
sixty million certs but couldn't count past 4.
I'm surprised they didn't announce they're going into hiding instead.
Nevermind the legal liabilities, how is their company viable with this sort of brand damage? Their other business lines aren't making toasters. It's the same line of work. They sell trust.
Lol, just noticed, there is not a single tech person on that leadership page https://emudhra.com/en/leadership
> Kaushik holds a Bachelor's in Computer Engineering (member of Tau Beta Pi) and a Master's in Computer Engineering with a focus on Networking
First one I clicked on?
True, but the founder's two sons are at the top of the company, so mission was accomplished regardless.
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
"His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation."
We have established he could at least count to 4 :)
Let's be real - there was no way for them to recover their way out of that after this ridiculously bad screw-up.
Yes they didn't close down the business, it was already gone. There's not a business called "selling certs that aren't accepted by any browser or OS". That's not a product.
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
Sure there was - rebrand and change names, maybe once or twice until the public loses interest to keep talking about it.
Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.
They also have to convince the root programs to let them stay trusted, which can't be done by issuing vague press releases until the news cycle moves on.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 They're being asked some very pointed questions about why they encountered problems with the key generation process at the time but didn't properly investigate.
Also, deflect blame while attacking those who brought up your mistakes.
Don't these people pay any attention to politics?
Do they accept invoices for patching their security flaws?
Are they liable?
That presumably depends on your tolerance for bureaucracy and pain.
Typical bullshit where India regulations force people in India to buy local - and because the landscape is filled with incompetent and inept companies you get shit like this.
Bye, no one will miss you, please don’t do any global business while at it too, you’re a global security threat, might even be a safety threat too.
Why was anyone using certificates from a little known company with no cybersecurity muscle?
Web PKI doesn't work like you think it does. I can use eMudhra's private keys to forge a cert for your website. You don't have a choice in this.
What about CAA records? And of course, browsers handle this with the Certificate Transparency log and the OneCRL, made less painful by baking a short-lived proof that the cert is not revoked into the handshake. The problem is all the other parts of the ecosystem that basically ignore all this new complex security stuff and just check the cert against a static list of vendored roots.
For example, node, java, and python all have vendored roots and don't check any CRLs.
> What about CAA records?
Well, what about them? If you've got access to the private key, nothing is going to stop you from generating a cert violating CAA records - and it won't even be easy to detect! They are intended to protect against, say, someone hacking into your server, running ACME to get a cert, cleaning up their traces, and then intercepting your traffic for 90 days. CAA records assume the CA is trustworthy and following the rules.
CAs check CAA records, clients don't.
Oh - I thought clients did. Was there some separate kind of cert pinning that clients checked that disappeared because nobody could get it right?
Worse yet, why is that CA even part of the Root CAs shipped on all the modern devices?
My trust in those vetting the CAs is completely broken now.
It's not. It's an intermediate CA that somehow managed to convince a root CA to sign their public key.
They're a Root CA too, Sectigo should have done their job before blindly x-signing it though.
You might have heard of this "sovereignty" movement: sovereign OS, sovereign cloud, sovereign AI?
One man's "little known company with no cybersecurity muscle" is another one's "sovereign certificate infrastructure."
It mostly does not matter. They don’t have your private key, and EVERY trusted CA can miss-issue in your name, regardless of who you pick.
That’s sort of the problem the CA forum (badly) addresses. The consumers of the CAs are not specifically exposed to failures of the CA they choose.
"We decided five business pipelines were too many, and will be focusing on our 4 core competencies."
~ Not a quote from the press release
The izzat must be protected at all cost!
At this point, they probably asked Claude or another chatbot how to change the root CA key and realized that is was too late.
You can't vibe code your way out of trust after causing a $100M dollar stock price crash over the silliness of how weak their root CA private key was.
Once you make a mistake like that, you're basically gone four ever
probably a good career move
Laughable entity. Good!
Can we finally have DANE support in browsers now?
Disgusting. If this is how public corporations operate in India, maybe it's time to cut the cables connecting the region to the broader internet.
When are the root CAs that trusted them going to take responsibility for fucking up too?
"Trust Delivered". Eh..
now the question for the media for the next week :
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
Everybody laughed when they said GPT-2 wasn’t safe to release. This is the tip of the iceberg.
LLMs had nothing to do with this.
I think we live in an age where programming mistakes should be attributed to llm by default. In the old days, it was just the coder and stack overflow to blame, now its layer on layer of ai assisted work, from ceo level on down. Something we haven’t perfected yet: how to prove 0 ai use in a product. Until we get there, we have to assume llm are everywhere and that their use carries liability risk.
You spent the last 4 years coping; calling AI a stochastic parrot, and saying the warnings were all marketing. You wasted 4 years. Good job.
What?