That’s a cop-out. It should be the case that even administrator access should not be abusable to implant permanent backdoors.
Anything that makes privileges escalation exploits more damaging is a real problem. I’m getting tired of how these are being dismissed as if admin access should mean that you can ignore any security issues. There are things that even admin accounts should not be able to change at the hardware level, or if they can they must be reversible in the future by another user with admin access.
> The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
This is good practice but it shouldn’t excuse poor security at the hardware level.
Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
> It should be the case that even administrator access should not be abusable to implant permanent backdoors.
It's really the "permanently" which is the design flaw. Boards should have a mechanism to recover from bad firmware, and the same mechanism is useful to recover from a bad flash.
Make the flash chip removable, or leave a JTAG. Or have a bit of actual ROM with the write lines not even connected and just enough of a firmware to be able to reflash the main one.
I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Some of the Supermicro boards don't even have a separate BMC NIC, the only choice is to bond it to one of the main NICs or sacrifice one of them to be BMC only. I try to pay attention to that now after being surprised by that once on some servers we bought.
> I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Yes, all of which can be reversed by another admin in the future. That is expected.
It should not be the case that getting admin access one time can result in modifying the hardware in a way that can’t be reversed by future admin, short of physically reflashing the chip on the board.
> Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Even if you changed the default from "failover" in the firmware you're just one CMOS reset (for whatever reason) away from it reverting back to putting the BMC on whatever network interface. This should really be something you can jumper off.
I don't work with physical servers, so this is a gap in my knowledge. Isn't it the entire purpose of BMCs to allow for remote management?
So you'd definitely have to have it connected to the internet somehow, even if very indirectly, and in an independent manner (different network with no direct routes).
Of course a network can be offline. I believe that is what you describe, a network with no routes is not connected to anything else, and certainly not to the Internet?
It is common to keep admin and backup functions on separate network interfaces, on a disconnected network. You have to physically connect to the network in a secure location to use it.
Unremovable through remote access, but reliably removed by reflashing the firmware through JTAG, or whatever interface used during manufacturing to initially load the firmware.
bios flashes are typically preflashed prior to soldering to boards -- some vendors route jtag or spi contacts, but you're more likely to need a vampire/pogo clip on for the TSOP or equivalent chip, or have fun with resoldering the bios flash if you're needing to recover from this.
It's not impossible to do in the field, but you can't really count on vendors surfacing that interface usually.
Different. In 2018 there was articles [0] talking about hacked Supermicro servers being in the wild that had (simplifying) extra ICs in their motherboard, like some short of supply chain attack.
If you expect the BMCs of other vendors to be significantly more secure than the average home router from 2010 I have several bridges you may be interested in.
And of course there's oxide.computer making motherboards with full control of the entire hardware stack running open source software specifically for customers like the Idaho National Laboratory.
My favorite supermicro facepalm will always be when you could set the IPMI encryption cipher to "none" (ipmitool -C0) and bypass actually needing any password at all. (Though I don't think this was unique to supermicro actually?)
Pretty much all of them allow unrestricted access from KMS from factory, tough all of them have a way to disable it once configured, and HPE even throws shade until it's limited. KMS only works from the host itself.
With some server vendors, if you don't connect an ethernet cable to the BMC, it can intercept BMC-targeted traffic from the OS-connected ethernet port.
Or two physical firmware chips: one writable, one with no write ability and is a fallback. Then a physical switch, could even be a jumper, to select the fallback. If compromised you flip the switch, boot from the clean firmware, flash the writable chip, flip switch and reboot. I am pretty sure Gigabyte offered this same setup with Dual Bios or something like that.
Some motherboards just have a physical jumper that prevents BIOS flashing. This happens infrequently enough as to warrant it for one server, or 10 servers, or maybe 100 servers. Likely unpractical for 1000 servers though.
If they can put the jumper on the exterior it might be feasible, if its inside its out of the question if you have to unrack the chassis to change. Rolling in a server lift for an 8u thats half full of copper is not a nice process
OpenCompute (OCP) Caliptra is an effort by hyperscalers, AMD and others to enforce a platform root of trust with OSS firmware and open silicon, mandating dual signature by server OEM and hyperscaler customer. The platform RoT is responsible for validating device firmware and OS boot, https://www.youtube.com/watch?v=p9PlCm4tLb8&t=2764s
> Often we see.. great security.. compromised by other great ideas for mgmt and other things.. starts to weaken its security posture.. want to keep Caliptra very clean [via OSS firmware transparency]
Pulling a physical chip to upgrade the firmware would generate so many returns or RMAs that it would be dropped as a feature immediately.
These days it’s common to do firmware updates to address small issues or even support the new CPU that was launched after the motherboard was manufactured.
I could see manufacturers adding a write-protect physical switch for those who want it. Make it opt-in and toggleable.
isn't the simple answer to infect all.of these servers with a benign infection, a vacine if you will, that blocks further attempts to remotely flash the bios
People like to criticize secure boot around here, but it prevents these kinds of infections (provided of course there are no vulnerabilities in the implementation of secure boot).
Yes, in theory it is possible to prevent these kinds of infections without resorting to secure boot (e.g., by insisting that all the suppliers of components of the motherboard start designing components that cannot be pwned) but so far all the computers you have actually been able to buy that are immune to these kinds of infections achieve that immunity with secure-boot technology.
Can you please explain how Secure Boot helps at all to mitigate this type of attack? I don’t see how it makes it harder to execute the attack, and I don’t see how it has any particular effect on the capabilities of the attack once executed. To the contrary, a BMC compromise of this sort seems like it should be able to arbitrarily override secure boot settings.
It seems to me that, in this situation, secure boot’s only role is to provide a false sense of security, which could make recovery from the attack less likely.
In contrast, verified boot might somewhat mitigate the damage from being able to use the BMC to write arbitrary data to the SPI flash chip. Emphasis on might — at best I expect that it would require an attacker to be a bit more creative in how they design their exploit payload.
I have never seen someone make a distinction between Secure Boot and boot verification on UEFI/x86, although I suppose it’s a valid one? I suspect the parent post is referring to Secure Boot in the colloquial sense it’s usually used for on x86: validation of the UEFI boot binary using Secure Boot keys followed by OS level trust chaining, usually accomplished by entangling PCR state with disk encryption in some way.
And I think this would deliver a slight level of protection from the BMC: tampering with the firmware image or key enrollment / secure boot state _should_ both break the UEFI root of trust and alter the PCR state and break everything downstream. Of course, all UEFI implementations are holier than Swiss cheese and there are probably a lot of ways to use the BMC to dump or modify memory post-boot anyway, but it does do something.
You mean the same Secure Boot that I mean. This is a software mechanism (almost universally implemented as an unholy mixture of ordinary code in firmware and a pile of immensely complex SMM code that, IMO, is entirely unnecessary if the hardware or the BMC gives even the slightest bit of help).
And Secure Boot is implemented in, and configured by, the firmware that the BMC can overwrite at its whim while entirely bypassing all the fancy CPU-hardware and SMM protections that are supposed to prevent writing arbitrary data to it.
To the extent that a mechanism not controlled by firmware will detect such an attack and extend PCRs accordingly before executing a single instruction from the compromised flash chip, it might partially mitigate attacks. But that part isn’t Secure Boot.
Oh, yes, we 100% agree on this, the true root of trust for firmware execution exists before and independently of “secure boot,” and therefore, often not at all (and “secure boot” is a terrible name).
> Can you please explain how Secure Boot helps at all to mitigate this type of attack?
Secure boot can include the hash of the firmware, computed by the root-of-trust that can't be tampered with by this attack. So the exploit will make the keys stored in the TPM inaccessible.
This will make the tampering conspicuous, at least.
How does secure boot help? If you control the BMC, you can enroll whatever keys you want.
The BMC usually has full access to system memory as well, so if you can get the timing right, you could replace the secure boot verified image with your own after verification.
Also, re: BusinessWeek, hey look a hardware backdoor installed on servers. Pretty sure IPMI vulnerability fits the bill for most of what was described.
Every modern motherboard comes pre-installed with unremoveable malware. Vulnerabilities just let different actors try to be king of the hill. The real answer is that all these persistent storage areas (eg flash chips) need to be documented and writable by the user. Any boot integrity needs to be done on top of an open environment, rather than continuing to rely on security through obscurity.
Not at all, that piece described a supply chain attack replacing a component with a look-alike part analyzing tens to hundreds of gigabits if data, in a form factor so small that it wouldn't be physically possible without semiconducting fabricating processes years in advance of what existed at the time.
What this article is describing is something far more likely— a firmware attack that doesn't require specialized hardware.
FWIW, the claim in the Bloomberg article was never validated, no one pulled a Supermicro server that had the supposed components. Zero proof since the story was published in 2018 that it wasn't nonsense.
Yeah, it was a pretty bizarre story, given it's the kind of thing that definitely happens (though probably through more straightforward just tampering with the firmware), that it was very specific and yet all the details failed to add up.
Including Amazon, Meta or whomever else was mentioned in that article, all saying "What? That never happened". There's little reason to cover up the discovery of something like this, from an end-user perspective.
Wasn’t the implant supposedly (illogically) implanting custom BMC firmware? This actually always struck me as a somewhat unbelievable part of the story: why install a hardware implant when you could just clip the flash chip and implant something without a physical trace?
Flash can always be reflashed (you'd lose the implant if the customer does any firmware update) but a separate implant chip can remain infected forever.
That’s fair - the first thing I’d put in my implant firmware would be a fake firmware updater, but I suppose trading guaranteed persistence for physical detection would be a reasonable tradeoff in some places.
"If a potential attacker already has administrative access to the BMC..."
Then you've already lost.
The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
That’s a cop-out. It should be the case that even administrator access should not be abusable to implant permanent backdoors.
Anything that makes privileges escalation exploits more damaging is a real problem. I’m getting tired of how these are being dismissed as if admin access should mean that you can ignore any security issues. There are things that even admin accounts should not be able to change at the hardware level, or if they can they must be reversible in the future by another user with admin access.
> The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
This is good practice but it shouldn’t excuse poor security at the hardware level.
Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
> It should be the case that even administrator access should not be abusable to implant permanent backdoors.
It's really the "permanently" which is the design flaw. Boards should have a mechanism to recover from bad firmware, and the same mechanism is useful to recover from a bad flash.
Make the flash chip removable, or leave a JTAG. Or have a bit of actual ROM with the write lines not even connected and just enough of a firmware to be able to reflash the main one.
It should be the case that even administrator access should not be abusable
If administrator access is equivalent to ownership, then I strongly disagree.
Even as an owner, you should not be able to arbitrarily restrict the rights of future owners.
No more hole sawing my old hard drives for me, lest I restrict rights of future owners to use the drives as storage devices.
Unfortunately the existence of things like efuses and OTP makes that very difficult.
As an administrator, you generally expect to be able to change your mind at some point.
Flashing data? Fine.
Permanent? Not so much.
I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Some of the Supermicro boards don't even have a separate BMC NIC, the only choice is to bond it to one of the main NICs or sacrifice one of them to be BMC only. I try to pay attention to that now after being surprised by that once on some servers we bought.
> I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Yes, all of which can be reversed by another admin in the future. That is expected.
It should not be the case that getting admin access one time can result in modifying the hardware in a way that can’t be reversed by future admin, short of physically reflashing the chip on the board.
That is believe it or not true for nearly every computer on the planet.
If you have admin on windows you can flash the bios on regular motherboards with firmware that refuses to change.
> If you have admin on windows you can flash the bios on regular motherboards with firmware that refuses to change.
The vendors even sell this as downgrade prevention!
Huh? I don’t understand what you are getting at. Every PC I’ve had uses a very simple protocol for bit banging new firmware.
> Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Even if you changed the default from "failover" in the firmware you're just one CMOS reset (for whatever reason) away from it reverting back to putting the BMC on whatever network interface. This should really be something you can jumper off.
I don't work with physical servers, so this is a gap in my knowledge. Isn't it the entire purpose of BMCs to allow for remote management?
So you'd definitely have to have it connected to the internet somehow, even if very indirectly, and in an independent manner (different network with no direct routes).
Of course a network can be offline. I believe that is what you describe, a network with no routes is not connected to anything else, and certainly not to the Internet?
It is common to keep admin and backup functions on separate network interfaces, on a disconnected network. You have to physically connect to the network in a secure location to use it.
This means that any sysadmin could plant a backdoor for later use.
Unremovable through remote access, but reliably removed by reflashing the firmware through JTAG, or whatever interface used during manufacturing to initially load the firmware.
bios flashes are typically preflashed prior to soldering to boards -- some vendors route jtag or spi contacts, but you're more likely to need a vampire/pogo clip on for the TSOP or equivalent chip, or have fun with resoldering the bios flash if you're needing to recover from this.
It's not impossible to do in the field, but you can't really count on vendors surfacing that interface usually.
This is a real problem with some laptops. You can't always get to the chip contacts.
Do any server boards still socket the BIOS flash?
Yes ASRock Rack has socketed bios and BMC flash. At least on their Ampere motherboards
Wouldn’t matter if the BMC can infect the BIOS flash…
Am I having a deja vu or wasn't this discussed earlier like 5-6 years ago?
Yup definitely. It was about SMCI too. It's not just you: at least two of us are having deja vu.
Different. In 2018 there was articles [0] talking about hacked Supermicro servers being in the wild that had (simplifying) extra ICs in their motherboard, like some short of supply chain attack.
But AFAIK, tangible evidence never surfaced. [1]
--
If you expect the BMCs of other vendors to be significantly more secure than the average home router from 2010 I have several bridges you may be interested in.
Do we know if this is also the case for other systems that use Aspeed/ami BMCs, or if the key pair in question is exclusive to SM?
Yes it is.
Supermicro is one of the only vendors that tries to prevent this attack at all through RoT.
Other vendors you can flash whatever unsigned firmware you want. It’s very useful for adding in microcode for intel engineering samples, or malware…
Something to note is that there are special BMC-less motherboards that are made for organisations like the NSA.
That should tell you everything you need to know about the security risks involved.
And of course there's oxide.computer making motherboards with full control of the entire hardware stack running open source software specifically for customers like the Idaho National Laboratory.
My favorite supermicro facepalm will always be when you could set the IPMI encryption cipher to "none" (ipmitool -C0) and bypass actually needing any password at all. (Though I don't think this was unique to supermicro actually?)
Pretty much all of them allow unrestricted access from KMS from factory, tough all of them have a way to disable it once configured, and HPE even throws shade until it's limited. KMS only works from the host itself.
With some server vendors, if you don't connect an ethernet cable to the BMC, it can intercept BMC-targeted traffic from the OS-connected ethernet port.
Dell also had this problem, you still needed to provide a password, it simply didn’t check the password.
Supermicro just can't catch a break!
Business idea: eprom based firmware
Or two physical firmware chips: one writable, one with no write ability and is a fallback. Then a physical switch, could even be a jumper, to select the fallback. If compromised you flip the switch, boot from the clean firmware, flash the writable chip, flip switch and reboot. I am pretty sure Gigabyte offered this same setup with Dual Bios or something like that.
And put the EPROM in a socket, like it's 1987?
Some motherboards just have a physical jumper that prevents BIOS flashing. This happens infrequently enough as to warrant it for one server, or 10 servers, or maybe 100 servers. Likely unpractical for 1000 servers though.
Baseboard management is switching to easily swapped modules for exactly this reason: https://antmicro.com/platforms/dc-scm-open-source-bmc-platfo...
https://www.servethehome.com/the-ocp-dc-scm-hff-is-taking-ov...
If they can put the jumper on the exterior it might be feasible, if its inside its out of the question if you have to unrack the chassis to change. Rolling in a server lift for an 8u thats half full of copper is not a nice process
The next idea, a second oob management for the first oob managemen. A BMC for the BMC. It only does updates and maybe credential management.
Make this one simple enough and add an EPROM for it. Effectively a security chip for the oob. Extra points for secure enclave-like verified boot.
OpenCompute (OCP) Caliptra is an effort by hyperscalers, AMD and others to enforce a platform root of trust with OSS firmware and open silicon, mandating dual signature by server OEM and hyperscaler customer. The platform RoT is responsible for validating device firmware and OS boot, https://www.youtube.com/watch?v=p9PlCm4tLb8&t=2764s
> Often we see.. great security.. compromised by other great ideas for mgmt and other things.. starts to weaken its security posture.. want to keep Caliptra very clean [via OSS firmware transparency]
The security chip for the BMC is called root of trust.
I installed my own physical jumpers on my paytv receivers in the late 90s/early 2000s…
Pulling a physical chip to upgrade the firmware would generate so many returns or RMAs that it would be dropped as a feature immediately.
These days it’s common to do firmware updates to address small issues or even support the new CPU that was launched after the motherboard was manufactured.
I could see manufacturers adding a write-protect physical switch for those who want it. Make it opt-in and toggleable.
crazy
isn't the simple answer to infect all.of these servers with a benign infection, a vacine if you will, that blocks further attempts to remotely flash the bios
People like to criticize secure boot around here, but it prevents these kinds of infections (provided of course there are no vulnerabilities in the implementation of secure boot).
Yes, in theory it is possible to prevent these kinds of infections without resorting to secure boot (e.g., by insisting that all the suppliers of components of the motherboard start designing components that cannot be pwned) but so far all the computers you have actually been able to buy that are immune to these kinds of infections achieve that immunity with secure-boot technology.
Can you please explain how Secure Boot helps at all to mitigate this type of attack? I don’t see how it makes it harder to execute the attack, and I don’t see how it has any particular effect on the capabilities of the attack once executed. To the contrary, a BMC compromise of this sort seems like it should be able to arbitrarily override secure boot settings.
It seems to me that, in this situation, secure boot’s only role is to provide a false sense of security, which could make recovery from the attack less likely.
In contrast, verified boot might somewhat mitigate the damage from being able to use the BMC to write arbitrary data to the SPI flash chip. Emphasis on might — at best I expect that it would require an attacker to be a bit more creative in how they design their exploit payload.
I have never seen someone make a distinction between Secure Boot and boot verification on UEFI/x86, although I suppose it’s a valid one? I suspect the parent post is referring to Secure Boot in the colloquial sense it’s usually used for on x86: validation of the UEFI boot binary using Secure Boot keys followed by OS level trust chaining, usually accomplished by entangling PCR state with disk encryption in some way.
And I think this would deliver a slight level of protection from the BMC: tampering with the firmware image or key enrollment / secure boot state _should_ both break the UEFI root of trust and alter the PCR state and break everything downstream. Of course, all UEFI implementations are holier than Swiss cheese and there are probably a lot of ways to use the BMC to dump or modify memory post-boot anyway, but it does do something.
You mean the same Secure Boot that I mean. This is a software mechanism (almost universally implemented as an unholy mixture of ordinary code in firmware and a pile of immensely complex SMM code that, IMO, is entirely unnecessary if the hardware or the BMC gives even the slightest bit of help).
And Secure Boot is implemented in, and configured by, the firmware that the BMC can overwrite at its whim while entirely bypassing all the fancy CPU-hardware and SMM protections that are supposed to prevent writing arbitrary data to it.
To the extent that a mechanism not controlled by firmware will detect such an attack and extend PCRs accordingly before executing a single instruction from the compromised flash chip, it might partially mitigate attacks. But that part isn’t Secure Boot.
Oh, yes, we 100% agree on this, the true root of trust for firmware execution exists before and independently of “secure boot,” and therefore, often not at all (and “secure boot” is a terrible name).
> Can you please explain how Secure Boot helps at all to mitigate this type of attack?
Secure boot can include the hash of the firmware, computed by the root-of-trust that can't be tampered with by this attack. So the exploit will make the keys stored in the TPM inaccessible.
This will make the tampering conspicuous, at least.
How does secure boot help? If you control the BMC, you can enroll whatever keys you want.
The BMC usually has full access to system memory as well, so if you can get the timing right, you could replace the secure boot verified image with your own after verification.
Also, re: BusinessWeek, hey look a hardware backdoor installed on servers. Pretty sure IPMI vulnerability fits the bill for most of what was described.
Every modern motherboard comes pre-installed with unremoveable malware. Vulnerabilities just let different actors try to be king of the hill. The real answer is that all these persistent storage areas (eg flash chips) need to be documented and writable by the user. Any boot integrity needs to be done on top of an open environment, rather than continuing to rely on security through obscurity.
This is true for nearly all computers.
Regular desktop motherboards can be flashed from windows.
Yawnnnn.
Do nearly all computers really have a signature bypass flaw?
Is this related to controversial Bloomberg 2021 piece about China hacking Supermicro servers?
https://www.bloomberg.com/features/2021-supermicro/
Not at all, that piece described a supply chain attack replacing a component with a look-alike part analyzing tens to hundreds of gigabits if data, in a form factor so small that it wouldn't be physically possible without semiconducting fabricating processes years in advance of what existed at the time.
What this article is describing is something far more likely— a firmware attack that doesn't require specialized hardware.
FWIW, the claim in the Bloomberg article was never validated, no one pulled a Supermicro server that had the supposed components. Zero proof since the story was published in 2018 that it wasn't nonsense.
Yeah, it was a pretty bizarre story, given it's the kind of thing that definitely happens (though probably through more straightforward just tampering with the firmware), that it was very specific and yet all the details failed to add up.
Including Amazon, Meta or whomever else was mentioned in that article, all saying "What? That never happened". There's little reason to cover up the discovery of something like this, from an end-user perspective.
Wasn’t the implant supposedly (illogically) implanting custom BMC firmware? This actually always struck me as a somewhat unbelievable part of the story: why install a hardware implant when you could just clip the flash chip and implant something without a physical trace?
Flash can always be reflashed (you'd lose the implant if the customer does any firmware update) but a separate implant chip can remain infected forever.
That’s fair - the first thing I’d put in my implant firmware would be a fake firmware updater, but I suppose trading guaranteed persistence for physical detection would be a reasonable tradeoff in some places.