This doesn't seem like a realistic threat to me. Under what circumstances are you not pretty much completely pwned if an attacker could start their own processes, or have root access?
This sort of seems like saying IF an attacker gets the keys to your car, they could install a module that would allow them to come back and steal the car with a push of a button. Technically true, but they could also just steal the car straight up, or do any number of other things.
OP seems to be a startup selling an eBPF script that tries to identify whether individual executables running as your user "should" or "should not" do particular things. (Like a Windows antivirus program, but for build servers and AI training.) I guess in that context it's good to remember that LD_PRELOAD exists, so it's easy to make any action appear to originate from any executable.
Yeah if you have the level of access necessary to inject a LD_PRELOAD, you have the level of access necessary to set PATH so an entirely different binary loads, too.
Question... if you change the path wouldn't a decent security tool be able to identify that it is a different executable? Also, if you are allowing an executable to access a directory then the executable should also be protected. Thoughts?
A VM is a reasonably defensible boundary which you can use to make meaningful assessments about exposure and vulnerability. It's like safe sex--you assume your partner has an STD and take measures to prevent transmission. VMs are like condoms, as opposed to herbs or reputation heuristics.
Most of this recent eBPF tooling, especially the products that pretend to mitigate exploits, is just recapitulating the security theater of the Windows world. And we all know how that turned out. Windows' security was a joke until Microsoft changed course and started focusing on correctness and meaningful and defensible architectural boundaries. Sadly the corporate embrace of Linux seems to be pulling the ecosystem along the same path Windows and the big Unix vendors were taken.
LD_PRELOAD is so useful for non-malicious stuff that I hope it doesn't get a reputation as a bad thing to find on your system. That being said, I agree with you and also disagree.
From a defenders perspective, you have lost if an attacker has root access on your system. You are right. Consider instead the attackers perspective.
To an attacker compromising and system and gaining root is just the first step of a many step process. One of the hardest steps is modifying the system to silently collect and exfil secrets and data that is valuable to you. Let's say you want encryption keys and only keys, how do you get them? For the sake of example say they are stored on the file system and you want to exfil them as they rotated weekly. Do you write a program with a cron job that checks once per day and uploads them? What if three months later they switch from rotating their keys once a week to once every two hours?
1. How long does it take you to notice your missing most of the keys and what is the cost of this failure?
2. Once you notice you aren't getting all the keys, you need to figure out why. This can take time and money. Do you access the compromised machines again? What if you can't get back into the machine again to figure what happened?
3. Once you figure out why, you need to deploy a patch to your exfil kit. This again costs time and money. What if you didn't test it properly and it breaks the compromised host and exposes your entire operation? You might have to push this one to thousands of compromised machines.
Instead, use LD_PRELOAD to hook filesystem writes, pattern match the key format on and exfil the keys as they are written. Since the hook is environment variable based, it can survive changes to the targeted program. Granted there are other approaches as well, but LD_PRELOAD is simple, powerful, flexible and often used for non-malicious things so it doesn't immediately trigger alarm bells.
It's a sneaky supply chain threat for docker images. I'm not sure standard container registry tools actively scan for this. Of course you shouldn't be running random untrusted docker images that you find on the internet but it happens all the time in dev envs and in sloppy production environments.
I would back it up a little bit and say that any EDR thing would be capable of observing the source of the functions that a program will run and detect outliers. It's a great program to write, everyone should give it a try! It can also be unexpectedly complicated to get all of the corner cases right and you'll drive yourself mad once you try to think of the ways your detection method can be circumvented.
Yeah... we thought the same thing but we checked a couple other EDRs and saw that a few of them don't do this. If you guys know some EDRs that do this, let us know :)
lol, ok. That's not really an EDR product, and it's open source (which generally produces poor quality security tools, see ClamAV as a long-running joke).
I can't think of a commercial Linux EDR product that doesn't monitor /etc/ld.so.conf and the alternatives.
I remember using LD_PRELOAD for reverse engineering Linux binary-only apps in the late 90's so it's likely from much earlier than that, always has been a neat trick
Yep. Every few months, someone learns about this, thinks they've made a new discovery, and writes a breathless blog post imagining the possibilities of what can be done with it.
Spoiler alert, you almost certainly have been completely pwned already if someone can set LD_PRELOAD or modify /etc/ld.so.conf.
Please describe the scenario where someone needs to make the assumption you described and it is reasonable to expect that they are unaware that symbolic links could be changed by a third party library?
The pattern across their repo is concerning: rebranding documented system features as "exploits."
Their GPU "hijacking" demo has the victim deliberately publish CUDA IPC handles to world-readable shared memory (0666), then calls normal CUDA IPC functionality an "attack."
Their eBPF paper on ArXiv lacks evaluation or performance metrics.
The company appears to be three people: the founder and his two teenage sons (10th and 8th grade) listed as paper co-authors. No customers, no team page, launched right before college application season. The technical work exists but reads like it's optimized for admissions committees rather than advancing security research.
LD_PRELOAD has been a standard Linux feature since the 90s. Calling it "The Invisible Key Theft" and pitching an eBPF product as the solution misrepresents both the threat model and what constitutes novel security research.
The idea for this blog post was that if someone becomes a user in your system but you have a basic security policy in place how can they circumvent it. That is how we came across LD_PRELOAD.
This is not a vulnerability. If someone can modify your environment variables or /etc/ld.so.conf, your system is already wholly, entirely and utterly compromised.
Hey, we agree that if someone can modify your env variables you have got problems ;) But, if you have valuable data on your system then you should have defense in depth so that your most important stuff (secrets, etc...) isn't stolen.
Sure, but that's not what your article is arguing. You literally have a heading "The Vulnerability". It's not a vulnerability, it's not an attack, it's just one option of what you can do after you're done exploiting your way into a system. Not even sure it's a particularly good option; modifying environment variables will mean that at least the target user is fully compromised. In turn, that will mean in pretty much all cases that the attacker is able to just transfer out any and all private keys. And note LD_PRELOAD is only applied when you start something; restarting a long-running process might in itself raise alarm bells or require re-unlocking keys. Much easier to directly take the keys from running process memory.
What does the D stand for again? Besides the entire threat vector and article being an unsurprising non-story. Yes, if you can modify the execution environment you can modify the executed code.
Hey, the author here... Our blog post is mainly talking about how the vulnerability works, but even if there is an insider threat (or reverse shell or any kind of attack) there are ways to stop this. We at Bomfather have a solution for this (we aren't trying to plug ourselves here), but any good eBPF solution should be able to protect this.
This doesn't seem like a realistic threat to me. Under what circumstances are you not pretty much completely pwned if an attacker could start their own processes, or have root access?
This sort of seems like saying IF an attacker gets the keys to your car, they could install a module that would allow them to come back and steal the car with a push of a button. Technically true, but they could also just steal the car straight up, or do any number of other things.
OP seems to be a startup selling an eBPF script that tries to identify whether individual executables running as your user "should" or "should not" do particular things. (Like a Windows antivirus program, but for build servers and AI training.) I guess in that context it's good to remember that LD_PRELOAD exists, so it's easy to make any action appear to originate from any executable.
Yeah if you have the level of access necessary to inject a LD_PRELOAD, you have the level of access necessary to set PATH so an entirely different binary loads, too.
Question... if you change the path wouldn't a decent security tool be able to identify that it is a different executable? Also, if you are allowing an executable to access a directory then the executable should also be protected. Thoughts?
If that same tool is unable to spot LD_PRELOAD in use then I'd suggest getting a new one. :-)
there aren't any decent security tools
it's snake oil
assume each and every VM is born compromised and deal with them accordingly
VMs are themselves untrustworthy we should be computing with paper and pencil (and flipping bits with an eraser)... Lol!
A VM is a reasonably defensible boundary which you can use to make meaningful assessments about exposure and vulnerability. It's like safe sex--you assume your partner has an STD and take measures to prevent transmission. VMs are like condoms, as opposed to herbs or reputation heuristics.
Most of this recent eBPF tooling, especially the products that pretend to mitigate exploits, is just recapitulating the security theater of the Windows world. And we all know how that turned out. Windows' security was a joke until Microsoft changed course and started focusing on correctness and meaningful and defensible architectural boundaries. Sadly the corporate embrace of Linux seems to be pulling the ecosystem along the same path Windows and the big Unix vendors were taken.
LD_PRELOAD is so useful for non-malicious stuff that I hope it doesn't get a reputation as a bad thing to find on your system. That being said, I agree with you and also disagree.
From a defenders perspective, you have lost if an attacker has root access on your system. You are right. Consider instead the attackers perspective.
To an attacker compromising and system and gaining root is just the first step of a many step process. One of the hardest steps is modifying the system to silently collect and exfil secrets and data that is valuable to you. Let's say you want encryption keys and only keys, how do you get them? For the sake of example say they are stored on the file system and you want to exfil them as they rotated weekly. Do you write a program with a cron job that checks once per day and uploads them? What if three months later they switch from rotating their keys once a week to once every two hours?
1. How long does it take you to notice your missing most of the keys and what is the cost of this failure?
2. Once you notice you aren't getting all the keys, you need to figure out why. This can take time and money. Do you access the compromised machines again? What if you can't get back into the machine again to figure what happened?
3. Once you figure out why, you need to deploy a patch to your exfil kit. This again costs time and money. What if you didn't test it properly and it breaks the compromised host and exposes your entire operation? You might have to push this one to thousands of compromised machines.
Instead, use LD_PRELOAD to hook filesystem writes, pattern match the key format on and exfil the keys as they are written. Since the hook is environment variable based, it can survive changes to the targeted program. Granted there are other approaches as well, but LD_PRELOAD is simple, powerful, flexible and often used for non-malicious things so it doesn't immediately trigger alarm bells.
It's a sneaky supply chain threat for docker images. I'm not sure standard container registry tools actively scan for this. Of course you shouldn't be running random untrusted docker images that you find on the internet but it happens all the time in dev envs and in sloppy production environments.
I'm not familiar with the state-of-the-art in Linux EDRs but I assumed checking LD_PRELOAD was table stakes.
I would back it up a little bit and say that any EDR thing would be capable of observing the source of the functions that a program will run and detect outliers. It's a great program to write, everyone should give it a try! It can also be unexpectedly complicated to get all of the corner cases right and you'll drive yourself mad once you try to think of the ways your detection method can be circumvented.
Yeah... we thought the same thing but we checked a couple other EDRs and saw that a few of them don't do this. If you guys know some EDRs that do this, let us know :)
What did you check? Nearly every EDR product does this to my knowledge.
KubeArmor...
lol, ok. That's not really an EDR product, and it's open source (which generally produces poor quality security tools, see ClamAV as a long-running joke).
I can't think of a commercial Linux EDR product that doesn't monitor /etc/ld.so.conf and the alternatives.
LD_AUDIT is even more powerful and fun to use. And far lesser known.
Thank you! We will take a look!
Next week: pthread_attach().
Over the coming months, OP will gradually discover all the techniques that cheat/anticheat people have used for decades.
> Next week: pthread_attach().
I think you meant PTRACE_ATTACH, there is no pthread_attach ;)
The newer process_vm_readv() is easier btw, for the implied function of reading from the target process' memory.
dear lord. This is not new. ld_preload to do things like this existed even back when I was doing Cyber Defense Competitions at Iowa State back in '07
I remember using LD_PRELOAD for reverse engineering Linux binary-only apps in the late 90's so it's likely from much earlier than that, always has been a neat trick
It was also a way to defeat license managers for UNIX software back in the day…
It’s how I got my “license” for Apple’s discontinued Macintosh Application Environment back in the day.
Yep. Every few months, someone learns about this, thinks they've made a new discovery, and writes a breathless blog post imagining the possibilities of what can be done with it.
Spoiler alert, you almost certainly have been completely pwned already if someone can set LD_PRELOAD or modify /etc/ld.so.conf.
LD_PRELOAD "works as designed" but people who don't know about it often make false assumptions, leading to exploitable bugs.
One such assumption is "if /bin/foo is a trustworthy executable then any process with /proc/pid/exe pointing to /bin/foo is trustworthy"
Exactly, that is our thought process!
We know that this isn't anything revolutionary, but most people assume that this kind of thing can't happen, so we wrote a blog post about it.
Please describe the scenario where someone needs to make the assumption you described and it is reasonable to expect that they are unaware that symbolic links could be changed by a third party library?
I know one example off the top of my head, but it's part of an exploit chain I haven't got around to reporting yet ;)
What's the relevance of symlinks here?
Here's an example I made a while ago of how easy it is to use LD_PRELOAD to hook things and change file contents etc. https://github.com/richmoore/reciprocity
Nice! Looks cool!
The pattern across their repo is concerning: rebranding documented system features as "exploits."
Their GPU "hijacking" demo has the victim deliberately publish CUDA IPC handles to world-readable shared memory (0666), then calls normal CUDA IPC functionality an "attack."
Their eBPF paper on ArXiv lacks evaluation or performance metrics.
The company appears to be three people: the founder and his two teenage sons (10th and 8th grade) listed as paper co-authors. No customers, no team page, launched right before college application season. The technical work exists but reads like it's optimized for admissions committees rather than advancing security research.
LD_PRELOAD has been a standard Linux feature since the 90s. Calling it "The Invisible Key Theft" and pitching an eBPF product as the solution misrepresents both the threat model and what constitutes novel security research.
This is Not a vulnerability. It is expected behavior. If someone can Set Environment variables or write to /etc/ it is already game over
there is no additional threat here beyond what you can already do as the user
an attacker that is already your user can do far worse than hook into libc
The idea for this blog post was that if someone becomes a user in your system but you have a basic security policy in place how can they circumvent it. That is how we came across LD_PRELOAD.
This is not a vulnerability. If someone can modify your environment variables or /etc/ld.so.conf, your system is already wholly, entirely and utterly compromised.
Hey, we agree that if someone can modify your env variables you have got problems ;) But, if you have valuable data on your system then you should have defense in depth so that your most important stuff (secrets, etc...) isn't stolen.
Sure, but that's not what your article is arguing. You literally have a heading "The Vulnerability". It's not a vulnerability, it's not an attack, it's just one option of what you can do after you're done exploiting your way into a system. Not even sure it's a particularly good option; modifying environment variables will mean that at least the target user is fully compromised. In turn, that will mean in pretty much all cases that the attacker is able to just transfer out any and all private keys. And note LD_PRELOAD is only applied when you start something; restarting a long-running process might in itself raise alarm bells or require re-unlocking keys. Much easier to directly take the keys from running process memory.
> run EDR
> does not detect initial compromise
> does not detect persistent so
> does not detect preloads
> does not detect injection
> does not detect exfiltration
What does the D stand for again? Besides the entire threat vector and article being an unsurprising non-story. Yes, if you can modify the execution environment you can modify the executed code.
What you take if you use a bad one?
How is someone supposed to deploy this ?
First, have remote shell.
Wait until you hear about $PATH, $LD_LIBRARY_PATH, $IFS, and so on - or the rest of the OS files in /etc and /lib aside from /etc/ld.preload.
Hey, the author here... Our blog post is mainly talking about how the vulnerability works, but even if there is an insider threat (or reverse shell or any kind of attack) there are ways to stop this. We at Bomfather have a solution for this (we aren't trying to plug ourselves here), but any good eBPF solution should be able to protect this.