This feels like something that’s a neat claim and will work against simple setups, but less accurate for more complicated scenarios (eg Tor). Then you’re really just relying on how accurate your knowledge of the proxies are.
Also, the readme has slightly incorrect logic I think:
> According to Special Relativity, information cannot travel faster than the speed of light. Therefore, if the round trip time (RTT) is 4ms, it's physically impossible for them to be farther than 2 light milliseconds away, which is approximately 600 kilometers.
It calls out the 33% for fiber but ignores that there’s not a straightline path between two points on the network and there could be wireless, cable, and DSL links somewhere on that hop.
Also, the controlled variable here is latency, not distance. Thus you can always increase latency through buffering and therefor you could be made to appear further than you are. And that buffering need not even be intentional - your perceived distance estimate will vary based upon queuing delays in intermediary depending on time of day (itself a fingerprint if you incorporate time-aware measurements, but a source of error if you don’t).
Fingerprinting is hard and I dislike the framing that it’s absolutely impossible to mask or that there’s not false positive and false negative error rates with the fingerprint.
About the straightline path I did think of that but apparently I forgot to address it when writing the README :p
The point I was trying to make is that if the RTT is low enough you can know the connection is being made from close, it's an upper bound, and making some assumptions you can get it lower, so it's not a way of knowing the exact distance but rather the max distance the connection can be made from. If someone is in Spain but they can't be more than 400km from Australia, something went terribly wrong somewhere hehe
In hindsight I think the issue with my explanation is that I was trying to explain the differences when fingerprinting two different protocols, but ended up going for a TCP-only approach since Fastly wouldn't expose to me the data I needed for the TLS and HTTP RTT. But in theory fingerprinting with protocol RTT difference where one protocol is proxied and the other is impossible to bypass, but this is only the theory.
I think I will edit the README in the future since I don't like how it turned out too much. Thanks for the feedback!
I think for stealth TCP proxies are more common since you can use your own TLS fingerprints and all of that, with something like an HTTP proxy you'd need to set up your requests to match with the TLS fingerprint that the proxy is using, although I guess the proxy could make the TLS look the same? There are other ways of detecting HTTP proxies like for example comparing with the RTT of websockets or something like that, the idea is that there will always be at least one thing with RTT from the proxy and at least the RTT for one thing from the client that must go trough the proxy, you measure the difference between the two and there you have it.
The most common method of proxying with residential proxies is still CONNECT tunnels and from my tests it catches a resi-proxy about 50% of the time. More with tuning of the score thresholds.
Every TCP proxy (that doesn't thwart this) is detectable :)
Countermeasure: pick some min-RTT >= the actual client RTT (you can do this as a TCP proxy by measuring client ping). Measure server RTT and artificially delay responses to be >= min-RTT. This will require an added delay during the handshake and ACKs, but no added delay for the response payloads.
Counter-countermeasure: the above may lead to TCP message types that don't make sense given a traditional TCP client state machine (e.g., delayed ACK would bundle ACK and PUSH but the system shows separate/simultaneous ACK and PUSH packets. Counter-counter-countermeasure is left to the reader.
I think you could also compare with TLS handshake timings, delay for client hello among other things. And you could also compare it with HTTP RTT, not to mention that you can do TCP fingerprinting and compare it with the TLS and HTTP fingerprint of the browser, you can also measure the IP TTL and ping, among many other things... What I mean is that there are a ton of things that can be done on both sides, but any company with enough people working at this and enough servers will surely make something miles away from my proof of concept, and they also have a lot of traffic to know what's baseline data and what isn't.
Neat demo. The unsettling part is how little signal you actually need: big CDNs and fraud teams already run much richer timing models than a simple min_rtt / rtt ratio. You can’t spoof away the speed of light, only add latency or jitter, and that itself becomes a fingerprint once you have enough traffic and a few global PoPs to compare from. So this doesn’t magically break L3 VPNs, but anyone relying on “just stick a TCP proxy in front and I’m anonymous/in-region” has been living with a pretty outdated threat model.
The minimal explanation is that TCP is "turned around" at a dumb proxy, but upper-layer protocols may go further before being turned around. Which is trivially avoidable by delaying the TCP response with the same timing as the upper-layer protocol (and doing so to the protocol above that, etc.)
The issue is that if HTTP is an extra 50ms than TCP for example, if you increase TCP by 50ms now HTTP is 100ms more. Basically it is always more no matter how much you increase it.
If the proxy can "see" the requests, then this isnt an issue because the headers can be trivially be modified.
The problem is that the proxies which are targets of identification - think proxies for large scale web scraping which use CONNECT tunnels - dont get to "see" the request.
Just in case someone tries to use it to make some kind of judgement about the traffic - there's a whole world behind legit or enforced proxies. Especially corporate environments will often tunnel all the traffic for compliance and audit reasons.
The difference in min TCP RTT and min RTT to respond to a websocket payload is a dead giveaway that there's a middlebox terminating TCP somewhere along the path. You can bypass this by sourcing your request within 30ms of wherever TCP is being terminated, anything under that threshold could be caused by regular noise and isn't a reliable fingerprint. Due to how many gateway's there are between you and a residential proxy exit node this makes fingerprinting them extremely easy.
I expect it won't be long until someone deploys the first proxy service that handles the initial CONNECT payload in the kernel before offloading packet forwarding to an eBPF script that will proxy packets between hosts at layer 3, making this fingerprinting technique obsolete. The cat and mouse game continues.
<html><body><h1>You don't seem to be using a TCP Proxy!</h1><p>(If you are using a VPN or any other kind of proxy that is not a TCP Proxy, this will not detect it)</p></body></html>
A request to a HTTPS target through a proxy will use a CONNECT request to establish a tunnel to the target.
This tunnel operates at layer 3, where the client sends TCP segments to the proxy, the server unpacks the segments and then repacks them into new segments to send to the end target. These new TCP segments will contain the timestamp of when they were created.
The HTTP request sent through those segments is unmodified, meaning it will contain the original timestamp from the client machine.
The newer timestamp on the TCP segments means there is a mismatch between the TCP RTT and the HTTP RTT.
When deployed on a popular server, one bit of "IP intelligence" this detector itself can gather is keep database of lowest-seen RTT per given source IP, maybe with some filtering - to cut out "faster-than-light" datapoints, gracefully update when actual network topology changes, etc.
That would establish a baseline, and from there, additional end-to-end RTT should become much more visible.
I imagine any big CDN implementing something like this could keep a database of all of this, combined with the old kind of IP intelligence and collecting not only RTT on other protocols like TLS, HTTP, IP (aka ping, and traceroutes too), TCP fingerprint, TLS fingerprint, HTTP fingerprint...
And with algorithms that combine and compare all these data points, I think very accurate models of the proxy could be made. And for things like credit card fraud this could be quite useful.
I like this. I could see this being extra useful for people not using CDN's if they could easily plug it into nginx, haproxy and such. Currently for proxies I look for the proxy headers and also use a list of known proxy IP's but that is obviously nowhere near as complete as what you built. It might also be interesting to test assorted configurations of SSH forwards and MitM TLS caching proxies such as Squid SSL Bump.
I guess for this to work best you'd build your own CDN and have as many servers as possible. I have always dreamed of an Open Source CDN managed by a nonprofit and dedicated to offering CDN services for free or for a reasonable cost.
If you did the timings by comparing to other protocols, like TLS or HTTP you could do this with a single server, but that's a bit more complex than doing it on the same protocol since you have to account for more stuff, but it could be done, at the end of the day, my idea with Aroma was mostly to prove that it's possible, thanks for the feedback btw!
Why would one want this? Are there particular situation(s) that it's desirable to detect a TCP proxy? Does presence of a TCP proxy indicate some adverserial behaviour? E.g. surveillance, censorship, a particular attack?
Also, something I haven't included on the README is that apart from testing with Tor, WARP and some other proxies. I did some testing with the free one-week trial of Brightdata's residential proxies, and it does detect them too!!!
It's done by checking the difference between the initial TCP RTT and the subsequent TCP RTTs, both of which can be retrieved from the Linux Kernel easily without the need for PCAPing. There is more info about how it is done on the README
It's set at a low threshold since I want to avoid blocking regular users at all costs, I think the detection can be improved a lot by using more data and not a single division to calculate the score, in this case it's a somewhat simple PoC.
Thanks for taking the time to test it, I really appreciate it!
It's a super cool tool, I've been wondering about an open source tool doing this since reading about the technique in one of Nikolai Tschacher's blog posts years ago (https://incolumitas.com/pages/about/).
There's a few ways to work around this, but I think it's one of the best signals available to detect low-effort/common proxy providers.
Oh I haven't seen that before, it's really cool, thank you for showing me that!
I want to clarify that the approaches are a bit different, they use IP intelligence too and this approach doesn't use any kind of websockets, which is a really good idea, and I have to admit I didn't think of that, but sadly it's not really possible to do it with Fastly.
Another big difference is that this could work with any TCP application, not only HTTP, and if you do it with HTTP/S you can know if it's a proxy or not on a request basis and totally passively, without adding any delay or changing the code of the app.
This feels like something that’s a neat claim and will work against simple setups, but less accurate for more complicated scenarios (eg Tor). Then you’re really just relying on how accurate your knowledge of the proxies are.
Also, the readme has slightly incorrect logic I think:
> According to Special Relativity, information cannot travel faster than the speed of light. Therefore, if the round trip time (RTT) is 4ms, it's physically impossible for them to be farther than 2 light milliseconds away, which is approximately 600 kilometers.
It calls out the 33% for fiber but ignores that there’s not a straightline path between two points on the network and there could be wireless, cable, and DSL links somewhere on that hop.
Also, the controlled variable here is latency, not distance. Thus you can always increase latency through buffering and therefor you could be made to appear further than you are. And that buffering need not even be intentional - your perceived distance estimate will vary based upon queuing delays in intermediary depending on time of day (itself a fingerprint if you incorporate time-aware measurements, but a source of error if you don’t).
Fingerprinting is hard and I dislike the framing that it’s absolutely impossible to mask or that there’s not false positive and false negative error rates with the fingerprint.
About the straightline path I did think of that but apparently I forgot to address it when writing the README :p
The point I was trying to make is that if the RTT is low enough you can know the connection is being made from close, it's an upper bound, and making some assumptions you can get it lower, so it's not a way of knowing the exact distance but rather the max distance the connection can be made from. If someone is in Spain but they can't be more than 400km from Australia, something went terribly wrong somewhere hehe
In hindsight I think the issue with my explanation is that I was trying to explain the differences when fingerprinting two different protocols, but ended up going for a TCP-only approach since Fastly wouldn't expose to me the data I needed for the TLS and HTTP RTT. But in theory fingerprinting with protocol RTT difference where one protocol is proxied and the other is impossible to bypass, but this is only the theory.
I think I will edit the README in the future since I don't like how it turned out too much. Thanks for the feedback!
By the way, it detects Tor, I tested it ;D
Do raw TCP proxies still get used often? I'd imagine most proxies you'd want to detect are full HTTP proxies and this formula won't detect those.
I suppose it's possible botnets ("residential proxies") may get detected this way if they're using SOCKS to forward requests?
Still, this looks like an interesting signal to add to a system like Anubis to increase the difficulty for suspicious traffic sources.
This does very reliably detect TOR traffic, though you can just download a list of exit nodes if that's what you want.
I think for stealth TCP proxies are more common since you can use your own TLS fingerprints and all of that, with something like an HTTP proxy you'd need to set up your requests to match with the TLS fingerprint that the proxy is using, although I guess the proxy could make the TLS look the same? There are other ways of detecting HTTP proxies like for example comparing with the RTT of websockets or something like that, the idea is that there will always be at least one thing with RTT from the proxy and at least the RTT for one thing from the client that must go trough the proxy, you measure the difference between the two and there you have it.
The most common method of proxying with residential proxies is still CONNECT tunnels and from my tests it catches a resi-proxy about 50% of the time. More with tuning of the score thresholds.
Every TCP proxy (that doesn't thwart this) is detectable :)
Countermeasure: pick some min-RTT >= the actual client RTT (you can do this as a TCP proxy by measuring client ping). Measure server RTT and artificially delay responses to be >= min-RTT. This will require an added delay during the handshake and ACKs, but no added delay for the response payloads.
Counter-countermeasure: the above may lead to TCP message types that don't make sense given a traditional TCP client state machine (e.g., delayed ACK would bundle ACK and PUSH but the system shows separate/simultaneous ACK and PUSH packets. Counter-counter-countermeasure is left to the reader.
I think you could also compare with TLS handshake timings, delay for client hello among other things. And you could also compare it with HTTP RTT, not to mention that you can do TCP fingerprinting and compare it with the TLS and HTTP fingerprint of the browser, you can also measure the IP TTL and ping, among many other things... What I mean is that there are a ton of things that can be done on both sides, but any company with enough people working at this and enough servers will surely make something miles away from my proof of concept, and they also have a lot of traffic to know what's baseline data and what isn't.
It's a complex but fun world we live in hehe
Neat demo. The unsettling part is how little signal you actually need: big CDNs and fraud teams already run much richer timing models than a simple min_rtt / rtt ratio. You can’t spoof away the speed of light, only add latency or jitter, and that itself becomes a fingerprint once you have enough traffic and a few global PoPs to compare from. So this doesn’t magically break L3 VPNs, but anyone relying on “just stick a TCP proxy in front and I’m anonymous/in-region” has been living with a pretty outdated threat model.
Thank you! There are other ways of detecting L3 VPNs, but I wanted to start with proxies since they do most of the damage.
The minimal explanation is that TCP is "turned around" at a dumb proxy, but upper-layer protocols may go further before being turned around. Which is trivially avoidable by delaying the TCP response with the same timing as the upper-layer protocol (and doing so to the protocol above that, etc.)
The issue is that if HTTP is an extra 50ms than TCP for example, if you increase TCP by 50ms now HTTP is 100ms more. Basically it is always more no matter how much you increase it.
Not if you receive the HTTP request from the client first, before any interaction with the end-host.
If the proxy can "see" the requests, then this isnt an issue because the headers can be trivially be modified.
The problem is that the proxies which are targets of identification - think proxies for large scale web scraping which use CONNECT tunnels - dont get to "see" the request.
Would a similar technique work for tunnels through QUIC?
Just in case someone tries to use it to make some kind of judgement about the traffic - there's a whole world behind legit or enforced proxies. Especially corporate environments will often tunnel all the traffic for compliance and audit reasons.
Yes, it's important to keep this in mind, thanks for your comment!
This is the core concept of how proxies are detected via services like https://layer3intel.com/tripwire or https://spur.us/monocle/
The difference in min TCP RTT and min RTT to respond to a websocket payload is a dead giveaway that there's a middlebox terminating TCP somewhere along the path. You can bypass this by sourcing your request within 30ms of wherever TCP is being terminated, anything under that threshold could be caused by regular noise and isn't a reliable fingerprint. Due to how many gateway's there are between you and a residential proxy exit node this makes fingerprinting them extremely easy.
I expect it won't be long until someone deploys the first proxy service that handles the initial CONNECT payload in the kernel before offloading packet forwarding to an eBPF script that will proxy packets between hosts at layer 3, making this fingerprinting technique obsolete. The cat and mouse game continues.
curl -x http://xxxxx:xxxxx@geo.iproyal.com:11202 -L https://aroma.global.ssl.fastly.net/
<html><body><h1>You don't seem to be using a TCP Proxy!</h1><p>(If you are using a VPN or any other kind of proxy that is not a TCP Proxy, this will not detect it)</p></body></html>
That's strange, could you try with "https://aroma.global.ssl.fastly.net/score"?
pardon my ignorance but it's a HTTP proxy not a TCP one. is not it? ... or is it considering that https upstream goes through "CONNECT" request?
A request to a HTTPS target through a proxy will use a CONNECT request to establish a tunnel to the target.
This tunnel operates at layer 3, where the client sends TCP segments to the proxy, the server unpacks the segments and then repacks them into new segments to send to the end target. These new TCP segments will contain the timestamp of when they were created.
The HTTP request sent through those segments is unmodified, meaning it will contain the original timestamp from the client machine.
The newer timestamp on the TCP segments means there is a mismatch between the TCP RTT and the HTTP RTT.
Very clever, I like it.
When deployed on a popular server, one bit of "IP intelligence" this detector itself can gather is keep database of lowest-seen RTT per given source IP, maybe with some filtering - to cut out "faster-than-light" datapoints, gracefully update when actual network topology changes, etc.
That would establish a baseline, and from there, additional end-to-end RTT should become much more visible.
First of all, thanks!
I imagine any big CDN implementing something like this could keep a database of all of this, combined with the old kind of IP intelligence and collecting not only RTT on other protocols like TLS, HTTP, IP (aka ping, and traceroutes too), TCP fingerprint, TLS fingerprint, HTTP fingerprint...
And with algorithms that combine and compare all these data points, I think very accurate models of the proxy could be made. And for things like credit card fraud this could be quite useful.
I like this. I could see this being extra useful for people not using CDN's if they could easily plug it into nginx, haproxy and such. Currently for proxies I look for the proxy headers and also use a list of known proxy IP's but that is obviously nowhere near as complete as what you built. It might also be interesting to test assorted configurations of SSH forwards and MitM TLS caching proxies such as Squid SSL Bump.
I guess for this to work best you'd build your own CDN and have as many servers as possible. I have always dreamed of an Open Source CDN managed by a nonprofit and dedicated to offering CDN services for free or for a reasonable cost.
If you did the timings by comparing to other protocols, like TLS or HTTP you could do this with a single server, but that's a bit more complex than doing it on the same protocol since you have to account for more stuff, but it could be done, at the end of the day, my idea with Aroma was mostly to prove that it's possible, thanks for the feedback btw!
Why would one want this? Are there particular situation(s) that it's desirable to detect a TCP proxy? Does presence of a TCP proxy indicate some adverserial behaviour? E.g. surveillance, censorship, a particular attack?
Surveillance, on the part of those who want to do this fingerprinting.
Also, something I haven't included on the README is that apart from testing with Tor, WARP and some other proxies. I did some testing with the free one-week trial of Brightdata's residential proxies, and it does detect them too!!!
If you like this then you will probably like "The Cuckoo's Egg: Tracking a Spy Through the Maze of Computer Espionage", a 1989 book by Clifford Stoll.
Also available as audiobook, and a documentary ("The KGB, The computer and Me"). https://www.youtube.com/watch?v=Xe5AE-qYan8
Thank you! Will check it out!
so will this detect residential proxies? how is that being done, I am getting hammered and its all legitimate normal ISP traffic.
It's done by checking the difference between the initial TCP RTT and the subsequent TCP RTTs, both of which can be retrieved from the Linux Kernel easily without the need for PCAPing. There is more info about how it is done on the README
To answer your first question, in my tests its around 50% of requests making it through.
Are you using a proxy? If you aren't that would be concerning, since false positives are way worse than false negatives.
If you are then it means the score is sometimes a bit lower and sometimes a bit higher than 0.1, which is the threshold for getting blocked.
If you want to know the exact score, you can check https://aroma.global.ssl.fastly.net/score
It's set at a low threshold since I want to avoid blocking regular users at all costs, I think the detection can be improved a lot by using more data and not a single division to calculate the score, in this case it's a somewhat simple PoC.
Thanks for taking the time to test it, I really appreciate it!
I'm testing using our residential proxies.
It's a super cool tool, I've been wondering about an open source tool doing this since reading about the technique in one of Nikolai Tschacher's blog posts years ago (https://incolumitas.com/pages/about/).
There's a few ways to work around this, but I think it's one of the best signals available to detect low-effort/common proxy providers.
Oh I haven't seen that before, it's really cool, thank you for showing me that!
I want to clarify that the approaches are a bit different, they use IP intelligence too and this approach doesn't use any kind of websockets, which is a really good idea, and I have to admit I didn't think of that, but sadly it's not really possible to do it with Fastly.
Another big difference is that this could work with any TCP application, not only HTTP, and if you do it with HTTP/S you can know if it's a proxy or not on a request basis and totally passively, without adding any delay or changing the code of the app.
But yeah, it's a really cool demo, thanks again!