Chinese AI companies are free to use any Nvidia GPU they want as long as the GPU is not physically located within China. It is entirely consistent with US export controls for Chinese AI companies to remotely use GPUs located in Singapore or Malaysia.
“In facilities across the region, Megaspeed rents Nvidia chips to Chinese tech giants such as Alibaba, an arrangement that does not violate current US export control regulations.” See [1].
Literally my first thought. Noone can stop them from training in data centers that are located elsewhere. It's not like some military hardware that needs to be physically shipped to China for them to make use of it. In fact why would they bother with Chinese compute providers if those are limited because of GPU export controls? They knew this for a long time and probably have been knocking on the door of providers in other countries for a while. Heck, they could be training inside the US. They already distribute their open model weights from the US.
The US definitely has the power to stop most of it. It just looks like the executive branch doesn’t yet have the legal authority to do it, but it’s in an advanced legislative state.
How are you going to monitor who is logging into a system, or who a job is being run for? If you want that level of control the machines should not be exported at all.
Datacenters are very large and not very mobile and full of very expensive export controlled equipment and training runs are large expensive contracts. This is not exactly a difficult context to enforce KYC rules in
Same way they do for Iran: if they catch you providing services to someone in Iran, they send you to jail for life. And they monitor all financial transactions so they'll notice.
They can do that to Americans. Harder for the rest of the world. How is the US going to monitor CIPS payments? How will they jail someone who never goes to the US? Just how much global jurisdiction is the US claiming?
In practice, the key ways that the US exerts influence outside of its own borders: (1) Sanction the company, which means any other company that does business with that company can also be sanctioned. (2) If enough companies are evading sanctions in a single country or the govt is helping those companies, then threaten to cut the country from USD settlement with the Federal Reserve, e.g., no USD FX allowed. That pretty much screws most economies. The "long arm" of the US govt is very, very, very long!
To be clear: I am not saying any of this is fair (or legal); I am only explaining how it works.
> How will they jail someone who never goes to the US?
The same way they "jail" UN-appointed judges that go against their masters' interests: de-platform them from all US-controlled payment and software platforms.
Yes, they can. They cannot buy things from Amazon or book through booking.com, as that French judge said, but they can do things that do not require American systems. Do you have any evidence they cannot have phones? What stops they shopping in person? Or booking directly with a hotel?
> He said: “We are going 30 years back in time. It’s like a time machine returning us to a pre-digital world,” he told France TV.
So another company in Asia is going to execute the batch and send the result back to China. And you can add as many indirection levels as you like. This is going to be impossible to track.
You'd just invert it. Make it so you have to be approved under export controls as a customer like they were gonna do for mythos (and like how mythos sort of still works) and make it an allowlist instead of a denylist
If that company is acting in defiance of sanctions, like any other company that acts as a middleman to circumvent sanctions, and it is caught, it will face severe consequences.
There's no gotcha here, companies the world over could just ignore any sanctions they wanted, and some try to, but they do so knowing they themselves are (potentially) breaking local and international laws.
You might have well asked "what if a company helps you by importing contraband instead of you doing it directly"
One of the key reasons that a country like the USA can enforce sanctions is that it can tell other countries that if their companies/citizens/etc break the sanctions, then the USA will put restrictions on those countries ability to trade with the USA, which is why only a few countries will trade with Iran, North Korea, and Russia
At some point those countries will also tire of the USA trying to tell them what to do and will tell the USA politely to fuck off. This is basically what India did with US sanctions on Russia and Iran.
Given the USA has completely torched its reputation amongst allies in the space of a year, I would expect this to happen more often.
Exactly right. The term I use is "economic network effect".
Does anyone know why does the US allows Canada to trade with Cuba? That one always stood out to me as a bit odd. My guess: Canada is the largest trade partner with the US, so the US probably has more to lose by sanctioning Canada.
Yes, that's called a treaty, or a trade agreement. Remember all the fuss about Investor/State Dispute Settlement clauses? Most countries have them with each other by now.
To add to this, if the provider suspects that a customer is facilitating sanction busting, or could reasonably be expected to have foreseen it, that's enough for a discussion with lawyers...
> sanctions have successfully applied to countries such as Russia, Iran, etc for decades.
Largely because other countries are willing to go along with it, and to the extent they are willing to do so.
Even NATO countries import Russian gas. The UK rolled back sanctions on Russia to import diesel earlier this year. Most of the rest of the world did not impose effective sanctions and lots of people trade with them (including huge economies like China).
The rest of the world is to dependent on China to want to get involved. The US might succeed in the current administrations attempt to split the world into new cold war spheres of influence (because people are scared of China too) but its not going to be easy, it will be painful, and its not going to get many countries to completely agree to what the US wants.
Not sure if you understand this, but nobody said they were, but, if you have a quick browse of the threads, it has been pointed out how they ensure cooperation from other countries.
- Electing someone USA doesn't like is not a crime.
Show me where anybody claimed anything close to that.
The truth is you've come here with a mindset based on a political ideology and that's blinded you to the objective discussion being held.
Further, the fact that I keep pointing out your argument is the same as the one where people claim laws are of no use because they are routinely broken does not mean anything other than I am telling you how pathetic your claim is.
You're going to make it the provider's problem, unleash the full wrath of the US Government on one or two who don't comply pour encourager les autres, and then it self-regulates(ish).
I was wondering why you wrote this part in French instead of English. Turns out it is a phrase that is sometimes loaned into English. It comes from the (French) writings of Voltaire.
Or our allies that we control. But in terms of how - remember the mythos launch? Im absolutely not saying it's necessarily a good idea, I'm just that it's not that hard if we want to do it.
They're also free to use Nvidia GPUs physically located in China. However, customs and border enforcement will try to stop them from moving GPUs physically located outside China to physical locations inside China. However, if they can bypass that, they can use the GPU.
Is there any international regulation that forces China to comply with arbitrary US export controls? If not I can't see why Chinese companies are not free to use any Nvidia GPU they can get their hands on period (How the US would then react towards China, the Chinese company and Nvidia is another question).
"A key player in getting Megaspeed off the ground was Alibaba, the Chinese e-commerce leader that has also been an aggressive player in AI, said people familiar with the negotiations at the time. The U.S. had begun imposing export controls on advanced chips in late 2022, preventing Nvidia from selling them directly to China over concerns they could boost China’s military and surveillance capabilities.
"Alibaba was looking for a partner that could help it access computing power from data centers stuffed with Nvidia chips located outside China, the people said."
The author of this piece is a SWE-turned-influencer "China hawk" from OpenAI (i.e. the primary advocate of export bans along with Anthropic), and this isn't an academic paper despite the theming.
I have no idea whether this is legit or not (there's no link to any code), I'm mentioning this because all the comments here are assuming truth without verifying the contents.
The US could impose secondary sanctions to close the loophole, or pressure these countries behind closed doors to not allow these data center uses. Since the only goal of the export restrictions is to hamper Chinese AI efforts, that would be a logical step. If there is one thing that is certain then it's that the current US administration doesn't care at all about open trade and international trade agreements.
The method seems sound and the findings correct. But I think many are missing that (as the research itself says) this doesn't prove any illegal behavior. It's perfectly legal for Chinese companies to train models on GPUs that they rent outside of China. It's also legal for companies to buy GPUs and install them in Malaysia with the intention of renting them to Chinese companies.
There isn't even an allegation that the Chinese companies lied about this. They simply didn't disclose which chips were used. In all cases where they did disclose the training chips, this research found the disclosures to be accurate.
The small print in TFA states the many limitations of their method, i.e. that it actually provides no evidence about which GPUs were really used for training, and much less about who was their owner and where they were located.
The method just strongly suggests that the initial values of the weights, before starting training, were generated with the NVIDIA cuRAND library on a Blackwell GPU (or on an AMD GPU, where the NVIDIA library is emulated for compatibility).
The training might have been done with thousands of GPUs of other types. It is plausible that the GPUs used for training or at least a part of them may have been of the same type with the one used for initializing the model, but it is not proven in any way.
A scenario that matches what is reported in the article, but in which there is only a single Blackwell GPU involved, is if the LLM was intialized on the workstation of some lead developer, who had the privilege to use the best GPU they had, and then it was transferred to a server cluster for training, where other kinds of GPUs were used.
Now that this has been published, it would be very simple in the future for any Chinese AI company to avoid this "fingerprinting", by just using a different PRNG, instead of cuRAND.
Yes it's simple to avoid this fingerprinting now that people know about it, but nobody knew about it before. I don't believe people are casually transferring snapshots between chip architectures without a really good reason, either. It's hard enough to coordinate these training runs without involving multiple chip architectures in a single run.
'Legal' and 'illegal' apply to Nvidia or other US companies facilitating these activities. Perfectly legal for Nvidia to supply GPUs to be rented by Chinese companies, or for Oracle to build datacenters for Chinese companies to rent, as long as they are installed outside of China.
You'd think so but the USA is also the country that's most advanced at getting its legal system to apply outside of it. E.g. they shut down the International Criminal Court.
I used to think the same, that export controls didn't work, until I realized that China has leapfrogged the West on every technology that wasn't subject to export controls. And the only ones where they're still behind are the ones under export controls.
It's insane how many articles have dropped in the past 24 hours that all have a similar tone. At this point it's hard to see this as anything other than a broad coordination from someone who wants to destroy Jensen/Nvidia.
There is a 1T IPO coming up. I'd be surprised if they didn't hand over plenty of bucks to PR companies, grass root campaigns, influencers and journalists.
if we're going paranoid mode, on 9th of November Trump said that "leader of one large company connected to artificial intelligence" was almost sent to prison by Ministry of Justice due to wrongdoings connected to Ai, and this threat caused immediate correction by that company
I guess fingers crossed & good luck to them? After making consumer hardware unaffordable due to their greed I guess that is a reasonable outcome & karma at work.
I definitely care about this much less than it seems I'm supposed to.
I think the frontier models are definitely in a healthy dynamic where it seems the near-but-not-quite performance of open-weight models are what is pushing the most innovation and price competition from OpenAI, Anthropic and co
Some people seem to think that American sanctions and export controls are some form of absolute justice, and that any violation is an affront to humanity.
If GPUs are going to be priced out of consumer reach anyway, I would rather see the GPUs used to create open weight models than just to make these AI companies richer.
Well the problem is you can't prove they are only being used to create open weight models. You can't prove they aren't also creating closed weight models with military applications beyond what they've publicly open sourced.
Why should I care that China is making some super powered hacking model? We already know they do state sponsored APTs. It's just motivation to care about security and red team your server code.
Their government is not really the type to violently take over other countries and impose their laws. Taiwan is a special case because they see it as a rebel state on their own land. The rest of us are probably better off with China being the top superpower than Trump's America.
> Their government is not really the type to violently take over other countries and impose their laws.
Taiwan would like a word. Taiwanese govt was the original government in China before CCP violently took over in the aftermath of WW2 with help from the Soviet Union.
China was also involved in destabilizing Vietnam by backing the communists and then after having a fall out with the communists there, proceeded to engage in border conflicts for almost two decades before normalizing relations.
China hasn't lifted a finger for Ukraine. It hasn't lifted a finger for Venezuela. It hasn't lifted a finger for Iran. Nobody knows what China's long term intentions are or how China will change after Xi Jinping's reign.
You don't surrender your culture's lead in tech willingly as it could be existential down the road.
> We recover traces of the models’ initial values, and then search for the hardware configurations and initial configurations that reproduce them
From my (very cursory, naive, layman's) glance at the summary, they're identifying and tracing some characteristics that specific hardware imparts into a model's initialized weights. But then concluding that this indicates the models were also "trained on" that same hardware.
But is that a valid assumption? Is it possible that a model was initialized on one system configuration, while the bulk of the training was performed on another?
In fact, it would seem reasonable for the "okay, I'm setting up this new model on my workstation, prepping it for training next month" pipeline step to be far removed from the "okay, now deploy the [initialized, but not yet trained] model across our giant fleet of beefy training machines".
yes. The USA hasn't force an embargo on China to other countries (e.g. by threatening to cut off Swift access). We aren't technically at war either, so its highly unlikely that the US would go that far.
What is surprising, I guess, is that the US allows GPUs to be exported at all to most countries. Since...as long as they are exported to a country that is friendly to china (which is basically most countries on the planet), China has access to GPUs on which to train with, making GPU purchasing restrictions mainly symbolic.
They dont need to threaten other countries, they can just pass a law saying that anyone anywhere in the world selling or renting gpus to chinese companies is committing a crime. Then they can sanction any company that does it anyway and that makes it a crime for any american company to sell gpus to that company. There have already been quite a few criminal cases around the export ban.
That would be unenforceable though knowing American arrogance they would definitely try. It’s like China trying to stop their rare earths from going to American uses by requiring all companies that import them to track their usage.
> The newest models we could read from Alibaba (Qwen), Zhipu (GLM), Tencent, Xiaomi and StepFun all read as Hopper-trained. Meituan says its LongCat-2.0 was trained entirely on “AI ASIC superpods,” and we found no NVIDIA fingerprint in the model weights it’s released.
Yeah makes sense. They are in transition. This article doesn't really tell us anything new (or interesting). There's nothing wrong w Chinese labs using GPUs that are already in the country. An export ban just means reduced import of new GPUs
Black & grey market devices marked up many times cost buyers heavily, while still suppressing motivation to develop sovereignty.
If nvidia selling to China tolerated for the realpolitik and gets undone by incoming powers to create an appearance of change, is it really the right thing to do? Is it the smart thing to do?
It's an interesting thought experiment, though, we will probably never know.
Why on earth would that be suppressing motivation to develop sovereignty? The whole thing makes developing sovereignty an existential priority. When one critical part of your AI technology stack is subject to whims of your adversary, it's not really fully your AI technology stack. Today they turn the tap partially on, but only unofficially, tomorrow they'll shut it off. Time to buy your own tap.
I don't know the exact reason, but all empirical evidence shows that it does. E.g. the whole of Europe still using Microsoft because they can, even though bits and pieces get cut off (like the ICC). If Microsoft cut off the whole of Europe, then Microsoft wouldn't have power over them any more.
Microsoft's days in many EU countries' government departments are numbered regardless of what Microsoft does. The understanding is there: a critical dependence on a product coming from a now-unreliable ally is a problem. It will, however, take many years for this to actually happen, because it's much easier said than done.
Yup, I think the same applies to traffic of arms regulations.. Private firms sell to countries they are 'allowed' to, passing it between organisations in a chain until they get to the original customer they couldn't sell to directly.. Cuts are taken along the way, and people making money look the other way..
When Xi visited, Trump offered to sell him US made weapons.
What's the chance that these GPUs got to China without Trump getting a cut somehow?
Pretty much zero. Trump sat on knowledge of the affair for an entire year. That's extremely guilty behavior, he was either extorting Huang or covering g for him.
The whole thing is the most corrupt thing we have ever seen at the presidential level.
I have run some large open models on neoclouds and the KYC has been minimal, even if the USG stops all GPU exports it will be extremely difficult to crack down on Chinese renting GPUs on Western servers.
There will be a black market of renting IDs to KYC and gain access to the chips, and then the hosts will monitor their servers for what is being trained, it will be a cat and mouse game.
We are headed for an increasingly unfree computation landscape and the politicians are going to make demands that will be hard to deliver.
Weird detail: some of these results are based on initial patterns in weights surviving through training. But isn't muon, which is used for many of these models, well-known to be able to consistently erase patterns in weights, unlike previous optimizers [1]?
This doesn't really call the overall results of the report in the question (the strongest finding is the perfectly matching weights for dead-weight rows which weren't touched by an optimizer at all), but it does suggest that the explanation in the twitter thread[2] of how this works is way too simplistic. Are the patterns in question somehow "special" and not erasable even with muon?
i dont know what is expected of export controls... they are never effective. they also find western chips in iranian drones etc. there is a huge gray area of resellers who resel to resellers etc. its pretty much uncontrollable....
During the Cold War, America used various shell companies to get valuable titanium out of Russia for building spy planes. If someone is willing to pay, smugglers will find a way.
> Simply banning things causes the price to go up. That alone is good enough to motivate the policy decision.
No twice.
First, banning them doesn't necessarily even make the price go up. If all you have to do to get the product is put someone else's name on the form, that raises the price by a number that rounds to zero. It probably costs your own companies more in bureaucracy and paperwork to try to enforce the pointless rules than it costs the other one to put a different name on the form.
Second, suppose it actually worked somehow. What happens then? The other country pays the higher price temporarily while throwing out some big government subsidies to develop a domestic version of the technology. Soon your own companies have a new foreign competitor in the world market which is receiving large government subsidies.
> That alone is good enough to motivate the policy decision.
This may work when its against an smaller. When deployed agianst a (near) peer - there usually isn't a price to high for a field of national security interest. China could get all the GPUs it wants / the US could get all the rare-earth magnets it wants over the objections of the other country.
> It can get all the GPUs it wants... at a higher price.
That's implicit in "there's no price too high". However higher the price is, the difference is still a rounding error when your budget is trillions of dollars.
I present this short-form documentary — or animated infographic, if you will — on how much governments are prudent in saving money in areas considered of vital national interests, denominated in GPUs instead of dollars: https://www.youtube.com/shorts/F6jMsMBVB34
Export controls inherently raise the cost of doing business with parties you don't like, the other party has to either invest more resources acquiring them, or make the decision to forego acquiring them. The grey market exists precisely because there's an opportunity to exploit this for profit. At the end of the day, when these effects of these export controls are discussed, they usually seem to revolve around export controls setting a specific program or effort back by a span of months, not in stopping a specific commodity from accessing another market whatsoever (excepting some hyper-specific components).
As Hezbollah discovered in 2024, covert supply-chain interdiction is very possible and I'd be surprised if the CIA didn't know what was going on, at least to some extent.
If you could rent compute to what is essentially a competitor and monitor everything about their code and models in the process, since you're still in control of the hardware, why would a company or in this case a country not take that competitive advantage?
Weights start as random values generated from a seed. Pytorch uses however many cores are available on the architecture to write out the initial values in parallel, so each seed + architecture pair has a particular signature. If I understand Will's technique correctly he's basically made a rainbow table of initial weight states. Training alters the initial weights, but if you find one seed + architecture pair that vastly more-closely matches the trained weights, you can infer it must have been the source of the initial weights.
Distillation might shape the weights similarly in some way to the source model, but that would come from the signal, not the noise. I think it would be very surprising if the initial untrained weights were encoded with any fidelity into the outputs such that training another randomly-initialized set of weights (likely of completely different size and shape) on them could leave a signature of the source model's initial state, does that make sense?
Not seeing how that could happen since all the "distillation" attacks i've seen are just token level, not logprob level, and definitely not vector level. So, I don't see how that could be the case.
1. access to the very best frontier models for hard problems
2. good enough open models from China, the US (I like poolside.ai, but lots of good choices), and even other countries like France that make smaller investments in Digital Sovereignty.
If the ‘super rich fucks’ in the US who have been making risky AI bets succeed in convincing the US government to curtail the use of small efficient models for easier problems, that will just hurt our economy.
@dang curious, is this post getting heavily downvoted or flagged?
I get that the geopolitical angle is sensitive, especially with the backdrop of today's big news - but I thought this was a really well-researched technical deep dive, right up HN's alley. (TIL: you could fingerprint gpu hardware just from an llm weights!)
I need to double check PyTorch code, I am not sure how robust this fingerprinting is without PyTorch "bug". Parallel philox based PRNG is a well-researched area and I am surprised (even if it is not secure-PRNG) it can fingerprinting like this. But obviously there are more people who are more knowledgeable on this can give their perspectives here.
Ah, that means the recent huge open-source AI advances, that we all benefit from, directly and indirectly, come from Jensen Huang's China boycott circumvention.
Which means, in future, we maybe cannot expect such large capability increases.
Cute trick but it's not exactly a secret, e.g. deepseek recently published their own inference engine -- and it only has kernels for SM100 (blackwell) and Huawei Ascend 950, and all its published performance results using blackwell.
No one found it remarkable a couple weeks ago, ... but now the fact that DS is using blackwell is apparently some super secret requiring clever RNG init stats???
"Specially configured AMD hardware, which is also restricted from export to China, is the only alternative we have identified; we have excluded it for DeepSeek-V4-Flash"
This sounds as if they are using something VERY fuzzy to "match" things, and if they are ALSO matching against completely different hardware which this makes it sounds like.... then I dont know if this means anything.
They match the pseudo-random generator from the NVIDIA library cuRAND.
AMD has a library that emulates it, for compatibility.
Their methodology is interesting, but it only proves access to a single Blackwell GPU, which would have been sufficient for initializing an LLM with random values, before the start of its training.
There is no evidence whatsoever about which GPUs were used for training, except the supposition that if they had access to 1 Blackwell, they might have had access to more of them. Such a supposition may be justified about the number of cockroaches in a kitchen, but less so about the components of a training cluster, which might be heterogeneous.
The LLM could have been initialized on the workstation of a developer, before being transferred to a server cluster, for training.
No, they distinguish AMD vs Nvidia based on rounding or some similar difference farther down in the stack, finding that the values that would differ based on chip vendor are all consistent with an Nvidia platform. Search for "slightly differently".
US frontier AI models carry the fingerprint of data acquired from illegal sources like Anna Archive, libgenesis, etc. and, (un?)surprisingly, nobody cares.
Only Anthropic was stupid enough to let be caught red-handed but get away with some ridiculously low penalty.
Chinese AI companies are free to use any Nvidia GPU they want as long as the GPU is not physically located within China. It is entirely consistent with US export controls for Chinese AI companies to remotely use GPUs located in Singapore or Malaysia.
“In facilities across the region, Megaspeed rents Nvidia chips to Chinese tech giants such as Alibaba, an arrangement that does not violate current US export control regulations.” See [1].
[1] https://www.atlanticcouncil.org/dispatches/beyond-the-spy-sc...
Literally my first thought. Noone can stop them from training in data centers that are located elsewhere. It's not like some military hardware that needs to be physically shipped to China for them to make use of it. In fact why would they bother with Chinese compute providers if those are limited because of GPU export controls? They knew this for a long time and probably have been knocking on the door of providers in other countries for a while. Heck, they could be training inside the US. They already distribute their open model weights from the US.
The US definitely has the power to stop most of it. It just looks like the executive branch doesn’t yet have the legal authority to do it, but it’s in an advanced legislative state.
How are you going to monitor who is logging into a system, or who a job is being run for? If you want that level of control the machines should not be exported at all.
Datacenters are very large and not very mobile and full of very expensive export controlled equipment and training runs are large expensive contracts. This is not exactly a difficult context to enforce KYC rules in
Same way they do for Iran: if they catch you providing services to someone in Iran, they send you to jail for life. And they monitor all financial transactions so they'll notice.
They can do that to Americans. Harder for the rest of the world. How is the US going to monitor CIPS payments? How will they jail someone who never goes to the US? Just how much global jurisdiction is the US claiming?
In practice, the key ways that the US exerts influence outside of its own borders: (1) Sanction the company, which means any other company that does business with that company can also be sanctioned. (2) If enough companies are evading sanctions in a single country or the govt is helping those companies, then threaten to cut the country from USD settlement with the Federal Reserve, e.g., no USD FX allowed. That pretty much screws most economies. The "long arm" of the US govt is very, very, very long!
To be clear: I am not saying any of this is fair (or legal); I am only explaining how it works.
How is the money going to end up in a US account via CIPS?
Why would it need to end up in a US account?
That's where Jensen Huang is
Does he own any data centres outside the US?
> How will they jail someone who never goes to the US?
The same way they "jail" UN-appointed judges that go against their masters' interests: de-platform them from all US-controlled payment and software platforms.
That is not jail.
if you are referring to people at the ICC, they are cut off from US services, but that is a long way from being in jail.
Can't travel, can't shop, can't have a cellphone or even a landline...
Yes, they can. They cannot buy things from Amazon or book through booking.com, as that French judge said, but they can do things that do not require American systems. Do you have any evidence they cannot have phones? What stops they shopping in person? Or booking directly with a hotel?
> He said: “We are going 30 years back in time. It’s like a time machine returning us to a pre-digital world,” he told France TV.
https://vinnews.com/2026/03/23/icc-judge-who-issued-netanyah...
If you would be so kind as to read your own source, it answers your questions
The same way that SAAS and IAAS (etc) providers do now for citizens of other countries under sanctions - IPs, Credit Cards
So another company in Asia is going to execute the batch and send the result back to China. And you can add as many indirection levels as you like. This is going to be impossible to track.
You'd just invert it. Make it so you have to be approved under export controls as a customer like they were gonna do for mythos (and like how mythos sort of still works) and make it an allowlist instead of a denylist
If that company is acting in defiance of sanctions, like any other company that acts as a middleman to circumvent sanctions, and it is caught, it will face severe consequences.
There's no gotcha here, companies the world over could just ignore any sanctions they wanted, and some try to, but they do so knowing they themselves are (potentially) breaking local and international laws.
You might have well asked "what if a company helps you by importing contraband instead of you doing it directly"
None of them are breaking local and international laws by doing so, unless a country's ruling says other countries laws supersede their own
One of the key reasons that a country like the USA can enforce sanctions is that it can tell other countries that if their companies/citizens/etc break the sanctions, then the USA will put restrictions on those countries ability to trade with the USA, which is why only a few countries will trade with Iran, North Korea, and Russia
At some point those countries will also tire of the USA trying to tell them what to do and will tell the USA politely to fuck off. This is basically what India did with US sanctions on Russia and Iran.
Given the USA has completely torched its reputation amongst allies in the space of a year, I would expect this to happen more often.
Exactly right. The term I use is "economic network effect".
Does anyone know why does the US allows Canada to trade with Cuba? That one always stood out to me as a bit odd. My guess: Canada is the largest trade partner with the US, so the US probably has more to lose by sanctioning Canada.
Yes, that's called a treaty, or a trade agreement. Remember all the fuss about Investor/State Dispute Settlement clauses? Most countries have them with each other by now.
To add to this, if the provider suspects that a customer is facilitating sanction busting, or could reasonably be expected to have foreseen it, that's enough for a discussion with lawyers...
Lol what consequences? Such shell companies can appear by the thousands every hour. Even if you ban 2 or 3 it doesn't matter.
Funny, sanctions have successfully applied to countries such as Russia, Iran, etc for decades...
They do try workarounds, and when those are caught there are consequences
You can sit back down now.
Edit: Is there going to be more "laws can be broken so let's not bother" arguments, because that's lame and boring
> sanctions have successfully applied to countries such as Russia, Iran, etc for decades.
Largely because other countries are willing to go along with it, and to the extent they are willing to do so.
Even NATO countries import Russian gas. The UK rolled back sanctions on Russia to import diesel earlier this year. Most of the rest of the world did not impose effective sanctions and lots of people trade with them (including huge economies like China).
The rest of the world is to dependent on China to want to get involved. The US might succeed in the current administrations attempt to split the world into new cold war spheres of influence (because people are scared of China too) but its not going to be easy, it will be painful, and its not going to get many countries to completely agree to what the US wants.
> They do try workarounds, and when those are caught there are consequences
Lol. Yeh we can all see how well the sanctions worked on Russia and Iran. Thanks for proving my point!
Again with the "laws are broken so let's not have them" argument.
Sanctions aren't laws. USA doesn't have universal jurisdiction. Electing someone USA doesn't like is not a crime.
- Sanctions aren't laws
Oh my poor sweet summer child, they are.
- USA doesn't have universal jurisdiction.
Not sure if you understand this, but nobody said they were, but, if you have a quick browse of the threads, it has been pointed out how they ensure cooperation from other countries.
- Electing someone USA doesn't like is not a crime.
Show me where anybody claimed anything close to that.
The truth is you've come here with a mindset based on a political ideology and that's blinded you to the objective discussion being held.
Further, the fact that I keep pointing out your argument is the same as the one where people claim laws are of no use because they are routinely broken does not mean anything other than I am telling you how pathetic your claim is.
Good day.
You're going to make it the provider's problem, unleash the full wrath of the US Government on one or two who don't comply pour encourager les autres, and then it self-regulates(ish).
> pour encourager les autres
I was wondering why you wrote this part in French instead of English. Turns out it is a phrase that is sometimes loaned into English. It comes from the (French) writings of Voltaire.
https://wordhistories.net/2016/10/22/pour-encourager-les-aut...
How so? They would need to forbid nvidia to sell to anybody but US customers
Or our allies that we control. But in terms of how - remember the mythos launch? Im absolutely not saying it's necessarily a good idea, I'm just that it's not that hard if we want to do it.
And regulate all the owners of nvidia hardware. Including, somehow, those who aren't in the USA.
how would that work?
They're also free to use Nvidia GPUs physically located in China. However, customs and border enforcement will try to stop them from moving GPUs physically located outside China to physical locations inside China. However, if they can bypass that, they can use the GPU.
This is the key statement. Lol
Is there any international regulation that forces China to comply with arbitrary US export controls? If not I can't see why Chinese companies are not free to use any Nvidia GPU they can get their hands on period (How the US would then react towards China, the Chinese company and Nvidia is another question).
No there isn't. If they have an Nvidia GPU, then they have it
I see that here also:
https://archive.ph/BiSrT
"A key player in getting Megaspeed off the ground was Alibaba, the Chinese e-commerce leader that has also been an aggressive player in AI, said people familiar with the negotiations at the time. The U.S. had begun imposing export controls on advanced chips in late 2022, preventing Nvidia from selling them directly to China over concerns they could boost China’s military and surveillance capabilities.
"Alibaba was looking for a partner that could help it access computing power from data centers stuffed with Nvidia chips located outside China, the people said."
The author of this piece is a SWE-turned-influencer "China hawk" from OpenAI (i.e. the primary advocate of export bans along with Anthropic), and this isn't an academic paper despite the theming.
I have no idea whether this is legit or not (there's no link to any code), I'm mentioning this because all the comments here are assuming truth without verifying the contents.
This post is from Will Depue
The supposed “China Hawk” you mention, on google searching, seems to be David Perdue.
Is there a chance you are confusing similar yet different last names as the same person?
No, the names are correct. OpenAI has quite a few SWEs who spend a lot of time advocating for positions on social media.
I don't understand. What difficulties would these Chinese companies face if they held their training overseas?
The US could impose secondary sanctions to close the loophole, or pressure these countries behind closed doors to not allow these data center uses. Since the only goal of the export restrictions is to hamper Chinese AI efforts, that would be a logical step. If there is one thing that is certain then it's that the current US administration doesn't care at all about open trade and international trade agreements.
The method seems sound and the findings correct. But I think many are missing that (as the research itself says) this doesn't prove any illegal behavior. It's perfectly legal for Chinese companies to train models on GPUs that they rent outside of China. It's also legal for companies to buy GPUs and install them in Malaysia with the intention of renting them to Chinese companies.
There isn't even an allegation that the Chinese companies lied about this. They simply didn't disclose which chips were used. In all cases where they did disclose the training chips, this research found the disclosures to be accurate.
The small print in TFA states the many limitations of their method, i.e. that it actually provides no evidence about which GPUs were really used for training, and much less about who was their owner and where they were located.
The method just strongly suggests that the initial values of the weights, before starting training, were generated with the NVIDIA cuRAND library on a Blackwell GPU (or on an AMD GPU, where the NVIDIA library is emulated for compatibility).
The training might have been done with thousands of GPUs of other types. It is plausible that the GPUs used for training or at least a part of them may have been of the same type with the one used for initializing the model, but it is not proven in any way.
A scenario that matches what is reported in the article, but in which there is only a single Blackwell GPU involved, is if the LLM was intialized on the workstation of some lead developer, who had the privilege to use the best GPU they had, and then it was transferred to a server cluster for training, where other kinds of GPUs were used.
Now that this has been published, it would be very simple in the future for any Chinese AI company to avoid this "fingerprinting", by just using a different PRNG, instead of cuRAND.
Yes it's simple to avoid this fingerprinting now that people know about it, but nobody knew about it before. I don't believe people are casually transferring snapshots between chip architectures without a really good reason, either. It's hard enough to coordinate these training runs without involving multiple chip architectures in a single run.
Does 'legal' an 'illegal' apply here? Surely, the USA legal system only apply in the USA?
'Legal' and 'illegal' apply to Nvidia or other US companies facilitating these activities. Perfectly legal for Nvidia to supply GPUs to be rented by Chinese companies, or for Oracle to build datacenters for Chinese companies to rent, as long as they are installed outside of China.
You'd think so but the USA is also the country that's most advanced at getting its legal system to apply outside of it. E.g. they shut down the International Criminal Court.
When did they shut it down?
There were cases being considered as recently as yesterday. https://www.icc-cpi.int/case-records
https://news.ycombinator.com/item?id=50479876
So not "shut down" but unsubstantiated reports that one country is in talks with the US about limiting its powers.
>USA is also the country that's most advanced at getting its legal system to apply outside of it
Yes it has nukes. However china also has nukes. That poses some legal limitation.
The USA didn't need nukes to get Venezuela.
Did they really "get" Venezuela ? What actually prevents them to start ignoring what US wants in a while ?
It needed them to avoid retaliation from the rest of the world.
it's funny because it convinces me that export controls don't work so why do them
I used to think the same, that export controls didn't work, until I realized that China has leapfrogged the West on every technology that wasn't subject to export controls. And the only ones where they're still behind are the ones under export controls.
The purpose of a system is what it does, not what it doesn't do.
It's like the war on drugs, just one more crackdown will fix everything this time.
Its more like the war on bugs. Just because you can't kill them all doesn't mean you shouldn't spray and lay pest control.
because they are bad
It's insane how many articles have dropped in the past 24 hours that all have a similar tone. At this point it's hard to see this as anything other than a broad coordination from someone who wants to destroy Jensen/Nvidia.
There is a 1T IPO coming up. I'd be surprised if they didn't hand over plenty of bucks to PR companies, grass root campaigns, influencers and journalists.
This is how the news works on a story of major importance. As details of corruption come out, they all get reported.
One bigger aspect of this is that it was all saved from being leaked until right after the election.
Somebody was playing partisan politics with national security.
One key question: did Trump recently sell NVDA? How much and when? He's known of this for more than a year.
if we're going paranoid mode, on 9th of November Trump said that "leader of one large company connected to artificial intelligence" was almost sent to prison by Ministry of Justice due to wrongdoings connected to Ai, and this threat caused immediate correction by that company
I guess fingers crossed & good luck to them? After making consumer hardware unaffordable due to their greed I guess that is a reasonable outcome & karma at work.
the Deep State has entered the chat.
I definitely care about this much less than it seems I'm supposed to.
I think the frontier models are definitely in a healthy dynamic where it seems the near-but-not-quite performance of open-weight models are what is pushing the most innovation and price competition from OpenAI, Anthropic and co
Some people seem to think that American sanctions and export controls are some form of absolute justice, and that any violation is an affront to humanity.
If GPUs are going to be priced out of consumer reach anyway, I would rather see the GPUs used to create open weight models than just to make these AI companies richer.
Well the problem is you can't prove they are only being used to create open weight models. You can't prove they aren't also creating closed weight models with military applications beyond what they've publicly open sourced.
Why should I care that China is making some super powered hacking model? We already know they do state sponsored APTs. It's just motivation to care about security and red team your server code.
Their government is not really the type to violently take over other countries and impose their laws. Taiwan is a special case because they see it as a rebel state on their own land. The rest of us are probably better off with China being the top superpower than Trump's America.
> Their government is not really the type to violently take over other countries and impose their laws.
Taiwan would like a word. Taiwanese govt was the original government in China before CCP violently took over in the aftermath of WW2 with help from the Soviet Union.
China was also involved in destabilizing Vietnam by backing the communists and then after having a fall out with the communists there, proceeded to engage in border conflicts for almost two decades before normalizing relations.
China hasn't lifted a finger for Ukraine. It hasn't lifted a finger for Venezuela. It hasn't lifted a finger for Iran. Nobody knows what China's long term intentions are or how China will change after Xi Jinping's reign.
You don't surrender your culture's lead in tech willingly as it could be existential down the road.
> We recover traces of the models’ initial values, and then search for the hardware configurations and initial configurations that reproduce them
From my (very cursory, naive, layman's) glance at the summary, they're identifying and tracing some characteristics that specific hardware imparts into a model's initialized weights. But then concluding that this indicates the models were also "trained on" that same hardware.
But is that a valid assumption? Is it possible that a model was initialized on one system configuration, while the bulk of the training was performed on another?
In fact, it would seem reasonable for the "okay, I'm setting up this new model on my workstation, prepping it for training next month" pipeline step to be far removed from the "okay, now deploy the [initialized, but not yet trained] model across our giant fleet of beefy training machines".
as the post mentions, the models could have been trained on 1. GPUs within China or 2. GPUs outside China for example datacenters in Singapore.
The 2nd option is perfectly legal and allowed by US government.
The 1st option is also perfectly legal and allowed by the US government. Only the export of GPUs is illegal, and only sometimes.
Is the U.S. government OK with this if the money to buy those GPUs and run inference on them comes from Chinese companies?
yes. The USA hasn't force an embargo on China to other countries (e.g. by threatening to cut off Swift access). We aren't technically at war either, so its highly unlikely that the US would go that far.
What is surprising, I guess, is that the US allows GPUs to be exported at all to most countries. Since...as long as they are exported to a country that is friendly to china (which is basically most countries on the planet), China has access to GPUs on which to train with, making GPU purchasing restrictions mainly symbolic.
They dont need to threaten other countries, they can just pass a law saying that anyone anywhere in the world selling or renting gpus to chinese companies is committing a crime. Then they can sanction any company that does it anyway and that makes it a crime for any american company to sell gpus to that company. There have already been quite a few criminal cases around the export ban.
That would be unenforceable though knowing American arrogance they would definitely try. It’s like China trying to stop their rare earths from going to American uses by requiring all companies that import them to track their usage.
Then china stops selling goods to USA for a couple of weeks until they realise there's more in life than GPUs.
> The newest models we could read from Alibaba (Qwen), Zhipu (GLM), Tencent, Xiaomi and StepFun all read as Hopper-trained. Meituan says its LongCat-2.0 was trained entirely on “AI ASIC superpods,” and we found no NVIDIA fingerprint in the model weights it’s released.
Yeah makes sense. They are in transition. This article doesn't really tell us anything new (or interesting). There's nothing wrong w Chinese labs using GPUs that are already in the country. An export ban just means reduced import of new GPUs
What if the gov tolerated gpu exports on purpose?
Black & grey market devices marked up many times cost buyers heavily, while still suppressing motivation to develop sovereignty.
If nvidia selling to China tolerated for the realpolitik and gets undone by incoming powers to create an appearance of change, is it really the right thing to do? Is it the smart thing to do?
It's an interesting thought experiment, though, we will probably never know.
> What if the gov tolerated gpu exports on purpose?
Given the track record of this administration, they were probably paid to tolerate said exports.
Indeed, Jensen 'gifted' $25m to the ballroom after Trump was briefed on the matter. Many sources, here's one: https://www.newsweek.com/trump-jensen-huang-spy-claim-nvidia...
Why on earth would that be suppressing motivation to develop sovereignty? The whole thing makes developing sovereignty an existential priority. When one critical part of your AI technology stack is subject to whims of your adversary, it's not really fully your AI technology stack. Today they turn the tap partially on, but only unofficially, tomorrow they'll shut it off. Time to buy your own tap.
I don't know the exact reason, but all empirical evidence shows that it does. E.g. the whole of Europe still using Microsoft because they can, even though bits and pieces get cut off (like the ICC). If Microsoft cut off the whole of Europe, then Microsoft wouldn't have power over them any more.
Microsoft's days in many EU countries' government departments are numbered regardless of what Microsoft does. The understanding is there: a critical dependence on a product coming from a now-unreliable ally is a problem. It will, however, take many years for this to actually happen, because it's much easier said than done.
That's what they say. What they actually do betrays otherwise.
It would suppress it because investing in this stuff is a very large short/mid term cost for long term gain.
This, providing access to the gpus but at a large markup plays into that dynamic.
Entities being myopic shouldn't be surprising.
Does the Chinese government strike you as an organization that's strapped for cash or infected by short-term thinking?
Yeah actually, they have engaged in tons of short term thinking to their detriment.
Look at their demographics issue, as well as things like the real estate bubble.
Their economy is below what it was supposed to be by a real margin.
Post-Trained To Reinforce Western Values
Yup, I think the same applies to traffic of arms regulations.. Private firms sell to countries they are 'allowed' to, passing it between organisations in a chain until they get to the original customer they couldn't sell to directly.. Cuts are taken along the way, and people making money look the other way..
When Xi visited, Trump offered to sell him US made weapons.
What's the chance that these GPUs got to China without Trump getting a cut somehow?
Pretty much zero. Trump sat on knowledge of the affair for an entire year. That's extremely guilty behavior, he was either extorting Huang or covering g for him.
The whole thing is the most corrupt thing we have ever seen at the presidential level.
> The whole thing is the most corrupt thing we have ever seen at the presidential level.
Bigger than Watergate by far
... So far
I have run some large open models on neoclouds and the KYC has been minimal, even if the USG stops all GPU exports it will be extremely difficult to crack down on Chinese renting GPUs on Western servers.
There will be a black market of renting IDs to KYC and gain access to the chips, and then the hosts will monitor their servers for what is being trained, it will be a cat and mouse game.
We are headed for an increasingly unfree computation landscape and the politicians are going to make demands that will be hard to deliver.
Matrix multiplication is a human right
Keep your laws off my weights.
Weird detail: some of these results are based on initial patterns in weights surviving through training. But isn't muon, which is used for many of these models, well-known to be able to consistently erase patterns in weights, unlike previous optimizers [1]?
This doesn't really call the overall results of the report in the question (the strongest finding is the perfectly matching weights for dead-weight rows which weren't touched by an optimizer at all), but it does suggest that the explanation in the twitter thread[2] of how this works is way too simplistic. Are the patterns in question somehow "special" and not erasable even with muon?
[1] https://docs.modula.systems/examples/weight-erasure/
[2] https://twitter.com/willdepue/status/2120992457630326841
i dont know what is expected of export controls... they are never effective. they also find western chips in iranian drones etc. there is a huge gray area of resellers who resel to resellers etc. its pretty much uncontrollable....
During the Cold War, America used various shell companies to get valuable titanium out of Russia for building spy planes. If someone is willing to pay, smugglers will find a way.
A lot safer than smuggling since re-exporters generally don't get arrested unless they travel to the US.
Simply banning things causes the price to go up. That alone is good enough to motivate the policy decision.
> Simply banning things causes the price to go up. That alone is good enough to motivate the policy decision.
No twice.
First, banning them doesn't necessarily even make the price go up. If all you have to do to get the product is put someone else's name on the form, that raises the price by a number that rounds to zero. It probably costs your own companies more in bureaucracy and paperwork to try to enforce the pointless rules than it costs the other one to put a different name on the form.
Second, suppose it actually worked somehow. What happens then? The other country pays the higher price temporarily while throwing out some big government subsidies to develop a domestic version of the technology. Soon your own companies have a new foreign competitor in the world market which is receiving large government subsidies.
Export controls are a failed policy.
> That alone is good enough to motivate the policy decision.
This may work when its against an smaller. When deployed agianst a (near) peer - there usually isn't a price to high for a field of national security interest. China could get all the GPUs it wants / the US could get all the rare-earth magnets it wants over the objections of the other country.
It can get all the GPUs it wants... at a higher price.
> It can get all the GPUs it wants... at a higher price.
That's implicit in "there's no price too high". However higher the price is, the difference is still a rounding error when your budget is trillions of dollars.
I present this short-form documentary — or animated infographic, if you will — on how much governments are prudent in saving money in areas considered of vital national interests, denominated in GPUs instead of dollars: https://www.youtube.com/shorts/F6jMsMBVB34
Export controls inherently raise the cost of doing business with parties you don't like, the other party has to either invest more resources acquiring them, or make the decision to forego acquiring them. The grey market exists precisely because there's an opportunity to exploit this for profit. At the end of the day, when these effects of these export controls are discussed, they usually seem to revolve around export controls setting a specific program or effort back by a span of months, not in stopping a specific commodity from accessing another market whatsoever (excepting some hyper-specific components).
> its pretty much uncontrollable
As Hezbollah discovered in 2024, covert supply-chain interdiction is very possible and I'd be surprised if the CIA didn't know what was going on, at least to some extent.
Yes or indeed supplying cards to the Chinese market with extras...
Would be nuts but unsurprising if the cia is selling subtlety broken gpus to china
If you could rent compute to what is essentially a competitor and monitor everything about their code and models in the process, since you're still in control of the hardware, why would a company or in this case a country not take that competitive advantage?
Not doubting that they have access to these.
But, could the fingerprints be due to distillation from Blackwell trained models rather than training?
Weights start as random values generated from a seed. Pytorch uses however many cores are available on the architecture to write out the initial values in parallel, so each seed + architecture pair has a particular signature. If I understand Will's technique correctly he's basically made a rainbow table of initial weight states. Training alters the initial weights, but if you find one seed + architecture pair that vastly more-closely matches the trained weights, you can infer it must have been the source of the initial weights.
Distillation might shape the weights similarly in some way to the source model, but that would come from the signal, not the noise. I think it would be very surprising if the initial untrained weights were encoded with any fidelity into the outputs such that training another randomly-initialized set of weights (likely of completely different size and shape) on them could leave a signature of the source model's initial state, does that make sense?
Not seeing how that could happen since all the "distillation" attacks i've seen are just token level, not logprob level, and definitely not vector level. So, I don't see how that could be the case.
Just further proof that export controls rarely restrict the ones they are designed to restrict
The purpose of a system is what it does, not what it doesn't do.
So the purpose of MS365 was to go down for 12+ hours?
Awesome job, most interesting analysis I've read these days.
This is all so stupid.
US businesses need two things:
1. access to the very best frontier models for hard problems
2. good enough open models from China, the US (I like poolside.ai, but lots of good choices), and even other countries like France that make smaller investments in Digital Sovereignty.
If the ‘super rich fucks’ in the US who have been making risky AI bets succeed in convincing the US government to curtail the use of small efficient models for easier problems, that will just hurt our economy.
@dang curious, is this post getting heavily downvoted or flagged?
I get that the geopolitical angle is sensitive, especially with the backdrop of today's big news - but I thought this was a really well-researched technical deep dive, right up HN's alley. (TIL: you could fingerprint gpu hardware just from an llm weights!)
I need to double check PyTorch code, I am not sure how robust this fingerprinting is without PyTorch "bug". Parallel philox based PRNG is a well-researched area and I am surprised (even if it is not secure-PRNG) it can fingerprinting like this. But obviously there are more people who are more knowledgeable on this can give their perspectives here.
Not too many years before the headline reads
"Western AI models using GPUs export controlled by China"
The decline of the USA and the West has been precipitous.
Ah, that means the recent huge open-source AI advances, that we all benefit from, directly and indirectly, come from Jensen Huang's China boycott circumvention.
Which means, in future, we maybe cannot expect such large capability increases.
Cute trick but it's not exactly a secret, e.g. deepseek recently published their own inference engine -- and it only has kernels for SM100 (blackwell) and Huawei Ascend 950, and all its published performance results using blackwell.
And these were completely openly disclosed as headline properties of the software: https://github.com/deepseek-ai/DeepInfer/tree/main/#performa...
No one found it remarkable a couple weeks ago, ... but now the fact that DS is using blackwell is apparently some super secret requiring clever RNG init stats???
So export controls are more like export friction
Are (ex-)OpenAI/Anthropic employees leading the charge here to protect their portfolios?
Ladies and gentlemen, we are now officially in Huang-gate.
My BS-meter just went through the roof:
"Specially configured AMD hardware, which is also restricted from export to China, is the only alternative we have identified; we have excluded it for DeepSeek-V4-Flash"
This sounds as if they are using something VERY fuzzy to "match" things, and if they are ALSO matching against completely different hardware which this makes it sounds like.... then I dont know if this means anything.
They match the pseudo-random generator from the NVIDIA library cuRAND.
AMD has a library that emulates it, for compatibility.
Their methodology is interesting, but it only proves access to a single Blackwell GPU, which would have been sufficient for initializing an LLM with random values, before the start of its training.
There is no evidence whatsoever about which GPUs were used for training, except the supposition that if they had access to 1 Blackwell, they might have had access to more of them. Such a supposition may be justified about the number of cockroaches in a kitchen, but less so about the components of a training cluster, which might be heterogeneous.
The LLM could have been initialized on the workstation of a developer, before being transferred to a server cluster, for training.
Such a great domain name by the way.
WAIT! Are you saying the giant plagiarism machine was built with…stolen IP?!?!
Title is deceptive; at most they've shown it carries the fingerprint of "a 148-CU GPU".
No, they distinguish AMD vs Nvidia based on rounding or some similar difference farther down in the stack, finding that the values that would differ based on chip vendor are all consistent with an Nvidia platform. Search for "slightly differently".
Land of the free*.
*conditions apply
clickbait written as research....
oh no, anyways
US frontier AI models carry the fingerprint of data acquired from illegal sources like Anna Archive, libgenesis, etc. and, (un?)surprisingly, nobody cares.
Only Anthropic was stupid enough to let be caught red-handed but get away with some ridiculously low penalty.
Well, d'uh...
I guarantee you this isn't an "accident". Whatever is being sent over there is loaded with its own special spyware lol