So after transforming multispectral satellite data into a 128-dimensional embedding vector you can play "Where's Wally" to pinpoint blackberry bushes? I hope they tasted good! I'm guessing you can pretty much pinpoint any other kind of thing as well then?
Yes it's very good fun just exploring the embeddings! It's all wrapped by the geotessera Python library, so with uv and gdal installed just try this for your favourite region to get a false-colour map of the 128-dimensional embeddings:
# for cambridge
# https://github.com/ucam-eo/geotessera/blob/main/example/CB.geojson
curl -OL https://raw.githubusercontent.com/ucam-eo/geotessera/refs/heads/main/example/CB.geojson
# download the embeddings as geotiffs
uvx geotessera download --region-file CB.geojson -o cb2
# do a false colour PCA down to 3 dimensions from 128
uvx geotessera visualize cb2 cb2.tif
# project onto webmercator and visualise using leafletjs over openstreetmap
uvx geotessera webmap cb2.tif --output cb2-map --serve
Downstream classifiers are really fast to train (seconds for small regions). You can try out a notebook in VSCode to mess around with it graphically using https://github.com/ucam-eo/tessera-interactive-map
The berries were a bit sour, summer is sadly over here!
This is all far outside of my wheel house but I'm curious if there's any way to use this for rocks and geology? Identifying dikes and veins on cliff sides from satellites would be really cool.
A major limitation is that most different rock types look essentially identical in visual+NIR spectral ranges. Things separate once you get out to SWIR bands. Sentinel2 does have some SWIR bands and it may work reasonably well with embeddings. But a lot of the signal the embeddings are going to be focused on encoding may not be the right features to distinguish rock types. Methods more focused specifically on the SWIR range are more likely to work reliably. E.g. simple band ratios of SWIR bands may give a cleaner signal than general purpose embeddings in this case.
Hyperspectral in the SWIR range is what you really want for this, but that's a whole different ball game.
Usually airplanes because the instruments are heavy. But yeah, that's the most common case. Hyperspectral sats are much rarer than aerial hyperspectral.
An interesting 30x30m satellite that recently launched and is giving back data last year is EnMAP https://www.enmap.org. Hooking that up to TESSERA is on our todo list as soon as we can get our mittens on the satellite data
It might work. TESSERA's embeddings are at a 10 metre resolution, so it might depend on the size of the features you are looking for. If those features have distinct changes in colour or texture over time or they scatter radar in different ways compared with their surroundings then you should be able to discriminate them.
The easiest way to test is to try out the interactive notebook and drop some labels in known areas.
Is there a way to cluster the embeddings spatially or look for patterns isolated so some dimensions? (Again, way out of my wheel house)
What I mean is a vein is usually a few meters wide but can be hundreds of meters long so ten meter resolution is probably not very helpful unless the embeddings can encode some sort of pattern that stretches across many cells.
It's possible to use embeddings as input to a convolutional network and then train that using labels. We've done that for at least one of the downstream tasks in the TESSERA paper: https://arxiv.org/abs/2506.20380 to estimate canopy height.
The downside of that approach is that you need to spend valuable labels on learning the spatial feature extraction during training. To fix that we're working on building some pre-trained spatial feature extractors that you should only need to minimally fine-tune.
I haven’t done this kind of thing since undergrad, but hyperspectral data is really frickin cool this way. Not only can you use spectral signatures to identify specific things, but also figure out what those things are made out of by unmixing the spectra.
For example, figure out what crop someone’s growing and decide how healthy it is. With sufficient temporal resolution, you can understand when things are planted and how well they’re growing, how weedy or infiltrated they are by pest plants, how long the soil remains wet or if rainwater runs off and leaves the crop dry earlier than desired. Etc.
If you’re a good guy, you’d leverage this data to empower farmers. If you’re an asshole, you’re looking to see who has planted your crop illegally, or who is breaking your insurance fine print, etc.
Hyperspectral data is really neat though it's worth pointing out that TESSERA is only trained on multispectral (optical + SAR) data.
You are very right on the temporal aspect though, that's what makes the representation so powerful. Crops grow and change colour or scatter patterns in distinct ways.
It's worth pointing out the model and training code is under an Apache2 license and the global embeddings are under a CC-BY-A. We have a python library that makes working with them pretty easy: https://github.com/ucam-eo/geotessera
> If you’re a good guy, you’d leverage this data to empower farmers. If you’re an asshole, you’re looking to see who has planted your crop illegally, or who is breaking your insurance fine print, etc.
How does using it to speculate on crop futures rank?
Every time someone explains the way short selling or speculative markets work, I have a “oh, I get it…” moment and then forget months later.
Same with insurance… socialized risk for our food supply is objectively good, and protecting the insurance mechanism from fraud is good. People can always bastardize these things.
It is complex. I was going to write out how it works in a simple way that everyone could understand - but then I realized that even though it would be a gross simplifications that are unrealistic, it still would be so complex that people would go "yep I understand that to every step", and then finish and not understand it. Every step alone makes perfects sense and is simple, but the total quickly gets complex.
Even calling this a speculative market is a gross simplification of the truth.
There are two sides hedging against bad harvests, the farmer that grows the crop, and the industry (cattle, ethanol, food oils, and others) that buys that crop. The farmer wants to get paid, and the industry wants to get their crop.
That's should be a pretty good usecase; if you do just a few labels manually of known hotsprings you should be able to find others quite quickly using the TESSERA interactive notebook. The embeddings capture the annual spectral-temporal signature, so a hotspring should be fairly distinctive vs the surroundings.
Well looks like they found a lot of brambles! Were there large areas without any bramble?
Cue dowsers, who successfully find water... but also who would anyway anywhere else because underground water isn't the underground river/pocket that people imagine and thus random chance by itself has high probability of finding water.
We did note several places during the trip that didn't contain bramble. The hotspot in the middle of the residential area was also entirely isolated.
For a proper evaluation you would need to be more methodological but as a sanity-check we were very happy with it.
One other thing to point out about the bramble model is that it is pixel-wise. That is each prediction is exclusively only what is within the 10 metre pixel (give or take the georeferencing error).
FarmLogs (YC 12) did exactly this. We used sat imagery in the near-infrared spectrum to determine crop health remotely. Modern farming utilizes a practice called precision ag - where your machine essentially has a map of zones on the field for where treatments are or aren't needed and controllers that can turn spray nozzles on/off depending on boundaries. We used sat imagery as the base for an automated prescription system, too. So a farmer can reduce waste by only applying fertilizer or herbicide in specific areas that need it.
> Can a model trained on satellite data really find brambles on the ground?
No, as per researcher, "However, it is obvious that most of the generated findings aren’t brambles" and obviously no.
All the model did was think they followed roads, all roads.
If it was oil and gas where people put in effort and their results where checked vs universities where meaningless citations matter and results are never confirmed, it would be more believable.
What they are asking is impossible, increasing the likelihood without silly hacks like it's not in rivers or on top of buildings is an interesting problem but out of scope for academics.
I was a lot more optimistic about Gabriel's model than he was. It is essentially a presence-only species distribution model where accuracy depends largely on assumptions around prevalence and which really needs some presence-absence data to calibrate.
As I mentioned in one of the other comments, the model is also only pixel-wise. That is, it is not using spatial information for predictions.
Not much detail on the method? Like what data it takes from iNaturalist - for example if it's taking in GPS coordinates of observations of brambles then it's not clear what there is for the ML model to do.
What detail was in the satellite images, was it taking signals of the type of spaces brambles are in, or was it just visually identifying bramble patches?
In the UK you get brambles in pretty much every non-cultivated green space. I wonder how well the classifier did?
Hi! You can find a bit more about Gabriel's model through some of his posts over the last few weeks: https://gabrielmahler.org/posts/
When it comes to the satellite images, the model actually used TESSERA (https://arxiv.org/abs/2506.20380) which is a model we trained to produce embeddings for every point on earth that encodes the temporal-spectral properties over a year.
Think of it like a compression of potentially fifty or a hundred observations of a particular point in earth down to a single 128 dimension vector.
We have a problem with Giant Hogweed and I was thinking about ways to identify hotspots. My guess is that standard satellite imagery, like Google Maps, probably isn’t good enough. To even check if this could work, you’d need high-res imagery (sub-meter), ideally multispectral, and some way to validate it on the ground. What steps should I take to verify if this is possible in a way this was done here?
https://github.com/ucam-eo/geotessera has an image showing our embedding coverage at the moment. Blue areas we have complete coverage for 2024, green areas we cover 2017-2024. We're slowly trying to populate everything 2017-2024 but the constraint is GPU and storage at the moment - each year takes ~20k GPU/200k CPU hours and requires storing and serving 200 terabytes of data. The world is big!
If there is an area you would like prioritised, there's an issue template on the geotessera github repo which we can use to move regions around in the processing queue.
Thanks for your explanation. For my region, 2024 coverage is already available, which should be sufficient to get started. After looking into the library, I just want to make sure I understand the workflow correctly: I would use the Tessera interactive map to mark known locations of Giant Hogweed, label them, and export as GeoJSON; then train a k-NN model, make predictions, and finally export the results as a GeoJSON back to the map. Does that sound right?
...and if you do build this @ensocode, feel free to open a PR to https://github.com/ucam-eo/geotessera and I'll incorporate it as an example in the repo.
That's actually a great idea! I wonder what kind of feature size would be needed though - TESSERA's embeddings are at a 10 metre resolution so for larger structures you might need some kind of spatial aggregation.
A model I have trained on ASTER and LANDSAT data has major difficulties identifying spots for agate hunting. Even after I've given it extra instruction such as looking only in volcanic terrain (with USGS map provided,) or focusing on mixed signals of hydrous silica and iron, checking near known fault zones in said volcanic areas, it still gave me results everywhere, and almost none matching my criteria.
Plants are a way different and more difficult ballgame (they like to mess up my satellite data) so as I read I am not surprised to see that this didn't really give proper results.
I read this and questioned the statistical methods 101. To say it works, one would also need to check for false positives. And such a check would pick up on "oh it's finding roads and there's a correlation between road and brambles."
Are you thinking of _new_ fresh water sources that emerge in recent years? If you have any candidate lat/lon where this might have happened, we can take a look at the 2024 and earlier embeddings to see if we can spot it.
There is the issue of just how visible truffles are from space though, if they grow under cover. That said, it may still work because you can find habitats that are very likely to have truffles. We've had some promising results looking at fungal biomass.
So after transforming multispectral satellite data into a 128-dimensional embedding vector you can play "Where's Wally" to pinpoint blackberry bushes? I hope they tasted good! I'm guessing you can pretty much pinpoint any other kind of thing as well then?
Yes it's very good fun just exploring the embeddings! It's all wrapped by the geotessera Python library, so with uv and gdal installed just try this for your favourite region to get a false-colour map of the 128-dimensional embeddings:
Because the embeddings are precomputed, the library just has to download the tiles from our server. More at: https://anil.recoil.org/notes/geotessera-pythonDownstream classifiers are really fast to train (seconds for small regions). You can try out a notebook in VSCode to mess around with it graphically using https://github.com/ucam-eo/tessera-interactive-map
The berries were a bit sour, summer is sadly over here!
This is all far outside of my wheel house but I'm curious if there's any way to use this for rocks and geology? Identifying dikes and veins on cliff sides from satellites would be really cool.
A major limitation is that most different rock types look essentially identical in visual+NIR spectral ranges. Things separate once you get out to SWIR bands. Sentinel2 does have some SWIR bands and it may work reasonably well with embeddings. But a lot of the signal the embeddings are going to be focused on encoding may not be the right features to distinguish rock types. Methods more focused specifically on the SWIR range are more likely to work reliably. E.g. simple band ratios of SWIR bands may give a cleaner signal than general purpose embeddings in this case.
Hyperspectral in the SWIR range is what you really want for this, but that's a whole different ball game.
> Hyperspectral in the SWIR range is what you really want for this, but that's a whole different ball game.
Are there any hyperspectral surveys with UAVs etc instead of satellites?
Usually airplanes because the instruments are heavy. But yeah, that's the most common case. Hyperspectral sats are much rarer than aerial hyperspectral.
An interesting 30x30m satellite that recently launched and is giving back data last year is EnMAP https://www.enmap.org. Hooking that up to TESSERA is on our todo list as soon as we can get our mittens on the satellite data
It might work. TESSERA's embeddings are at a 10 metre resolution, so it might depend on the size of the features you are looking for. If those features have distinct changes in colour or texture over time or they scatter radar in different ways compared with their surroundings then you should be able to discriminate them.
The easiest way to test is to try out the interactive notebook and drop some labels in known areas.
Is there a way to cluster the embeddings spatially or look for patterns isolated so some dimensions? (Again, way out of my wheel house)
What I mean is a vein is usually a few meters wide but can be hundreds of meters long so ten meter resolution is probably not very helpful unless the embeddings can encode some sort of pattern that stretches across many cells.
It's possible to use embeddings as input to a convolutional network and then train that using labels. We've done that for at least one of the downstream tasks in the TESSERA paper: https://arxiv.org/abs/2506.20380 to estimate canopy height.
The downside of that approach is that you need to spend valuable labels on learning the spatial feature extraction during training. To fix that we're working on building some pre-trained spatial feature extractors that you should only need to minimally fine-tune.
almost definitely!
I haven’t done this kind of thing since undergrad, but hyperspectral data is really frickin cool this way. Not only can you use spectral signatures to identify specific things, but also figure out what those things are made out of by unmixing the spectra.
For example, figure out what crop someone’s growing and decide how healthy it is. With sufficient temporal resolution, you can understand when things are planted and how well they’re growing, how weedy or infiltrated they are by pest plants, how long the soil remains wet or if rainwater runs off and leaves the crop dry earlier than desired. Etc.
If you’re a good guy, you’d leverage this data to empower farmers. If you’re an asshole, you’re looking to see who has planted your crop illegally, or who is breaking your insurance fine print, etc.
Hyperspectral data is really neat though it's worth pointing out that TESSERA is only trained on multispectral (optical + SAR) data.
You are very right on the temporal aspect though, that's what makes the representation so powerful. Crops grow and change colour or scatter patterns in distinct ways.
It's worth pointing out the model and training code is under an Apache2 license and the global embeddings are under a CC-BY-A. We have a python library that makes working with them pretty easy: https://github.com/ucam-eo/geotessera
> If you’re a good guy, you’d leverage this data to empower farmers. If you’re an asshole, you’re looking to see who has planted your crop illegally, or who is breaking your insurance fine print, etc.
How does using it to speculate on crop futures rank?
Every time someone explains the way short selling or speculative markets work, I have a “oh, I get it…” moment and then forget months later.
Same with insurance… socialized risk for our food supply is objectively good, and protecting the insurance mechanism from fraud is good. People can always bastardize these things.
It is complex. I was going to write out how it works in a simple way that everyone could understand - but then I realized that even though it would be a gross simplifications that are unrealistic, it still would be so complex that people would go "yep I understand that to every step", and then finish and not understand it. Every step alone makes perfects sense and is simple, but the total quickly gets complex.
Even calling this a speculative market is a gross simplification of the truth.
It is good to enable people to hedge against bad harvests.
There are two sides hedging against bad harvests, the farmer that grows the crop, and the industry (cattle, ethanol, food oils, and others) that buys that crop. The farmer wants to get paid, and the industry wants to get their crop.
Yes! TESSERA is very new so we're still exploring how well it works for various things.
We're hoping to try it with a few different things for our next field trip, maybe some that are much harder to find than brambles.
I've wondered this about finding hot springs.
That's should be a pretty good usecase; if you do just a few labels manually of known hotsprings you should be able to find others quite quickly using the TESSERA interactive notebook. The embeddings capture the annual spectral-temporal signature, so a hotspring should be fairly distinctive vs the surroundings.
Video of the notebook in action https://crank.recoil.org/w/mDzPQ8vW7mkLjdmWsW8vpQ and the source https://github.com/ucam-eo/tessera-interactive-map
Well looks like they found a lot of brambles! Were there large areas without any bramble?
Cue dowsers, who successfully find water... but also who would anyway anywhere else because underground water isn't the underground river/pocket that people imagine and thus random chance by itself has high probability of finding water.
We did note several places during the trip that didn't contain bramble. The hotspot in the middle of the residential area was also entirely isolated.
For a proper evaluation you would need to be more methodological but as a sanity-check we were very happy with it.
One other thing to point out about the bramble model is that it is pixel-wise. That is each prediction is exclusively only what is within the 10 metre pixel (give or take the georeferencing error).
FarmLogs (YC 12) did exactly this. We used sat imagery in the near-infrared spectrum to determine crop health remotely. Modern farming utilizes a practice called precision ag - where your machine essentially has a map of zones on the field for where treatments are or aren't needed and controllers that can turn spray nozzles on/off depending on boundaries. We used sat imagery as the base for an automated prescription system, too. So a farmer can reduce waste by only applying fertilizer or herbicide in specific areas that need it.
> Can a model trained on satellite data really find brambles on the ground?
No, as per researcher, "However, it is obvious that most of the generated findings aren’t brambles" and obviously no.
All the model did was think they followed roads, all roads.
If it was oil and gas where people put in effort and their results where checked vs universities where meaningless citations matter and results are never confirmed, it would be more believable.
What they are asking is impossible, increasing the likelihood without silly hacks like it's not in rivers or on top of buildings is an interesting problem but out of scope for academics.
https://gabrielmahler.org/environment/ai/ml/%F0%9F%A6%94/202...
For the "However, it is obvious that most of the generated findings aren’t brambles"
I was a lot more optimistic about Gabriel's model than he was. It is essentially a presence-only species distribution model where accuracy depends largely on assumptions around prevalence and which really needs some presence-absence data to calibrate.
As I mentioned in one of the other comments, the model is also only pixel-wise. That is, it is not using spatial information for predictions.
isn't this the same findings as the old "we trained to identify huskies, but instead we identified snow" problem?
Not much detail on the method? Like what data it takes from iNaturalist - for example if it's taking in GPS coordinates of observations of brambles then it's not clear what there is for the ML model to do.
What detail was in the satellite images, was it taking signals of the type of spaces brambles are in, or was it just visually identifying bramble patches?
In the UK you get brambles in pretty much every non-cultivated green space. I wonder how well the classifier did?
Interesting project.
Hi! You can find a bit more about Gabriel's model through some of his posts over the last few weeks: https://gabrielmahler.org/posts/
When it comes to the satellite images, the model actually used TESSERA (https://arxiv.org/abs/2506.20380) which is a model we trained to produce embeddings for every point on earth that encodes the temporal-spectral properties over a year.
Think of it like a compression of potentially fifty or a hundred observations of a particular point in earth down to a single 128 dimension vector.
Happy to answer any other questions.
We have a problem with Giant Hogweed and I was thinking about ways to identify hotspots. My guess is that standard satellite imagery, like Google Maps, probably isn’t good enough. To even check if this could work, you’d need high-res imagery (sub-meter), ideally multispectral, and some way to validate it on the ground. What steps should I take to verify if this is possible in a way this was done here?
I would try https://github.com/ucam-eo/tessera-interactive-map , this is relatively easy to get started with and has a nice interface for labeling.
https://github.com/ucam-eo/geotessera has an image showing our embedding coverage at the moment. Blue areas we have complete coverage for 2024, green areas we cover 2017-2024. We're slowly trying to populate everything 2017-2024 but the constraint is GPU and storage at the moment - each year takes ~20k GPU/200k CPU hours and requires storing and serving 200 terabytes of data. The world is big!
If there is an area you would like prioritised, there's an issue template on the geotessera github repo which we can use to move regions around in the processing queue.
Thanks for your explanation. For my region, 2024 coverage is already available, which should be sufficient to get started. After looking into the library, I just want to make sure I understand the workflow correctly: I would use the Tessera interactive map to mark known locations of Giant Hogweed, label them, and export as GeoJSON; then train a k-NN model, make predictions, and finally export the results as a GeoJSON back to the map. Does that sound right?
So the interactive map should do this workflow for you. You place points and it will run the knn classifier over the landscape for you.
If you want to go further you can export the GeoJSON and then run it through any machine learning pipeline you like.
...and if you do build this @ensocode, feel free to open a PR to https://github.com/ucam-eo/geotessera and I'll incorporate it as an example in the repo.
The in-person verification of hotspots was good, but in-person verification of non-hotspots was not done, and might be difficult.
Seems like it could be pretty useful for archaeology as well.
That's actually a great idea! I wonder what kind of feature size would be needed though - TESSERA's embeddings are at a 10 metre resolution so for larger structures you might need some kind of spatial aggregation.
As a hobby project, I was looking into using LiDAR data to view archeological points of interest in Switzerland: https://github.com/r-follador/delta-relief
It would be interesting to overlay TESSERA data there, although the resolution is of course very different.
Related, I think? Satellite + AI = finding things, not sure if similar beyond that
https://www.pnas.org/doi/10.1073/pnas.2407652121
A model I have trained on ASTER and LANDSAT data has major difficulties identifying spots for agate hunting. Even after I've given it extra instruction such as looking only in volcanic terrain (with USGS map provided,) or focusing on mixed signals of hydrous silica and iron, checking near known fault zones in said volcanic areas, it still gave me results everywhere, and almost none matching my criteria.
Plants are a way different and more difficult ballgame (they like to mess up my satellite data) so as I read I am not surprised to see that this didn't really give proper results.
If it can find sloes it's going to make sloe gin foragers very very angry. Generally when they find a usable crop they don't share it.
Brambles are blackberries. Sloes are from Blackthorn bushes. They are different plants but probably are in the same location!
That location being the bottom of my garden.
Teacher says "I hope you brought enough for everyone"
I read this and questioned the statistical methods 101. To say it works, one would also need to check for false positives. And such a check would pick up on "oh it's finding roads and there's a correlation between road and brambles."
> So it turns out that there's a lot of bramble between the community center and entrance to Milton Country Park.
> In every place we checked, we found pretty significant amounts of bramble.
[Shocked Pikachu face]
> Stopping to take a photo of a very photogenic bee
Show us the bee!
https://www.inaturalist.org/observations/316277035, needs some id help :-)
The whole-earth embeddings are interesting. Wonder if it'd be any good for looking for fresh water sources in the desert.
Are you thinking of _new_ fresh water sources that emerge in recent years? If you have any candidate lat/lon where this might have happened, we can take a look at the 2024 and earlier embeddings to see if we can spot it.
can it find me truffles?
If you have some GPS locations of truffles, you could use the notebook Anil mentioned here https://news.ycombinator.com/item?id=45378855 and give it a go.
There is the issue of just how visible truffles are from space though, if they grow under cover. That said, it may still work because you can find habitats that are very likely to have truffles. We've had some promising results looking at fungal biomass.
[dead]