With media ingestion this is called "eager" processing. Historically for things like pulling thumbnails for images / video and pre-generating common sizes for things. This follows the same pattern and makes all the sense in the world. My only concern is that due to the non deterministic nature of LLMs new models will reveal new information about your data.
For example you might identify a car in an image but the context is the car running a red light. A new model might pick that up while an old one doesn't. These context adjustments might sometimes require you to rerun your LLM processing or potentially have a one to many relationship for multiple runs so you can take the best of or combine results.
Actual usage will also reveal most commonly used assets and you can target the ones that are most trafficked and save a ton on processing that way.
> we don't send images to the model at query time. We describe each image once, at indexing time, with a cheap vision model, store the descriptions as text, and retrieve them alongside ordinary text chunks
This is what I've been doing in my Obsidian infodump for a while. If I know that an image is important, I generate a text description (Mermaid if possible, English if not) and paste it after the image in a block. This lets agents see the image if they don't really see it. Though my process is manual, the improvements in outcomes for agents that rely on text search/retrieval is very real and is worth it.
Well I don't know if this one has been getting by others too but I have been doing this since 2 years ago and it works really well. Except the fact that for the documents I had to chunk containing these images I had to chase the authors(multiple of them) to update the relevant captions for their images.
It is cost efficient than multi-modal. Lesser ingestion time altogether. Only part is that if the retrieval query is a question which can be answered only after looking at the image, then this architecture would need some little modification.
With media ingestion this is called "eager" processing. Historically for things like pulling thumbnails for images / video and pre-generating common sizes for things. This follows the same pattern and makes all the sense in the world. My only concern is that due to the non deterministic nature of LLMs new models will reveal new information about your data.
For example you might identify a car in an image but the context is the car running a red light. A new model might pick that up while an old one doesn't. These context adjustments might sometimes require you to rerun your LLM processing or potentially have a one to many relationship for multiple runs so you can take the best of or combine results.
Actual usage will also reveal most commonly used assets and you can target the ones that are most trafficked and save a ton on processing that way.
> we don't send images to the model at query time. We describe each image once, at indexing time, with a cheap vision model, store the descriptions as text, and retrieve them alongside ordinary text chunks
This is what I've been doing in my Obsidian infodump for a while. If I know that an image is important, I generate a text description (Mermaid if possible, English if not) and paste it after the image in a block. This lets agents see the image if they don't really see it. Though my process is manual, the improvements in outcomes for agents that rely on text search/retrieval is very real and is worth it.
What does Mermaid text description of an image mean?
Descriptions of images that are charts or diagrams to start with?
Well I don't know if this one has been getting by others too but I have been doing this since 2 years ago and it works really well. Except the fact that for the documents I had to chunk containing these images I had to chase the authors(multiple of them) to update the relevant captions for their images. It is cost efficient than multi-modal. Lesser ingestion time altogether. Only part is that if the retrieval query is a question which can be answered only after looking at the image, then this architecture would need some little modification.
That's smart. Just the other day, I was thinking about how I would solve images/graphs/rich PDF stuff in a RAG system. Now I know more, thanks!
How descripting is the caption that you obtain?
So you include colour, shapes, etc?
why not a multimodal embedding model?
Article says this misses important details, eg data that might be in the image.
very bad take. with most modern multomodal models you get way better performance then going to text first
视觉分块思路可以。但图文混合的图怎么办？CLIP认风格，不认结构意图。
We have it in our open-source framework, in case anyone wants to deploy it:
https://github.com/Qbix/AI/blob/6753f6e453908682401f49760002...
https://github.com/Qbix/AI/blob/main/config/observations.jso...
wrote it up here a few months ago: https://community.safebots.ai/t/building-cultural-infrastruc...
That cookie popup just makes me wanna leave and never come back
I think they've fixed it now.
Thanks! Yep fixed