Tabular Foundation Models (TFMs) logopeterindia.net
Knowledge Hub

Multimodal Embedding Models

Multimodal embeddings for text, images, video, audio and documents: hosted APIs, open-weight and visual document retrieval models.

34 topics•7 categories•Updated October 2026

What Are Multimodal Embedding Models?

5 entries

Multimodal embedding models map text and other content, such as images, screenshots, PDF pages, video and audio, into one shared vector space so that a text query can retrieve an image or a page, and the reverse. They are used for visual document retrieval, image and video search, and multimodal RAG. Details below come from each provider's documentation, model card or repository; descriptions are not rankings, so test on your own content.

01Embedding ModelsText and multimodal embedding APIs and open models, on this site. 02Reranker ModelsRerankers for retrieval precision, on this site. 03Multimodal Language ModelsMultimodal LLMs, on this site. 04Vision Language ModelsVLMs that read images and documents, on this site. 05Vector DatabasesStores for embeddings, on this site.

Hosted Multimodal Embedding APIs

7 entries

Managed models, described from each provider's own documentation.

06Google Gemini Embedding 2Google documents it as the first multimodal embedding model in the Gemini API, mapping text, images, video, audio and documents into one space with 8,192 input tokens, up to 6 images, 120 seconds of video and a PDF of up to 6 pages per request, and 128 to 3072 dimensions. 07Cohere Embed 5Cohere lists embed-v5.0-pro and embed-v5.0-fast for text, images and mixed inputs such as PDFs, with 128k-token context and 256 to 2048 dimensions; embed-v4.0 is the earlier model. 08Voyage multimodal embeddingsVoyage lists voyage-multimodal-3.5 for interleaved text and visual data, with 32,000-token context and 256 to 2048 dimensions. Video input is supported only by the 3.5 model, and voyage-multimodal-3 is the earlier model. 09Amazon Nova Multimodal EmbeddingsAWS describes a single model that embeds text, documents, images, video and audio into a unified space, with 8K tokens or 30 seconds of video or audio per input, dimensions of 3072, 1024, 384 or 256, and asynchronous segmentation of long files. 10Amazon Titan Multimodal EmbeddingsAWS documents Titan multimodal embedding models on Amazon Bedrock for text and image search. 11TwelveLabs MarengoTwelveLabs describes Marengo as an embedding model that analyzes visual, audio and text information; version 3.5 is described as covering video, audio, images, text and documents. 12Jina Embeddings APIJina describes multimodal, multilingual, long-context embeddings for search, RAG and agent applications through an API.

Open-Weight Multimodal Embedding Models

7 entries

Models whose weights can be downloaded. Licences are from each Hugging Face model card; read the licence before commercial use.

13Qwen3-VL-Embedding-8BAlibaba Qwen model for text, images, screenshots and video with 32k context and output sizes from 64 to 4096 (Apache-2.0); a 2B size is also available. 14EmbeddingGemma 2Google DeepMind model that embeds text, images, video and audio into one 768-dimension space, with Matryoshka truncation down to 128 (Apache-2.0). 15SigLIP 2Google image-text model family with a sigmoid loss; the so400m-patch14-384 checkpoint has about 1.1B parameters (Apache-2.0). 16OpenCLIPOpen-source implementation of CLIP for training and running contrastive image-text models. 17CLIP (OpenAI)OpenAI repository for Contrastive Language-Image Pretraining, which predicts the most relevant text snippet for an image. 18LAION CLAPContrastive Language-Audio Pretraining for audio-text embeddings. 19ImageBindMeta research project that learns one embedding space across several modalities.

Visual Document Retrieval Models

5 entries

Models that embed page images directly, avoiding separate OCR and layout steps.

20ColPali (colpali-engine)Code for ColPali, ColQwen2 and ColSmol late-interaction visual document retrievers. The repository says colpali-engine is deprecated and recommends Sentence Transformers v6 for new projects. 21ColQwen2.5Visual document retrieval model from the vidore organization; its model card lists the MIT licence, and the base model has its own terms. 22ColNomic Embed Multimodal 7BNomic visual document retrieval model (Apache-2.0). 23ViDoRe LeaderboardHugging Face space for exploring the ViDoRe visual document retrieval benchmarks by metric and model. 24ColPali paperPaper describing efficient document retrieval with vision language models.

Models With Restricted Licences

3 entries

Open-weight models whose licences limit commercial use or use a custom licence. Check the terms before building a product on them.

25Jina Embeddings v4About 4B parameters for text, images and visual documents with single-vector (2048, truncatable) and multi-vector (128) outputs. The model card says the correct licence is the Qwen Research License, not CC BY-NC 4.0 as first listed. 26Jina CLIP v2About 0.9B-parameter multilingual image-text model under CC BY-NC 4.0. 27NVIDIA Llama NemoRetriever ColEmbed 3BVisual document retrieval model with about 4.4B parameters listed under a custom licence; read the licence file.

Research Papers

2 entries

Foundational papers for multimodal embeddings.

28CLIP paperLearning Transferable Visual Models From Natural Language Supervision. 29SigLIP paperSigmoid Loss for Language Image Pre-Training.

Frequently Asked Questions

What is a multimodal embedding model?

A model that converts text and other content, such as images, screenshots, PDF pages, video or audio, into vectors in one shared space. A text query can then retrieve images or pages, and an image can retrieve related text.

When do I need multimodal embeddings instead of text embeddings?

When the answer lives in images, charts, slides, scanned pages, video or audio and extracting text would lose information. For plain text corpora, a text embedding model is usually enough and often cheaper.

How is visual document retrieval different from OCR plus text embeddings?

Models such as ColPali embed page images directly, so layout, tables and figures are kept. The ColPali repository describes this as removing the need for complex OCR and layout-recognition pipelines. An OCR pipeline can still help when exact text is needed; see the OCR Models page.

What are single-vector and multi-vector (late interaction) embeddings?

Single-vector models produce one vector per input, which suits standard vector databases. Late-interaction models such as ColPali produce many vectors per page and score token-level matches, which can be more precise but needs more storage and a database that supports multi-vector search. Jina Embeddings v4 offers both.

Can I mix embeddings from different multimodal models?

No. Each model has its own embedding space, so changing models means re-embedding the content; Google notes this for the move from gemini-embedding-001 to Gemini Embedding 2. Some families, such as Cohere Embed 5, document a shared space across their own variants.

Are open-weight multimodal embedding models free for commercial use?

Not always. Qwen3-VL-Embedding, EmbeddingGemma 2, SigLIP 2 and ColNomic list Apache-2.0, while Jina CLIP v2 lists CC BY-NC 4.0 and Jina v4 and NVIDIA ColEmbed use other licences. Read each model card.

How current is this page?

Links and model details were checked on 31 October 2026 against provider documentation, Hugging Face model cards and repositories. Provider descriptions follow their own wording and are not rankings, and benchmark scores are not reported because they change.