When Your Search Needs to Go Beyond Text: BGE-VL and the Era of Visual Embeddings

You've built a RAG pipeline that works great for documents and text queries. But what happens when your users want to search for images using text descriptions, or find text passages that match a specific photograph? The standard embedding models break down completely here, because they live in a text-only world. BGE (BAAI General Embedding) has been a go-to toolkit for text retrieval and RAG for a while, and now with the introduction of BGE-VL, they've stepped into the multimodal space—handling text-to-image, image-to-text, and even combined image-and-text queries without requiring separate models or complex orchestration.

What It Does

BGE is a one-stop retrieval toolkit built for search and RAG (Retrieval-Augmented Generation) applications. It's a collection of embedding models, tools, and research projects maintained by BAAI (Beijing Academy of Artificial Intelligence), available under the MIT license. The flagship release as of March 2025 is BGE-VL, a multimodal embedding model that handles visual search combinations of all kinds—text-to-image, image-to-text, image-and-prompt-to-image, text-to-image-and-text, and more.

The project lives at github.com/FlagOpen/FlagEmbedding and includes a model zoo hosted on Hugging Face, with the BGE series models collected under a dedicated Hugging Face collection. The toolkit also includes supporting infrastructure like MegaPairs, a massive synthetic dataset that was used to train BGE-VL, and a growing set of tutorials and documentation at bge-model.com.

Beyond multimodal capabilities, the project has released other specialized embedding models: bge-en-icl which incorporates in-context learning by accepting task-relevant query-response examples, and bge-multilingual-gemma2 based on Google's Gemma-2-9b model for multilingual retrieval tasks. There's also MemoRAG, a research project pushing toward what they call "RAG 2.0" with memory-inspired knowledge discovery.

Why It's Cool

The obvious headline grabber is BGE-VL's support for "any visual search combination." But let's talk about what that actually means in practice.

• One model, many query types. Most multimodal systems force you to decide upfront: are you doing text-to-image retrieval or image-to-text? BGE-VL handles both, plus hybrids like "find images similar to this one but with a specific color palette" (image-and-prompt-to-image) or "find text passages that describe this image" (image-to-text). You don't need to chain together multiple models or write glue code.

• MIT license, no restrictions. Both academic and commercial use are explicitly permitted. That's increasingly rare in the embedding model space, where many strong models are locked behind research-only or restricted licenses. If you're building a product, this matters.

• The MegaPairs dataset is open too. The synthetic dataset used to train BGE-VL is released alongside the model, which is a thoughtful move. It means you can understand what the model was trained on, and potentially use that dataset for your own fine-tuning or evaluation work.

• Active community and documentation. They've built a dedicated documentation site (bge-model.com), maintain a WeChat group for community discussion, and have started a tutorials section that they promise to actively update. The project also tracks leaderboard performance through C-MTEB on Hugging Face, so you can compare models objectively.

• Specialized variants for specific needs. If you don't need multimodal, the bge-en-icl model offers in-context learning for text embeddings—you provide query-response examples to guide the embedding toward your specific task. And the multilingual Gemma2-based model covers languages that many other embedding models neglect.

How to Try It

The fastest way to get started is through Hugging Face. The BGE series models are collected at huggingface.co/collections/BAAI/bge-66797a74476eb1f085c7446d. You can load any model using the transformers library:

from transformers import AutoModel, AutoTokenizer

model_name = "BAAI/bge-vl-base"  # or any other BGE variant
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

For local installation, clone the repository:

git clone https://github.com/FlagOpen/FlagEmbedding.git
cd FlagEmbedding
pip install -e .

Then check the quick start section in the README for specific usage examples. The repository also contains research projects under the research/ directory with their own documentation and code.

If you want to dive deeper, the dedicated documentation site at bge-model.com centralizes model cards, tutorials, and best practices. And if you're interested in the training data behind BGE-VL, the MegaPairs dataset and paper are linked from the README.

Final Thoughts

BGE-VL fills a real gap in the embedding model landscape. Most retrieval pipelines today are text-only, and the few multimodal options that exist are either closed-source or licensed restrictively. BGE gives you a solid, permissively-licensed alternative that covers both text and visual modalities in a single model. It's best for developers building search or RAG systems that need to handle mixed media—think product search, visual documentation retrieval, or any app where users might query with images or expect image results. The project is actively maintained with a clear roadmap, and the release of supporting resources like MegaPairs and documentation shows they're thinking beyond just shipping a model. If you've been avoiding multimodal retrieval because it felt like too much complexity, this is worth a look.

Follow @githubprojects for more developer tools and open source projects.