NVIDIA's Cosmos3-H-Dreams turns clinical cholecystectomy video into an interactive surgical simulator on one RTX PRO 6000, with 28 control dimensions and two open checkpoints
A 16B-parameter base, distilled into four-step and two-step students, ships under the NVIDIA Open Model License. NVIDIA calls it a research simulator, not a validated system for patient care.
To our knowledge, this is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.
Comprehensive take
If you build or evaluate surgical-robotics policies, this is an open world-model recipe worth benchmarking against your numerical simulator, but treat 'interactive' as unproven until you have measured latency yourself and had clinicians grade the physics. Everyone else can wait for surgeon-graded evaluations.
Use cases
Policy evaluation before hardware
Replay candidate control policies against generated cholecystectomy scenes to find failure modes before touching a physical system, which is the model card's stated primary use.
Teleoperated exploration
Drive the simulator from a VR headset or other controller to inspect whether instruments follow commands and tissue responds coherently.
Synthetic data generation
Generate action-conditioned video for downstream training, keeping in mind it is visually plausible rather than biomechanically validated.
Adapt to your own embodiment
Use the published teacher fine-tuning and Self-Forcing distillation recipe on other synchronised video-plus-action datasets.
Specs
| Base model | Cosmos3-H-Embodied-Nano, 16B parameters, pretrained on Open-H-Embodiment source |
|---|---|
| Action space | 28 dimensions: pose and gripper for both arms, energy controls, thumbstick axes and buttons source |
| Training data format | 832 x 480 video, actions aligned at 10 Hz; teacher horizon extended from 24 to 72 frames (2.4 to 7.2 s) source |
| Released checkpoints | Four-step student and a separately trained two-step student source |
| Runtime hardware | Single NVIDIA RTX PRO 6000 GPU via FlashDreams streaming inference source |
| License | NVIDIA Open Model License (license:other on the Hub) source |
Insights
- Axion reading: the durable idea is pairing Self-Forcing distillation with KV-cache consolidation, which treats error accumulation in long rollouts as both a training problem and a memory problem.
- Axion caution: no frame-rate, latency or surgeon-graded accuracy figures appear in the post, so 'real time' and 'physically accurate tissue deformation' remain NVIDIA's descriptions until independently measured.
What actually shipped
NVIDIA's Cosmos3-H-Dreams is a surgical world model: give it one endoscopic frame, a short history of robot commands and a stream of future commands for both arms, and it generates the resulting video autoregressively, block by block. The checkpoint went up on Hugging Face on 19 October 2026 and NVIDIA's team published the write-up on 20 October. It is the successor to Cosmos-H-Dreams, which covered dVRK tabletop suturing; this version moves to clinical cholecystectomy recordings from the CMR Surgical Versius platform.
The recipe is worth reading because it is reproducible. The base is Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. NVIDIA specialises a bidirectional teacher on curated cholecystectomy video and kinematics, first on 24 future frames and then on 72 (2.4 and 7.2 seconds at 10 Hz). Distillation then turns the teacher into a causal student: causal teacher forcing first, then Self-Forcing, where the student trains on its own generated context, guided by the teacher and a learned critic. Two students are released: a four-step model and a separately trained two-step model. The post says plainly that fewer denoising steps buy speed at the cost of visual quality, so the pair is a dial, not a ranking.
The numbers that matter
The control interface is 28 action dimensions: pose and gripper for both arms, plus energy controls, thumbstick axes and thumbstick buttons. Training video is 832 x 480 with actions aligned at 10 Hz, which NVIDIA notes is distinct from runtime throughput. Streaming runs through FlashDreams, NVIDIA's accelerated streaming-inference library, with an adapted FadeMem module that consolidates the growing key/value cache during long rollouts. The headline hardware claim is that it runs interactively on a single NVIDIA RTX PRO 6000 GPU.
What the post does not give us, in the sections we reviewed, is a frames-per-second or latency figure. "Interactive" and "real time" are NVIDIA's descriptions; if you plan to put this in a control loop, measure it yourself on your own hardware before you believe either word.
Why practitioners should care
The usual way to test a surgical-robotics policy is a numerical simulator that models geometry, tissue deformation and contact. Cosmos3-H-Dreams takes the other route: it learns visual dynamics straight from synchronised video and kinematics. That is attractive because tissue under traction, shifting reflections, occlusion and smoke are painful to model explicitly. The cost is that a plausible-looking rollout is not a physically validated one, and NVIDIA says so itself: visual realism is "a starting point for validation, not proof of biomechanical accuracy."
The model card lists the intended uses as surgical-robotics policy evaluation in simulation, teleoperated world-model exploration and synthetic data generation. It ships under the NVIDIA Open Model License, and NVIDIA publishes a step-by-step recipe for teacher fine-tuning and Self-Forcing distillation if you want to adapt it to another dataset or embodiment. Controller integrations must match the training action conventions and normalisation.
Limits to keep in view
NVIDIA's own caveat is the important one: this is a research simulator, not a validated system for patient care, and the roadmap items (full-procedure coverage, coupling with numerical simulation, evaluation with surgeons, more than two controlled instruments, closed-loop autonomous policies) are described as research directions, not established clinical capabilities. The claim of being the first real-time generative surgical simulator trained on clinical cholecystectomy recordings with live kinematic control is NVIDIA's, prefaced by "to our knowledge," and we have not independently verified it. We also have no surgeon-graded accuracy results to report, because the post does not include them.