XRT
Unity
WebRTC
!!
!!
!!
!!
XRT is a humanoid robot teleoperation interface for VR headsets that allows users to control robots using inside-out body tracking via wireless connection.
Existing robot teleoperation interfaces (both hardware and software) are not ideal, and the field lacks a de-facto teleop stack:
- Most tend to be outdated and unmaintained.
- Many rely on wired connections. Having wires around user and the robot is suboptimal for not only safety, but also the quality of motion you collect.
- Many rely on physical controllers {hover: that looks like this:link} instead of hand tracking, which allows users more intuitive control of dextrous robot hands.
Gaming was the primary driver of research and adoption of VR/AR technology, and we can look towards them for some inspiration and get a feel for the SoTA:
- Virtual Desktop, which is one of the most popular VR apps of all time, performs real-time encoding of video (from PC) and decoding (in VR) to essentially implement a virtual computer monitor with no perceived latency (which requires something like <80ms) purely over Wi-Fi!
- VR user interfaces—like the one in Meta OS—makes great use of hand gestures, to allow a wider range of actions than would be possible with on-screen UI.
- This grants users more control affordances without bombarding with many buttons on the screen or obstructing the primary activity.
It’s a no-brainer that it would be great to bring some of those innovations into robotics—and that’s exactly why XRT was started. More concretely, our vision was:
- Be completely open source.
Screenshots [HIDE]
| Screenshot | Description | Why |
|---|---|---|
| someone using XRT (3rd person) | ||