← Back to Projects

XRT

Unity
WebRTC

!!Whitepaper

!!Website

!!GitHub (Client)

!!GitHub (Server)

XRT is a humanoid robot teleoperation interface for VR headsets that allows users to control robots using inside-out body tracking via wireless connection.

Existing robot teleoperation interfaces (both hardware and software) are not ideal, and the field lacks a de-facto teleop stack:

  • Most tend to be outdated and unmaintained.
  • Many rely on wired connections. Having wires around user and the robot is suboptimal for not only safety, but also the quality of motion you collect.
  • Many rely on physical controllers {hover: that looks like this:link} instead of hand tracking, which allows users more intuitive control of dextrous robot hands.

Gaming was the primary driver of research and adoption of VR/AR technology, and we can look towards them for some inspiration and get a feel for the SoTA:

  • Virtual Desktop, which is one of the most popular VR apps of all time, performs real-time encoding of video (from PC) and decoding (in VR) to essentially implement a virtual computer monitor with no perceived latency (which requires something like <80ms) purely over Wi-Fi!
  • VR user interfaces—like the one in Meta OS—makes great use of hand gestures, to allow a wider range of actions than would be possible with on-screen UI.
    • This grants users more control affordances without bombarding with many buttons on the screen or obstructing the primary activity.

It’s a no-brainer that it would be great to bring some of those innovations into robotics—and that’s exactly why XRT was started. More concretely, our vision was:

  • Be completely open source.

Screenshots [HIDE]

Screenshot Description Why
someone using XRT (3rd person)