GitHub - qrafty-ai/teleop_xr: Transforms your VR/AR headset into a powerful, precise robot controller

3 min read Original article ↗

TeleopXR

Transform your VR/AR headset into a powerful, precise robot controller. TeleopXR provides a lightweight, installation-free teleoperation interface with low-latency video streaming and full WebXR state tracking. Check full documentation here. TeleopXR Cover

Key Features

  • đŸ•ļī¸ VR/Passthrough: Seamlessly switch between fully immersive VR and high-fidelity AR Passthrough modes, allowing you to choose between total focus and situational awareness.
  • 📡 WebRTC Video Streaming: Get ultra-low latency, real-time video feedback directly in the headset, providing a near-instantaneous visual link to your robot's perspective.
  • 🤖 Robot Visualization: Benefit from real-time 3D visualization of the robot model, ensuring your digital twin is always perfectly synchronized with the physical robot's state.
  • 🕹ī¸ Realtime Teleoperation based on Whole-Body IK: Achieve precise and intuitive control through advanced Whole-Body Inverse Kinematics, enabling complex coordination with minimal effort.

IK Demo

🚀 Quick Start (Demo)

Use the built-in demo to verify connectivity and visualize the XR state data in real-time.

Installation

Basic installation (teleop mode only):

With IK support:

The IK solver requires additional dependencies. Install with:

pip install teleop-xr
# Install IK dependencies from PyPI
pip install spatialmath-python>=1.1.15 gitpython>=3.1.46 xacro>=2.1.1 filelock>=3.20.3 viser>=1.0.21
# Install pyroki and ballpark from GitHub (not available on PyPI)
pip install git+https://github.com/chungmin99/pyroki.git
pip install git+https://github.com/chungmin99/ballpark.git

Note: pyroki and ballpark are not available on PyPI, so pip install teleop-xr[ik] will not work. You must install them manually from GitHub as shown above.

Alternatively, if you have npm installed, you can also install the whole package from source:

pip install "teleop-xr[ik]@git+https://github.com/qrafty-ai/teleop_xr"

Running the Demo

Modes

The demo supports two operation modes:

  • Teleop Mode (Default): Visualizes raw XR state data and button events. 
    python -m teleop_xr.demo --mode teleop
  • IK Mode: Enables the high-performance IK solver (configured for Unitree H1 by default). Requires IK dependencies installed. 
    python -m teleop_xr.demo --mode ik

Usage

  1. Open the displayed URL (https://<ip>:4443) in your headset.
  2. Enter VR mode.
  3. Observe the live state data and event logs in your terminal.

📖 Documentation

For detailed guides on integrating TeleopXR into your own projects, including the Generic Python API and ROS2 Interface, please visit our official documentation website:

👉 https://qrafty-ai.github.io/teleop_xr/generic/

Development

For developers contributing to TeleopXR or customizing the frontend:

Prerequisites

  • Python 3.10+ with pip
  • uv (recommended for development)
  • Node.js & npm (for WebXR frontend)

Setup

  1. Clone the repository:

    git clone https://github.com/qrafty-ai/teleop_xr.git
cd teleop_xr

  2. Install Python dependencies:

    Option A: Using uv (recommended)

    Option B: Using pip

    pip install -e .

# For IK support, install additional dependencies:
pip install spatialmath-python>=1.1.15 gitpython>=3.1.46 xacro>=2.1.1 filelock>=3.20.3 viser>=1.0.21
pip install git+https://github.com/chungmin99/pyroki.git
pip install git+https://github.com/chungmin99/ballpark.git

  3. Build the WebXR frontend:

    cd webxr
npm install
npm run build

    (The build output will be used by the Python server)

  4. Run from source:

    # From the root directory
# With uv:
uv run python -m teleop_xr.demo

# Or with pip:
python -m teleop_xr.demo

Note on IK Dependencies

The IK solver requires pyroki and ballpark, which are not on PyPI. During development with uv, these are automatically installed from git. For pip-based installations, install them manually from GitHub as shown above.

Acknowledgments

This project is forked from SpesRobotics/teleop. We are grateful for their foundational work in creating a WebXR-based teleoperation solution.

We also leverage powerful libraries for robotics:

  • Pyroki: For high-performance, differentiable Inverse Kinematics and collision checking.
  • Ballpark: For robust collision geometry generation and sphere decomposition.

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.