Current Data Landscape

1. Real-World Data

Real-world data consists of sensor outputs such as camera images, LiDAR, radar scans, audio, and haptic feedback collected directly from physical robots operating in authentic scenarios. These data are invaluable for training models to manage real-world uncertainty and variability.

• Current Status: Extremely limited in scope and quantity; typically fragmented datasets obtained from small-scale deployments or laboratory environments, comprising less than 0.1% of the training data.

• Challenges:

  • High cost and logistical complexity of data collection (robot operation, maintenance, sensor calibration).

  • Privacy and regulatory constraints limiting data collection in public and private spaces.

  • Variability in sensor accuracy and data quality.

2. Simulated Data

Simulated data is generated using virtual environments, where digital twins replicate physical scenarios to train and test AI models. Examples include NVIDIA Isaac Sim, CARLA, and Unity-based simulators.

• Current Status: Widely available and extensively used in training, yet comprising less than 1% combined with real-world data.

• Challenges:

  • Sim-to-real transfer gaps arising from simplified physics, lighting discrepancies, and sensor inaccuracies in simulation.

  • Difficulties in simulating human interactions and nuanced behaviors realistically.

  • Computational costs associated with high-fidelity simulations.

3. Internet Data

Internet-sourced data includes unstructured textual, image, and video content from websites, social media, and online repositories. Approximately a 10 billion video clips of datasets can be obtained from the internet, significantly bolstering the training data for foundational AI models.

• Current Status: Vast quantities accessible and dominant, contributing about 99% of the training data.

• Challenges:

  • Lacks structured, scenario-specific context required for direct application.

  • Typically noisy and inconsistent in quality.

  • Ethical and copyright considerations.

4. Task-Specific Data

Task-specific data involves highly focused datasets tailored explicitly for particular tasks, typically gathered through reinforcement learning (RL) and human-guided training.

• Current Status: Extremely limited, constituting roughly 0.01% of the training data.

• Challenges:

  • High cost and time-intensive human involvement.

  • Difficulty scaling across diverse tasks and scenarios.

Why Internet-Only Training Falls Short

Internet-scale training provides a strong prior for vision and language understanding. But it's ungrounded: models know “what a cup is” but not how it feels, how to grasp it, or how it might break. This results in hallucinated actions or infeasible plans in real-world use.

Large VLA models like RT-2, PaLM-E, and OpenVLA have shown that a small fraction of real-world robot interaction data, when weighted properly, can drastically improve performance. But to go further, we need much more:

• Embodied experience: billions of multimodal steps that include action-perception loops, not just static media.

• Task coverage: thousands of household activities and their compositional variants.

• Diverse environments and embodiments: not just one lab robot in one kitchen, but dozens of robots across thousands of homes.

What Data We Really Need for Embodied Intelligence

A roadmap toward general-purpose home robots demands strategic investment in these five types of data:

1. Task Demonstration Trajectories
   Sequences of sensor inputs and actions that show how a robot completes a task, either through teleoperation or autonomous learning. Example: Open X-Embodiment collected 1M+ real demos across 22 robots and 500+ skills.

2. Language-Conditioned Episodes
   Task demonstrations paired with human instructions, enabling models to map language to perception and action.

3. Object Interaction Data
   Detailed manipulation of deformable, fragile, transparent, and unfamiliar objects with varied affordances.

4. Failure and Long-Tail Edge Cases
   Data that shows what happens when things go wrong—and how recovery happens. Essential for robustness and safety.

5. Cross-Embodiment and Multi-Environment Data
   Coverage across different robot hardware, home layouts, lighting conditions, cultural contexts, and more.

From 1% to 80%: Reshaping the Data Mix

Today’s models train on a data mix of ~99% internet content and ~1% real or simulated robot interaction. To achieve generality:

• Shift toward 50–80% embodied data, including real-world and high-fidelity simulation.

• Use internet data as a base, but emphasize grounding through task-specific episodes.

Simulation platforms like Habitat, AI2-THOR, and RoboCasa can generate vast, photorealistic, labeled data with varied layouts, tasks, and agents—faster and safer than physical trials.

Techniques to Expand and Improve Data

• Self-supervised interaction: Let robots “play” without explicit rewards, learning physics through predictive modeling.

• Synthetic augmentation: Use generative models to diversify scenes, objects, and instructions, amplifying data.

• Cross-modal annotation: Automatically label videos or sensor data with instructions, object labels, and spatial relations using VLMs.

• Data curation and curriculum: Focus training on diverse, high-value, and underperforming tasks. Don’t just scale—target wisely.

Designing Scalable Embodied Data Pipelines

A high-quality VLA training pipeline should:

• Aggregate real robot demos across labs, formats, and platforms (like Open X-Embodiment).

• Standardize episodes for unified cross-task and cross-robot training.

• Continuously collect new data via deployed agents, with automatic filtering and labeling.

• Integrate simulation and real-world trials, using a feedback loop to refine both.

How Much Data Is Enough? GPT-Scale Estimates for Robotics

To match the leap seen in GPT-3/4 for language, embodied AI needs:

• 100M to 1B task episodes, including tens of millions of real/simulated robot interactions.

• 10^9+ action steps, especially with language and visual grounding.

• 10,000+ distinct tasks and environments (kitchens, bathrooms, bedrooms, clutter, lighting variance).

• Billions of parameters in VLA models, trained with multi-GPU infrastructure over months.

With 1,000 robots collecting 10 demos/hour, 10M real demos/year is achievable. Combined with scalable simulation, that unlocks internet-comparable scale.

Beyond Data Quantity: Closing the Loop on Alignment

Raw data won’t suffice. We also need:

• Reinforcement learning from human feedback (RLHF) to align behavior with human values.

• Task success + safety reward models to ensure robustness.

• Clarification and personalization interfaces so the robot can ask, adapt, and learn from user intent.

Home robots must not only “do the thing” but do it safely, politely, and as intended—especially in unpredictable environments.

Conclusion: The Road to Embodied GPT

Achieving a GPT-level leap in embodied AI is less a question of models and more a challenge of data design at scale. We must:

• Treat task-grounded data as the new pretraining priority.

• Build shared, open, scalable pipelines for collecting, curating, and evaluating it.

• Optimize for generality, safety, and long-tail robustness—not just average success rate.

If GPT was built on billions of diverse text tokens, its robotic counterpart must be built on billions of diverse embodied moments.

The ingredients are here: foundation models, high-fidelity simulators, multi-robot labs, and generative tools. Now it’s about putting them together—with a data engine worthy of the task.

2025/06/06