What are the shortcomings of LLMs for modeling reality and why should different sensory data be used instead?

Question

Accepted Answer

LLMs (Large Language Models) have several shortcomings when it comes to modeling reality, primarily due to their limitations in handling sensory data and understanding the physical world. Here are the key points summarized with relevant timestamps:

1. **Lack of Understanding of Physical Reality**:
   - LLMs are unable to handle visual data effectively. They can operate in the space of concepts but struggle with visual representations ([[00:17:00]] - [[00:17:26]]). This limitation hinders their ability to truly understand and model the physical world.

2. **Primitive Characteristics**:
   - LLMs lack persistent memory, reasoning, and planning capabilities. These are essential characteristics for intelligent systems to understand and interact with the physical world ([[00:10:00]] - [[00:10:25]]).

3. **Training and Representation Issues**:
   - Although there are LLMs with vision extensions, they are often seen as hacks and are not trained end-to-end to handle visual data effectively. Consequently, these systems do not understand intuitive physics or common sense reasoning about physical space ([[00:18:00]] - [[00:18:56]]).

4. **Need for Grounding in Reality**:
   - Intelligence needs to be grounded in reality, which LLMs currently lack. They need a more concrete understanding of the world, which can be facilitated through learning from richer environments, either physical or simulated ([[00:14:00]] - [[00:14:23]] and [[00:13:30]] - [[00:13:57]]).

5. **Use of Sensory Data**:
   - To overcome these shortcomings, integrating sensory data such as images, video, and audio into the training process of LLMs is crucial. This approach can help LLMs develop a better understanding and make more informed decisions based on a broader range of inputs ([[00:18:03]] - [[00:18:25]]).

By addressing these limitations and incorporating different sensory data, LLMs can evolve to better model and interact with reality, providing more accurate and contextually relevant outputs.