๐๐ข๐ฌ๐ฎ๐๐ฅ๐ข๐ณ๐๐ญ๐ข๐จ๐ง ๐จ๐ ๐๐ก๐จ๐ฎ๐ ๐ก๐ญ ๐๐ฅ๐ข๐๐ข๐ญ๐ฌ ๐๐ฉ๐๐ญ๐ข๐๐ฅ ๐๐๐๐ฌ๐จ๐ง๐ข๐ง๐ ๐ข๐ง ๐๐๐ซ๐ ๐ ๐๐๐ง๐ ๐ฎ๐๐ ๐ ๐๐จ๐๐๐ฅ๐ฌ(Paper by microsoft)
Imagine a scenario where youโre listening to a story. As you follow along, your mind naturally starts to visualize the the unseen objects, actions, and their relationships. This mental process is often referred to as the โMindโs Eye.โ Now, consider how language models can perform similar spatial reasoning tasks.
Spatial reasoning involves the ability to visualize and reason about relationships in a three-dimensional environment. Itโs the skill that allows us to comprehend and navigate our surroundings. Recently, Microsoft introduced a paper that explores this concept in LLMs.
To evaluate spatial awareness and the LLMโs ability to understand spatial relationships, several tests have been conducted:
๐๐๐ญ๐ฎ๐ซ๐๐ฅ ๐๐๐ง๐ ๐ฎ๐๐ ๐ ๐๐๐ฏ๐ข๐ ๐๐ญ๐ข๐จ๐ง
In this approach, the model is tested on its ability to determine its current location based on a set of steps. Imagine a 4x4 square grid where objects are placed. For instance, if you start in the bottom-left corner and move right, you encounter a car; moving up, you find a chair, and so on.
After informing the LLM about the objects on the grid, we ask it to perform specific actions (e.g., โgo up, then left, then rightโ). The model must then identify the current location of the specified object. At each reasoning step, the LLM visualizes the grid and performs the requested action.
2)๐๐ข๐ฌ๐ฎ๐๐ฅ ๐๐๐ฏ๐ข๐ ๐๐ญ๐ข๐จ๐ง
The visual navigation task presents a synthetic 2D grid world to the LLM. The challenge is to navigate using visual cues. The LLM generates navigation instructions (left, right, up, down) to reach a destination from a starting point while avoiding obstacles.
This task involves two sub-tasks: route planning (finding a path) and next step prediction (using multi-hop spatial reasoning).
3)๐๐ข๐ฌ๐ฎ๐๐ฅ ๐๐ข๐ฅ๐ข๐ง๐
In this challenge, called polyomino tiling, a 6x6 grid is filled with colored shapes (e.g., green squares, yellow triangles). The LLM must fit a specific shape (e.g., a 3x1 red line) vertically within the grid. This tests the modelโs ability to comprehend, organize, and reason with shapes in a confined area.
๐๐ข๐ฌ๐ฎ๐๐ฅ๐ข๐ณ๐๐ญ๐ข๐จ๐ง ๐จ๐ ๐๐ก๐จ๐ฎ๐ ๐ก๐ญ
The key idea behind VOT is to visualize the state after each reasoning step. Instead of simply generating an answer, the LLM performs a visualization at each reasoning stage. This approach helps improve the accuracy of the modelโs results.
For a clearer understanding, refer to the image below, which illustrates the three tasks mentioned earlier and how VOT is applied.
If you like what you see, hit the follow button! You can also find me on LinkedIn, and we can follow each other there too. ๐
