∞-THOR — Beyond Needle(s) in the Embodied Haystack

Abstract

∞-THOR is a framework for long-horizon embodied tasks that pushes long-context understanding in embodied AI. It provides:

A generation framework for synthesizing scalable, reproducible and unlimited long-horizon trajectories.
Needle(s) in the Embodied Haystack (NiEH), an embodied QA task where multiple clues scattered across an extended trajectory test an agent's long-context reasoning.
A dataset and benchmark suite of complex tasks spanning hundreds of environment steps, each paired with ground-truth action sequences.

To make agents capable of this, we explore architectural adaptations — interleaved Goal-State-Action modeling, context extension, and Context Parallelism — to equip LLM-based agents for extreme long-context reasoning and interaction. Our results highlight the difficulty of the benchmark and give insight into training strategies and model behaviour under long-horizon conditions.

0

steps in the trajectory playing behind this page

0

tokens of visual context it costs an agent

0

chained subgoals, no resets in between

∞

length the generator can keep going to

the length problem

How long is long-horizon?

Existing embodied benchmarks use episodes of a few dozen steps, which rarely require an agent to retain information across the episode. ∞-THOR trajectories run for thousands of steps, so the observations relevant to the final task typically fall outside the context window of current models.

ALFRED expert demo

≈50 steps

∞-THOR FloorPlan218

2287 steps · ≈3316K tokens

The generator keeps sampling

∞ — run it until you run out of disk

Context is counted the same way as in the paper — roughly 1.45K tokens per interleaved state–action step once a trajectory is converted into the LLM input space. Nothing here is stitched or fast-forwarded: every step is one executed low-level action inside a single continuous episode.

the entire FloorPlan218 episode — 2287 steps, one continuous run drag / hover to scrub

Keyframes spanning the entire FloorPlan218 trajectory

t=0

t = 0t = 1143t = 2287

1 · the generator

Unlimited trajectories

Instead of collecting long demonstrations, we grow them. The generator samples a new task from what the agent can currently reach, plans it symbolically, and executes it in AI2-THOR — appending subgoal after subgoal to a single continuous episode.

01 / SAMPLE

Pick a reachable goal

A goal type, pickup object, movable receptacle and target receptacle are sampled from the objects the agent can actually see and reach in the current scene state.

02 / PLAN

Solve it with PDDL

The task is compiled into a PDDL problem and solved by a Metric-FF planner into a high-level plan, then expanded into low-level navigation and interaction actions.

03 / EXECUTE

Run it in AI2-THOR

Actions are executed in the simulator. Every step logs which objects were visible, pickupable and open — the record that later becomes the haystack.

04 / REVALIDATE

Replay from step 0

The full episode is replayed and object IDs repaired. If it no longer reproduces, the last subgoal is discarded and the agent is teleported back to the last valid state.

One command grows trajectories in any floor plan; a second deterministically replays a saved episode into frames at any resolution. Details in the repository README.

2 · the benchmark

Needle(s) in the Embodied Haystack

Text needle-in-a-haystack tests hide one sentence in a wall of prose. NiEH is harder in two ways: the clues are multiple and scattered, and the haystack is embodied — a stream of egocentric frames interleaved with the actions that produced them. To answer, an agent has to recall what it saw, where it was, and what it did, hundreds of steps ago.

Single-evidence NiEH question types — **Single-evidence.** The answer is recoverable from one observation somewhere in the trajectory — the agent must find that one frame among thousands.

Multi-evidence NiEH question types — **Multi-evidence.** Several temporally distant observations have to be combined — counting, comparing or ordering events that are hundreds of steps apart.

3 · interactive evaluation

Final goals depend on distant observations

Each test episode ends with a synthetic final task deliberately built from objects the agent encountered at distant, unrelated moments. Below, the tomato is seen at t = 17 and the counter top at t = 560; the goal arrives at t = 670. Completing the task therefore requires retaining observations across the full episode, rather than reasoning over recent context alone.

Long-horizon task example from infini-THOR — **Figure.** A trajectory and long-horizon task generated by ∞-THOR. The final goal (“put the tomato on the counter top”) requires recalling both the tomato (t=17) and the counter (t=560). Context size is the input token length once the trajectory is converted into the LLM input space.

Online evaluation: long-horizon execution remains an open challenge

Beyond QA, agents are evaluated online in AI2-THOR on the final long-horizon task of each test trajectory — acting in the environment with the preceding hundreds of steps as their history. Results are reported for fine-tuned Qwen2.5-VL.

Method	Go to	Pick up	Total
Memory-Augmented (Text)	10.8	1.5	6.1
Memory-Augmented (Image, Top-20)	6.2	0.0	3.1
Interleaved Goal-State-Action (32K)	11.3	1.5	6.4
Interleaved Goal-State-Action (128K)	18.5	2.3	9.2
+ Dynamic Scaling	20.0	2.3	9.9
+ YaRN (×4)	23.1	2.6	11.5

Success rate (%) on the long-horizon task; 32K / 128K is the context length used during fine-tuning. Two things stand out. Interleaved beats retrieval — processing the continuous history works better than searching a memory of it. And more context is worth a lot: 32K → 128K nearly doubles the total, with YaRN pushing it to 11.5%. But the Put column is zero everywhere — it is the last stage of the task, and agents almost never survive the earlier ones to reach it. Long-horizon embodied control is wide open.

4 · results

Long context is still an open problem

Feeding the entire trajectory to a VLM and asking a NiEH question. Models whose pretraining context cannot span the episode do not degrade gracefully — they collapse to zero. Even the strongest models lose most of their accuracy the moment an answer needs two distant observations instead of one.

Model	Single-evidence	Multi-evidence
LLaVA-OneVision (7B)	0.00%	0.00%
DeepSeek-VL (7B)	0.00%	0.00%
Qwen2.5-VL (7B)	47.35%	36.60%
Gemini 2.0 Flash	67.36%	30.94%

LLaVA-OneVision and DeepSeek-VL fail to handle contexts beyond their pretraining limits. Note that the best single-evidence model is not the best multi-evidence one: retrieving one needle and reasoning over several are different abilities.

training for it

Context Parallelism, up to 128K

We ship a torchtitan-based trainer with Tensor, Data and Context Parallelism so a 7B VLM can be fine-tuned on sequences long enough to hold a real trajectory.

Config	Parallelism	Sequence length	Use case
ft_llava_ov_7B_tp4_dp2	TP4 × DP2	32K	default 8-GPU setup
ft_llava_ov_7B_tp2_cp2_dp2	TP2 × CP2 × DP2	64K	long-context training
ft_qwen_25vl_7B_infini_cp8	CP8 × FSDP	32K	8-way context parallelism
ft_qwen_25vl_7B_infini_cp16	CP16 × FSDP	128K	very long sequences

5 · the data

More trajectories, all of them long

Every episode below was generated by the same script, in a different floor plan, with no human in the loop. Hover to play.

FloorPlan230

1451steps

27subgoals

2103Ktokens

FloorPlan218

1653steps

30subgoals

2396Ktokens

FloorPlan323

1002steps

24subgoals

1452Ktokens

FloorPlan210

889steps

25subgoals

1289Ktokens

6 · get started

Everything is open

Generation framework, NiEH benchmark, long-horizon training data, model checkpoints and the torchtitan-based training stack — all public. The README walks through evaluation, generation and training end to end.

Code & README 🤗 Training data 🪡 NiEH benchmark 🐳 Docker image

cite

BibTeX

@article{kim2025infinithor,
  title   = {Beyond Needle(s) in the Embodied Haystack: Environment,
             Architecture, and Training Considerations for Long Context Reasoning},
  author  = {Kim, Bosung and Ammanabrolu, Prithviraj},
  journal = {arXiv preprint arXiv:2505.16928},
  year    = {2025},
  url     = {https://arxiv.org/abs/2505.16928}
}