EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory

Abstract

Long-horizon egocentric memory transforms continuous first-person video and audio into a searchable record of past experiences. We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent. To address both bottlenecks, we introduce EgoCITE (Egocentric Context-augmented Indexing and Time-aware Evidence retrieval), a long-horizon agentic memory framework for egocentric QA. EgoCITE comprises three components. EgoScheme uses local multimodal context to turn fragmentary video captions and speech transcripts into self-contained atomic memory indices. EgoIndex organizes complementary action, activity, utterance, and conversation representations into searchable multi-view memory indices at multiple granularities. EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved evidence. We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench in terms of answer accuracy and target-event retrieval alignment. EgoCITE improves accuracy over agentic memory baselines by at least 4.4–14.2% while achieving 36× lower cost than long-context LLM agents.

Motivations

We evaluate existing long-horizon egocentric memory systems and identify two fundamental limitations.

Insight 1: Long-horizon egocentric agentic search is limited by what is indexed.

Memory indices are typically built directly from short, independently generated captions and speech transcripts that omit the local context needed to resolve people, objects, actions, and elliptical utterances. Because context discarded before indexing remains unavailable to downstream agentic search, memory construction must use local context to produce self-contained atomic memory indices.

10.1%of WorldMM triples contain unresolved references

17.4%of LoCoMo observations contain verbatim quotes

Insight 2: Retrieval must model temporal intent in addition to semantic relevance.

Memory recall questions express temporal intent through recency (“last time”), habits (“usually”), or explicit time (“yesterday afternoon”). Existing systems rank primarily by semantic similarity and treat time as passive metadata, allowing semantically plausible but temporally incorrect indices to fill the top-k.

76%of EgoLifeQA questions contain temporal intent

48%of EgoMem questions contain temporal intent

Failure cases in existing memory construction and retrieval systems. — **Figure 2.** The workflow and failure cases of existing memory construction and retrieval approaches.

Methodology

EgoCITE integrates context-augmented atomic memory index construction, multi-view egocentric memory indexing, and time-aware retrieval of atomic memory indices.

EgoIndex Multi-View Egocentric Memory Indexing

EgoIndex separates experience along two axes: physical behavior versus spoken interaction, and fine- versus coarse-grained semantics. This results in four views: action, activity, utterance, and conversation.

The four index views, their components, construction specifications, and examples.
View	Component	Specification	Example
Action	HOI	Require a named person, a specific object, and direct physical handling.	“I fasten the shelf bracket with a screwdriver.”
	Gesture	Require a named person and a meaningful expression.	“Jake laughs loudly.”
	Movement	Require a named person and a named destination that the person reaches.	“Shure enters the living room.”
Utterance	—	Require a named speaker, a speech act, and self-contained content with resolved pronouns and ellipsis.	“Jake tells Tasha that dinner is ready.”
Activity	—	Require an immediate goal, specific participants, an object, and a location.	“Alice, Tasha, and I prepare dinner in the kitchen.”
Conversation	—	Require participants and a specific, coherent topic across captions.	“Jake and I debate which model to use for the demo video.”

Multi-view memory

EgoIndex offers two advantages: (1) Multi-granularity retrieval. Fine- and coarse-grained atomic memory indices support queries at different abstraction levels. (2) Separation of interaction types. Physical behavior and spoken interaction capture complementary aspects of an experience.

EgoRetrv Time-Aware Atomic Memory Retrieval

EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved atomic memory indices. The drafting agent estimates the time query from the question using commonsense temporal knowledge and indices retrieved in prior rounds, updating it across rounds. The sampling agent orders the timestamped index pool chronologically and, for “last,” prioritizes the latest timestamps while retaining related indices needed to distinguish answer choices.

Question

“Who used the black signature pen last time?”

Asked at DAY1 13:48

Drafting agent

Time-aware retrieval

Semantic query: signature pen Time query: DAY1 13:48 The agent’s prediction of when the event most likely occurred.

Database Harness

Retrieval scoring

Relevance Score =Semantic Relevance×Time Relevance

Top-K ranking

Index pool

Working memory

earlierlatest

The sampling agent selects the latest index based on the question.

Sampling agent

Time-aware curation

Response agent

Final response

Caption: “I see Shure is writing with a signature pen.”

Answer: Shure

Prioritize recall

The drafting agent issues multi-round semantic, memory-view, and time queries to EgoIndex, retrieves top-ranked timestamped atomic memory indices, and accumulates them in a persistent index pool.

QuestionWhat was I doing the first time Tasha made cupcakes?Asked at DAY6 14290806

Memory view queryAction

Semantic queryI fry eggs

Time queryDAY1 19000000 to DAY1 20300000

#24Semantic onlyOutside action top-20

Time-awareInside action top-20#4

Without temporal relevanceSemantic relevance

1DAY4 11030500–11033000I fry the eggs in the frying pan0.758semantic

2DAY1 19533000–19535900I agree that we should fry the egg0.686semantic

3DAY4 11083100–11085800I say that the eggs are fried eggs0.680semantic

24DAY1 19460100–19463000 GT overlapI crack eggs into the pan0.594semantic

With temporal relevanceSemantic relevance × time relevance = score

1DAY1 19533000–19535900I agree that we should fry the egg0.686 × 1.000 = 0.686semantic × time relevance

2DAY1 14003100–14010000I say that preserved eggs can be fried0.635 × 0.951 = 0.604semantic × time relevance

3DAY1 19533000–19535900Lucia suggests that we fry the egg0.599 × 1.000 = 0.599semantic × time relevance

4DAY1 19460100–19463000 GT overlapI crack eggs into the pan0.594 × 1.000 = 0.594semantic × time relevance

Why the rank changes: The target lies inside the requested time window and keeps full temporal relevance, while the DAY4 semantic match is downweighted from 0.758 to 0.588 and falls from #1 to #5; the target consequently enters the action view's top-20 retrieval budget.

QuestionWho was I with the last time I stood on the escalator in the supermarket?Asked at DAY5 16062618

Memory view queryActivity

Semantic querygo supermarket escalator

Time queryDAY3 17050000 to DAY3 17200000

#21Semantic onlyOutside activity top-10

Time-awareInside activity top-10#4

Without temporal relevanceSemantic relevance

1DAY3 16373100–16385800Shure, Tasha, Alice, and I head to Hema supermarket for barbecue shopping from the mall escalator0.501semantic

2DAY5 15511800–15523000Jack, Tasha, Lucia, and I head into the two-floor supermarket to inspect its selection0.462semantic

3DAY1 18263000–18295300I prepare the group to leave with carts, bags, and bills at the supermarket exit0.453semantic

21DAY3 17130200–17163000 GT overlapShure, Tasha, Alice, Lucia, Katrina, and I carry the groceries out of Hema toward the upper mall exit0.373semantic

With temporal relevanceSemantic relevance × time relevance = score

1DAY3 16373100–16385800Shure, Tasha, Alice, and I head to Hema supermarket for barbecue shopping from the mall escalator0.501 × 0.996 = 0.499semantic × time relevance

2DAY3 15423000–15452900I walk with Tasha toward the mall through the residential streets after leaving the supermarket0.401 × 0.987 = 0.396semantic × time relevance

3DAY3 17000000–17022700Lucia, Tasha, Shure, Alice, Katrina, and I compare yogurt, douzhi, bread, and grocery prices in the supermarket aisle0.385 × 1.000 = 0.385semantic × time relevance

4DAY3 17130200–17163000 GT overlapShure, Tasha, Alice, Lucia, Katrina, and I carry the groceries out of Hema toward the upper mall exit0.373 × 1.000 = 0.373semantic × time relevance

Why the rank changes: The answer-bearing group activity lies inside the drafting agent's “last time” window and keeps full temporal relevance; downweighting out-of-window supermarket matches moves it 17 places into the activity view's top-10 retrieval budget.

QuestionWhen was the last time Wonky's Milk appeared in front of me?Asked at DAY7 13200913

Memory view queryAction

Semantic querymilk carton

Time queryDAY1 11000000 to DAY1 16000000

#26Semantic onlyOutside action top-20

Time-awareInside action top-20#8

Without temporal relevanceSemantic relevance

1DAY3 21350300–21353000I move the milk carton on the table0.585semantic

2DAY2 11180000–11182700I place another carton of milk near the table0.580semantic

3DAY5 20080000–20083000Tasha picks up a carton of milk from the table0.546semantic

26DAY1 14103400–14105700 GT overlapI put the Wangzai milk on top of the big box0.466semantic

With temporal relevanceSemantic relevance × time relevance = score

1DAY2 11180000–11182700I place another carton of milk near the table0.580 × 0.936 = 0.543semantic × time relevance

2DAY2 11190100–11193000I place a carton of milk on one side of the table0.542 × 0.936 = 0.507semantic × time relevance

3DAY2 11180000–11182700I move a carton of milk0.541 × 0.936 = 0.506semantic × time relevance

8DAY1 14103400–14105700 GT overlapI put the Wangzai milk on top of the big box0.466 × 1.000 = 0.466semantic × time relevance

Why the rank changes: The target lies inside the DAY1 time window and keeps full temporal relevance, while the closest DAY3 milk match is downweighted from 0.585 to 0.440 and falls from #1 to #11; the target consequently enters the action view's top-20 retrieval budget.

Experiments

We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench in terms of answer accuracy and target-event retrieval alignment.

Metrics:

1. Accuracy: the percentage of questions for which the response agent selects the ground-truth choice from the candidate answers.

2. Retrieval Hit Rate: the percentage of questions whose retrieved memory indices temporally overlap the target time needed to answer the question, using a maximum retrieval budget of 15 memory indices for every baseline before question answering.

Top values per metric 1st Best 2nd Second 3rd Third Scroll horizontally to inspect all benchmarks

Answer accuracy and retrieval hit rate across EgoLifeQA, EgoMem, and EgoR1-Bench. The three highest distinct values in each accuracy and hit-rate metric are ranked.
Method	Input tokens	Accuracy (%)			Retrieval Hit Rate (%)
Method	Input tokens	EgoLifeQA	EgoMem	EgoR1-Bench	EgoLifeQA	EgoMem	EgoR1-Bench
Long-context LLM agents
GPT-5.4 agent	783k	54.9	75.1	71.0	—	—	—
Gemini-3.1-Pro agent	804k	60.2	78.3	72.7	—	—	—
Agentic memory baselines
EgoRAG	16k	34.3	59.5	49.5	4.3	41.1	7.6
A-MEM	6k	42.9	58.8	52.0	15.4	41.4	24.7
VideoRAG	21k	46.5	68.6	64.7	2.0	17.4	6.0
WorldMM-Qwen	87k	46.4	74.8	66.3	31.0	72.6	38.3
WorldMM-GPT	134k	49.6	68.1	64.0	34.5	66.9	40.0
EgoCITE-Qwen (ours)	34k	61.5	79.9	73.0	43.4	86.8	59.0
EgoCITE-GPT (ours)	32k	63.8	79.2	75.7	49.6	89.6	62.7

Per-category QA accuracy on EgoLifeQA and EgoMem.

EgoLifeQA accuracy over an increasing memory horizon.

EgoLifeQA accuracy, latency, and normalized cost trade-offs.

Temporal Questions

Accuracy is evaluated on 2,223 EgoLifeQA questions containing explicit temporal cues, which constitute 76% of the benchmark.

Result: EgoCITE-GPT reaches 62.6% accuracy, outperforming Gemini-3.1-Pro by 3.9 percentage points and WorldMM-GPT by 15.5 percentage points.

Insight: EgoCITE captures temporal intent more effectively than both long-context LLM agents and existing agentic memory baselines.

Accuracy on the 2,223-question temporal-aware EgoLifeQA subset. Higher values are better; the three highest distinct values are ranked.
Method	Accuracy (%) Higher is better.
Long-context LLM agents
GPT-5.4 agent	51.6
Gemini-3.1-Pro agent	58.7
Agentic memory baselines
EgoRAG	34.8
A-MEM	42.2
VideoRAG	43.4
WorldMM-Qwen	43.8
WorldMM-GPT	47.1
EgoCITE
EgoCITE-Qwen (ours)	60.9
EgoCITE-GPT (ours)	62.6

Habitual Questions

Accuracy is evaluated on habitual questions in EgoLifeQA and multiple-evidence questions in EgoMem, both of which require retrieving information across multiple timestamps rather than from a single moment.

Result: EgoCITE-GPT reaches 64.3% accuracy on habitual questions and 80.3% on multiple-evidence questions, outperforming agentic memory baselines by 5.8–29.9 and 3.0–22.7 percentage points, respectively.

Insight: EgoRetrv aggregates and retrieves information across long temporal horizons while requiring more than 24× fewer input tokens than long-context LLM agents.

Accuracy on habitual EgoLifeQA questions and multiple-evidence EgoMem questions. Higher values are better; the three highest distinct values in each column are ranked.
Method	Accuracy (%) Higher is better.
Method	HabitualEgoLifeQA	Multiple evidenceEgoMem
Long-context LLM agents
GPT-5.4 agent	63.9	83.8
Gemini-3.1-Pro agent	59.8	81.2
Agentic memory baselines
EgoRAG	34.4	57.6
A-MEM	44.3	61.6
VideoRAG	58.5	70.7
WorldMM-Qwen	49.5	77.3
WorldMM-GPT	53.6	64.6
EgoCITE
EgoCITE-Qwen (ours)	58.8	78.2
EgoCITE-GPT (ours)	64.3	80.3

Retrieval rounds

The EgoCITE-GPT ablation varies retrieval depth across one, three, and five rounds and reports retrieval hit rate and average QA accuracy on EgoLifeQA, with curation applied only to the three- and five-round variants.

Result: The five-round variant reaches a 49.6% hit rate and 63.8% average accuracy, improving over the one-round variant by 13.1 and 5.1 percentage points, respectively.

Insight: Most gains occur within three rounds, where hit rate rises by 12.1 points and accuracy by 3.9 points, showing that EgoCITE progressively refines retrieval and recovers additional relevant atomic memory indices.

Retrieval-round ablation for EgoCITE-GPT on EgoLifeQA. Higher values are better; the three highest distinct values in each row are ranked.
Metric	1 roundNo curation	3 roundsCurated	5 roundsCurated
Retrieval and overall QA
Hit rate (%)	36.5	48.6	49.6
Average accuracy (%)	58.7	62.6	63.8
QA accuracy by category (%)
EntityLog	61.9	66.1	67.4
EventRecall	54.7	58.6	59.5
HabitInsight	60.1	60.1	64.3
RelationMap	62.6	65.4	65.6
TaskMaster	61.0	71.0	71.4

Examples

Source windowA1_JAKE · DAY3 · 12:38:01–12:38:29

Katrina and Lucia plan a flower pressing test in the courtyard.

Jake walks toward the courtyard at 12:38:03

Jake and Katrina enter the courtyard at 12:38:14

Multimodal caption

I walk past the whiteboard and continue forward, heading outside. “Should we put this in a dry place?” Katrina asks. “It will press and dry the flowers faster,” she explains. I open the door and step into the courtyard. “Let's make another batch today,” Katrina says. “Remember we had a few extra flowers on the first day? Let's cut them and press them. See how it turns out. Because we have two days.” I approach the table. “I want to test it in advance,” Katrina adds. Lucia chimes in, “Let's see when to stop.”

ActionFine-grained physical behavior

I open the door to the courtyard
I walk into the courtyard
I approach the table in the courtyard

UtteranceFine-grained spoken interaction

Katrina asks whether the flowers should go in a dry place
Katrina suggests making another batch of pressed flowers today
Katrina suggests cutting the extra flowers and pressing the flowers

ActivityCoarse-grained physical behavior

Katrina plans a flower pressing test with Lucia in the courtyard beside the outdoor table

ConversationCoarse-grained spoken interaction

Katrina, Lucia, and I discuss drying extra flowers today to test the pressing process in advance

Four complementary views: EgoIndex preserves entering the courtyard, flower-pressing suggestions, the flower-pressing activity, and its discussion.

Source windowA1_JAKE · DAY7 · 15:33:00–15:33:28

The group discusses flat stones while practicing stone skipping.

Alice and Tasha handle stones beside the outdoor table at 15:33:03

The group discusses stones at the outdoor table at 15:33:14

Multimodal caption

Alice says, “If it's too flat, you can't grab it, hard to grab.” I reply, “Indeed, too flat is hard to grab,” as I throw a stone. I watch Tasha pick up stones. I adds, “Yeah, flat for stone skipping. Skipping, skipping stones. What's stone skipping? Really skipping stones.” I lose a stone and point in the direction where the stones are being thrown. I exclaims, “Look, I skipped it twice,” and laughs. I laugh happily, adjust my glasses, and place my right hand on the table before throwing another stone. I asks, “How many times, twice?” I lose another stone and put my leg down. Alice warns, “If you hit me, I'll tell you. I'll catch you with these glasses. Don't leave today. I'm telling you,” as I lower my head to pick up another stone.

ActionFine-grained physical behavior

I throw a stone into the water
Tasha picks up stones from the table
I throw another stone into the water
I pick up another stone from the table

UtteranceFine-grained spoken interaction

Alice explains that flat stones are hard to grab for stone skipping
I ask Alice what stone skipping means
I say that the stone skipped twice
I ask Alice whether the stone skipped twice

ActivityCoarse-grained physical behavior

I practice stone skipping with flat pebbles beside the water near the table

ConversationCoarse-grained spoken interaction

Alice, Tasha, and I talk about which stones work for skipping and try skipping them

Four complementary views: EgoIndex preserves individual stone throws, the stone-skipping practice activity, questions about stone skipping, and the group's discussion of which stones work.

Source windowA1_JAKE · DAY1 · 11:10:00–11:10:30

Passing a phone around so each person can mark the timestamp.

Jake holds a phone near the dining table at 11:10:04

Jake shows the timestamp on his phone at 11:10:15

Multimodal caption

“Come on, everyone mark it. Each person marks it,” I say. I put the phone in front of everyone and hand it to Katrina. “Pass it along and mark it, hey. You can see it, right?” I ask. Katrina hands the phone to Alice, who then passes it to Tasha. “Alright, all done marking,” I say. The phone is handed back to me. “Okay,” I say. I turn off my phone. “Turn it on,” I instruct myself, but instead put away my phone. I walk up to the small tripod. “So today we're just going to discuss a bit. Discuss this morning. What should we do on our last day? And then maybe... Hmm...” I say as I hold a morning meeting with everyone. I continue the meeting discussion with everyone.

WorldMMOpenIE triples

Issue: The handoff triples omit the purpose of marking a timestamp.

LoCoMoObservations

Jake handed his phone to Katrina to pass along for marking. Jake received his phone back and turned it off before putting it away. Jake walked to a small tripod and led a morning meeting about what to do on the last day.

Issue: Individual observations compound multiple actions, reducing the precision of similarity-based retrieval.

EgoCITEAtomic memory indices

UtteranceI tell everyone to mark the timestamp on my phone. ActionI hand my phone to Katrina. ActionKatrina hands my phone to Alice. ActionAlice hands my phone to Tasha.

Advantage: EgoScheme preserves the marking intent while separating each handoff into its own searchable atomic memory index.

Source windowA1_JAKE · DAY1 · 11:24:00–11:24:30

Jake puts boxes down before the group looks for a pen.

Jake walks toward the dining table at 11:24:15

Multimodal caption

I hold the box to my chest and walk forward, turning right before placing it in the corner. “This thing, you can put these down,” I says. Shure adds, “Let's assemble it later, let's write first.” I turn the box with both hands as I continue walking. “It's not a big problem, right?” I asks. “Not a big problem,” he confirms, while Shure repeats, “Not a big problem, not a big problem. It won't affect my calligraphy.” I place the box in the corner and turn around, walking toward the dining table. “Where's the pen?” I asks. Shure echoes, “Where's the pen?” and remarks, “Good question,” which I agrees with. I walk left into my room and look at the table, then glance back at the bed. “Ask him, ask AI, where's the pen?” Shure suggests. “Where's the pen, where's the pen?” I repeats. Lucia chimes in, “Looking for my iPhone?” and laughs. I turn right and walk out of the room, continuing forward as I asks again, “Yeah, where's the pen?” and Lucia starts, “Looking for my...”

WorldMMOpenIE triples

Issue: The triples preserve incomplete and context-dependent speech such as “Let's assemble it” and “Looking for my...”

LoCoMoObservations

Jake carried a box against his chest, placed it in the corner, and said those things could be put down. Jake went into his room, looked around, and came back out still asking for the pen. Shure suggested assembling the item later and writing first.

Issue: “Those things” and “the item” remain generic, while several distinct behaviors are merged into single observations.

EgoCITEAtomic memory indices

ActionI place the box in the corner of the wall. UtteranceI tell Shure these boxes can be put down. UtteranceShure suggests assembling the box later. UtteranceLucia asks me whether I am looking for Lucia's iPhone.

Advantage: EgoScheme replaces generic references with named people and objects, then decouples the actions and utterances.

Source windowA1_JAKE · DAY1 · 12:52:31–12:53:00

Katrina asks whether all packages are outside; Jake answers “Yes.”

Jake walks through the entryway at 12:52:34

Multimodal caption

I stand still, turn my head to the left, and walk forward into the entryway. I continue walking through the hall while Katrina asks, “Are all the packages outside?” I reply, “Yes.” I adds, “Today, a few more arrived. But the housekeeper will still come to deliver them.” I pick up a pole and speak while holding it.

WorldMMOpenIE triples

Issue: “[I | reply | Yes]” is not self-contained; its meaning is lost when retrieved without the preceding question.

LoCoMoObservations

Jake replied, “Yes.” when asked if all packages were outside. Jake said, “Today, a few more arrived. But the housekeeper will still come to deliver them.” Jake picked up a pole and spoke while holding it.

Issue: The observation preserves the elliptical answer verbatim instead of expressing the confirmed proposition directly.

EgoCITEAtomic memory indices

UtteranceKatrina asks about the packages outside. UtteranceI reply that the packages are outside. UtteranceI say that a few more packages arrived today. UtteranceI add that the housekeeper will deliver more packages later.

Advantage: EgoScheme resolves “Yes” into a complete proposition and separates the exchange into searchable atomic memory indices.

Citation

BibTeX entry for the arXiv preprint.

arXiv BibTeX

@misc{egocite2026,
  title  = {EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory},
  author = {Zhang, Le and Sun, Ke},
  year   = {2026},
  eprint = {2608.12627},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2608.12627}
}

Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory

Abstract

Motivations

Insight 1: Long-horizon egocentric agentic search is limited by what is indexed.

Insight 2: Retrieval must model temporal intent in addition to semantic relevance.

Methodology

EgoIndex Multi-View Egocentric Memory Indexing

Multi-view memory

Self-contained atomic memory indices

EgoRetrv Time-Aware Atomic Memory Retrieval

Drafting agent

Database Harness

Index pool

Sampling agent

Response agent

Prioritize recall

Improve precision

Experiments

Explore EgoCITE results

Category accuracy

Memory scaling

Efficiency

Temporal Questions

Habitual Questions

Retrieval rounds

Examples

Citation