A2X: An end-to-end framework for assessing agent and environment interactions in multimodal human trajectory prediction

Samuel S. Sohn*, Mihee Lee, Seonghyeon Moon, Gang Qiao, Muhammad Usman, Sejong Yoon, Vladimir Pavlovic, Mubbasir Kapadia

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


In recent years, human trajectory prediction (HTP) has garnered attention in computer vision literature. Although this task has much in common with the longstanding task of crowd simulation, there is little from crowd simulation that has been borrowed, especially in terms of evaluation protocols. The key difference between the two tasks is that HTP is concerned with forecasting multiple steps at a time and capturing the multimodality of real human trajectories. A majority of HTP models are trained on the same few datasets, which feature small, transient interactions between real people and little to no interaction between people and the environment. Unsurprisingly, when tested on crowd egress scenarios, these models produce erroneous trajectories that accelerate too quickly and collide too frequently, but the metrics used in HTP literature cannot convey these particular issues. To address these challenges, we propose (1) the A2X dataset, which has simulated crowd egress and complex navigation scenarios that compensate for the lack of agent-to-environment interaction in existing real datasets, (2) evaluation metrics that convey model performance with more reliability and nuance, and (3) a guideline for future data acquisition in HTP. A subset of the proposed metrics are novel multiverse metrics, which are better suited for multimodal models than existing metrics. The dataset is available at: https://mubbasir.github.io/HTP-benchmark.

Original languageEnglish
Pages (from-to)130-140
Number of pages11
JournalComputers and Graphics
StatePublished - Aug 2022

Bibliographical note

Publisher Copyright:
© 2022 Elsevier Ltd


  • Datasets
  • Evaluation metrics
  • Human trajectory prediction

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • General Engineering
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'A2X: An end-to-end framework for assessing agent and environment interactions in multimodal human trajectory prediction'. Together they form a unique fingerprint.

Cite this