EVGeoQA: Benchmark for Testing LLMs in Dynamic Geospatial Search with Dual Goals
EVGeoQA is a specialized dataset for testing large language models (LLMs) in dynamic multi-objective search scenarios. Each query is tied to real user coordinates and combines two goals: EV charging and a nearby activity like grabbing coffee. GeoRover—an agentic evaluation system with four geospatial tools—exposes LLM weaknesses in long-distance searches, where Hits@2 drops from 50% to 38%.
Task Setup and Differences from Static Benchmarks
Traditional GSQA benchmarks like GeoQA201 or MapQA stick to static fact retrieval, ignoring real-world dynamics. EVGeoQA models mobile EV charging scenarios, where wait times prompt users to multitask. Formally, query Q is anchored to user coordinates L_u. The goal is a station S that satisfies:
- Primary objective: charging access.
- Secondary objective: proximity to a POI (coffee shop, store) within walking distance.
The dataset covers Hangzhou (megacity), Qingdao, and Linyi (growing cities). User locations are generated via K-Means clustering based on population density and road networks, minimizing spatial bias.
Dataset Statistics Table:
| City | Stations | Locations | POI | QA Pairs |
|----------|----------|-----------|-----|----------|
| Hangzhou | 258 | 997 | 25 | 19940 |
| Qingdao | 165 | 995 | 23 | 14162 |
| Linyi | 157 | 997 | 21 | 14416 |
GeoRover: Agent Architecture for Evaluation
GeoRover implements multi-step search via interactive tools:
- get_poi — retrieve points of interest by type and radius.
- get_stations — find charging stations in a given area.
- get_trajectory — agent's movement history.
- calculate_distance — distance and route metrics.
The agent iteratively plans actions, synthesizes trajectories, and outputs the optimal station. This tests LLMs' planning, active exploration, and generalization skills.
Experiments with GPT-4o, Claude-3.5-Sonnet, and Llama-3.1-405B reveal performance degradation at larger radii. Emergent ability: LLMs generalize from past trajectories without explicit instructions, boosting efficiency.
LLM Performance Analysis
Testing on EVGeoQA uncovers key patterns:
- High accuracy (Hits@2 ~50%) within 5 km.
- Drop to 38% at 20+ km due to long-term planning errors.
- Strong tool use for subtasks, but weak spatial info integration.
Compared to baseline GSQA, EVGeoQA better reveals limits of embodied AI.
Key Takeaways
- EVGeoQA shifts focus from facts to dynamic planning tied to L_u with dual goals.
- GeoRover's tools simulate real searches, testing iterative reasoning.
- LLMs struggle with distant searches but show trajectory generalization.
- Dataset realism via K-Means with population density.
- Benchmark ideal for embodied AI and geospatial agents.
— Editorial Team
No comments yet.