Overview
Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents' ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models struggle with complex scientific tool-use. Even for a leading model like GPT-5, success rates drop sharply from 58.8% to 34.6% as interaction horizons extend, primarily due to failures in multi-step workflow execution. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific agents. The code and data are released in https://github.com/CMarsRover/SciAgentGYM.
SciAgentGym provides an integrated execution environment comprising four components: a toolkit of scientific tools, a filesystem for intermediate artifacts, scientific databases, and a Python interpreter. Each task runs in an isolated instance with its own registered tools and filesystem, ensuring reproducibility and avoiding cross-task contamination.
SciAgentGym covers scientific tools from physics, chemistry, materials science, life science, and astronomy. The toolkit includes optics, mechanics, electromagnetism, thermodynamics, analytical chemistry, computational chemistry, crystallography, spectroscopy analysis, structural biology, and mass spectrometry.
SciAgentGym provides a standardized evaluation pipeline. It automatically infers and loads required tools, executes agent-generated tool calls in the environment, extracts the final answer from the response, and compares it against the gold answer with flexible matching.
We evaluate closed-source and open-source models under both without-tools and with-tools settings. Tool usage consistently improves performance, while the difficulty breakdown shows that long-horizon scientific tool-use remains challenging for current LLM agents.
| Model | Overall | By Subject (w/ Tools) | By Difficulty (w/ Tools) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| w/o Tools | w/ Tools | Delta | SPL | Phys. | Chem. | Mat. | Life | L1 | L2 | L3 | |
| Closed-Source Models | |||||||||||
| GPT-5 | 32.3 | 41.3 | +9.0 | 0.24 | 46.3 | 43.8 | 28.6 | 32.3 | 58.8 | 38.4 | 34.6 |
| Grok-4-1 | 30.4 | 40.3 | +9.9 | 0.25 | 47.2 | 38.2 | 32.4 | 30.0 | 50.0 | 43.9 | 28.6 |
| Claude-Sonnet-4 | 22.4 | 35.9 | +13.5 | 0.19 | 39.4 | 39.5 | 27.0 | 25.0 | 57.4 | 36.5 | 20.3 |
| Gemini-2.5-Flash | 28.5 | 32.7 | +4.2 | 0.21 | 38.3 | 32.4 | 28.6 | 17.2 | 48.0 | 33.1 | 22.1 |
| Gemini-2.5-Pro | 24.8 | 32.6 | +7.8 | 0.21 | 37.3 | 35.1 | 26.5 | 18.8 | 54.2 | 33.6 | 17.3 |
| O3 | 26.6 | 32.0 | +5.4 | 0.26 | 35.5 | 37.3 | 32.4 | 6.5 | 47.9 | 31.4 | 23.1 |
| O4-mini | 27.8 | 31.1 | +3.3 | 0.24 | 31.2 | 35.5 | 30.6 | 20.0 | 51.0 | 35.0 | 11.7 |
| Gemini-2.5-Pro-Think | 28.9 | 28.8 | -0.1 | 0.19 | 33.3 | 28.9 | 21.2 | 21.9 | 51.0 | 28.8 | 14.5 |
| GPT-4o | 17.1 | 18.7 | +1.6 | 0.14 | 21.3 | 20.5 | 8.6 | 16.0 | 36.0 | 17.4 | 9.2 |
| Open-Source Large Models (>30B) | |||||||||||
| GLM-4.6V | 26.0 | 30.9 | +4.9 | 0.25 | 30.9 | 37.5 | 22.2 | 18.8 | 48.8 | 27.5 | 22.2 |
| Qwen3-VL-235B-Think | 24.4 | 28.0 | +3.6 | 0.16 | 30.6 | 29.5 | 22.9 | 22.6 | 46.8 | 27.4 | 17.4 |
| Qwen3-VL-235B-Inst | 23.0 | 23.9 | +0.9 | 0.16 | 28.1 | 26.5 | 5.0 | 17.2 | 57.1 | 22.6 | 4.6 |
| Qwen3-VL-32B-Think | 24.4 | 27.9 | +3.5 | 0.17 | 33.0 | 31.2 | 8.8 | 22.6 | 45.3 | 27.3 | 16.9 |
| Qwen3-VL-32B-Inst | 22.8 | 27.4 | +4.6 | 0.15 | 31.8 | 29.3 | 20.0 | 16.1 | 47.1 | 27.3 | 14.5 |
| Open-Source Small & Medium Models (≤30B) | |||||||||||
| Qwen3-VL-8B-Inst | 18.4 | 23.4 | +5.0 | 0.09 | 24.0 | 28.6 | 7.1 | 24.1 | 44.4 | 25.2 | 7.0 |
| SciAgent-8B | 23.3+4.9 | 30.1+6.7 | +6.8 | 0.16 | 33.0+9.0 | 35.2+6.6 | 9.1+2.0 | 31.0+6.9 | 45.5+1.1 | 30.5+5.3 | 20.3+13.3 |
| Qwen3-VL-4B-Inst | 17.0 | 19.7 | +2.7 | 0.10 | 23.8 | 20.6 | 10.3 | 13.3 | 48.8 | 16.5 | 8.3 |
| SciAgent-4B | 17.4+0.4 | 25.2+5.5 | +7.8 | 0.13 | 28.4+4.6 | 28.4+7.8 | 14.7+4.4 | 19.4+6.1 | 46.5-2.3 | 24.3+7.8 | 14.5+6.2 |
| Pixtral-12B | 7.8 | 7.2 | -0.6 | 0.07 | 7.5 | 6.3 | 5.9 | 10.0 | 16.0 | 5.0 | 5.1 |
| Average | 23.3 | 28.3 | +4.9 | 0.18 | 31.6 | 30.8 | 19.0 | 20.1 | 47.4 | 28.0 | 16.4 |
@article{shen2026sciagentgym,
title={SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents},
author={Shen, Yujiong and Yang, Yajie and Xi, Zhiheng and Hu, Binze and Sha, Huayu and Zhang, Jiazheng and Peng, Qiyuan and Shang, Junlin and Huang, Jixuan and Fan, Yutao and Tong, Jingqi and Dou, Shihan and Zhang, Ming and Bai, Lei and Yin, Zhenfei and Gui, Tao and Ma, Xingjun and Zhang, Qi and Huang, Xuanjing and Jiang, Yu-Gang},
journal={arXiv preprint arXiv:2602.12984},
year={2026}
}