NeurIPS 2026

RobustPruner: Decoupled Relevance and Uncertainty for Efficient Visual Token Pruning in MLLMs

Hanwei Zhu*,1 Junhan Fu*,2 Xi Zhang1,# Jiamang Wang3 Weisi Lin1,#

1 Nanyang Technological University   2 The Chinese University of Hong Kong (Shenzhen)   3 Alibaba Group

* Equal contribution   # Corresponding authors

Paper (Coming Soon) Code

Overview

Abstract

Multimodal large language models (MLLMs) represent images as long sequences of visual tokens, making inference costly and often redundant. The central challenge is to prune these tokens aggressively without removing evidence required for reliable reasoning. Existing pruning methods typically rank tokens by estimated importance or redundancy, implicitly assuming that low-scored tokens are safe to discard. This assumption is fragile when small, occluded, or fine-grained visual cues are decisive and stable, and it becomes especially brittle under semantics-preserving prompt perturbations. We present RobustPruner, a decoder-integrated framework for prompt-robust and uncertainty-aware visual token pruning. RobustPruner predicts query-conditioned relevance, uses a determinantal point process (DPP) to construct a diverse candidate subset, and then applies uncertainty-guided refinement within that subset. By decoupling relevance and uncertainty, the method preserves ambiguous but potentially decisive evidence while remaining computationally efficient. Inserted into intermediate decoder layers, RobustPruner performs one-shot pruning and reduces KV-cache and memory costs. Across three open-source MLLMs and diverse vision-language benchmarks, RobustPruner consistently achieves a stronger accuracy–efficiency–robustness trade-off than prior pruning baselines, with especially clear gains on fine-grained and text-rich tasks. On Qwen-2.5-VL-7B, RobustPruner retains 95.22% of the original performance while reducing GPU memory, prefill FLOPs, and KV cache size to 91.43%, 50.97%, and 20.57% of the baseline.

Our code is available at https://github.com/oscakywalker/RobustPruner_2026.

Figure 1: example of stable token selection under a rephrased question, robustness radar chart, and efficiency comparison
Figure 1. Motivation, robustness and efficiency of RobustPruner. (a) Equivalent question phrasings make CDPruner and VScan lose key evidence; RobustPruner keeps the answer stable. (b) The radar chart compares robustness-aware performance across five vision-language benchmarks. (c) Qwen-2.5-VL-7B results compare GPU memory, prefill FLOPs, KV cache and retained performance.

Why uncertainty matters

Empirical Analysis

Attention-based uncertainty. Attention importance spreads across salient regions, while instability under equivalent prompts clusters around thin boundaries, small structures and partially occluded content; attention magnitude alone does not reveal reliability.

Similarity-based uncertainty. Query–visual similarity gives a sharper relevance map, yet sparse uncertainty peaks remain around secondary objects and ambiguous cues, showing that relevance and uncertainty capture different information.

Figure 2: image and token importance and uncertainty maps at left; token-wise score plots at right
Figure 2. Importance and uncertainty under semantics-preserving textual perturbations. (a) The example image and four heat maps show where attention-based and similarity-based importance and uncertainty concentrate. (b) Token-wise plots show that uncertainty spikes on a sparse subset of tokens and does not simply follow importance.

Pipeline

How RobustPruner Works

1

Query Anchor Token

A token initialized from the instruction summarizes the query and visual context at the pruning layer.

2

Query-Conditioned Relevance Prediction

A lightweight head scores each visual token using its interaction with the query anchor and visual features.

3

DPP Candidate Selection

A DPP builds a candidate pool that balances query relevance with diversity among visual tokens.

4

Uncertainty-Guided Refinement

The least uncertain candidates are kept to meet the final visual-token budget.

Figure 3: RobustPruner pipeline from image and query through the query anchor, relevance prediction, DPP selection and uncertainty refinement
Figure 3. One-shot pruning at an intermediate decoder layer, followed by decoding with the retained tokens.

Evaluation

Performance Analysis

Across three MLLMs, RobustPruner preserves strong benchmark performance at 10%, 20% and 30% visual-token retention. The comparisons below reproduce the paper's results.

Qwen-2.5-VL-7B

Table 1: Qwen-2.5-VL-7B benchmark comparison at 10, 20 and 30 percent token retention
Performance comparison on Qwen-2.5-VL-7B at three retention ratios.

LLaVA-OneVision-1.5-8B

Table 7: LLaVA-OneVision-1.5-8B benchmark comparison at 10, 20 and 30 percent token retention
Performance comparison on LLaVA-OneVision-1.5-8B at three retention ratios.

LLaVA-v1.5-7B

Table 8: LLaVA-v1.5-7B benchmark comparison at 10, 20 and 30 percent token retention
Performance comparison on LLaVA-v1.5-7B at three retention ratios.

Robustness under Prompt Perturbations

On Qwen-2.5-VL-7B, the paper evaluates an original question and five equivalent rewrites across AI2D, ChartQA, GQA, MME and OCRBench; RobustPruner retains more of the full-model performance than the compared pruning baselines.

Table 9: robustness results for Qwen-2.5-VL-7B across five benchmarks and five prompt perturbations at 20 percent token retention
Robustness under perturbations on Qwen-2.5-VL-7B across five benchmarks (retain 20% tokens).

Inference cost

Efficiency Analysis

At 10% visual-token retention on AI2D, RobustPruner uses about 10× fewer visual tokens, roughly halves prefill FLOPs and cuts KV cache size by nearly 5× on both tested backbones. GPU memory and time per question improve more modestly because the frozen backbone and the decoder's earlier layers still contribute to cost.
Our goal is not intended to maximize raw speed in isolation, but to provide a substantially stronger accuracy–efficiency trade-off under aggressive pruning

Table 4: visual-token count, prefill FLOPs, KV cache, GPU memory, latency and score for two models with and without RobustPruner
Efficiency on the AI2D test set with 10% of visual tokens retained.

Qualitative results

More Visualization Samples

Under equivalent question phrasings, RobustPruner more consistently retains patches near the queried person, animal feature or spatial relation.

Figure 6: three visual-token pruning examples comparing CDPruner, VScan and RobustPruner under original and rephrased questions
Figure 4. Retained visual tokens for original and perturbed queries; red boxes indicate answer-critical regions.

Get in touch

Contact

Citation

BibTeX

@inproceedings{zhu2026robustpruner,
  title = {RobustPruner: Decoupled Relevance and Uncertainty for Efficient Visual Token Pruning in MLLMs},
  author = {Zhu, Hanwei and Fu, Junhan and Zhang, Xi and Wang, Jiamang and Lin, Weisi},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2026},
}