Description
MergeOver develops efficient inference methods for recursive Vision Transformers, with a focus on deployment on resource-constrained hardware. Vision Transformers process images as sequences of tokens. Recursive weight-sharing reduces model parameters by reusing processing blocks, while token merging reduces the sequence length processed by attention and feed-forward blocks. MergeOver combines these compression methods to reduce memory and computation demands while controlling prediction accuracy, without retraining the underlying model.
The initial implementation integrates Token Merging (ToMe) into the Sliced Recursive Transformer (SReT). The project addresses the constraints of combining these methods, evaluates their behaviour across processors and workloads, and develops merging and reconstruction strategies suited to different deployment requirements.
Research Approach
Combining token merging with recursive processing requires consistent token counts and spatial organisation across reused blocks. MergeOver records merge operations in an Unmerge tracking stack so that features can be restored where required. Merge-rate adjustment respects the model's merging constraints, while token-mass tracking remains synchronised when spatial features are rearranged. A stage-wise single-shot schedule merges tokens at the first block of each stage, then keeps the sequence length fixed during that stage's recursive iterations.
Evaluation considers prediction accuracy, peak activation memory, throughput, and latency across different processors and batch sizes. The resulting trade-offs depend on the execution platform and workload, so compression settings must be assessed against actual deployment objectives.
Initial Evaluation
Against the unmodified SReT baseline, ImageNet-1K evaluation of the selected MergeOver configuration shows a top-1 accuracy reduction of 1.47 percentage points. GPU peak activation memory falls by 37.3% at batch size 1 and 38.4% at batch size 16. GPU throughput decreases by 21.7% at batch size 1 and increases by 21.7% at batch size 16. On a Raspberry Pi 5 ARM CPU, latency falls by 2.4% and 17.6% at those batch sizes.
These results establish a baseline for combining token merging with hierarchical recursive Transformers without retraining. They also expose the importance of merging overhead: fewer tokens do not guarantee faster inference for every workload.
Future Research Directions
Future development focuses on the interaction between compression decisions, execution hardware, and image content. The aim is to develop merging and reconstruction strategies that improve resource use while keeping prediction accuracy and runtime overhead within deployment limits.
Hardware-aware design-space exploration
Automated design-space exploration will investigate token-merging schedules against hardware-specific objectives. Latency, memory use, throughput, and accuracy guide the selection of reproducible configurations suited to a target processor and workload.
Input-adaptive token merging
Input-adaptive merging will investigate a lightweight runtime policy that selects merge rates from intermediate model signals. The aim is to adapt compression to image redundancy and difficulty, improving the computation-accuracy trade-off while controlling decision overhead.
Spatially aware token reconstruction
Spatially aware reconstruction will investigate how merged features can be restored using position, similarity, neighbourhood, or merge-assignment information. The goal is to preserve useful spatial information while balancing prediction accuracy and inference cost without retraining the underlying model.