Abstract
Adapter modules have emerged as a parameter-efficient method for fine-tuning large pre-trained models to downstream tasks. However, adapter training can suffer from shortcut learning, where the model exploits spurious correlations in the training data rather than learning robust, generalizable features. We propose a novel approach to prevent shortcuts in adapter training by explicitly providing the shortcuts during training. By exposing the model to the shortcuts it might otherwise exploit, we force the adapter to learn more robust representations that go beyond simple pattern matching. Our method demonstrates improved generalization and robustness across various benchmarks while maintaining the parameter efficiency of standard adapter training.