Knowledge Distillation for Hybrid CNN–Vision Transformer Architectures: A Systematic Review
DOI:
https://doi.org/10.5281/zenodo.23208860Keywords:
Knowledge distillation; vision transformer; hybrid architecture; model compression; cross-architecture distillation; edge deployment.Abstract
Hybrid architectures that combine convolutional neural networks (CNNs) with Vision Transformers (ViTs) have become a dominant design pattern in computer vision, pairing the local inductive bias of convolution with the global relational reasoning of self-attention. This dual advantage comes at a computational cost that frequently excludes hybrid models from edge and mobile deployment, motivating a growing body of work that applies knowledge distillation (KD) either to compress hybrid CNN-ViT models directly or to transfer their heterogeneous knowledge into lightweight students of a different architecture family. Distilling from or into a hybrid model is fundamentally harder than distilling between architecturally homogeneous networks, because it requires reconciling two qualitatively different internal representations: spatially local convolutional feature maps and globally relational attention weights. This review synthesizes more than seventy studies published between 2015 and 2026 that address this problem. The literature is organized along three axes: the direction of cross-architecture knowledge transfer (CNN-to-hybrid, hybrid-to-CNN, hybrid-to-hybrid, and ViT-to-ViT within hybrid families), the type of knowledge transferred (response-based, feature-based, attention-based, and relation-based), and the application domain (general vision, medical imaging, remote sensing, object detection and autonomous driving, agriculture, and action recognition or robotics). A comparative synthesis table of representative methods is provided, and six persistent open challenges are identified: the absence of a unified theory of heterogeneous feature alignment, unexamined hyperparameter sensitivity in multi-component losses, domain-fragmented and incomparable evaluation protocols, scarce validation on physical edge hardware, shallow integration with complementary compression techniques, and underreporting of negative or boundary results. A concrete research agenda is proposed to address these gaps. The review is intended to orient researchers designing distillation pipelines for hybrid vision architectures and to consolidate a literature that currently spans disconnected, domain-specific silos.
References
Aboelenin, S., Elbaz, S., Ammar, H., & Amin, K. (2025). A hybrid CNN-Vision Transformer ensemble for apple and corn leaf disease classification. [Study cited from thesis context as anchor architecture example].
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV).
Chen, K., et al. (2019). Learning lightweight pedestrian detector with hierarchical knowledge distillation. arXiv:1909.09325.
Chen, X., Cao, Q., Zhong, Y., & Tao, D. (2022). DearKD: Data-efficient early knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., & Liu, Z. (2022). Mobile-Former: Bridging MobileNet and transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021). CoAtNet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems (NeurIPS).
d'Ascoli, S., Touvron, H., Leavitt, M., Morcos, A., Biroli, G., & Sagun, L. (2022). ConViT: Improving vision transformers with soft convolutional inductive biases. Journal of Statistical Mechanics: Theory and Experiment.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR).
Emerald Publishing (2024). Efficient knowledge distillation for remote sensing image classification: A CNN-based approach (KDE-Net). International Journal of Web Information Systems, 20(2), 129.
Fang, J., Lin, H., Chen, X., & Zeng, K. (2022). A hybrid network of CNN and Transformer for lightweight image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International Journal of Computer Vision, 129(6), 1789-1819.
Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jégou, H., & Douze, M. (2021). LeViT: A vision transformer in ConvNet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Hao, Z., Guo, J., Han, K., Hao, Y., Xu, C., & Wang, Y. (2023). One-for-All: Bridge the gap between heterogeneous architectures in knowledge distillation. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2310.19444.
Hatamizadeh, A., Heinrich, G., Yin, H., Tao, A., Alvarez, J. M., Kautz, J., & Molchanov, P. (2023). FasterViT: Fast vision transformers with hierarchical attention. arXiv:2306.06189.
Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019). A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. NeurIPS Deep Learning and Representation Learning Workshop. arXiv:1503.02531.
IEEE (2022). A CNN-Transformer knowledge distillation for remote sensing scene classification. IEEE Xplore, document 9884099.
IET Cyber-Systems and Robotics (2024). Efficient knowledge distillation for hybrid models: A vision transformer-convolutional neural network to convolutional neural network approach for classifying remote sensing images (HMKD-Net). 6, e12120.
Kim, J., Park, S., & Kwak, N. (2018). Paraphrasing complex network: Network compression via factor transfer. Advances in Neural Information Processing Systems (NeurIPS).
Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., Wang, Y., & Ren, J. (2022). EfficientFormer: Vision transformers at MobileNet speed. Advances in Neural Information Processing Systems (NeurIPS).
Lin, S., Xie, H., Wang, B., & Wang, G. (2022). Knowledge distillation via the target-aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Liu, Y., et al. (2022). Cross-architecture knowledge distillation (CAKD). [As characterized in subsequent cross-architecture distillation literature].
Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S. W., Anwer, R. M., & Khan, F. S. (2022). EdgeNeXt: Efficiently amalgamated CNN-transformer architecture for mobile vision applications. In Proceedings of the European Conference on Computer Vision Workshops (ECCVW). arXiv:2206.10589.
Mehta, S., & Rastegari, M. (2022). MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer. International Conference on Learning Representations (ICLR).
Nandhini, S., & Usha, K. (2024). Lightweight deep learning models for agricultural edge deployment: A systematic review. [Domain synthesis reference for edge-deployment thresholds].
Park, N., & Kim, S. (2022). How do vision transformers work? International Conference on Learning Representations (ICLR).
Park, W., Kim, D., Lu, Y., & Cho, M. (2019). Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Peng, Z., Huang, W., Gu, S., Xie, L., Wang, Y., Jiao, J., & Ye, Q. (2021). Conformer: Local features coupling global representations for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Perera, S., Erzurumlu, Y., Gulati, D., & Yilmaz, A. (2024). MobileUNETR: A lightweight end-to-end hybrid vision transformer for efficient medical image segmentation. arXiv:2409.03062.
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). FitNets: Hints for thin deep nets. International Conference on Learning Representations (ICLR).
ScienceDirect (2024). Optimized hybrid knowledge distillation framework for cardiac MRI classification. S0952197624015884.
ScienceDirect (2025). Exploring the synergies of hybrid convolutional neural network and Vision Transformer architectures for computer vision: A survey. S0952197625000570.
ScienceDirect (2025). LW-CTrans: A lightweight hybrid network of CNN and Transformer for 3D medical image segmentation (incl. LiteMobileV3 distillation). S1361841525000921.
ScienceDirect (2025). CMKD-Net: A cross-modal knowledge distillation method for remote sensing image classification. S0273117725003333.
ScienceDirect (2024). TSD-DETR: A lightweight real-time detection transformer of traffic sign detection for long-range perception of autonomous driving. S0952197624016944.
Springer (2025). A knowledge distillation method based on detection transformer for farmland obstacle detection. Complex & Intelligent Systems.
SPIE (2025). Lightweight TransUNet with knowledge distillation for efficient medical image segmentation.
Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML).
Tian, Y., Krishnan, D., & Isola, P. (2020). Contrastive representation distillation. International Conference on Learning Representations (ICLR).
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2021). Training data-efficient image transformers & distillation through attention. In Proceedings of the International Conference on Machine Learning (ICML).
Tung, F., & Mori, G. (2019). Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS).
Wang, G., Li, K., et al. (2024). Object detection of lightweight transformer based on knowledge distillation. Journal of System Simulation, 36(11).
Wang, L., & Yoon, K. J. (2021/2022). Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6), 3048-3068.
Wang, Y., et al. (2024). KD-DETR: Knowledge distillation for detection transformer with consistent distillation points sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021). CvT: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022). TinyViT: Fast pretraining distillation for small vision transformers. In Proceedings of the European Conference on Computer Vision (ECCV).
Xu, H., et al. (2022). Knowledge distillation for tomato leaf disease detection: 163-fold compression via CNN ensemble to ShuffleNetV2. [Domain application reference, agricultural KD].
Yang, Z., Li, Z., Zeng, A., Li, Z., Yuan, C., & Li, Y. (2024). ViTKD: Practical guidelines for feature-based knowledge distillation for vision transformers. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). arXiv:2209.02432.
Zagoruyko, S., & Komodakis, N. (2017). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. International Conference on Learning Representations (ICLR).
Zhang, J., Peng, H., Wu, K., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022). MiniViT: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Zhao, B., Cui, Q., Song, R., Qiu, Y., & Liang, J. (2022). Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Additional corpus references (cross-architecture, domain-application, and survey studies identified during the systematic search but summarized collectively in text rather than individually authored-and-dated due to preprint/anonymized venue status at time of review):
Cross-Architecture Distillation Made Simple with Redundancy Suppression. arXiv:2507.21844 (2025).
Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures (MLDR-KD). arXiv:2502.06189 (2025).
Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillation. arXiv:2410.12342 (2024).
Revisiting Cross-Architecture Distillation: Adaptive Dual-Teacher Transfer for Lightweight Video Models. AAAI. arXiv:2511.09469 (2026).
Distilling Knowledge from CNN-Transformer Models for Enhanced Human Action Recognition. arXiv:2311.01283 (2023).
X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning. arXiv:2601.11269 (2026).
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification. arXiv:2407.07516 (2025).
MedLiteNet: Lightweight Hybrid Medical Image Segmentation Model. arXiv:2509.03041 (2025).
LightBTSeg: A Lightweight Breast Tumor Segmentation Model via Dual-Path Joint Knowledge Distillation. arXiv:2311.11086 (2023).
Hierarchical Semantically Aligned Heterogeneous Knowledge Distillation (HSAH-KD) for crop leaf disease recognition. ScienceDirect S0952197625000570 (2025).
Lightweight Remote Sensing Scene Classification on Edge Devices via Knowledge Distillation and Early-exit. arXiv:2507.20623 (2025).
Applications of Knowledge Distillation in Remote Sensing: A Survey. arXiv:2409.12111 (2024).
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning. arXiv:2506.09429 (2025).
Survey on Remote Sensing Scene Classification: From Traditional Methods to Large Generative AI Models. arXiv:2603.26751 (2026).
Knowledge Distillation in Object Detection: A Survey from CNN to Transformer. PMC12788226 (2025).
SO-DETR: Leveraging Dual-Domain Features and Knowledge Distillation for Small Object Detection. arXiv:2504.11470 (2025).
Image Recognition with Online Lightweight Vision Transformer: A Survey. arXiv:2505.03113 (2025).
Knowledge Distillation in Vision Transformers: A Critical Review. arXiv:2302.02108 (2023).
Knowledge Distillation in Visual Algorithms: A Survey. Chinese Journal of Computer Research and Development (CRAD) (2025).
A Survey of the Vision Transformers and Their CNN-Transformer Based Variants. arXiv:2305.09880 (2023).
Teacher-Student Architecture for Knowledge Distillation: A Survey. arXiv:2308.04268 (2023).
Theoretical Perspectives on Knowledge Distillation: A Review. WIREs Computational Statistics (2025).
A Lightweight Distillation CNN-Transformer Architecture for Remote Sensing Image Super-Resolution. Taylor & Francis (2023).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Muhammad Hamza Adam, Abdullahi Ishaq, Suleiman Ibrahim, Ali Shuaibu Babaa (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright. Licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. https://creativecommons.org/licenses/by/4.0/ (CC BY 4.0 deed)
