I am currently a fourth-year direct Ph.D. student at the College of Computer Science and Technology, Zhejiang University. My research primarily focuses on the generalizability of multimodal large language models and vision-language models, with a prior concentration on unsupervised domain adaptation and domain generalization. Before joining Zhejiang University, I earned my undergraduate degree at Beijing University of Chemical Technology. Currently, I am very fortunate to be advised by Prof. Chao Wu.

I am on the job market now and will be graduated in the 2025 summer. I am open to both academic and industrial positions! Please contact me if you have matched positions.

Contact

Email: didi_zhu at zju dot edu dot cn / one dot invisible dot numb at gmail dot com

Research Interest

• Robustness of Multi-modal Large Language Models
• Generalization of Vision-Language Models
• Unsupervised Domain Adaptation

Recent News

• 05 / 2024: One first-author paper has been accepted to KDD 2024.
• 05 / 2024: One first-author paper has been accepted to ICML 2024.
• 08 / 2023: One paper about prompt learning is online.
• 07 / 2023: One first-author paper has been accepted to ICCV 2023.
• 07 / 2023: One first-author paper has been accepted to ACM Multimedia 2023.
• 05 / 2023: One paper has been accepted to KDD 2023.
• 11 / 2022: One paper has been accepted to IEEE Transactions on Big Data.
• 10 / 2022: I have passed the Mid-term Assessment of Doctoral Program.
• 10 / 2022: I am presented with Outstanding Postgraduate Student Award on 2021-2022 by Zhejiang University.
• 05 / 2022: One paper has been accepted to IJCAI 2021 FL Workshop.

Publications

Refereed Publications

• Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models. [paper] Didi Zhu, Zhongyisun Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, Kun Kuang ICML 2024
• Neural Collapse Anchored Prompt Tuning for Generalizable Vision-Language Models. [paper] Didi Zhu, Zexi Li, Min Zhang, Junkun Yuan, Jiashuo Liu, Kun Kuang, Chao Wu KDD 2024
• Universal domain adaptation via compressive attention matching. [paper] Didi Zhu, Yinchuan Li, Junkun Yuan, Zexi Li, Kun Kuang, Chao Wu ICCV 2023
• Generalized Universal Domain Adaptation with Generative Flow Networks. [paper] Didi Zhu, Yinchuan Li, Yunfeng Shao, Jianye Hao, Fei Wu, Kun Kuang, Jun Xiao, Chao Wu ACM Multimedia (MM) 2023
• Ensemble federated adversarial training with non-iid data. [paper] Shuang Luo, Didi Zhu, Zexi Li, Chao Wu International Joint Conferences on Artificial Intelligence (IJCAI) FL workshop 2021
• Quantitatively Measuring and Contrastively Exploring Heterogeneity for Domain Generalization. [paper] Yunze Tong, Junkun Yuan, Min Zhang, Didi Zhu, Keli Zhang, Fei Wu, Kun Kuang KDD 2023
• Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching. [paper] Zexi Li, Jiaxun Lu, Shuang Luo, Didi Zhu, Yunfeng Shao, Yinchuan Li, Zhimeng Zhang, Yongheng Wang, Chao Wu IEEE Transactions on Big Data

Projects

• Catastrophic Forgetting in Multi-Modal Large Language Models

  • Sep 2023-Fed 2024 in Tencent Youtu Lab
  • Pioneered the first comprehensive exploration and revelation of catastrophic forgetting in MLLMs such as InstructBLIP and LLaVa.
  • Addressed the issue through an innovative training-free model grafting technique.

• Prompt Tuning in Large Vision-Language Models Based on Neural Collapse

  • April 2023-Aug 2023 in DCD lab, Zhejiang University
  • The first exploration of large vision-language models through the lens of neural collapse in deep learning theory.
  • Tackle class imbalance in generalization tasks for large vision-language models by leveraging neural collapse theory.

• Out-of-Distribution Detection based on Self-Attention Mechanism

  • Dec 2022-Mar 2023 in Huawei Noah’s Ark Lab
  • Addressed the issue of inconsistent source-target label spaces in Universal Domain Adaptation directly using self-attention in ViT.
  • This research has been accepted at ICCV 2023.

• Unifying Domain Adaptation Variants with Label Heterogeneity based on GFlowNet

  • May 2022-Nov 2022 in Huawei Noah’s Ark Lab
  • Introduced a comprehensive problem called Generalized Universal Domain Adaptation, achieving a unification of all Domain Adaptation sub-problems involving label heterogeneity.
  • Implemented an exploration-aware active learning strategy based on Generative Flow Networks to effectively address GUDA.
  • This research has been accepted at ACM Multimedia 2023.