About Me

I am currently a second-year Ph.D. student at Show Lab, National University of Singapore (NUS), supervised by Prof. Mike Shou. Previously, I was an AI researcher at Netease Fuxi AI Lab, working with Dr. Lincheng Li. I received my bachelor’s degree and master’s degree from the Image Processing Center of Beihang University, supervised by Prof. Zhenwei Shi, also working closely with Prof. Zhengxia Zou.

My current research interests lie in multimodal and GenAI, including customizing and improving foundational generation models.

News

• 2024.07: MotionDirector got accepted for oral presentation at ECCV 2024.
• 2023.10: Foundational T2V generation model Show-1 released!
• 2023.07: Invited talk at OPPO Research Institute, “Text-Driven Avatar Auto-Creation”.
• 2023.04: A collection of papers on video diffusion models: Awesome-Video-Diffusion .
• 2023.03: Our paper for avatar auto-creation was accepted by CVPR 2023.
• 2022.08: One first-authored paper selected as ESI Highly Cited Paper.
• 2022.07: Castle in the Sky accepted by IEEE Transactions on Image Processing.
• 2022.01: Awarded the Outstanding Graduates of Beijing.
• 2021.09: Awarded the National Scholarship.

Selected Publications

NeurIPS 2024

EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models
The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
Rui Zhao, Hangjie Yuan, Yujie Wei, Shiwei Zhang, Yuchao Gu, Lingmin Ran, Xiang Wang, Jay Wu, David Zhang, Yingya Zhang, Mike Zheng Shou.
[arXiv][Hugging Face] [Github ]

ECCV 2024

MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Proceedings of the European Conference on Computer Vision, 2024
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, Mike Zheng Shou
[Project Page][arXiv] [Github ]
🎙️ Oral Presentation, Acceptance Rate: 2.3%.
🤗 Featured in Hugging Face “Spaces of the Week 🔥” trending list.
📃 Featured in “Top 40 most cited papers of ECCV 2024” list by AIR-SUN.

IJCV 2024

Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation
International Journal of Computer Vision, 2024
David Junhao Zhang^†, Jay Zhangjie Wu^†, Jia-Wei Liu^†, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, Mike Zheng Shou (^†equal contribution)
[Project Page][arXiv] [Github ]

CVPR 2023

Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creation
The IEEE Conference on Computer Vision and Pattern Recognition, 2023
Rui Zhao, Wei Li, Zhipeng Hu, Lincheng Li, Zhengxia Zou, Zhenwei Shi, and Changjie Fan
[PDF]

Featured apps: Justice Mobile (The first plausible solution for text-driven game characters auto-creation.).
Media coverage: [NewScientist] Character creator AI puts Barack Obama – or anyone – in a video game || [游戏日报] 网易伏羲“文字捏脸”方向论文入选CVPR会议，称已应用于《逆水寒手游》 || CVPR 2023 | 网易伏羲5篇论文入选 包含文字捏脸等业内首创工作 || [AUTOMATON] Researchers report technological breakthroughs in using AI to “make characters by asking with words”. (in Japanese, for English version, please refer to reprint 1 and reprint 2) || [Nazology] "Character creation support AI" that creates the ideal face specified in the sentence is now available! .

TIP 2022

Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos
IEEE Transactions on Image Processing, 2022
Zhengxia Zou, Rui Zhao, Tianyang Shi, Shuang Qiu, and Zhenwei Shi
[PDF][Project Page] [Github ]

Featured apps: Weights & Biases, a ML developer tool with 100,000+ practitioners.

Media coverage: [TNW] This open-source AI tool can make your video spectacular with sky replacement effects || [Better Programming] The Top 10 Trending ML Projects of 2020.

TGRS 2021

High-resolution remote sensing image captioning based on structured attention
IEEE Transactions on Geoscience and Remote Sensing, 2021
Rui Zhao, Zhenwei Shi, and Zhengxia Zou
[PDF]

🏆️ ESI Highly Cited Paper*
* received enough citations to place in the top 1% of the academic field of Geosciences based on publication year. [Clarivate]

Selected Honors

• 2022 The Outstanding Graduates of Beijing (Top 1%)
• 2021 National Scholarship, Ministry of Education of China (Top 1%)
• 2019 The Outstanding Graduates of Beijing (Top 1%)
• …

Academic Service

• Conference Reviewer: ICCV 2023, CVPR 2024-2025, ECCV 2024, NeurIPS 2024, ICLR 2025.

• Journal Reviewer: IEEE Transactions on Geoscience and Remote Sensing (Q1); ISPRS Journal of Photogrammetry and Remote Sensing (Q1); IEEE Geoscience and Remote Sensing Letters (Q1); IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (Q1); Remote Sensing (Q1); International Journal of Digital Earth (Q1); IEEE Internet of Things Journal (Q1).