About Me
I am a postdoc at Stanford, advised by Prof. Yejin Choi. I got my PhD degree in computer science at UCLA. I was fortunate to be co-advised by two of the best advisors in the world, Prof. Ying Nian Wu and Prof. Kai-Wei Chang. I was also supervised by Prof. Song-Chun Zhu in my early PhD years. Previously, I was an undergraduate in the Department of Electrical Engineering in Shanghai Jiao Tong University, advised by Prof. Xinbing Wang and Prof. Weinan Zhang. I'm generally super grateful for all of my past and current advisors.
Research Highlights
I'm dedicated to building general-purpose embodied agents that could actively explore and interact with the 3D physical world, and perform common sense reasoning within the embodied environment. Specifically, I think the critical aspects of building such embodied agents reside in:- Building 3D world model.
- Large embodied foundation models .
- Visual Common Sense Reasoning.
News
- 2025/01: SlowFast-VGen is accepted by ICLR as Spotlight with scores of 8-8-8-6!
- 2023/12: I gave a talk on "Building Embodied 3D Foundation Models" at Nvidia!
- 2023/09: 3D-LLM is accepted by NeurIPS as Spotlight!
- 2023/06: Successfully hosted the second Machine Visual Common Sense Workshop at CVPR!
Publications
EMBODIED WEB AGENTS: Bridging Physical-Digital Realms for Integrated Agent Intelligence- Yining Hong*; Rui Sun*; Bingxuan Li†; Xingcheng Yao†; Maxine Wu†; Alexander Chien†; Da Yin; Ying Nian Wu; Zhecan James Wang; Kai-Wei Chang
arXiv 2025 [Project Page] [Paper] [Data] [Code]
3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model
- Wenbo Hu; Yining Hong; Yanjun Wang; Leison Gao; Zibu Wei; Xingcheng Yao; Nanyun Peng; Yonatan Bitton; Idan Szpektor; Kai-Wei Chang
CVPR 2025 Foundation Models Meet Embodied Agents Workshop (Best Paper Award) [Project Page] [Paper]
SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation
- Yining Hong; Beide Liu; Maxine Wu; Yuanhao Zhai; Kai-Wei Chang; Lingjie Li; Kevin Lin; Chung-Ching Lin; Jianfeng Wang; Zhengyuan Yang††; Yingnian Wu††; Lijuan Wang††
ICLR 2025 (Spotlight, Score 8-8-8-6) [Project Page] [Paper]
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
- Yining Hong; Zishuo Zheng; Peihao Chen; Yian Wang; Junyan Li; Chuang Gan
CVPR 2024 [Project Page] [Paper]
Compositional VLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
- Junyan Li; Delin Chen; Yining Hong; Zhenfang Chen, Peihao Chen; Yikang Shen; Chuang Gan
ICLR 2024 [Project Page] [Paper] [Code & Data]
Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules
- Zhenfang Chen; Rui Sun; Wenjun Liu; Yining Hong; Chuang Gan
ICLR 2024 [Project Page] [Paper] [Code] [Data]
3D-LLM: Injecting the 3D World into Large Language Models
- Yining Hong; Haoyu Zhen; Peihao Chen; Shuhong Zheng, Yilun Du, Zhenfang Chen, Chuang Gan
NeurIPS2023 (Spotlight) [Project Page] [Paper] [Code & Data]
3D Concept Learning and Reasoning from Multi-View Images
- Yining Hong; Chunru Lin; Yilun Du; Zhenfang Chen; Joshua B. Tenenbaum; Chuang Gan
CVPR2023 [Project Page] [Paper] [Code & Data]
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
- Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, Chuang Gan
AAAI2024 [Paper]
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
- Qing Li; Siyuan Huang; Yining Hong; Yixin Zhu; Ying Nian Wu; Song-Chun Zhu
ICLR2023 [Project Page] [Paper] [Code] [Data]
3D Concept Grounding on Neural Fields
- Yining Hong; Yilun Du; Chunru Lin; Joshua Tenenbaum; Chuang Gan
NeurIPS2022 [Paper] [Code & Data]
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
- Yining Hong; Kaichun Mo; Li Yi; Leonidas Guibas; Antonio Torralba; Joshua Tenenbaum; Chuang Gan
CVPR2022 [Paper] [Code & Data]
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
- Yining Hong; Li Yi; Joshua B. Tenenbaum; Antonio Torralba; Chuang Gan
NeurIPS2021 [Paper] [Code & Data]
VLGrammar: Grounded Grammar Induction of Vision and Language
- Yining Hong; Qing Li; Song-Chun Zhu; Siyuan Huang
ICCV2021 [Paper] [Code & Data]
Learning by Fixing: Solving Math Word Problems with Weak Supervision
- Yining Hong; Qing Li; Daniel Ciao; Siyuan Huang; Song-Chun Zhu
AAAI2021 [Project Page] [Paper] [Code] [Slides]
SMART: A Situation Model for Algebra Story Problems via Attributed Grammar
- Yining Hong; Qing Li; Ran Gong; Daniel Ciao; Siyuan Huang; Song-Chun Zhu
AAAI2021 [Project Page] [Paper] [Code & Data] [Slides]
Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
- Qing Li; Siyuan Huang; Yining Hong; Song-Chun Zhu
ECCV2020 (Oral) [Paper]
Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
- Qing Li; Siyuan Huang; Yining Hong; Yixin Chen; Yingnian Wu; Song-Chun Zhu
ICML2020 [Project Page] [Code]
Best Paper Award in ICML 2020 Workshop on Bridge Between Perception and Reasoning: Graph Neural Networks & Beyond
Academic Reader: An Interactive Question Answering System on Academic Literatures
- Yining Hong; Jialu Wang; Yuting Jia; Weinan Zhang; Xinbing Wang
AAAI2019 (Demo) [Project Page]
Awards & Honors
- Two Sigma PhD Fellowship Award UCLA Internal Nomination, 2024.
- CVPR PhD Consortium Award, 2024.
- Rising Stars in Computational and Data Sciences, 2024.
- Baidu Scholarship (10 recipients worldwide), 2022.
- Snap Fellowship Honorable Mention, 2022.
- China National Scholarship (Top 0.2%), 2018.
Invited Talks
- 2023/04 Invited talk at Umass Amherst
- 2023/02 Invited talks on "Building Embodied 3D Foundation Models" at Apple, USC and CUHK
- 2024/01 Invited talks on "Building Embodied 3D Foundation Models" at Tsinghua University, Peking University
- 2023/12 Invited talks "Building General-Purpose Embodied Foundation Models" at Nvidia, Shanghai Jiao Tong University and Shanghai AI Lab
- 2022/4 Invited talk "When Structure-Based Representations Meet Cognitive Reasoning" at UT Austin
- 2022/3 Invited talk "Part-based Conceptual, Relational, and Physical Reasoning" at AI Time.
- 2021/08 Invited talk "Grounded Grammar Induction of Vision and Language" at AI Drive.
As A Musician
Outside of research, I'm a multi-instrumentalist, composer, and metalhead. I'm the featured organist, keyboard section leader, pipa soloist, ondes martenot soloist in an orchestra (ondes martenot is a very rare instrument - only 60 ondes in total worldwide; pipa is a Chinese instrument similar to guitar). I also play the pipe organ in a local church; the keyboard and guitar in a rock band; occasionally play the saxophone, melodica (highly underrated instrument!), drums, harp, Konghou (Chinese harp), in casual jams. (Favorite music genres: metal, jazz, classic, rock. Favorite book. Favorite Band: Dream Theater. Favorite Album: Metropolis Pt 2.)I've been playing the piano since I was five, and I was the president of the piano association in Shanghai Jiao Tong University. I can do fast sightreading on staff / lead sheet, and have rich recording / performing experience on piano, organ and pipa. I'm a member of American Guild of Organists and my organ teacher is Prof. Christoph Bull from UCLA. (Stay tuned! Gonna release a solo organ album soon.)
Right now I'm really into improvisation and composition. I feel tons of music inside me and the urge to express it. Improvisation also helps me understand more about human planning ^^
2025/6, I played the electric guitar / banjo / keyboard / melodica / percussion set and sang in CVPR 2025 reception.