About Me
I am currently a fourth-year PhD student in computer science at UCLA. I'm co-advised by Prof. Chuang Gan from MIT-IBM Watson AI Lab, Prof. Ying Nian Wu and Prof. Kai-Wei Chang from UCLA. I was also under the supervision of Prof. Song-Chun Zhu from 2019 to 2021. Previously, I was an undergraduate in the Department of Electrical Engineering in Shanghai Jiao Tong University.
Research Highlights
I'm dedicated to building general-purpose embodied agents that could actively explore and interact with the 3D physical world, and perform common sense reasoning within the embodied environment. Specifically, I think the critical aspects of building such embodied agents reside in:- Building 3D world model.
- Large embodied foundation models .
- Visual Common Sense Reasoning.
News
- 2023/12: I gave a talk on "Building Embodied 3D Foundation Models" at Nvidia!
- 2023/09: 3D-LLM is accepted by NeurIPS as Spotlight!
- 2023/06: Successfully hosted the second Machine Visual Common Sense Workshop at CVPR!
Publications
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World- Yining Hong; Zishuo Zheng; Peihao Chen; Yian Wang; Junyan Li; Chuang Gan
CVPR 2024 [Project Page] [Paper]
Compositional VLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
- Junyan Li; Delin Chen; Yining Hong; Zhenfang Chen, Peihao Chen; Yikang Shen; Chuang Gan
ICLR 2024 [Project Page] [Paper] [Code & Data]
Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules
- Zhenfang Chen; Rui Sun; Wenjun Liu; Yining Hong; Chuang Gan
ICLR 2024 [Project Page] [Paper] [Code] [Data]
3D-LLM: Injecting the 3D World into Large Language Models
- Yining Hong; Haoyu Zhen; Peihao Chen; Shuhong Zheng, Yilun Du, Zhenfang Chen, Chuang Gan
NeurIPS2023 (Spotlight) [Project Page] [Paper] [Code & Data]
3D Concept Learning and Reasoning from Multi-View Images
- Yining Hong; Chunru Lin; Yilun Du; Zhenfang Chen; Joshua B. Tenenbaum; Chuang Gan
CVPR2023 [Project Page] [Paper] [Code & Data]
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
- Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, Chuang Gan
AAAI2024 [Paper]
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
- Qing Li; Siyuan Huang; Yining Hong; Yixin Zhu; Ying Nian Wu; Song-Chun Zhu
ICLR2023 [Project Page] [Paper] [Code] [Data]
3D Concept Grounding on Neural Fields
- Yining Hong; Yilun Du; Chunru Lin; Joshua Tenenbaum; Chuang Gan
NeurIPS2022 [Paper] [Code & Data]
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
- Yining Hong; Kaichun Mo; Li Yi; Leonidas Guibas; Antonio Torralba; Joshua Tenenbaum; Chuang Gan
CVPR2022 [Paper] [Code & Data]
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
- Yining Hong; Li Yi; Joshua B. Tenenbaum; Antonio Torralba; Chuang Gan
NeurIPS2021 [Paper] [Code & Data]
VLGrammar: Grounded Grammar Induction of Vision and Language
- Yining Hong; Qing Li; Song-Chun Zhu; Siyuan Huang
ICCV2021 [Paper] [Code & Data]
Learning by Fixing: Solving Math Word Problems with Weak Supervision
- Yining Hong; Qing Li; Daniel Ciao; Siyuan Huang; Song-Chun Zhu
AAAI2021 [Project Page] [Paper] [Code] [Slides]
SMART: A Situation Model for Algebra Story Problems via Attributed Grammar
- Yining Hong; Qing Li; Ran Gong; Daniel Ciao; Siyuan Huang; Song-Chun Zhu
AAAI2021 [Project Page] [Paper] [Code & Data] [Slides]
Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
- Qing Li; Siyuan Huang; Yining Hong; Song-Chun Zhu
ECCV2020 (Oral) [Paper]
Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
- Qing Li; Siyuan Huang; Yining Hong; Yixin Chen; Yingnian Wu; Song-Chun Zhu
ICML2020 [Project Page] [Code]
Best Paper Award in ICML 2020 Workshop on Bridge Between Perception and Reasoning: Graph Neural Networks & Beyond
Academic Reader: An Interactive Question Answering System on Academic Literatures
- Yining Hong; Jialu Wang; Yuting Jia; Weinan Zhang; Xinbing Wang
AAAI2019 (Demo) [Project Page]
Awards & Honors
- CVPR PhD Consortium, 2024.
- Rising Stars in Computational and Data Sciences, 2024.
- Baidu Scholarship (10 recipients worldwide), 2022.
- Snap Fellowship Honorable Mention, 2022.
- China National Scholarship (Top 0.2%), 2018.
Invited Talks
- 2023/04 Invited talk at Umass Amherst
- 2023/02 Invited talks on "Building Embodied 3D Foundation Models" at Apple, USC and CUHK
- 2024/01 Invited talks on "Building Embodied 3D Foundation Models" at Tsinghua University, Peking University
- 2023/12 Invited talks "Building General-Purpose Embodied Foundation Models" at Nvidia, Shanghai Jiao Tong University and Shanghai AI Lab
- 2022/4 Invited talk "When Structure-Based Representations Meet Cognitive Reasoning" at UT Austin
- 2022/3 Invited talk "Part-based Conceptual, Relational, and Physical Reasoning" at AI Time.
- 2021/08 Invited talk "Grounded Grammar Induction of Vision and Language" at AI Drive.
Misc.
Outside of research, I'm a multi-instrumentalist. I play the piano, pipa (a Chinese instrument), and harpsichord in an orchestra; the pipe organ in a local church; the keyboard and guitar in a rock band. I also occasionally play the saxophone and melodica (highly underrated instrument!) in casual jazz jams. (Favorite music genres: metal, jazz, classic, rock. Favorite book.)I've been playing the piano since I was five, and I was the president of the piano association in Shanghai Jiao Tong University. I can do fast sightreading on staff / lead sheet, and have rich recording / performing experience on piano and pipa.
Right now I'm really into improvisation and composition. I feel tons of music inside me and the urge to express it.