|
I am a 5th-year Ph.D. student at the NLP Lab at Fudan University, advised by Prof. Xipeng Qiu and Prof. Xuanjing Huang. Previously, I had an internship at Shanghai AI Laboratory (2023), Alibaba DAMO Academy (2022), and Amazon Shanghai AI Lab (2019-2020). My research interests lie in the field of Machine Learning and Natural Language Processing, particularly in pre-trained large language models and their optimization-, inference-, and data-efficient methods. Reach out to me over email: txsun19@fudan.edu.cn. CV / Google Scholar / Github / Twitter / OpenMOSS |
|
|
|
Full list of papers can be found at Google Scholar / Semantic Scholar / DBLP / ORCID (*: Equal contribution) |
|
Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, Xipeng Qiu arXiv, 2402.12201 pdf / blog on OpenMOSS Sparse dictionary learning has been a rapidly growing technique in mechanistic interpretability to attack superposition and extract more human-understandable features from model activations. We ask a further question based on the extracted more monosemantic features: How do we recognize circuits connecting the enormous amount of dictionary features? We propose a circuit discovery framework alternative to activation patching. |
|
Qinyuan Cheng*, Tianxiang Sun*, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, Xipeng Qiu arXiv, 2401.13275 pdf / code / blog on OpenMOSS We ask the question "Can AI assistants know what they don't know and express them through natural language?" To answer this question, we construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets. Then we align the assistant with its corresponding Idk dataset and observe whether it can refuse to answer its unknown questions after alignment. |
|
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, Xipeng Qiu ICML, 2022 (Spotlight) pdf / code / slides We propose a promising and practical scenario, Language-Model-as-a-Service (LMaaS), where users cannot access model parameters and gradients but can only access language models' output probability. For such a scenario, we propose the black-box tuning to optimize continuous prompts via derivative-free optimization. |
|
Tianxiang Sun, Xiangyang Liu, Xipeng Qiu, Xuanjing Huang Machine Intelligence Research, 2022 (Invited Paper) pdf / project / slides Recent years have witnessed a trend of paradigm shift in a variety of NLP tasks, which is to solve a task that is originally performed with a paradigm (e.g., sequence labeling) with another paradigm (e.g., machine reading comprehension). |
|
Xiangyang Liu*, Tianxiang Sun*, Junliang He, Jiawen Wu, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, Xipeng Qiu NAACL, 2022 (Oral Presentation) pdf / code / benchmark / slides We propose a benchmark, ELUE (Efficient Language Understanding Evaluation), for efficient NLP models and a strong baseline/backbone pre-trained model, ElasticBERT. |
|
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, Zheng Zhang COLING, 2020 pdf / code / slides We pre-train a model called CoLAKE for jointly learning language and knowledge representation by unifying language and knowledge into word-knowledge graphs. |
|
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, Xuanjing Huang SCIENCE CHINA Technological Sciences, 2020 (Invited Paper, Most Influential Paper of SCTS in 2020) We provide a comprehensive survey of pre-trained models (PTMs) for NLP, ranging from non-contextual word embeddings to state-of-the-art language models. This is a hands-on guide for understanding, using, and developing PTMs for various NLP tasks. |
|
Tianxiang Sun*, Yunfan Shao*, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, Xuanjing Huang AAAI, 2020 (Oral Presentation) pdf / code / slides We propose a new parameter sharing mechanism for multi-task learning, sparse sharing, which allocates a subnet for a task based on lottery ticket hypothesis. The sparse sharing successfully avoids negative transfer between tasks. |
|
|
|
project led by Tianxiang Sun MOSS is a conversational language model like ChatGPT. It is capable of following users' instructions to perform various natural language tasks including question answering, generating text, summarzing text, generating code, etc. MOSS is also able to challenge incorrect premises, and reject inappropriate requests. Here is a brief introduction to MOSS. |
|
maintained by Tianxiang Sun Pre-trained large language models (LLMs) such as GPT-3 are usually released as a service instead of open sourcing model weights. We call this scenario "Language-Model-as-a-Service (LMaaS)", where users can access the powerful LLMs through their inference APIs. We maintain a curated list of papers that fit into this scenario. |
|
|
Student Seminar Co-Chair
Reviewer / Program Committee Member
|
|
Design and source code from Jon Barron's website |