About
I am a postdoctoral researcher at ETH Zurich, advised by Ryan Cotterell. My research lies at the intersection of natural language processing and machine learning. I use formal methods—including formal logic, formal language theory, and circuit complexity—to analyze and interpret modern deep learning models, and to guide the development of new architectures and algorithms inspired by these formal perspectives.
Selected Publications
- Characterizing the Expressivity of Local Attention in Transformers
- Probability Distributions Computed by Hard-Attention Transformers
- Characterizing the Expressivity of Fixed-Precision Transformer Language Models
- Unique Hard Attention: A Tale of Two Sides
- What Do Language Models Learn in Context? The Structured Task Hypothesis
- A Transformer with Stack Attention
- Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
- Probing via Prompting
- Differentiable Subset Pruning of Transformer Heads
- Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models
Education
-
PhD in Computer Science, ETH Zurich
Doctoral Fellow at the ETH AI Center
Dissertation: Understanding and Extending the Expressive Power of Transformers
Advisor: Ryan Cotterell2021 - 2026 -
Master's in Data Science, ETH Zurich2018 - 2021
-
Bachelor's in Electronic and Communication Engineering, City University of Hong Kong2014 - 2018