[LG] A mathematical perspective on Transformers
A Mathematical Perspective on Transformers: This paper presents a mathematical framework for analyzing Transformers, focusing on their interpretation as interacting particle systems. The study reveals
[LG] Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks
This paper provides a mathematical explanation for the emergence of Fourier features in learning systems, such as neural networks. It demonstrates that if a neural
[CL] Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
Y Tian, B Peng, L Song, L Jin, D Yu, H Mi, D Yu [Tencent AI Lab] (2024)
- LLMs still struggle with complex reasoning and planning tasks. Advanced prompting and fine-tuning
[LG] How Transformers Learn Causal Structure with Gradient Descent
E Nichani, A Damian, J D. Lee [Princeton University] (2024)
- The paper studies how transformers learn causal structure through gradient descent when trained on a novel in-context learning
[LG] Mixture of A Million Experts
X O He [Google DeepMind] (2024)
- The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows.
- Sparse
[LG] A Mathematical Guide to Operator Learning
Operator learning aims to discover properties of an underlying dynamical system or partial differential equation (PDE) from data. Here, we present a step-by-step guide to operator learning. We explain the
[CL] Tokenization Is More Than Compression
The paper challenges a widely accepted notion in Natural Language Processing (NLP) that fewer tokens result in better downstream task performance. The authors introduce a new tokenizer, PathPiece, designed to
[IR] Meta Knowledge for Retrieval Augmented Large Language Models
L Mombaerts, T Ding, A Banerjee, F Felice... [Amazon Web Services] (2024)
- The paper introduces a novel data-centric workflow for retrieval augmented large language models (RAG),
[LG] Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory
X Niu, B Bai, L Deng, W Han [Huawei Technologies Co., Ltd] (2024)
- Increasing model size does not always improve performance. The scaling laws do not explain this
[CL] A Closer Look into Mixture-of-Experts in Large Language Models
- Experts act like fine-grained neurons. The gate embedding determines expert selection while the gate projection matrix controls neuron activation. Their heatmaps are correlated,
[LG] Fine-tuning with Very Large Dropout
J Zhang, L Bottou [New York University & Facebook AI Research] (2024)
- It is common to pre-train models on large datasets and then fine-tune them on smaller datasets for specific tasks. This violates the
[LG] ExpeL: LLM Agents Are Experiential Learners
A Zhao, D Huang, Q Xu, M Lin, Y Liu, G Huang [Tsinghua University] (2023)
- ExpeL is an autonomous agent that learns from experiences without parameter updates, compatible with proprietary LLMs.
- During
[CL] Understanding Emergent Abilities of Language Models from the Loss Perspective
Z Du, A Zeng, Y Dong, J Tang [Zhipu AI & Tsinghua University] (2024)
- Recent studies question if emergent abilities in LMs are exclusive to large models, as smaller models
[LG] Mechanics of Next Token Prediction with Self-Attention
Y Li, Y Huang, M. E Ildiz, A S Rawat, S Oymak [University of Michigan & Google Research NYC] (2024)
- Self-attention learns to retrieve high-priority tokens from the input sequence and creates a
[LG] Training trajectories, mini-batch losses and the curious role of the learning rate
M Sandler, A Zhmoginov, M Vladymyrov, N Miller [Google Research] (2023)
#MachineLearning
#ML
#AI
[LG] An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization
- Diffusion models have achieved tremendous success in generative modeling and sample generation in computer vision, audio, reinforcement
[LG] DPO Meets PPO: Reinforced Token Optimization for RLHF
- This paper models RLHF as an MDP, offering a token-wise characterization of LLM's generation process. It theoretically demonstrates advantages of token-wise MDP over sentence-wise bandit
[LG] FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information
D Hwang [Google LLC] (2024)
- Adam optimizer can be considered as an approximation of natural gradient descent through the use of diagonal empirical Fisher
[CL] Toward a Theory of Tokenization in LLMs
N Rajaraman, J Jiao, K Ramchandran [UC Berkeley] (2024)
- Transformers trained on data from certain simple high-order Markov processes fail to learn the underlying distribution and instead predict characters
[CV] Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion
J Tian, L Aggarwal, A Colaco, Z Kira, M Gonzalez-Franco [Georgia Institute of Technology & Google] (2023)
- Proposes DiffSeg, an unsupervised zero-shot
[CL] Large Language Models for Generative Information Extraction: A Survey
Application of Large Language Models (LLMs) in generative information extraction. The study explores the remarkable capabilities of LLMs in text understanding and generation,
[LG] Deep Networks Always Grok and Here is Why
A I Humayun, R Balestriero, R Baraniuk [Rice University] (2024)
- Grokking, where generalization occurs long after achieving near zero training error, is more widespread than previously thought. It occurs not
[CL] LoRAMoE: Revolutionizing Mixture of Experts for Maintaining World Knowledge in Language Model Alignment
S Dou, E Zhou, Y Liu, S Gao, J Zhao... [Fudan University] (2023)
- Vanilla supervised fine-tuning (SFT) with a massive amount of instruction data
[LG] A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold
Y Fei, X Wei, Y Liu, Z Li, M Chen [East China Normal University] (2023)
#MachineLearning
#ML
#AI
[LG] Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
B Pan, Y Shen, H Liu, M Mishra, G Zhang, A Oliva, C Raffel, R Panda [MIT & University of Toronto] (2024)
- MoE models usually require 2-4x more parameters
[CL] Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
J Wieting, J H. Clark, W W. Cohen, G Neubig... [Google Research & CMU & University of California San Diego, San Diego] (2022)
#MachineLearning
#ML
#AI
#NLP
#NLProc
[CL] Diffusion-LM Improves Controllable Text Generation
X L Li, J Thickstun, I Gulrajani, P Liang, T B. Hashimoto [Stanford University] (2022)
#MachineLearning
#ML
#AI
#NLP
#NLProc
[LG] Model Stock: All we need is just a few fine-tuned models
D Jang, S Yun, D Han [NAVER AI Lab] (2024)
- Fine-tuned models with different random seeds lie on a thin shell in weight space layer-wise.
- Proximity of weights to the center of this thin
[LG] Unsupervised Manifold Linearizing and Clustering
T Ding, S Tong, K H R Chan, X Dai, Y Ma, B D. Haeffele [Johns Hopkins University & UC Berkeley] (2023)
#MachineLearning
#ML
#AI
#CV
[CV] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
S Tong, Z Liu, Y Zhai, Y Ma, Y LeCun, S Xie [New York University & UC Berkeley] (2024)
- The paper finds that multimodal large language models (MLLMs) still exhibit systematic
[CL] Mixture of LoRA Experts
X Wu, S Huang, F Wei [Microsoft Research Asia] (2024)
- LoRA (Low-Rank Adaptation) is an effective technique for fine-tuning large pre-trained models. Composing multiple trained LoRAs can enhance performance across tasks.
-
[CL] The pitfalls of next-token prediction
G Bachmann, V Nagarajan [ETH Zürich & Google Research] (2024)
- Next-token prediction has become a core part of modern language models, but there is a growing belief that it cannot truly model human thought and
[CL] Tree Search for Language Model Agents
J Y Koh, S McAleer, D Fried, R Salakhutdinov [CMU] (2024)
- This paper proposes a search algorithm for language model agents to improve performance on multi-step web automation tasks. Language models struggle
[CV] Tails Tell Tales: Chapter-Wide Manga Transcriptions with Character Names
R Sachdeva, G Shin, A Zisserman [University of Oxford] (2024)
- The paper introduces Magiv2, a model that can generate high-quality chapter-wide manga transcripts with named
[LG] Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
- Common transformer architectures like BERT and GPT-2 structurally cannot represent additive models like linear models or generalized additive
[LG] A Survey of Mamba
- Mamba is an emerging deep learning architecture that has demonstrated remarkable success across diverse domains due to its powerful modeling capabilities and computational efficiency.
- Mamba is inspired by classical
[LG] Masked Autoencoders are PDE Learners
A Zhou, A B Farimani [CMU] (2024)
- Masked autoencoders can learn useful latent representations for PDEs through self-supervised pretraining on unlabeled spatiotemporal data. This allows them to improve
[LG] A Survey on the Memory Mechanism of Large Language Model based Agents
- The memory module is a key component that differentiates agents from original large language models (LLMs), enabling agent-environment interactions.
- Memory serves
[LG] Riemannian Residual Neural Networks
I Katsman, E M Chen, S Holalkere, A Asch, A Lou, S Lim, C D Sa [Yale University & Cornell University & Stanford University & University of Central Florida] (2023)
- The paper proposes a novel and principled
[CL] Latent Attention for Linear Time Transformers
The paper introduces the "Latte Transformer," a new transformer model that reduces the time complexity of the attention mechanism from quadratic to linear by employing latent vectors. This approach
[LG] Learning Iterative Reasoning through Energy Diffusion
Y Du, J Mao, J B. Tenenbaum [MIT] (2024)
- This paper introduces iterative reasoning through energy diffusion (IRED), a novel framework for learning to reason for various tasks by formulating
[CL] What Do Language Models Learn in Context? The Structured Task Hypothesis
- The paper examines three hypotheses on how LLMs perform in-context learning: task selection, meta-learning, and structured task composition.
- It invalidates task
[CL] Datasets for Large Language Models: A Comprehensive Survey
The survey delves into the significance of datasets for Large Language Models (LLMs) and their foundational role in LLM advancement. It emphasizes that high-quality datasets are essential
[LG] Gradient Networks
S Chaudhari, S Pranav, J M. F. Moura [CMU] (2024)
- The paper introduces gradient networks (GradNets) to directly parameterize gradients of arbitrary functions. GradNets have architectural constraints to ensure they correspond to
[CL] ReLoRA: High-Rank Training Through Low-Rank Updates
V Lialin, N Shivagunde, S Muckatira, A Rumshisky [University of Massachusetts Lowell] (2023)
- Motivation: Understanding why overparametrized models with billions of parameters are needed for
[LG] Understanding Diffusion Models by Feynman's Path Integral
The paper introduces a novel approach to understanding score-based diffusion models by employing Feynman's path integral formalism, traditionally used in quantum physics. It provides a
[LG] How to Boost Any Loss Function
R Nock, Y Mansour [Google Research] (2024)
- The paper presents a boosting algorithm that can optimize essentially any loss function without using first-order information. This is in contrast to traditional boosting
[CL] MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining
J Portes, A Trott, S Havens, D King, A Venigalla, M Nadeem, N Sardana, D Khudia, J Frankle [MosaicML × Databricks] (2023)
- Proposes MosaicBERT, a modified BERT architecture optimized
[CL] How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning
S Dutta, J Singh, S Chakrabarti, T Chakraborty [IIT Delhi & IIT Bombay] (2024)
- Despite superior reasoning ability with Chain-of-Thought (CoT) prompting, the
[CL] Text Quality-Based Pruning for Efficient Training of Language Models
V Sharma, K Padthe, N Ardalani, K Tirumala… [FAIR, Meta] (2024)
- The paper proposes a novel method to numerically evaluate text quality in large unlabeled NLP datasets in a
[LG] Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations
R Csordás, C Potts, C D. Manning, A Geiger [Stanford University] (2024)
- RNNs learn non-linear representations to store and generate sequences,
[LG] Learning a Decision Tree Algorithm with Transformers
Y Zhuang, L Liu, C Singh, J Shang, J Gao [Microsoft Research & UC San Diego] (2024)
- The paper proposes MetaTree, a transformer-based model that learns to generate decision trees.
- MetaTree is
[CL] sDPO: Don't Use Your Data All at Once
D Kim, Y Kim, W Song, H Kim, Y Kim, S Kim, C Park [Upstage AI] (2024)
- They propose stepwise DPO (sDPO), which divides available preference datasets and uses them step-by-step, instead of all at once like
[CL] Transformer Layers as Painters
Q Sun, M Pickett, A K Nain, L Jones [Emergence AI] (2024)
- The paper investigates the internal workings of transformer layers in large pre-trained language models by conducting empirical studies on frozen models.
-
[LG] The Topos of Transformer Networks
M J Villani, P McBurney [King’s College London] (2024)
- The transformer neural network architecture has achieved great success in natural language processing tasks. This paper provides a theoretical analysis of the
[LG] What Do We Maximize in Self-Supervised Learning And Why Does Generalization Emerge?
R Shwartz-Ziv, R Balestriero, K Kawaguchi, Y LeCun [New York University & Facebook] (2023)
#MachineLearning
#ML
#AI
[CL] Instruction Tuning for Large Language Models: A Survey
S Zhang, L Dong, X Li, S Zhang, X Sun, S Wang, J Li, R Hu, T Zhang, F Wu, G Wang [Zhejiang University & ] (2023)
- Instruction tuning (IT) is a crucial technique to enhance
[LG] Denoising Autoregressive Representation Learning
Y Li, J Bornschein, T Chen [Google DeepMind] (2024)
- The paper proposes Denoising Autoregressive Representation Learning (DARL), which uses a decoder-only Transformer to predict image patches
[CL] Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?
G Yona, R Aharoni, M Geva [Google Research] (2024)
- The paper proposes the notion of "faithful response uncertainty", which measures how well a language model
[CL] A Language Model's Guide Through Latent Space
D v Rütte, S Anagnostidis, G Bachmann, T Hofmann [ETH Zurich] (2024)
- The paper systematically compares linear probing and guidance techniques on hidden representations in large language models (LLMs)
[LG] Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think
L Sernau [Google DeepMind] (2024)
- Infinite width models like NTKs historically underperform finite models. This is attributed to lack of feature learning.
-
[LG] A Survey on LoRA of Large Language Models
- LoRA is an effective parameter efficient fine-tuning paradigm that updates dense layers of neural networks with pluggable low-rank matrices.
- LoRA is computationally efficient and provides good
[LG] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
T Dao, A Gu [Princeton University & CMU] (2024)
- This paper shows theoretical connections between structured state space models (SSMs),
[LG] Information-Theoretic Foundations for Machine Learning
- The paper provides a mathematical framework using Bayesian statistics and information theory to analyze the fundamental limits of machine learning performance.
- It establishes a
[LG] Continual Learning of Large Language Models: A Comprehensive Survey
- Large language models (LLMs) have shown promise for artificial general intelligence (AGI), but face challenges adapting to new data distributions, tasks, and users without
[LG] Manifold Diffusion Fields
A A. Elhag, J M. Susskind, M A Bautista [Apple] (2023)
- This paper proposes a method called Manifold Diffusion Fields (MDF) to learn generative models on Riemannian manifolds.
- It utilizes insights from spectral
[LG] On the Anatomy of Attention
N Khatri, T Laakkonen, J Liu, V Wang-Maścianica [Quantinuum] (2024)
- The paper introduces a category-theoretic diagrammatic formalism to systematically relate and reason about machine learning models.
- The diagrams
[CL] Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations
Z Ma, Z Wang, J Chai [University of Michigan] (2024)
- This work introduces a trial-and-demonstration (TnD) learning framework to examine the
[LG] Attention as an RNN
L Feng, F Tung, H Hajimirsadeghi, M O Ahmed, Y Bengio, G Mori [Borealis AI] (2024)
- Transformers brought breakthroughs in sequence modeling but are inefficient for low-resource settings due to quadratic complexity.
- This work
[CL] Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
Y Zhang, R Sun, Y Chen, T Pfister… [Google Cloud AI Research & Penn State University] (2024)
- Chain-of-Agents (CoA) is a multi-agent LLM collaboration framework for solving
[LG] Learning Associative Memories with Gradient Descent
V Cabannes, B Simsek, A Bietti [Meta AI & Flatiron Institute] (2024)
- The paper studies the training dynamics of associative memory models trained with cross-entropy loss. It shows the dynamics can
[CL] AgentInstruct: Toward Generative Teaching with Agentic Flows
- Synthetic data is becoming increasingly important for accelerating the development of language models, both large and small.
- Despite several successful use cases, researchers
[LG] What makes an image realistic?
L Theis [Google DeepMind,] (2024)
- Quantifying realism remains challenging despite progress in generating realistic data. A good generative model alone is insufficient.
- Probability and typicality fail to quantify
[CL] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
J Pfau, W Merrill, S R. Bowman [New York University] (2024)
- Let's Think Dot by Dot shows that transformers can use meaningless filler tokens like '......' to solve complex
[LG] Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
A Antoniades, X Wang, Y Elazar, A Amayuelas, A Albalak, K Zhang, W Y Wang [Universig of Califormia, Sonta Barbara & Allen Institute for Al] (2024)
-
[LG] Vision language models are blind
P Rahmanzadehgervi, L Bolton, M R Taesiri, A T Nguyen [Auburn University] (2024)
- Large language models with vision capabilities (VLMs) are scoring high on vision-understanding benchmarks but may not actually be
[CL] LLM Pruning and Distillation in Practice: The Minitron Approach
S T Sreenivas, S Muralidharan, R Joshi, M Chochowski… [NVIDIA] (2024)
- The paper presents a comprehensive report on compressing Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B
[CV] LookupViT: Compressing visual information to a limited number of tokens
- LookupViT introduces a novel Multi-Head Bidirectional Cross-attention (MHBC) module that enables effective information flow with significant computational savings.
-
[LG] AutoEval Done Right: Using Synthetic Data for Model Evaluation
P Boyeau, A N. Angelopoulos, N Yosef, J Malik, M I. Jordan (2024)
- Autoevaluation using AI-labeled synthetic data can significantly reduce the need for human annotations when evaluating
[LG] RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
A Setlur, S Garg, X Geng, N Garg… [CMU & Google DeepMind] (2024)
- Training on model-generated synthetic data is promising for finetuning LLMs on math
[CL] Automata-based constraints for language model decoding
T Koo, F Liu, L He [Google DeepMind] (2024)
- Language models are often expected to generate strings in some formal language, but this is not guaranteed, especially with smaller LMs. Tuning
[CL] A Primer on the Inner Workings of Transformer-based Language Models
- The paper provides a concise technical introduction to interpretability techniques used to analyze Transformer-based language models, focusing on the generative decoder-only
[CL] Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
E Zelikman, G Harik, Y Shao, V Jayasiri, N Haber, N D. Goodman [Stanford University & Notbad AI Inc] (2024)
- Quiet-STaR is a generalization of STaR that allows language models
[LG] Attention as a Hypernetwork
S Schug, S Kobayashi, Y Akram, J Sacramento, R Pascanu [ETH Zürich] (2024)
- Transformers can generalize to novel compositional problem instances whose parts were seen during training. This paper aims to understand the
[LG] A Study of Optimizations for Fine-tuning Large Language Models
- Fine-tuning large language models is computationally intensive due to high memory requirements.
- Techniques like gradient checkpointing, low rank adaptation, ZeRO
[AI] How Far Are We From AGI
- The paper provides a comprehensive overview of the current state of AI research and progress towards AGI, covering internal capabilities, interface to the external world, and underlying systems.
- It proposes
[CL] When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
R Kamoi, Y Zhang, N Zhang, J Han, R Zhang [Penn State University] (2024)
- Prior work often does not define research questions on self-correction in
[CL] T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
B Deiseroth, M Brack, P Schramowski, K Kersting, S Weinbach [IPAI & Technical University Darmstadt] (2024)
- Tokenizers require dedicated training which
[LG] Understanding the performance gap between online and offline alignment algorithms
Y Tang, D Z Guo, Z Zheng, D Calandriello… [Google DeepMind] (2024)
- The paper investigates the performance gap between online and offline alignment algorithms for
[CL] LitLLM: A Toolkit for Scientific Literature Review
LitLLM is a toolkit for conducting scientific literature reviews. It addresses the limitations of existing tools that use Large Language Models (LLMs) by operating on Retrieval Augmented
[CL] Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models
- Vision-and-language navigation (VLN) is gaining increasing research attention with the rise of foundation models like BERT, GPT-3, and CLIP. This
[CL] Understanding Transformers via N-gram Statistics
T Nguyen [Google DeepMind] (2024)
- The paper studies how transformer LLM predictions depend on context by approximating them with simple N-gram based rules formed from training data statistics.
- An
[CL] A Survey of Multimodal Large Language Model from A Data-centric Perspective
- Multimodal large language models (MLLMs) extend traditional LLMs by integrating multiple modalities like text, vision, audio, etc. MLLMs require diverse,
[LG] Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
- Model merging combines multiple models into one model at the parameter level, eliminating the need to store all models separately like ensemble
[CL] Patch-Level Training for Large Language Models
- Patch-level training is introduced to improve the training efficiency of large language models, which reduces sequence length by compressing multiple tokens into patches.
- During patch-level