A non-profit research lab focused on interpretability, alignment, and ethics of artificial intelligence.
Creators of GPT-J, GPT-NeoX, Pythia, and VQGAN-CLIP
Over the past two and a half years, EleutherAI has grown from a group of hackers on Discord to a thriving open science research community. Today, we are excited to announce the next step in our evolution: the formation of a non-profit research institute.
The most common question we get about our models is "will X fit on Y GPU?" This, and many more questions about training and inferring with LLMs, can be answered with some relatively easy math.
By
@QuentinAnthon15
,
@BlancheMinerva
, and
@haileysch__
Everyone knows that transformers are synonymous with large language models… but what if they weren’t? Over the past two years
@BlinkDL_AI
and team have been hard at work scaling RNNs to unprecedented scales. Today we are releasing a preprint on our work
What do LLMs learn over the course of training? How do these patterns change as you scale? To help answer these questions, we are releasing a Pythia, suite of LLMs + checkpoints specifically designed for research on interpretability and training dynamics!
📕Today, we'd like to draw attention to the EleutherAI cookbook! ()
The cookbook contains practical details and utilities that go into working with real models!
Such as: 🧵
As part of our work to democratize and promote access to language model technology worldwide, the Polyglot team at EleutherAI is conducting research on multilingual and non-English NLP. We are excited to announce their first models: Korean LLMs with 1.3B and 3.8B parameters.
We have been getting emails from confused individuals trying to access recently. That webpage doesn’t exist, because we don’t have an API. One of them finally clued us in to why: apparently ChatGPT suggests it for trying out our models.
Excited to share our new paper, Lessons From The Trenches on Reproducible Evaluation of Language Models!
In it, we discuss common challenges we’ve faced evaluating LMs, and how our library the Evaluation Harness is designed to mitigate them 🧵
ggml is a deeply impressive project, and much of its success is likely ascribable to
@ggerganov
's management. Managing large-scale branching collaborations is a very challenging task (one we hope to improve at!), and Georgi deserves huge props for how he handles it.
The ggml roadmap is progressing as expected with a lot of infrastructural development already completed
We now enter the more interesting phase of the project - applying the framework to practical problems and doing cool stuff on the Edge
We applaud
@Meta
’s continued push to openly license their models with
#Llama2
having the most permissive license yet. However we are extremely sad to see Meta continue to spread misinformation about the licensing of the model: LLaMA 2 is not open source
We’re excited to be collaborating on a new *resource release* to help provide an on-ramp for new open model developers: the Foundation Model Development Cheatsheet!
A common meme in the AI world is that responsible AI means locking AIs up so that nobody can study their strengths and weaknesses. We disagree: if there is going to be LLM products by companies like OpenAI and Google then independent researchers must be able to study them.
The EMNLP camera-ready version of
@RWKV_AI
is now available on arXiv! Congrats again to
@BlinkDL_AI
@eric_alcaide
@QuentinAnthon15
and the rest of the team on the first successful scaling of RNNs to the ten billion parameter regime!
A 🧵
The Foundation Model Transparency Index by
@StanfordCRFM
purports to be an assessment of how transparent popular AI models are. Unfortunately its analysis is quite flawed in ways that minimize its usefulness and encourage gamification
The latest paper in EleutherAI's close collaboration with
@mark_riedl
's lab on computational storytelling shows how to use a CLIP-like contrastive model to guide the generation of natural language stories to meet human preferences.
If you have substantially contributed to a ML training that required multiple compute nodes, we would like to interview you!
Email contact
@eleuther
.ai with your resume, details about the training run, and a short description of your current interests.
More jobs coming soon!
Our recent blog post contained a meme about code golfing, inspired by a paper that bragged about reaching 80%+ on ImageNet with code that fit in a tweet. In the past 24 hours we've received five emails with code beating Gao (2021), with the current record holder being 260 bytes:
How can we talk about the way AI chat bots behave without falling into false anthropomorphic assumptions? In our latest paper we explore role-play as a framework for understanding chatbots without falsely ascribing human characteristics to language models
Congratulations to our friends at
@allen_ai
on joining (along with EleutherAI and
@llm360
) the tiny club of organizations that have trained a large language model with:
1. Public training data
2. Partially trained checkpoints
3. Open source licensing on model weights
OLMo is here! And it’s 100% open.
It’s a state-of-the-art LLM and we are releasing it with all pre-training data and code. Let’s get to work on understanding the science behind LLMs. Learn more about the framework and how to access it here:
A few days ago we had the first meeting of our newest reading group, focusing on Mixture of Expert (MoE) models. Check out the recording, and drop by our discord server to join the next meeting!
Last year, many people at
@AiEleuther
worked on an project to improve on
@CollinBurns4
's CCS method for eliciting latent knowledge from LLMs.
We were unable to improve on CCS, but today we're publishing the proposed method and negative empirical results.
We are discussing ramping up our public education efforts. What are topic(s) regarding LLMs and other large scale AI technologies that you would like to see more lay-accessible blog posts, infographics, etc. about?
The first major codebase to come out of
@carperai
, our Reinforcement Learning from Human Feedback (RLHF) lab. Previous work by
@OpenAI
and
@AnthropicAI
has made it clear that RLHF is a promising technology, but a lack of released tools and frameworks makes using it challenging
Today, we’re launching Aya, a new open-source, massively multilingual LLM & dataset to help support under-represented languages. Aya outperforms existing open-source models and covers 101 different languages – more than double covered by previous models.
We are very excited to share the results of our collaboration with
@farairesearch
on developing tooling for understanding how model predictions evolve over the course of training. These ideas are already powering our ELK research, so expect more soon!
Ever wonder how a language model decides what to say next?
Our method, the tuned lens (), can trace an LM’s prediction as it develops from one layer to the next. It's more reliable and applies to more models than prior state-of-the-art. 🧵
We believe that building a robust, interoperable research community requires collaboration.
@huggingface
has been doing a phenomenal job organizing multilateral collaborations and we're excited to continue to participate.
Congrats to
@haileysch__
and the entire
@BigCodeProject
!
Introducing: 💫StarCoder
StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. It can be prompted to reach 40% pass
@1
on HumanEval and act as a Tech Assistant.
Try it here:
Release thread🧵
We’ve trained and released Llemma, strong base LMs for mathematics competitive with the best similar closed+unreleased models.
We hope these models + code will serve as a powerful platform for enabling future open Math+AI research!
We release Llemma: open LMs for math trained on up to 200B tokens of mathematical text.
The performance of Llemma 34B approaches Google's Minerva 62B despite having half the parameters.
Models/data/code:
Paper:
More ⬇️
Amazing news for our close partner in research and major donor,
@huggingface
. We've be thrilled to work with HF on projects like BLOOM and the Open LLM Leaderboard, and are excited to continue to work with them to advance open AI research and the open source ecosystem.
Great to see
@CerebrasSystems
build on top of Pile and releasing these open source! Cerebras-GPT is Chinchilla optimal up to 13B parameters. A nice compliment to our Pythia suite, allowing for the comparison of the effect of different training regimes on model behavior
🎉 Exciting news! Today we are releasing Cerebras-GPT, a family of 7 GPT models from 111M to 13B parameters trained using the Chinchilla formula. These are the highest accuracy models for a compute budget and are available today open-source! (1/5)
Press:
Building a ChatGPT-like LLM might be easier than anyone thought. At
@Databricks
, we tuned a 2-year-old open source model to follow instructions in just 3 hours, and are open sourcing the code. We think this tech will quickly be democratized.
We are excited to join other leaders in artificial intelligence in partnering with
@NSF
to launch the National AI Research Resource (NAIRR), a shared infrastructure that will promote access to critical resources necessary to power AI research.
NSF and its partners are proud to launch the National AI Research Resource pilot.
Its goal? To democratize the future of
#AI
research & development by offering researchers & educators advanced computing, datasets, models, software, training & user support.
As models become larger and more unwieldy, auto-interp methods have becoming increasingly important. We are excited to be releasing the most comprehensive auto interp library to enable wider research on this topic.
Sparse autoencoders recover a diversity of interpretable features but present an intractable problem of scale to human labelers. We build new automated pipelines to close the gap, scaling our understanding to GPT-2 and LLama-3 8b features.
@goncaloSpaulo
@jacobcd52
@norabelrose
Reinforcement Learning from Human Feedback is an allegedly powerful technology for language models, but one that has so far been kept out of the hands of most researchers. We are thrilled to be working on bringing the ability to study and evaluate these model to the mainstream.
Kyle is one of four members of our community without a PhD with their first first-author paper under review currently! We view providing this training and mentorship as an important part of our public service.
We are grateful to EleutherAI for permitting access to their compute resources for initial experiments. The welcome and open research community on the EleutherAI Discord was especially helpful for this project and my growth as a scientist. 😊
Another day, another math LM bootstrapping their data work off of the work done by OpenWebMath and Llemma teams. This makes the three in the past week!
Open data work is 🔥 Not only do people use your data, but high quality data work has enduring impact on data pipelines.
AutoMathText
Autonomous Data Selection with Language Models for Mathematical Texts
paper page:
dataset: .
To improve language models' proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy
Releasing data is amazing, but tools like these that help people make sense of the data is arguably an even more important step forward for data transparency. We're thrilled to see our community continue to lead by example when it comes to in transparent releases.
An essential blocker to training LLMs on public domain books is not knowing which books are in the public domain. We're working on it, but it's slow and costly... if you're interested in providing support reach out!
@BlancheMinerva
@rom1504
Indeed, these would be *extremely* valuable data resources.
The databases on are, unfortunately, ununified and the records themselves seem anemic.
Somewhat odd considering USPTO has flagship datasets (available via NAIRR). Greater financial incentives?
A very interesting analysis from
@ZetaVector
looks at how the most cited papers each year break down.
We’re especially proud of this statistic: Almost 20% of papers with EleutherAI authors were in the top 100 most cited papers of their year.
Full report:
We are excited to see torchtune, a newly announced PyTorch-native finetuning library, integrate with our LM Evaluation Harness library for standardized, reproducible evaluations!
Read more here:
Blog:
Thread:
torchtune provides:
- LLM implementations in native-PyTorch
- Recipes for QLoRA, LoRA and full fine-tune
- Popular dataset-formats and YAML configs
- Integrations with
@huggingface
Hub,
@AiEleuther
Eval Harness, bitsandbyes, ExecuTorch and many more
[3/5]
Interested in Mixture-of-Experts models but don't want to train it from scratch? Check out the latest from
@arankomatsuzaki
, who spent his internship
@GoogleAI
figuring out how to convert existing dense models to MoE ones.
We have released "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints"!
Our method converts a pretrained dense model into a MoE by copying the MLP layers and keeps training it, which outperforms continued dense training.
(1/N)
Very glad to see this. “Public release” of models doesn’t mean much at this scale if you can’t provide a free API, as almost nobody can afford to deploy the model.
Great work by
@huggingface
and
@Azure
making this happen and keeping it supported
most NLP researchers had a very minimal understanding of the engineering undertaking required to train such models or their capabilities & limitations. We started as a ragtag group nobody had heard of, and within a year had released the largest OSS GPT-3-style model in the world.
@Meta
@ylecun
@paperswithcode
@huggingface
@StabilityAI
If “open source” is to mean anything, we must stand with
@OpenSourceOrg
and call out corporate misinformation.
You don’t need to license your models open source. It may even be the best choice to *not* do so. But if you don’t, you shouldn’t lie and say you did.
A really phenomenal deep dive into LLM evaluations and good illustration of why
1. Real life applications should be evaluated in the deployment context
2. Open access to models and evaluation code is essential for understanding the claims made in papers
𝗗𝗼𝗻’𝘁 𝗯𝗹𝗶𝗻𝗱𝗹𝘆 𝘁𝗿𝘂𝘀𝘁 𝘁𝗵𝗲 𝗢𝗽𝗲𝗻 𝗟𝗟𝗠 𝗟𝗲𝗮𝗱𝗲𝗿𝗯𝗼𝗮𝗿𝗱!
We used
@try_zeno
to explore the Open LLM Leaderboard data. Spoiler: Unless you use LLMs for multiple choice questions, these benchmarks aren’t that helpful.
Zeno Report:
Claiming that you can match a transformers’ performance is nothing new, and plenty of other papers put forth that claim. What makes RWKV special is that we actually train models up to 14B params and show consistently competitive performance with token-matched transformers!
A little over a month ago,
@Vermeille_
showed up in our discord server with a simple question: can CFG be applied to LLMs?
Probably, but the devil’s in the details. So we sat down to figure those details out.
Check out his new paper for more ⬇️⬇️⬇️
We borrowed CFG from vision and run it with LLMs. We get increased control, and benchmarks increases similar to a model twice the size. Ready for all your models (incl. chatbots!) : no special training or fine tuning required.
thx
@AiEleuther
!
Congrats to everyone who won one of these grants. The open source community desperately needs more funding so that people can be *professional* open source engineers and researchers, lest the only end-game be a closed-source job.
[New program] a16z Open Source AI Grants
Hackers & independent devs are massively important to the AI ecosystem.
We're starting a grant funding program so they can continue their work without pressure to generate financial returns.
It’s been a pleasure to watch Theodore’s ideas develop over the past two years. Definitely check out his paper one finetuning LLMs into “text-to-structure” models, and how to use them to design tools that are useful to architects.
It's finally out! After almost 2 years of delay, our paper on Architext, the first, open-source, language model trained for Architectural design, is now on arxiv.
In the unlikely event you're curious to read it, you can find it here:
Quick thread ↓
This will enable us to do much more, and we look forward to building a world class research group for public good! Lead by Stella Biderman
@BlancheMinerva
as Executive Director and Head of Research, Curtis Huebner as Head of Alignment, and Shiv Purohit as Head of Engineering.
RNNs struggle to scale because of how they parallelize, but making the time decay of each channel data-independent, we are able to parallelize RWKV the same way transformers are during training! After training, it can be used like an RNN for inference.
We're excited to announce that
@lintangsutawika
and
@haileysch__
will be at ICML 2024 in Vienna on July 22 to present a tutorial on "Challenges in Language Model Evaluations"!
Website:
This is some really phenomenal work out of
@StanfordHAI
. Evaluation work, like data work, is a massively understudied and undervalued. But work like this has far more impact than a half dozen medium papers about minor tweaks to transformer architectures
Two lessons we learned through HELM (Sec 8.5.1; ): 1. CNN/DM and XSum reference summaries are worse than summaries generated by finetuned LMs and zero-/few-shot large LMs. 2. Instruction tuning, not scale, is the key to “zero-shot” summarization.
Interested in studying formal proofs with LLMs? Check out ProofNet, a new benchmark for theorem proving and autoformalization of undergraduate-level mathematics by
@zhangir_azerbay
@haileysch__
and others. Follow-up work is already in progress!
How good are language models at formalizing undergraduate math? We explore this in
"ProofNet: autoformalizing and formally proving undergraduate-level mathematics"
Thread below. 1/n
Congrats to
@StabilityAI
and their collaborators. We are excited to see people continuing to push for non-English non-Chinese LLM research, and thrilled that they're finding our libraries including GPT-NeoX and lm-eval useful! To get started on your next LLM project, check out 👇
Today, we are releasing our first Japanese language model (LM), Japanese StableLM Alpha.
It is currently the best-performing openly available LM created for Japanese speakers! ↓
@databricks
GPT-J-6B might be “old” but it’s hardly slowing down. Month after month it’s among the most downloaded GPT-3 style models on
@huggingface
, and no billion+ param model has ever come close (“gpt2” is the 125M version, not the 1.3B version).
@BigscienceW
This is just the beginning of our work on non-English and multilingual NLP. We have a 6B Korean model currently training, and plans to expand to East Asian and Nordic language families next! Keep an eye on our GitHhub or stop by
#polyglot
on our Discord!
As access to LLMs has increased, our research has shifted to focus more on interpretability, alignment, ethics, and evaluation of AIs. We look forward to continuing to grow and adapt to the needs of researchers and the public
Check out our latest work at
EleutherAI is excited to collaborate with NIST in its newly formed AI Safety Institute Consortium (AISIC) to establish a new measurement science for safe AI systems. See the official announcement here:
#AISIC
@NIST
@CommerceGov
RWKV substantially lags behind S4 on the long range arena benchmark, as well as subsequent work by
@_albertgu
et al.
@HazyResearch
such as SGConv and Mamba. It remains to be seen if that's a killer for NLP applications. Note that the scores are nearly identical for the text task.
Very cool work!
@_albertgu
has been pushing on state-space models for some time now and the release of billion-parameter scale models is a big step forward for this line of work.
We look forward to the community testing the models out!
Quadratic attention has been indispensable for information-dense modalities such as language... until now.
Announcing Mamba: a new SSM arch. that has linear-time scaling, ultra long context, and most importantly--outperforms Transformers everywhere we've tried.
With
@tri_dao
1/
Just a few days into
@huggingface
and
@LambdaAPI
's Whisper fine-tuning event and have already seen huge breakthroughs in multilingual ASR. Very smart people working on this. Here's the SOTA whisper-based module I fine-tuned for Portuguese 🇧🇷🇵🇹
We were very happy with the reception to our researchers
@lintangsutawika
and
@haileysch__
's ICML tutorial, "Challenges in LM Evaluation", this past week!
For all those who requested it, the slides are now available at . Enjoy!
We are thrilled to share the latest in our collaboration with
@EnricoShippole
and
@NousResearch
on sequence length extension.
We're now pushing sequence lengths that will enable work in malware detection and biology that are currently hamstrung by sequence length limitations!
Transparency about whose data is contained in datasets is an essential first step towards establishing meaningful provenance and consent. We applaud
@BigCodeProject
's efforts in this regard and look forward to implementing similar techniques for our future datasets.
The first RNN in transformers! 🤯
Announcing the integration of RWKV models in transformers with
@BlinkDL_AI
and RWKV community!
RWKV is an attention free model that combines the best from RNNs and transformers.
Learn more about the model in this blogpost:
Even after only 55% of the training,
@BlinkDL_AI
’s multilingual RWKV “World” model is the best open source Japanese LLM in the world!
Check out the paper:
Code and more models can be found at:
The JPNtuned 7B
#RWKV
World is the best open-source Japanese LLM 🚀Runner: Model (55% trained, finishing in a few days): More languages are coming🌍RWKV is 100% RNN
If you’re attending
#ACL2023NLP
or
#icml2023
don't miss our seven exciting papers on crosslingual adaption of LLMs, the Pythia model suite, novel training methodologies for LLMs, data trusts, and more!
🧵
Interested in practical strategies to continually pre-train existing models on new data? Take a look at the recent paper between
@AiEleuther
and
@irinarish
's CERC lab as a part of our joint INCITE grant!
Interested in seamlessly updating your
#LLM
on new datasets to avoid wasting previous efforts & compute, all while maintaining performance on past data? Excited to present Simple and Scalable Strategies to Continually Pre-train Large Language Models! 🧵 1/N
The world has changed quite a lot since we first got started. When EleutherAI was founded, the largest open source GPT-3-style language model in the world had 1.5B parameters. GPT-3 itself was not available for researchers to study without special access from OpenAI, and
This is deeply necessary work and a heroic effort by Shayne et al.
"This is the best NLP data work of 2023."
@BlancheMinerva
"If there's anything less glamorous yet higher-impact in ML than looking at the data, it's doing due diligence on licensing."
@haileysch__
📢Announcing the🌟Data Provenance Initiative🌟
🧭A rigorous public audit of 1800+ instruct/align datasets
🔍Explore/filter sources, creators & license conditions
⚠️We see a rising divide between commercially open v closed licensed data
🌐:
1/
Interested in our recent paper "LLeMA: An Open Language Model For Mathematics"? Check out this summary by
@unboxresearch
Or dig into our work directly
Paper:
Code:
I can imagine a future where advanced mathematics has completely changed. What makes math challenging today is the ability to learn abstract technical concepts, as well as the ability to construct arguments that solve precise logical problems.
[article: ]
It was great to see a lot of excitement about attention-free models
@NeurIPSConf
! We had great conversations with many people interested in next-gen architectures for language models.
Pic from Systems for foundational models and foundation models for systems by Chris Re
@arankomatsuzaki
has graduated from explaining other peoples’ papers in Discord and on Twitter to doing it at conferences when the author misses their poster session
Aran Komatsuzaki giving walkthroughs of the codeRL paper before the author arrives. After 10 minutes of SBFing his way into answering poster questions he revealed he was not the author and everyone lost their mind (Poster 138
#NeurIPS2022
)
Benchmark results show that the models have performance comparable to or better than the best publicly available Korean language models, including Facebook's 7.5B xGLM and Kakao Brain's 6.0B koGPT model. We do not show
@BigscienceW
's BLOOM models as they are not trained in Korean
EleutherAI is blessed to have ungodly amounts of compute for a research non-profit. Part of that blessing though is a responsibility to develop things that are interesting and useful not just to us, but to the many researchers who wouldn’t have been able to do this themselves.
We are currently using these models to investigate a variety of phenomena (expect initial papers within the month!), but are making the models public now because we believe that these models will be widely useful to the NLP community writ large and don't want to make others wait
We present the first compute-optimal scaling laws analysis of a large RNN, finding highly predictable scaling across runs. Unfortunately we don't sample densely enough to estimate the optimal token-per-parameter, but we plan to in future work.
Excellent and timely reminder from the FTC. The question, as always, is whether the USG will be able to bring itself to levy meaningful penalties that actually deter illegal behavior by companies.
h/t:
@emilymbender
We are also introducing DeeperSpeed v2.0, which will be synced with the latest upstream DeepSpeed. It also provides GPT-NeoX-specific bugfixes and features and additional optimizations specific to EleutherAI's HPC providers (
@StabilityAI
@CoreWeave
@ORNL
)
Parameter-efficient methods revolutionized the accessibility of LLM fine-tuning, but can they do pre-training? Today at NeurIPS Workshop on Advancing Neural Network Training we present ReLoRA — the first PEFT method that can be used for LLMs at scale!
We envision a world where "safety" isn't dictated by model developers but is something that downstream deployers have agency over. For a small step in this direction, check out the latest work by
@lcastricato
@haileysch__
@BlancheMinerva
and their collaborators.
PINK ELEPHANTS! 🐘 Now, don’t think about it.
Chatbots also find this supremely difficult.
Ask one of the most popular open source models NOT to talk about pink elephants, and it will fail 34% of the time.
In our new paper, we address this problem.
1/N
We just crossed 100,000 organizations on HF!
Some of my favorites:
- The MLX community for on-device AI:
- The
@AiEleuther
org with over 150+ datasets:
- The
@Bloomberg
org to show big financial institutions can use the hub:
The biggest issue with the FMTI is that it's not what it purports to be: instead of focusing on transparency, most of the questions are closer to "being a good product." An extremely transparent LLM can score as low as 30/100 on the FMTI! See how here:
If you're interested in eliciting and editing knowledge in neural networks, don't miss
@norabelrose
's talk on her recent and up-coming research. These ideas form one of the core interpretability research areas at EleutherAI.
Thank you to
@norabelrose
who gave an engaging presentation on Concept Erasure and Elicit Latent Knowledge to our open science community this week. ✨
Thanks
@oohaijen
and
@jonas_kg
for hosting.
📹 Catch the replay here
Instead, it shoehorns questions about "impact", "risks" and "mitigation" under the umbrella of "transparency." These are important things, certainly. But they're not transparency and pretending they are muddles the conversation about responsible AI.
RWKV isn’t without its flaws. While we do approximately match the performance of transformers, our anecdotal experience is that it’s more sensitive to prompts and struggles to incorporate very long range information more than traditional transformers do.
A couple months ago, EAI started an internal speaker series for alignment researchers. Checkout the transcript and slides of the first talk, featuring Steve Byrnes here:
We're thrilled to be sponsoring
@cv4ecology
with >12,000 A6000-hours of compute.
Sharing innovations in ML beyond "core ML" applications is an essential and underfunded job that we are proud to play a part in. The world needs better ecological work more than another LLM.
Week 1 (of 3) in the books at
#CV4Ecology2023
!
We're working hard while still enjoying the California sunshine, and celebrating our wins as they come 😀!
@QuentinAnthon15
@BlancheMinerva
@haileysch__
This is the first in a series of blog posts on implementation details for large scale distributed DL that are far too often skimmed over in papers and articles. Stay tuned for more, including how to choose your parallelization and a deeper dive on FLOPs, latency, and perf metrics
New feature in the
@AiEleuther
's Harness (lm-eval): chat templates for
@huggingface
models!
You can now:
- evaluate chat models fairly, in a turn-by-turn fashion
- and specify system prompts to tailor the model behavior!