tomaarsen @tomaarsen Twitter profile | Pikagi

Pikagi

tomaarsen

@tomaarsen

1,521

Followers

192

Following

94

Media

403

Statuses

Sentence Transformers, SetFit & NLTK maintainer Machine Learning Engineer at 🤗 Hugging Face

Netherlands

https://t.co/Y4DfYcksyl

Joined December 2023

Don't wanna be here? Send us removal request.

Pinned Tweet

@tomaarsen

tomaarsen

4 months

‼️Sentence Transformers v3.0 is out! You can now train embedding models with multi-GPU training, bf16 support, loss logging, callbacks & much more. I also release 50+ datasets to train on & much more. Learn how to use the new Trainer here: Details in 🧵

Tweet card media

Training and Finetuning Embedding Models with Sentence Transformers v3

9

101

418

Last Seen Profiles

@helpme_thebigt

@MadeBuyAI

@Asanagi_Shione

@kellethebrat

@queen__2

@petanque0

@rajmasudforhad

@lRgIxtHLPl33052

@NeeKoos2

@afrouze

@yr_mtks

@dul_turkporno

@emillycomvc22

@thomasladegard

@Shwg_2030

@HeiressSarada

@sfuhellenic

@Roberto24397701

@azozsaaf

@evilAUX

@thisisalwaysit

@petervanee

@thisisalwaysit

@Damianstr8

@DorisLolle16528

@bokeplokalmalam

@menjibar4

@bokeplokalmalam

@popdouglasTD

@Igrou_31

@ohhstone_

@Estelle__K74

@eimikosmoothie

@OxfordByzantine

@WillHen20006463

@AliElmouss31416

@tomaarsen

tomaarsen

6 months

Embedding Quantization is here! 25x speedup in retrieval; 32x reduction in memory usage; 4x reduction in disk space; 99.3% preservation of performance🤯 The sky is the limit. Read about it here: More info in 🧵

Tweet card media

Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval

5

93

375

@tomaarsen

tomaarsen

7 months

🔥Sentence Transformers v2.4.0 is released! It introduces Matryoshka Embedding models (training & inference), 2 new state-of-the-art loss functions, prompt templates, instructor model support & more. See the🧵

6

93

375

@tomaarsen

tomaarsen

4 months

78.17% -> 93.40% accuracy by finetuning an embedding dataset on a small synthetic dataset with Sentence Transformers v3! Great work 👏

@vanstriendaniel

Daniel van Strien

@vanstriendaniel

4 months

Thanks to @tomaarsen 's work in the latest Sentence Transformers release, training custom models is easier than ever. With improved training support and synthetic data for fine-tuning, you can build a model in less than a day. Example here👇:

Tweet media one

1

8

77

0

11

144

@tomaarsen

tomaarsen

7 months

Biggest release of the week: just released 'mxbai-embed-large-v1', a top embedding model outperforming all equivalently sized models such as 'bge-large-en-v1.5'. It's Apache-2.0 licensed, i.e. commercially viable. Model link: 🧵1/5

Tweet media one

3

22

140

@tomaarsen

tomaarsen

10 months

Thrilled to share that I've joined @huggingface 🤗 as a Machine Learning Engineer tasked with maintaining the awesome Sentence Transformers project! It's high time to bring modern training functionality to finetuning embedding models!⚡️

5

11

125

@tomaarsen

tomaarsen

6 months

GLiNER has new Apache 2.0 models for efficient, cheap and high quality information extraction. > 2 English models: small (152M) and medium (195M) > 1 Multilingual model (288M) > 25x-50x smaller and faster than any 7b LLM > 3 lines of code Demo on CPU:

Tweet media one

3

33

123

@tomaarsen

tomaarsen

8 months

The long-awaited Sentence Transformers v2.3.0 is now released! It contains a ton of bug fixes, performance improvements, loading custom models, more efficient loading, a new strong loss function & more! Check out the release notes: Or this 🧵below:

Tweet card media

Release v2.3.0 - Bug fixes, improved model loading & Cached MNRL · UKPLab/sentence-transformers

This release focuses on various bug fixes & improvements to keep up with adjacent works like transformers and huggingface_hub. These are the key changes in the release: Pushing models to the Hu...

5

24

107

@tomaarsen

tomaarsen

6 months

Alongside our blogpost on Embedding Quantization, we released a useful demo showcasing that it allows for <0.1s retrieval across all of Wikipedia (41 million texts) while using 32x less memory than normal retrieval. E.g. <10GB memory rather than 160GB 💸

Tweet media one

3

9

96

@tomaarsen

tomaarsen

6 months

Big update for the Massive Text Embedding Benchmark (MTEB) intended to simplify finding a good embedding model! Model filtering, search, memory usage, model size in parameters. The updated leaderboard: Details in 🧵:

Tweet media one

2

22

91

@tomaarsen

tomaarsen

4 months

@1littlecoder My take on this

Tweet media one

5

3

88

@tomaarsen

tomaarsen

4 months

Synthetic data used to improve an embedding model from 78.1% accuracy -> 93.4% accuracy with Sentence Transformers finetuning! Learn how to do this yourself ⏬ Great work, Daniel!

@vanstriendaniel

Daniel van Strien

@vanstriendaniel

4 months

Do you need a dataset to train a custom sentence transformer model? I've created a pipeline for using an LLM to create a synthetic dataset you can directly use for fine-tuning/training a Setence Transformers model. *Link in next tweet

Tweet media one

3

19

102

5

9

84

@tomaarsen

tomaarsen

7 months

released 3 state-of-the-art open text reranker models: fully Apache 2.0 & outperforming current top models such as bge-reranker-large and cohere-embed-v3 on BEIR datasets. All models are ready to use on the @huggingface Hub! More details in 🧵

Tweet media one

3

18

65

@tomaarsen

tomaarsen

6 months

Snowflake has just shook up the MTEB Retrieval leaderboard with 5 new model releases: > 23, 33, 110, 137 & 335M parameters > Apache 2.0 license > SOTA performance for their sizes/speeds > 512 & 8192 seq length Technical report coming soon. Model link:

Tweet media one

3

8

60

@tomaarsen

tomaarsen

6 months

2 new tiny Apache 2.0 reranker models just got released by @JinaAI_ . Despite their small size/latency, they perform competitively on benchmarks, reportedly outperforming bge-reranker-base and mxbai-rerank-base on MTEB Retrieval. Models: Details in 🧵

Tweet media one

3

15

60

@tomaarsen

tomaarsen

4 months

Sentence Transformers just reached 14k stars! Just in time for the upcoming v3.0 update 👀It'll be the biggest update since the inception of the project. More details coming soon!

Tweet media one

0

0

43

@tomaarsen

tomaarsen

4 months

Just published Sentence Transformers v3.0.1: the first patch release since v3 from last week. It introduces gradient checkpointing, pushing model checkpoints to Hugging Face while training, model card improvements and fixes. Release notes: Details in 🧵

Tweet card media

Release v3.0.1 - Patch introducing new Trainer features, model card improvements and evaluator...

This patch release introduces some improvements for the SentenceTransformerTrainer, as well as some updates for the automatic model card generation. It also patches some minor evaluator bugs and a ...

1

2

37

@tomaarsen

tomaarsen

6 months

Sentence Transformers v2.7.0 is out! Featuring a new loss function, easier Matryoshka model inference & evaluation, CrossEncoder improvements & Intel Gaudi2 Accelerator support. Release notes: Or read the details in 🧵

Release v2.7.0 - CachedGISTEmbedLoss, easy Matryoshka inference & evaluation, CrossEncoder, Intel...

This release introduces a new promising loss function, easier inference for Matryoshka models, new functionality for CrossEncoders and Inference on Intel Gaudi2, along much more. Install this versi...

1

8

37

@tomaarsen

tomaarsen

6 months

The last Sentence Transformers release introduced GISTEmbedLoss by @avsolatorio , which allows for training models that outperform those trained by the wildly popular in-batch negatives loss (MultipleNegativesRankingLoss). Learn about it in this 🧵(links at the bottom):

Tweet media one

3

8

33

@tomaarsen

tomaarsen

2 months

Recently, @mixedbreadai and @deepset_ai collaborated on a SOTA German text embedding model, outperforming multilingual-e5-large and jina-embeddings-v2-base-de. Link: Details: - 478M parameters: small enough to run on CPU and GPU 🧵

Tweet media one

1

7

34

@tomaarsen

tomaarsen

10 months

📈 There's almost 4000 open source Sentence Transformers models on the Hugging Face Hub right now! Open source for the win❤️

Tweet media one

5

8

34

@tomaarsen

tomaarsen

3 months

Absolutely loving the flexibility of Sentence Transformers v3 for training embedding models - allows for much easier paper reproductions.

Tweet media one

2

2

29

@tomaarsen

tomaarsen

10 months

🤗 The long-awaited full release of SetFit is finally out! SetFit v1.0.0 brings an all-new Trainer, TrainingArguments, logging, evaluation, integrations, callbacks, model cards, docs & more! 1/6

1

9

27

@tomaarsen

tomaarsen

7 months

Matryoshka embedding models can produce useful embeddings of various dimensions, which can heavily speed up downstream tasks like retrieval (e.g. for RAG). Check out our blogpost with all of the details:

Tweet card media

🪆 Introduction to Matryoshka Embedding Models

1

3

20

@tomaarsen

tomaarsen

4 months

You can now finetune embeddings models with Sentence Transformers & AutoTrain without writing any code! Works locally, in Google Colab, on any cloud or via Hugging Face Spaces. Check it out!

@abhi1thakur

abhishek

4 months

🚨 NEW TASK ALERT 🚨 AutoTrain now supports fine-tuning of sentence transformer models 💥 Now, you can improve and customize your RAG or retrieval models without writing a single line of code 🤗 ✅ Supports multiple types of sentence transformers training and finetuning ✅ CSV /

Tweet media one

3

12

50

1

4

19

@tomaarsen

tomaarsen

3 months

I'm absolutely loving these new dataset size markers on @huggingface datasets 👏

Tweet media one

1

3

20

@tomaarsen

tomaarsen

5 months

@bo_wangbo @JinaAI_ Also, I would like to implement ColBERT training into Sentence Transformers (based on the HF Trainer with MultiGPU, bf16, callbacks + integrations, useful model cards, etc.), so I'm looking forward to your findings & promising loss functions there.

1

2

19

@tomaarsen

tomaarsen

6 months

@huggingface @mixedbreadai This quantization is included in today's v2.6.0 Sentence Transformers release. Read the release notes here:

Tweet card media

Release v2.6.0 - Embedding Quantization, GISTEmbedLoss · UKPLab/sentence-transformers

This release brings embedding quantization: a way to heavily speed up retrieval & other tasks, and a new powerful loss function: GISTEmbedLoss. Install this version with pip install sentence-tr...

1

1

16

@tomaarsen

tomaarsen

7 months

We also reduced the dependencies & made many more small changes. See the release notes for all of this information in more detail: I'm looking forward to seeing your models pop up on the Hub!🤗See you in the next release!

Release v2.4.0 - Matryoshka models, SOTA loss functions, prompt templates, INSTRUCTOR support ·...

This release introduces numerous notable features that are well worth learning about! Install this version with pip install sentence-transformers==2.4.0 MatryoshkaLoss (#2485) Dense embedding mode...

1

0

16

@tomaarsen

tomaarsen

10 months

SetFit is extremely well suited for zero-shot text classification, and often outperforms much larger (and slower) zero-shot models on the Hugging Face Hub! Check out our new how-to guide for zero-shot text classification here:

Tweet media one

1

2

15

@tomaarsen

tomaarsen

7 months

CoSENTLoss is a new drop-in replacement of the popular CosineSimilarityLoss, while produces a stronger training signal. AnglELoss is another variant which uses a different similarity function to avoid vanishing gradients. See the loss docs for more info:

1

0

15

@tomaarsen

tomaarsen

7 months

@mixedbreadai released a🪆2D Matryoshka text embedding model. Such models have two notable properties: Adaptable embedding size & adaptable layer counts. It allows you to speed up both inference & all post-processing (e.g. retrieval). Model link: See 🧵

Tweet media one

1

4

14

@tomaarsen

tomaarsen

4 months

An excellent training script for embedding models via the new Sentence Transformers v3 by @mrm8488 . Try it out yourself!

@mrm8488

Manu Romero

4 months

🚀 Just out: Sentence-Transformers 3 is transforming the game! Kudos to @tomaarsen for the stellar update. 🌟 🔥 NEW FEATURE: Train your own Matryoshka embedding models! Want to dive in? I've set up a Colab notebook to get you started right away. Check it out and start creating

0

12

83

1

3

13

@tomaarsen

tomaarsen

6 months

@huggingface @mixedbreadai The future of search is int8 & binary.

1

0

12

@tomaarsen

tomaarsen

10 months

The new SetFit v1.0.0 release also brings SetFitABSA: Few-Shot Aspect Based Sentiment Analysis! ABSA is like Sentiment Analysis, except it tells you which parts people were happy/unhappy about, it's extremely useful! Check out the blogpost: 1/3

Tweet card media

SetFitABSA: Few-Shot Aspect Based Sentiment Analysis using SetFit

1

3

12

@tomaarsen

tomaarsen

6 months

The Massive Text Embedding Benchmark (MTEB) is being extended to become massively multilingual. Everyone is invited to contribute & co-author an upcoming publication. 📜 Details:

@KCEnevoldsen

Kenneth Enevoldsen

6 months

🚀 Exciting News! We're launching MMTEB, the Multilingual Massive Text Embedding Benchmark. A community initiative to make text embeddings more inclusive & diverse. Join us in expanding the coverage of NLP to a wide range of languages! 🌍 #MMTEB #NLP

2

8

42

0

3

12

@tomaarsen

tomaarsen

4 months

Since yesterday's Sentence Transformers v3.0 update, distributed training of embedding models (for RAG, retrieval, semantic similarity, etc.) is now a breeze. You can expect some serious speedups when scaling the number of GPUs. Usage in 🧵

Tweet media one

1

1

12

@tomaarsen

tomaarsen

6 months

. @huggingface and @mixedbreadai announce embedding quantization: a post-processing technique for embeddings that results in massive cuts in costs for retrieval. E.g., rather than needing 200GB, we can search Wikipedia in this demo with just 5GB of RAM: 🧵

Tweet card media

Quantized Retrieval - a Hugging Face Space by sentence-transformers

2

2

12

@tomaarsen

tomaarsen

7 months

@1littlecoder Embedding models are explicitly trained such that cosine similarity becomes a strong measure of semantic similarity. For all real-world embedding models, the findings of this paper do not apply at all. You can keep safely using cosine similarity.

1

0

11

@tomaarsen

tomaarsen

7 months

Recently, consensus has developed that larger sequence lengths result in notably worse embeddings, so this model uses a more reasonable 512. 🧵3/5

1

1

10

@tomaarsen

tomaarsen

4 months

@n0riskn0r3ward @nvidia To not get you too excited, there are a few concerns with this model at this point. The 1st place is mostly due to its high score on classification (87.35 vs 81.49 for #2 ), which is because it scores unexpectedly high on a few of the datasets, notably the EmotionClassification.

Tweet media one

2

0

11

@tomaarsen

tomaarsen

4 months

AutoTrain now supports finetuning embedding models using Sentence Transformers! With other words: embedding models for your data without having to write any code. Details: More in 🧵

Tweet card media

How to Fine-Tune Custom Embedding Models Using AutoTrain

1

1

10

@tomaarsen

tomaarsen

8 months

🔥By applying optimum-intel we can get a 3.5x increase in throughput for SetFit text classification models on CPUs. It applies quantization using the Intel Neural Compressor (INC), resulting in higher throughput on CPUs than with torch on GPUs. Notebook:

Tweet media one

2

2

10

@tomaarsen

tomaarsen

4 months

This v3.0 release has been the biggest in the Sentence Transformers history (13k lines changed, 292 files updated), and I'm very excited to see it come to fruition. I'm very much looking forward to seeing your finetuned models on @huggingface 🧵

1

1

9

@tomaarsen

tomaarsen

4 months

@eugeneyan It seems rather challenging to access the underlying ClueWeb22 dataset (), but I would love love love to get this dataset on the Hub and in here:

Tweet card media

Embedding Model Datasets - a sentence-transformers Collection

1

1

10

@tomaarsen

tomaarsen

5 months

@orionweller @srchvrs @n0riskn0r3ward @spacemanidol @memray0 @Quantum_Stat @bo_wangbo Sentence Transformers v3 will completely overhaul the training loop. I'm sure it'll include just about anything you'd need, from Multi-GPU w. DDP, GradCache losses, W&B/Tensorboard integration, extensive model card generation, FA2 on various models (more coming soon), etc.

1

0

9

@tomaarsen

tomaarsen

4 months

@jobergum My favourite part: The all-new automatically generated model cards:

tomaarsen/mpnet-base-all-nli-triplet · Hugging Face

2

0

9

@tomaarsen

tomaarsen

5 months

@bo_wangbo @JinaAI_ Looking forward to those papers! Especially your plans for v3 & collecting parallel data. On the topic of the latter, I just reformatted 10 Parallel Sentence datasets for easy use with Sentence Transformers v3:

Tweet card media

Hugging Face – The AI community building the future.

1

0

8

@tomaarsen

tomaarsen

7 months

We also support Prompt Templates now! Useful for those models that always need prompts before the text (e.g. "query: ..." or "Represent this sentence for searching relevant passages: "). Learn more about it here:

1

0

9

@tomaarsen

tomaarsen

4 months

@_philschmid Link to docs 🤗:

0

0

9

@tomaarsen

tomaarsen

4 months

... - Improved callback support + an excellent Weights & Biases integration - Gradient checkpointing, gradient accumulation - Improved model card generation - Resuming from a training checkpoint without performance loss - Hyperparameter Optimization and much more! 🧵

Tweet media one

1

1

7

@tomaarsen

tomaarsen

4 months

5️⃣ Dataset Release To help you out with finetuning models, I've released 50+ ready-to-go datasets that can be used with training or finetuning embedding models. Check them out here: 🧵

Tweet card media

Embedding Model Datasets - a sentence-transformers Collection

1

0

8

@tomaarsen

tomaarsen

4 months

@huggingface And stay tuned for future updates, I've got big plans and plenty of motivation to make 'em happen. Check out the repository to keep up to date or to submit issues/feature requests/pull requests:

Tweet card media

GitHub - UKPLab/sentence-transformers: State-of-the-Art Text Embeddings

State-of-the-Art Text Embeddings. Contribute to UKPLab/sentence-transformers development by creating an account on GitHub.

0

0

8

@tomaarsen

tomaarsen

6 months

🔍 Time for some sneak previews! Soon, models trained/finetuned with Sentence Transformers will automatically include detailed model cards! In this 🧵I'll show what's included: - Model Details, e.g. base model, sequence length, output dimensionality, training datasets, language.

Tweet media one

1

1

8

@tomaarsen

tomaarsen

7 months

Additionally, we now support the popular INSTRUCTOR models, such as . Check out the documentation on how to use these models:

Tweet card media

hkunlp/instructor-large · Hugging Face

1

0

8

@tomaarsen

tomaarsen

7 months

@llama_index @LangChainAI @deepset_ai Check out the release blogpost by @mixedbreadai to learn more details: 🧵5/5

Tweet card media

Open Source Strikes Bread - New Fluffy Embedding Model - Blog

Our English embedding model provides state-of-the-art performance among other efficiently sized models. It outperforms closed source models like OpenAI's text-embedding-v3.

www.mixedbread.ai

0

2

7

@tomaarsen

tomaarsen

9 months

SetFit v1.0.2 is out to fix some v1.0 release bugs: incorrect model cards when using custom metrics, multi-output mixed with predict_proba, the "unique" sampler, and predicting polarities of gold aspect spans in SetFit ABSA models. Check the repo here:

Tweet card media

GitHub - huggingface/setfit: Efficient few-shot learning with Sentence Transformers

Efficient few-shot learning with Sentence Transformers - huggingface/setfit

1

0

7

@tomaarsen

tomaarsen

8 months

📉 A new "Cached" variant of the powerful Multiple Negatives Ranking Loss allows normal hardware to get performance that used to only be viable on multi-gpu clusters. 🐎 Community Detection is now much faster (7x speedup at 500k sentences 🤯)

1

0

7

@tomaarsen

tomaarsen

6 months

Would you look at that, in the meantime @urchadeDS uploaded a third English model: You know models are fresh when they're still being created during the announcements😄

Tweet card media

urchade/gliner_large-v2.1 · Hugging Face

0

0

7

@tomaarsen

tomaarsen

10 months

Extremely excited to have contributed this implementation! I really think Attention Sinks will be huge to bring constant memory & constant fluency to LLMs.

1

0

7

@tomaarsen

tomaarsen

5 months

@jobergum Including ONNX export 😉it's all on the todo-list

1

0

7

@tomaarsen

tomaarsen

6 months

@huggingface @mixedbreadai We can preserve 97% of retrieval performance with scalar (int8) quantization without rescoring (99.3% with rescoring) and 96.45% with binary quantization with rescoring (92.5% without). Note: Rescoring is extremely cheap with this approach. Learn more about it in the blogpost. 🧵

Tweet media one

1

0

6

@tomaarsen

tomaarsen

6 months

@jobergum A shame - they're very useful. Full float32 retrieval just feels like throwing money away now.

1

0

6

@tomaarsen

tomaarsen

4 months

@andersonbcdefg Every day I wake up in fear that I'll be bested by Sentence Transformest

0

0

6

@tomaarsen

tomaarsen

4 months

2️⃣ Similarity Score Not sure how to compare embeddings? you can now use `model.similarity(embeddings1, embeddings2)` to get similarity scores immediately. Model authors can specify a similarity score, so you don't have to worry which function to use. 🧵

Tweet media one

1

0

6

@tomaarsen

tomaarsen

6 months

@huggingface @mixedbreadai This allows you to use much smaller and smaller cloud instances, while decreasing your latency by an order of magnitude! And all that without notable reductions in retrieval accuracy due to a clever rescoring approach by Yamada et al. () 🧵

Tweet media one

1

0

6

@tomaarsen

tomaarsen

6 months

@iamrobotbear @ClementDelangue They do, the new models (v2.1) are all Apache 2.0, which allows for commercial use: - - - -

Tweet card media

urchade/gliner_large-v2.1 · Hugging Face

0

0

6

@tomaarsen

tomaarsen

8 months

⬆ Uploading Models to the HF hub with 'save_to_hub'. ⬇ Downloading Models from the HF hub now downloads only the required files. ⚙ Custom Models (e.g. the Jina AI 8k models) are now loadable with `trust_remote_code=True`.

1

0

5

@tomaarsen

tomaarsen

10 months

@alignment_lab @HaihaoShen @IntelAI @MosheWasserblat @huggingface Definitely:

Tweet card media

Efficient Few-Shot Learning Without Prompts

Recent few-shot methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label-scarce settings. However, they are...

1

1

5

@tomaarsen

tomaarsen

6 months

@LangChainAI I'm very curious about use cases combining the new @SnowflakeDB retrieval models with these new rerankers. I'd be glad to hear community findings on that. I'm also curious to read more about whether the models work well at larger sequence lengths.

0

0

4

@tomaarsen

tomaarsen

4 months

1️⃣ Training Refactor Embedding models can now be trained using an extensive trainer with a lot of powerful features: - MultiGPU Training (Data Parallelism (DP) and Distributed Data Parallelism (DDP)) - bf16 training support; loss logging - Evaluation datasets + evaluation loss 🧵

Tweet media one

4

0

5

@tomaarsen

tomaarsen

7 months

For convenient usage, the model has been integrated with Sentence Transformers. Due to this integration, the model also comes with day 0 support for @llama_index , @LangChainAI , @deepset_ai haystack & many other frameworks for RAG. 🧵4/5

Tweet media one

1

0

4

@tomaarsen

tomaarsen

4 months

And many more changes! See the full release notes here: Additionally, the entire documentation has been overhauled. We recommend that you check it out: 🧵

Release v3.0.0 - Sentence Transformer Training Refactor; new similarity methods; hyperparameter...

This release consists of a major refactor that overhauls the training approach (introducing multi-gpu training, bf16, loss logging, callbacks, and much more), adds convenient similarity and similar...

1

0

5

@tomaarsen

tomaarsen

8 months

As of Sentence Transformers v2.3, you should now be able to effectively push models to the Hugging Face Hub! In this example, I finetuned MPNet-base on the AllNLI dataset using the new Cached MNRL loss. Feel free to have a look at the final model here:

Tweet media one

0

0

5

@tomaarsen

tomaarsen

6 months

@llama_index @LangChainAI @deepset_ai 4️⃣ Model size: Rather than being displayed in GB, we now show the model size in the number of parameters. This is a much more standard metric for measuring model size. 4/🧵

Tweet media one

1

0

5

@tomaarsen

tomaarsen

8 months

🪶 Removed the "torchvision" dependency for a smaller installation. Check out the full release notes here: Expect many more updates in the future as I pick up development of the project!

Tweet card media

Release v2.3.0 - Bug fixes, improved model loading & Cached MNRL · UKPLab/sentence-transformers

This release focuses on various bug fixes & improvements to keep up with adjacent works like transformers and huggingface_hub. These are the key changes in the release: Pushing models to the Hu...

0

0

5

@tomaarsen

tomaarsen

4 months

@jobergum It's turned out to be quite valuable - most users are unaware of the similarity metric options and/or what the model was designed for.

1

0

5

@tomaarsen

tomaarsen

6 months

@prem_k @jobergum Here you go:

Tweet card media

Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval

1

0

5

@tomaarsen

tomaarsen

4 months

@osanseviero I can't complain

Tweet media one

0

1

5

@tomaarsen

tomaarsen

8 months

Just released Sentence Transformers v2.3.1 with a niche bug fix. The bug only affected users that used a model that 1) is local, 2) contains a Normalize module and 3) but does not contain the directory required by the Normalize module. Release notes:

Tweet media one

0

0

4

@tomaarsen

tomaarsen

6 months

@xhluca Certainly. Our demo () uses exact search just to prove that binary search is fast, but using approximate search can certainly 10x the search speed at a tiny cost in performance. We recommend HNSW for big enough use cases, actually.

Tweet card media

Quantized Retrieval - a Hugging Face Space by sentence-transformers

1

1

4

@tomaarsen

tomaarsen

5 months

@jxmnop @orionweller @srchvrs @n0riskn0r3ward @spacemanidol @memray0 @Quantum_Stat @bo_wangbo @Muennighoff Sentence Transformers v3 should indeed allow for DDP, bf16, GradCache & accelerate all at the same time. Should come out soon, all that remains is the docs & example rewrites. Pre-release branch:

0

0

4

@tomaarsen

tomaarsen

6 months

@osanseviero Notably, you cannot use DBRX to improve any other **large** language model, but perhaps that means that you can still use it to e.g. label or generate training data for non-large language models. So, perhaps you can still use DBRX to help finetune classifiers or embedding models.

0

0

3

@tomaarsen

tomaarsen

4 months

@_philschmid Ooh, I can imagine that an embedding model finetuned on this dataset could be valuable.

1

0

4

@tomaarsen

tomaarsen

3 months

@bo_wangbo @Robro612 The MTEB team is also interested in pursuing a MTEB-lite for this exact purpose, e.g. Perhaps there's some room for collaboration?

Paper segment: Task selection · Issue #837 · embeddings-benchmark/mteb

The goal of this segment is to create meaningful benchmark subsets with a minimal set of tasks. I believe the steps are as follows: construct an experimental subset. If people agree I can construct...

2

0

3

@tomaarsen

tomaarsen

7 months

The model is 335M params, equivalent to other -large models like bge-large-en-v1.5, which is still viable on CPU. mxbai-embed-large-v1 outperforms all equivalently sized models on MTEB, and proprietary ones like OpenAI's text-embedding-3-large and Cohere-embed-english-v3.0. 🧵2/5

1

0

2

@tomaarsen

tomaarsen

6 months

@llama_index @LangChainAI @deepset_ai There's more changes planned for the future, also in terms of multilinguality! Stay tuned for that. For now, feel free to explore the leaderboard:

Tweet card media

MTEB Leaderboard - a Hugging Face Space by mteb

0

0

4

@tomaarsen

tomaarsen

6 months

If you missed our blogpost, here it is again: Well worth a read if you're into retrieval; I think this method should not be underestimated.

Tweet card media

Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval

0

1

4

@tomaarsen

tomaarsen

6 months

@Nils_Reimers Looks great; the cost reduction is particularly appealing. I'm also personally interested in the performance of the 3-phase vs binary search + int8 rescoring. Also well done on the thorough release (GitHub; PyPI; Demo; cost breakdown).

0

0

3

@tomaarsen

tomaarsen

3 months

@ArthurCamara @Robro612 @bo_wangbo @nandan__thakur @lukemerrick_ @spacemanidol Sentence Transformers just added some extensive hard negatives mining functionality that should do well for datasets of up to a few million pairs: It might come in useful for you! For example: - -

tomaarsen/natural-questions-hard-negatives · Datasets at Hugging Face

1

0

4

@tomaarsen

tomaarsen

6 months

@llama_index @LangChainAI @deepset_ai 1️⃣ Model filtering: b) You can now filter on model size: Useful for filtering away models that are much too large/slow. 2️⃣ Model search: With ~250 models it can become difficult to find out where certain models stand. Search should make this much easier! 3/🧵

Tweet media one

1

0

4

@tomaarsen

tomaarsen

5 months

@NirantK @huggingface @nomic_ai This is excellent! It should work OOB with the upcoming Sentence Transformers v3 release. with all 5 losses that work with (anchor, positive, negative) triplets from here:

1

0

4

@tomaarsen

tomaarsen

6 months

1️⃣ Model filtering: a) You can now filter on model type: Open, Proprietary and Sentence Transformers compatible. The latter is useful for determining if a model will work with Sentence Transformers, @llama_index , @LangChainAI , @deepset_ai Haystack, etc. 2/🧵

1

0

3

@tomaarsen

tomaarsen

8 months

@davidbstein1957 Only a cool 582 days. Expect the next release to take less long😄

Tweet media one

0

0

3

@tomaarsen

tomaarsen

7 months

The model can be used directly with Sentence Transformers. See also the Sentence Transformer documentation on how to train 2d Matryoshka models: Also, feel free to read the paper that introduced 2d Matryoshka models:

0

0

3

@tomaarsen

tomaarsen

4 months

@hamishogilvy @penberg But if you really want better performance, you can store the int8 versions of the document embeddings. This still saves 4x storage space and gives you essentially all performance. This is used in our demo of quantized retrieval:

Tweet card media

Quantized Retrieval - a Hugging Face Space by sentence-transformers

0

0

3

@tomaarsen

tomaarsen

3 months

@bo_wangbo Looking forward to those!

0

0

3

@tomaarsen

tomaarsen

4 months

@LoubnaBenAllal1 without-twitter

0

0

3

@tomaarsen

tomaarsen

6 months

@JagersbergKnut @huggingface @mixedbreadai Actually, embedding quantization is different than model quantization😉It doesn't speed up inference (like model quantization), but is a post-processing on embedding vectors to convert them from float32 to int8 or binary. But yes, it works on all embeddings, so also on those.

1

0

1

@tomaarsen

tomaarsen

6 months

@n0riskn0r3ward Heya! Apologies, this is an oversight from before I moved the demo. It is supposed to link to

Tweet card media

app.py · sentence-transformers/quantized-retrieval at main

0

0

3