I'm proud to share our latest work on object detection using transformers.
Check it out! Our github features a colab with minimal implementation to play with. (1/4)
Blog post:
Code:
Paper:
We are releasing Detection Transformers (DETR), an important new approach to object detection and panoptic segmentation. It’s the first object detection framework to successfully integrate Transformers as a central building block in the detection pipeline.
Happy to present our work on generalization in Multi-Agent RL at
@NeurIPSConf
next week. Spotlight at 4:35pm in track 3 and Poster
#194
on Tuesday.
Paper:
Blog post:
Source code:
"A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning" by
@alcinos26
et al. is going to be presented at NeurIPS next week, come talk to us!
Wow, someone made a web demo of DETR! Happy to see people playing around with it already 🙂 You can see for yourself that adding NMS doesn't improve DETR's detections.
@plotlygraphs
We were super impressed by the new Detection Transformers (DETR) library, so we made this Dash GUI from start 2 finish in ~200 lines of Python
Code:
Demo:
cc:
@PyTorch
@facebookai
@szagoruyko5
@fvsmassa
Awesome work by
@HugoTouvron
@quobbe
@alexsablay
@fvsmassa
@hjegou
, M. Douze to ablate and democratize transformers for image classification. No need for extra data, augmentation+distillation suffices! Comes with code built on
@wightmanr
's excellent Timm library. Check it out!
I'm happy to share our latest work on training competitive Visual Transformers using only ImageNet.
Code and pre-trained models:
Paper:
Check it out! (1/2)
Check out our latest work on multimodal understanding through modulated detection. We extend DETR to accept text as input, and show that this allows to tackle downstream vision+language tasks. 1/4
Excited to share our work on Modulated Detection for End-to-End Multi-Modal Understanding. MDETR detects any objects that are mentioned in free-form text. 1/5
Website:
Code:
Paper:
Colab:
In Panoptic segmentation, our solution treats "things" (countable foreground objects) and "stuff" (background classes) exactly the same way. This novel unified approach allows the transformer to globally reason about both kind of entities. (4/4)
We provide an end-to-end pipeline competitive with Faster-RCNN, without needing any custom layers or prior knowledge (NMS, anchors, ROI-align,…), in true deep-learning spirit. We obtain 42 AP with the same number of parameters, inference time, and half the compute (GFLOPS)(2/4)
If you work on fine grained vison-language understanding or open world detection, check out our new benchmark TRICD! As the name implies, we expect it to fool the best current models 🙂
Check out our new fine-grained vision and language understanding task (CPD) and associated benchmark - TRICD! 📢
Contextual Phrase Detection (CPD) is a single task that subsumes object detection and phrase grounding.
Paper:
Website:
We show that the architecture and formulation can be easily extended to related tasks, such as panoptic segmentation where we achieve SOTA performance. We hope that a fully differentiable, easy to use detection pipeline will also benefit other downstream tasks (eg. VQA) (3/4)
Amazing work from my colleagues at FAIR! A new open source (Apache 2.0) segmentation model that can segment literally *anything* (try the demo!) as well as a dataset of 1B (!!) segments on 10M images. By
@kirillov_a_n
@nikhilaravi
and many others
Today we're releasing the Segment Anything Model (SAM) — a step toward the first foundation model for image segmentation.
SAM is capable of one-click segmentation of any object from any photo or video + zero-shot transfer to other segmentation tasks ➡️
Built using awesome libraries from
@huggingface
and
@wightmanr
on top of
@PyTorch
, huge props to the open-source ecosystem! It truly accelerates research and enables bolder ideas to be tried out.
@phillip_isola
I find this graph misleading. Stable diffusion itself is trained on billions of paired data. Shouldn't the red curve be shifted by that much to the right?
I just gave this new tool a try, and my first impressions are very good! Could save a lot of time while doing literature review, and potentially uncover unforeseen links between research sub-communities. And the UI is pretty slick. Give it a go and see how it works for you :)
After a long beta, we are launching!
Connected Papers is a unique, visual tool to help researchers and applied scientists find and explore papers relevant to their field of work.
@phillip_isola
Training on SD output is a form of distillation. Similar to your argument, clip models trained on LAION also already exist. Is there any evidence that distillation from a generative model as you did is superior to distilling from a contrastive one, eg the aforementioned clip?
@tdietterich
@gileshooker
What about dropping the paper format, and have the contributions look like Github Pull-Requests to said wiki? Would force authors to insert their work in existing knowledge without wasting efforts on re-exposing (eg background sect.) facts already better explained somewhere else.
Thank you all for your questions during the Q&A, unfortunately we didn't have time to cover all of them. Join us in the zoom poster session on Wednesday 6-8 am or 2-4 pm (utc+1) if you want to chat.
Last but not least, we show that vision+language pre-training offer a way to tackle the long tail of object detection, and show good performance on the rare categories of LVIS dataset, even with a fraction of the data.
You can play around with your own images in our colab! 4/4
We show significant improvement over state-of-the-art on various tasks, such as Referring Expression Comprehension/Segmentation and grounded detection on Flickr30k Entities (+12% accuracy over previous SOTA) 2/4
We also fine-tune it on GQA, and obtain competitive performance. Notably, the detection of the objects mentioned in the question offers some insights on the model's reasoning process. In the illustration, the question was "What color is the train?" 3/4
@wightmanr
@NielsRogge
@facebookai
ViT and variants struggle with the kind of resolutions we need in detection. SWIN, on the other hand, really delivers both with DETR variants and otherwise. See for eg
I'm hiring Master interns at FAIR Paris to work on code generation, to work with me and the awesome CodeGen team (
@b_roziere
,
@jadecopet
,
@jnsgehring
,
@adiyossLC
et al.). We do Code Llama and research. Candidate at and send me an email or message.
@giffmana
RL for Object Detection has been around for a while. You might want to cite "Learning Globally Optimized Object Detector via Policy Gradient" Rao et al,CVPR 2018 but IIRC there are others.
@francoisfleuret
@chriswolfvision
You still need to aggregate information from all nodes, hence infinite depth to handle arbitrarily long sequences. See for a set of tasks where vanilla transformers do poorly. Ironically, the proposed fix makes transformer look like RNNs again.
@drew_jaegle
@gberta227
@icmlconf
Nice work! I'd like to point out that the arch is quite similar to our decoder in DETR, with the same high level goal (dim reduction), your latent array being ~equivalent to our object queries. Maybe you could consider mentioning it in the related work if you publish a revision?
@drew_jaegle
@gberta227
@icmlconf
Woops, my phone had cached the previous revision of the paper 😅 But I feel you, I wish related work and acknowledgements didn't count towards the page limit, I don't see any good reason to limit content in those.
@quocleix
@hieupham789
@lmthang
@ZihangDai
@QizheXie
Impressive results! I'm curious about the use of hard pseudo labels in this work, as it intuitively makes backprop more challenging. The NoisyStudent paper claimed hard and soft perform equally well, do you have any intuition why it's not the case here?
@donglixp
@giffmana
@NielsRogge
@OpenAI
That's interesting, thanks for sharing! I'm curious, have you also tried non-parametric methods (maybe Kmeans?) to tokenize the patches?
@giffmana
It would interesting to investigate if the rl step qualitatively changes the predictions (does it meaningfully move the boxes? Does it improve calibration?) or if it's just hacking AP metric(eg spamming boxes always improve AP without improving "quality" or downstream usefulness)
@srush_nlp
@henripal
It doesn't help that the "official" pytorch named_tensors are a barely usable implementation that requires verbose ugliness to workaround the rough edges and glaring holes... Sadly the project seems mostly abandoned