@AlexTamkin
This is from buggy implementations in standard libraries, **not a fundamental mathematical issue**. For large factors, buggy implementations are essentially equivalent to naive subsampling. The libraries will be fixed...eventually.
Attached image from
Ask any graphics or signal processing person: *not antialiasing* and ignoring Nyquist sampling theorem is a bug
pip install antialiased-cnns
import antialiased_cnns
model = antialiased_cnns.resnet50(pretrained=True)
Now your convnet is antialiased 😃
Amazing GenAI results don't just come from thin air! They're a reflection of the underlying training data.
Can we specifically identify the highly influential data?
See our ICCV work on Data Attribution w/
@ShengYuWang6
,
@junyanz89
, A. A. Efros:
1/
I switched groups in my 3rd year, pushed hard but still missed 2 paper deadlines. The whole PhD thing was not looking very good.
After deciding not to submit to CVPR that night, my advisor said "let's chat". I thought I was in trouble...1/2
Overall, I had a really positive PhD experience. Now that I’ve reached 10k followers (!), I’m ready to share my secret:
You must choose an advisor who be kind and not toxic for 5+ years. Oh, and you need decide this based on a few hours of interviews. Don’t pick wrong! 1/
I was included on MIT
@techreview
's Innovators Under 35 list (at a youthful 34.9)!
My appreciation to the wonderful collaborators, mentors, and labmates at
@AdobeResearch
and
@berkeley_ai
for the past decade
1/
"I'm so grateful to my labmates and mentors, who are world-class researchers," shares Senior Research Scientist
@rzhang88
on his name being added to
@techreview
's prestigious list of top Innovators Under 35 for his achievements in gen AI & image forensics.
When I started working on this 5 years ago, I often used family photos to test. My grandma (who recently passed) always got a kick out of it. Funnily, this image started propagating into other people's papers/presentations
Happy we've improved
@Photoshop
Colorize! Try it out
Over here crying my eyes out with this new color changing Neural Photoshop filter.
I colorized my late grandma and grandpa's wedding photo 6 years ago in college. It took me a total of 5hrs to do so. The image below, took 2 minutes with one click of a button
#AdobeMAX
#photoshop
He said, "Richard, you have all the skills to be a great researcher. Don't worry, keep doing what you're doing, and it will all be okay".
A few months later, the colorization paper happened. It all ended up okay. My wonderful advisor and labmates saved my career. 2/2
Somehow, our ECCV'16 project website got popular and started propagating around .
Four years later, I have gotten around to cleaning it up. Hope folks find it helpful: . Originally made by most helpful postdoc ever,
@phillip_isola
Modern convnets ignore the Nyquist sampling criterion, making them unstable. Come see how simple antialiasing can make your net more stable, accurate, and robust! 3pm tomorrow (Wed) in Seaside Ballroom.
#ICML2019
Download the latest
@Photoshop
to play with Landscape Mixer, based on our Swapping Autoencoder NeurIPS '20 work (first author Taesung Park). A lot of work from
@AdobeResearch
+ Neural Filters teams to get it into production!
Come check out pix2pix-zero at
#SIGGRAPH2023
!
Our twist on traditional image translation -- by leveraging pretrained txt2img models, we can define tasks (e.g., cat→dog) on-the-fly!
Talk by
@GauravTParmar
at 2pm today (Mon, 8/7), Petree Hall D
Webpage:
Short on GPUs/time/data/$, but still want to train a GAN?
In "Ensembling Off-the-shelf Models for GAN Training", we show GANs can be "Vision-Aided" with pre-trained nets as discriminators.
A
#CVPR2022
(oral) by
@nupurkmr9
,
@elishechtman
,
@junyanz89
Web:
@jbhuang0604
🤫 Shh
Ok...since we're sharing
- Thu is the best; you get the whole weekend
- Fri is the worst; you don't appear until Sun, when (a) nobody looks (b) it gets wiped to oblivion by busy Mon
Summary
10:59am PT Thu -> best spot on best day
11:01am PT Thu -> worst spot on worst day
@soumithchintala
Fortunately, this is a property of libraries, like
@PyTorch
, not implementing downsampling correctly, not a math/ML issue. This has created some issues in our ecosystem, for example benchmarking GANs . Perhaps we can reexamine + fix 🙂!
Complaining on the internet: I request
@openreviewnet
use normal reviewer numbering (R1, R2, R3) instead of random hashes (8FhW, tCqP, mJaY).
Cross-referencing uninterpretable characters slows down CVPR rebuttals/meta-reviewing, with no discernible benefit, as far as I can tell
I think being an AC helped me write more useful reviews. The CVPR AC training video will help you better understand the process, whether you're an author and/or reviewer:
I don't think we should use "e", like 1e-5 for 10^-5, in papers. My understanding is that it's for calculators limited to 8-segment displays, and it's no longer 1997.
At this week's
#ECCV2020
,
@AdobeResearch
is presenting new work on research topics ranging from shape estimation to image synthesis, image relighting, geometry processing, facial synthesis, and many more!
#ComputerVision
I will talk about "Style and Structure Disentanglement for Image Manipulation" @ AIM workshop
#ECCV2020
Hope to see
- West coast insomniacs: 1:30am PT
- East coast early risers: 4:30am ET
- folks from non-American continents: 930 UTC+1
Rip the image from the pdf & zoom in. We see clear stippling. Ironically, our computers (or eyesight) antialiases it, so we don't see it when zoomed out or at low-res
Hope this is good motivation to learn signal processing 😀
You don't want a random dog. You want *your* dog. Come see how to find and edit it in BigGAN!
Transforming and Projecting Images into Class-conditional Generative Networks
w/ Huh (+his dog)
@junyanz89
@AaronHertzmann
#ECCV2020
@ 4pm PT/7pm ET/00 UTC+1
First in-person talk in 2.5 years! We'll discuss making generative models faster+higher resolution. So see you at the NTIRE (1:30pm Rm 218) and AICC (4:10pm, Rm 208-210)
#CVPR22
workshops.
Come see some 9MPix morphs! Can't see that properly over Zoom or Twitter
Come visit our poster; we are lonely! We will discuss how contrastive learning can help as a structured loss for image translation problems
Webpage:
#ECCV2020
page:
Soon, will we be able to tell synthetic GenAI imagery from real?
In "Online Detection of AI-Generated Images", we replay history, studying if leveraging today's models can help detect tmrw's unseen models
Come check out our
#ICCV2023
Workshop talk, tmrw Oct 2, 8:50am in Rm W07!
@jon_barron
Yea I found it here:
@inproceedings
{barron2021general,
title={A general and adaptive decaying learning schedule},
author={Barron, Jonathan T},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
year={2021}
}
Come check out our
#CVPR
posters tmrw (Thu) morning, 10-12:30!
- 4a: Ensembling Off-the-Shelf Models for GAN Training (*also an oral at 8:30*)
- 76a: Spatially-Adaptive Multilayer Selection for GAN Inversion & Editing
- 77a: On Aliased Resizing & Surprising Subtleties in GAN Eval
We trained an audio distance metric on human perceptual judgments. To my vision friends: lpips for audio
w/ P. Manocha, A. Finkelstein, N. J. Bryan, G. J. Mysore, Z. Jin
@interspeech20
"pip install dpam" to try it!
Talk:
Github:
Or if you want an AI solution (that works worse), convert to grayscale and colorize. I just converted our 2016-17 models from caffe to
@PyTorch
yesterday!
python demo_release.py -i [[image_path.jpg]]
Also see this
#BMVC2020
oral
Delving Deeper into Anti-Aliasing in ConvNets
Zou, Xiao, Yu, Lee
They show antialiasing also helps instance and semantic segmentation, in addition to classification accuracy
Realizing that cleaning old code is probably more useful than writing new (not clean) code
pip install lpips
import lpips
loss_fn = lpips.LPIPS()
loss = loss_fn(img0, img1)
Shared some experiences about working in GenAI at
@berkeley_ai
+
@AdobeResearch
, including
- responding to changes in the past decade
- slowing down to speed up
- coming from a different field
Check out
@techreview
TR35 Festival!
Submit 8-page
@ICCVConference
paper
--> Bloats to 9-page arxiv
--> Squish back down to 8 pages
--> Receive instructions saying camera ready is 9 pages
Doh 😂
Unconditional GANs are now used in the real-world! E.g.,
@Photoshop
Neural Filters. However, they can be slow: StyleGAN2 is 36x more MACs than ResNet50. Our AnycostGAN enables fast previews at lower computational budgets.
From
@jilin_14
(awesome summer intern) et al at
#CVPR2021
Try out the demo and Colab of Anycost GAN: . Our method provides consistent outputs at various computational budgets, paving the way for interactive image synthesis and editing. (w/
@rzhang88
, Frieder Ganz,
@SongHan_MIT
,
@junyanz89
) (1/2)
@jaakkolehtinen
Bilinear downsampling with F.interpolate in PyTorch
(Top) Directly downsampling the original by 2x, 4x, ... 32x aliases
(Bottom) Recursively applying 2x downsampling looks okay
They should be equivalent, but it seems that 2x is safe and other factors are not
Guest appearance in Yagiz Aksoy+Richard Zhang's (
@richardzhangsfu
) Intro to Graphics:
Antialiasing is common practice in classic graphics+vision, but not in modern convnets... yet: pip install antialiased-cnns
Also, the course material is excellent!
I'll be speaking at the
#NeurIPS2023
ML for Creativity & Design Workshop today at 1:30pm CT (11:30am PT).
The talk will be about "Incentivizing Opt-in & Enabling Opt-out for Text-to-Image Models". Come check it out!
2020 was the year in which *neural volume rendering* exploded onto the scene, triggered by the impressive NeRF paper by Mildenhall et al. I wrote a post as a way of getting up to speed in a fascinating and very young field and share my journey with you:
Applications to
@AdobeResearch
2021 fellowship open on 11/12! Join us for an informational webinar on 11/11:
I have collaborated with 2020 winners, Taesung Park and
@TamarRottShaham
, and won back in 2017, which financed my Honda Civic purchase
To nurture the next generation of computer scientists,
@AdobeResearch
annually awards fellowships to PhD students. Learn about 2020 fellows / interns, and watch for 2021 fellowship applications, coming soon!
Will your CVPR 202X synthesis method be detectable? If it uses a convnet, we think so! Give it a try: . I recently ran it on 3 submissions.
#CVPR2020
Oral:
Poster today
@4
-6pm, tmrw
@4
-6am PT
w/ Wang
@oliverwang81
@andrewhowens
Efros
Agreed, paper & presentation are very different mediums!
I redraw plots and build them, step-by-step. I ALWAYS spend ~5-10x longer than I expect 😂
I also learned from
@CarlDoersch
to manually animate bar plots in ppt
How to present a line plot?
Line plots are effective for describing the relationship between two variables of interests.
Unfortunately, most junior students would simply copy&paste the figure from the paper in their talk and cause much confusion. 😕
Let's break it down ... 🧵
#CVPR2021
in 15min!
(10382) Ensembling With Deep Generative Views;
@lucyrchai
et al
(2535) Anycost GANs for Interactive Image Synthesis and Editing;
@jilin_14
et al
(1509) Spatially-Adaptive Pixelwise Networks for Fast Image Translation;
@TamarRottShaham
et al
See you there!
We are excited to have many new PhD students joining our research community this year! Looking forward to all the discussion, collaboration and fun we'll have together.
Fast image translation (4x-18x faster) using 2 streams: a shallow pixel-wise net at full-res, using parameters predicted at low-res
@TamarRottShaham
, Gharbi, myself,
@elishechtman
, Michaeli
#CVPR2021
Website:
Paper:
ASAPNet was accepted to
#CVPR2021
! Extremely fast image to image translation (18x faster than baselines) with hyper network and implicit functions. With Michael Gharbi,
@rzhang88
,
@elishechtman
and Tomer Michaeli.
project:
abs:
We can try attributing synthetic Stable Diffusion images to the LAION training set.
But note that there's still a generalization gap from customization → full attribution. We also haven't tackled compositionality.
In other words, there's a LOT left to do!
4/
I'll be speaking at 3pm PT on Analyzing CNN Artifacts in Discriminative and Generative Models:
Come check out our just accepted CVPR paper: "CNN-generated images are surprisingly easy to spot...for now" (w/ S.Y. Wang, O. Wang,
@andrewhowens
, A. A. Efros)!
The use of contributor data for GenAI systems is top of mind, with real-world implications. Solving attribution is a critical piece for recognizing contributors
See the recent WaPo op-ed from Joseph Gordon Levitt. We even got a shout-out afterwards!
/End
Seems some technologists think it’s impossible to tell which pieces of an AI’s training data influenced which outputs, and therefore which humans would deserve AI Residuals.
But some think it’s possible. Here’s a senior research scientist at Adobe. Thanks for this, Richard!
Disentangling the contribution from >>100M images & obtaining ground truth attribution is super hard!
So we take an initial step. By tuning models towards exemplars using "customization" methods, we can create ground truth training-synthetic pairs, by construction.
2/
(1/2)
Can GANs be aggressively scaled up? We tried!
We present GigaGAN for text-to-image synthesis. It inherits things we like – a disentangled latent space and fast run-time. GigaGAN can generate 512px images in 0.13 sec and produce 4K images.
We detect when a face has been warped by Photoshop and even try to "undo" it. Come see "Detecting Photoshopped Faces by Scripting Photoshop" tomorrow (Fri), 3:30-6pm, Poster
#93
.
#iccv2019
with S.Y. Wang, A. Owens, O. Wang, A. A. Efros.
We offer this as an Attribution by Customization (AbC) benchmark, w/ >18,000 models and >4M images.
This lends itself naturally to a statement for attribution — given a synthetic image, what % chance is a given image the exemplar?
See paper for more dataset & method details
3/
I have been digitizing my annotations on some paper printouts📜& spotted a couple of gems
🧐a president in the acknowledgements
😎a meme in the citations...
so if anyone ever tells you your paper is informal due to something similar just👀
How much editing is done on magazines? This is running our Photoshop Unwarping forensics tool, from back in 2019!
I did a demo here (onstage with
@mulaney
!):
Paper:
w/
@ShengYuWang6
@oliver_wang2
@andrewhowens
, Alexei A. Efros
We asked our 2021
@AdobeResearch
Women-in-Technology scholars what truly sparks their curiosity about the fields they are studying. Check out what they shared, and don’t miss the opportunity to apply for the 2022 program!
#AdobeWiT
#Scholarship
First day of ballot dropoffs, and
@SteveKerr
was there!!!!!
My favorite growing up in Chicago!! Responsible for my earliest sports memory: . I gawked a bit and then ran away 😅. I will make a better plan for next time.
Everybody please
#VOTE
Last
#ECCV2020
event for me! I will be speaking about "Detecting Generated Imagery, Deep and Shallow" (...and then also showing an image manipulation algorithm 😬) at the SenseHuman workshop
3:20pm PT / 6:20pm ET / 2320 UTC+1
Click Session 2
Come join!
@docmilanfar
Worse, "invariance" is overloaded 🙄
- shift(F(x))==F(shift(x)) is "invariance" in signal processing and "equivariance" in ML
- F(x)==F(shift(x)) is "invariance" in ML
Signal processing came first, but for communication, I just use ML nomenclature when in ML environments
Introducing AI Paygrades ()! Statistics of industry offers for AI jobs. The goal is to reduce information assymetry so candidates can make informed decisions and negotiate better. Submit your information and spread the word! With
@abhshkdz
.
Our LPIPS CVPR18 evaluated how well deep activations reflect human perceptual similarity judgments. We did not investigate issues when backpropping it. Check out this paper from
@jaakkolehtinen
on how ensembling helps prevent LPIPS from being "attacked"
E-LPIPS: Adversarial attacks on neural image similarity metrics, and how to fix them by random ensembling. The resulting geometric properties are perhaps my favorite research finding so far! Paper: Code:
@AaltoCS
@FCAI_fi
@NvidiaAI
Here's an overview video (5 min) highlighting our work in perception, generation, and forensics:
I'm excited to continue pushing to build a healthy ecosystem around GenAI, making it fair and transparent for consumers, creators, and contributors alike
2/2
3/7 We found that deep contrastive embedding models achieve neural prediction accuracy that equals or exceeds that of supervised models, in cortical areas V1, V4, and IT all along the ventral pathway. Best-matching algorithm:
@MattNiessner
Seems folks will individually determine if x% covid possibility is okay and make a personal judgment call. For me, it's definitely worth it (especially if it's < 100 °F and a direct flight away)!
I was wondering how this year's covid-positive folks feel:
@Ar_Douillard
I once had a heuristically set hyperparameter of T=1/6. Unfortunately, after release, I found a silly bug in my code (log base 10 in one place, e^x in another). I had to absorb it into the constant to create T=ln(10)/6
🤦♂️🤦♂️🤦♂️