|| assistant prof
@UMontreal
|| leading the systems neuroscience and AI lab (SNAIL) ||
@Mila_Quebec
||
#NeuroAI
|| vision and learning in brains and machines
Our lab 🐌now has a website and a logo:
Check out our website to learn more about our approach and research goals.
Big thanks to the great artist and neuroscientist
@MariaZamfirPhD
for helping with the logo.
🚨News🚨Absolutely thrilled to announce that, starting January 2023, I will be an Assistant Professor in the department of psychology at
@UMontreal
.
I will be developing a NeuroAI research lab focused on visual perception and learning in artificial and biological neural networks
Check out our new paper:
Can a single neural network explain specialized pathways of the visual system?
TLDR; yes, but only if you train with a self-supervised predictive loss function!
Here is a summary of our paper: (1/n)
I’ll be recruiting 1-2 graduate students for Fall 2025 to work on visual learning generalization in humans and artificial neural networks.
If you’re interested, apply!
Check out our lab website () and reach out if you need more info.
RT please.
📷 Meet our student community! Interested in joining Mila? Our annual supervision request process for admission in the fall of 2025 is starting on October 15, 2024. More information here
Meta is announcing the AI Research SuperCluster (RSC), our latest AI supercomputer 💻 for AI research. RSC will allow our researchers to do new, groundbreaking experiments in
#AI
. Learn more about RSC and the important role it will play:
Neuroscientists should closely follow the mechanistic interpretability efforts in AI and the debates around it, if they’re not already.
Eg see how much the example below (+ the whole thread) is relevant to causal manipulation experiments in the brain.
Second big issue: if there are multiple causes of the same effect, it would be very easy to miss both of them if we only ablate one at a time. A related concept is preemption: one cause prevents another from having any effect, and so we miss the preempted cause.
“Here, we demonstrate that predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rule”
Neuro-AI folks, how do you keep up with both neuro and AI literature? What’s your strategy? It’s a full-time job!!!
(And yes, giving up is always an option)
I very much agree with
@markdhumphries
that evolution doesn't care what we call a brain area. But, what evolution might care about is modularity as an efficient way of expanding a constantly adapting system.
A 🧵:
By far, one of the best analysis I’ve ever read on academia as a business model, the role of postdocs as its horsepower, and how twitter can break down the whole system for good.
Can Twitter Save Science? - Alex Danco's Newsletter
Representational drift in the mouse visual cortex; by Daniel Deitch, Alon Rubin, Yaniv Ziv
The intrinsic structure of movies representation doesn't change over time despite drift in the coding of the neurons supporting it.
Octopus arms show photostatic response to light, and most interestingly, the motor response is channeled through the central nervous system (not local reflexes).
So it’s kinda like having an 👁 in your palm, and moving your arm based on what your hand “sees”.
It’s hard to fully understand the importance of her work unless you’re doing science in a developing country where universities and research institutes can’t afford outrageous subscription fees.
We're still hyped about last week's RIOTS, where we had Alexandra Elbakyan (
@ringo_ring
) share her thoughts on open knowledge, copyright and
#SciHub
.
If you've missed it (or want to rewatch it):
📺
Huge thanks again to
@mariiabocharova
for translating
Not all visual features are treated equally in brains or artificial neural networks; some are favored by more neurons.
What are the behavioural and learning consequences of these biased representations?
I discuss this question in a new blog post:
(1/4)
This work from
@FieteGroup
looks pretty cool:
Showing how structure in the sensory cortex (hierarchy, topography, etc.) can emerge from self-organizing dynamics and spontaneous activity on a cortical sheet.
Looking forward to reading this one.
“our results suggest that the ventral visual cortex, like DNN models, provides a texture-like basis set of features, and that further neural computations, perhaps downstream of IT, are necessary to account for the shape selectivity of visual perception.”
Prestige seeking behaviors are built into the fabric of academia (and its evaluation systems). Pre-pub peer review & journals have evolved to serve this need in academics. Demolishing it will leave an empty space that will be filled by alternatives that are not necessarily better
Watching
#AlphaFold2
unfolding, it’s hard to resist the question of what’d be the neuroscience problem that could be pushed toward solving by throwing engineering and compute resources at it. Some might argue that neuro isn’t even at the stage of having such well defined question
Had heard it from others, but now it’s firsthand experience. The joy of hanging out with these young and fun scientists is the best part of running a lab.
The first SNAIL social event 🐌
"Dissociation in neuronal encoding of object versus surface motion in the primate brain"
Showing long-range motion direction selectivity in V4 (up to 29% of neurons, using neuropixel recordings)
A figure in a recent paper by
@talia_konkle
and Alvarez () reopened an old question for me. The figure shows that an AlexNet architecture with group normalization has more similar representations to the brain than one with batch normalization.
Science has definitely become more inclusive with the virtual conferences.
Someone who hasn't dealt with visa offices can't see the true point of virtual conferences.
When live conferences do come back, I do think we should keep virtual participation as an option, to allow broader participation. But I feel like all-virtual conferences are slowly strangling academia.
Interested in joining Mila's research community? Our annual supervision request process for new Mila students is starting this Sunday October 15, 2023.
For more information ➡️
New paper
@UMontreal
@Mila_Quebec
(In collab with Chris Pack's lab
@TheNeuro_MNI
@mcgillu
)
We studied Visual Perceptual Learning (VPL) in distinguishing stimuli that have "asymmetric" representations in the visual cortex.
How does this asymmetry affect VPL? 🧵(1/10)
mini two-photon microscope for fast, high-res, multiplane ca imaging of over 1,000 neurons at a time in freely moving mice:
"A novel technique for successive imaging across multiple, adjacent FOVs enabled recordings from more than 10,000 neurons ..." 🤯
Interesting, new paper from Cichy lab
"The spatiotemporal neural dynamics of object location representations in the human brain" by
@MonikaGraumann
et al.
Object loc emerges in the ventral areas ~ 150 ms later on a high clutter background.
The next generation of big neuro projects could look something like GPT-JT-6B.
A community based, distributed, large-scale model, optimized to align with all the neural/behavioral data that we could put together.
What % of the neuro community would get on board with that?
This enables geo-distributed computing across cities or even countries. Now everyone can BYOC (“Bring Your Own Compute”) and join the training fleet to contribute to the open-source development. The scheduling algorithm makes no assumption about the device types.
3/
Plenty of gems for new PIs in this note by
@behrenstimb
:
Especially liked this one: “If you ever become Buzsaki, Deisseroth, or Dolan you can have people that work for you. Until then, you work for them.”
Another cool work on texture / shape bias but in large vision-language models:
“we find that VLMs are often more shape-biased than their vision encoders, indicating that visual biases are modulated to some extent through text in multimodal models”
“our results suggest that the ventral visual cortex, like DNN models, provides a texture-like basis set of features, and that further neural computations, perhaps downstream of IT, are necessary to account for the shape selectivity of visual perception.”
Super excited that both
@ShahabBakht
and I's papers were accepted in NeurIPS as spotlights! Hope you're ready for a whirlwind tour of how the brain solves motion! Tweeprint here 👇
It’s quite possible that mouse and monkey V1s operate at different processing stages, as suggested in this super interesting review paper by Jon Kaas et al.:
Another study highlighting that visual findings from mice don't generalize to primates.
2 thoughts:
1) It's fascinating that mice have no orientation columns - architectural constraints?
2) It's crazy nobody has directly shown orientation columns in humans
I’ll be talking about Self-Supervised Learning and how it can be used for modeling and studying the development of animals’ visual systems: what have been done and what should we do next.
Join us if you’re interested.
I'm very happy to have the opportunity of building my lab in Montreal, one of the best and fastest growing NeuroAI ecosystems in the world. I'll be recruiting trainees at all levels. If you're interested in working at the intersection of neuroscience and AI, drop me a message.
Lastly, to all Iranian students that are affected by the recent events: if you're considering to apply for graduate programs, please let me know if I can be of any help.
Eve Marder at
@neuromatch
4.0 panel discussion: "If every single postdoc is convinced that he or she should have bona fides in machine learning, what does it tell us about herds?"
I’ve migrated to Neuromatch’s mastodon instance.
A great community of neuro and NeuroAI people already there. Join us if you haven’t already.
(And for those who wonder if instances matter: yes - local timeline is where you see the difference)
With prompts in Farsi,
#stablediffusion
generates a lot of beautiful Islamic architectures and Persian carpets, which have nothing to do with the content of the prompts.
"An oil pastel painting of a skeptical researcher absolutely amazed by what he's seeing on a computer" (AI-generated image created by DALL-E with a prompt I wrote, top 1 sample)
Deep learning studies are getting closer and closer to monkey ephys: spend ~ 6-12 months train the first one, look at the data, if found anything interesting, go for the second one.
The Deep Learning Devil is in the Details. I love this work from
@IrwanBello
and collaborators in which they show how training "tricks" improve ~3% absolute accuracy on ImageNet, progress equivalent to years of developments and research!
Paper:
Number of Vision Science Society (VSS) abstracts with 'neural networks' in title since 2001
Looks like a paradigm shift ... And no sign of saturation!
Something happened in 2016 - maybe Tensorflow?!
Or lagged response to
@dyamins
et al (2014)?
Interpreting any failure of AI models as a failure of their "understanding" has proven to be a mistake.
Soon, a model will generate accurate gymnastics videos, and we will have to choose between accepting the AI's understanding of human body motion or moving the goalposts.
CW: Body Horror?
This AI video attempt to show gymnastics is one of the best examples I have seen that AI doesn’t actually understand the human body and it’s motion but is just regurgitating available data. (Which appears to be minimal for gymnastics)
@fchollet
The hard part is to get your arXiv preprint seen by as many ppl as possible. That’ll have a huge effect on the “adoption” part. One could put brilliant work on arXiv but might not survive the overly skewed attention competition out there, especially given corporates’ PR machinery
2/ GaLore takes a novel approach compared to methods like LoRA and ReLoRA. Instead of doing low-rank projection in weight space (W = W_0 + A
@B
) and baking this into the weights every T steps, GaLore performs low-rank projection in gradient space: G = P @ G' @ Q^T.
Dear (prof first name),
I have long been a fan of your research. Here is what your logically next paper would be (without saying it) and I would love to work on it and am ideally prepared for it. Can we talk?
Yours,
(Your first name)
I wonder if people with “better” passports are aware of the “passport” effect. Do they consider it in their evaluations? It’s a real thing and it can hugely affect one’s career.
If I had known earlier, I could have done this before. They didn't direct me to the appointment page until I finished the documents. Emailed the embassy and got a usual response.
F*** this s***. I don't miss physical conferences. They are for people with better passports.
Google’s new Pathways project is a great example of what corporate labs are good at: scaling up a simple, promising idea and pushing it to its limits.
Small academic labs should be good at developing those simple ideas and demonstrating their potentials.
David Sussillo mentioned this old paper by Zipser and Anderson in his talk
@neuroAIworkshop
at
#NeurIPS
:
It’s perhaps one of the first examples of comparing representations in the brain and artificial neural nets.
This result by SimCLRv2 (a self-supervised model) is impressive.
With only 1% of ImageNet labels, SimCLR2 is as good as the fully supervised ResNet-50.
I love it every time
@AnthropicAI
mentions the link between their exciting work and prior work in neuroscience. It's not something you see frequently in AI these days
Our research builds on work on sparse coding and distributed representations in neuroscience, disentanglement and dictionary learning in machine learning, and compressed sensing in mathematics.
Our two spotlight papers (by
@ShahabBakht
and
@patrickmineault
) at
#NeurIPS2021
showing how ANNs can develop dorsal stream representations now have their websites up (with links to the updated papers and code):
Enjoyed reading this paper by Yaoda Xu and Maryam Vaziri-Pashkam:
Careful analysis of representational similarities between 6 areas of human visual cortex and 14 different ImageNet-trained ANNs.
Early areas can be ‘fully’ explained, but not higher areas
On the emergence of specialized streams in neural networks:
1/ specialization can emerge without task-specific optimization. Self-supervised learning, as shown in the recent work by
@dfinz
et al and our previous work, can lead to emerged specialized streams.
I know that people who follow me here are not doing that for politics. But this is not politics!
Last time this happened, thousands were arrested and hundreds - if not more - were killed on the streets of Iran - A preparation step for another bloodbath.
#مهسا_امینی
⚠️ Confirmed: Real-time network data show a nation-scale loss of connectivity on MCI (First Mobile),
#Iran
's leading mobile operator, and Rightel; the incidents come amid widespread protests over the death of
#MahsaAmini
📵
📰 Background:
These findings from
@TimKietzmann
lab are among the most exciting I’ve seen recently. Especially, if you’re into predictive coding, make sure you read this paper.
I also wrote a preview on why I believe this is a very interesting and important paper:
Excited that our paper "Predictive coding is a consequence of energy efficiency in recurrent neural networks" is now out in
@Patterns_CP
! Much has changed since preprint, so check it out:
Work with the fantastic
@hellothere_ali
,
@nasiryahm
, and
@marcelge
!
Happy to see our paper finally out.
Perceptual improvements caused by Visual Perceptual Learning are highly specific to the training stimulus.
In this paper, we showed that the specificity of VPL depends on the complexity of the training stimulus.
This rule also applies spatially not just temporarily.
If you’re a researcher in tech (or any field) and you’re not reading anything five metres away from you (metaphorically speaking), your own work will be forgotten five metres from you.
If you’re a researcher in tech and you’re not reading anything more than five years old, you can be sure your own work will be forgotten five years from now.
@neuroecology
@DynamicEcology
“Neural mechanisms for interacting with a world full of action choices” by Cisek and Kalaska
For me, it broke down the perception/action dichotomy by using a neuronal interpretation of Gibsonian affordances.
Finally ... humans playing Atari during fMRI
"Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments"
So, if we regularize an ANN to have similar representations as those of mouse visual cortex, the ANN becomes more robust to adversarial attacks.
Here is the most interesting part of the paper for me:
Behavior shapes retinal motion statistics during natural locomotion
Another great work from Hayhoe lab!
Describing statistical properties of the retinal motion patterns during walking, and how they're affected by gaze location and behavior.
For folks who aren't able to attend in-person, we are excited to be able to stream all talks this year on our youtube channel:
Papers can be found here:
Please share!
Anyone wondering how ANNs and deep learning have advanced our understanding of the brain should read this paper.
I was fortunate to see these results at various stages of their development, and they only got better and better.
1/13 Heavily updated preprint!
We show that the contextual information encoded in Large Language Models (LLMs) is beneficial for modelling the complex visual information extracted by the brain from natural scenes. 🧵
Join us in this exciting talk series.
I’ll delve into the recent advances in self-supervised learning in AI and their applications in modeling the visual system.
GACs have become my favourite way of getting an updated view of a comp neuro/cog field.
Whoever came up with the idea deserves the Nobel peace prize.
This one from
#CCN2021
was great:
As much as I was shocked by the ignorance of the original consciousness tweet, I'm way more shocked by how it's turned my timeline upside down.
This level of influence and reach can definitely be put to a better use.
I believe it’s because neuroscience lacks an easy-to-use metric to measure transformative-ness.
There is no list of major open questions that everyone agrees on. We’re divided into small subfields (for better or worse), busy with our own domain-specific questions. (1/2)
The lack of replies to this reinforces my own belief: neuroscience has come up with many cool results in the past few decades, but none that are truly transformative
There are so many interesting observations in this paper:
The most interesting one imo:
1- "[CLIP] scores close to humans across all of our metrics presented most strikingly in terms of error consistency"
Generalization in data-driven models of primary visual cortex
"We demonstrate that a representation pre-trained on thousands of neurons from various animals and scans generalizes to neurons from a previously unseen animal"
#ICLR2021_submission
Beautiful work presenting a generative model of brain connectome!
Easy to imagine all the interesting things that can be done with this approach to bring evolution and development into NeuroAI.
So happy to share! 🎉
Recently, we’ve seen 🔥 work using generative models to elucidate candidate principles of neural connectivity.
Our v2: Inspired by redundancy reduction, our new model can generate both the topology & weights of the connectome:
I’m amazed by how easily you can break LLMs by including some simple relational queries or info in the prompt. No matter how good they’re in generating high quality texts or images, they easily fail in dealing with relations. And that, imho, says a lot about their shortcomings
Imo, one of the most influential works in computational (vision) neuroscience.
It was my first encounter with an optimization-based approach to visual coding which was a tempting alternative to the more common tuning-curve-ology approach
Oldies but goldies: B. Olshausen, D. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images, 1996. Showed that dictionary learning on natural image produces wavelet-like atoms.
A very useful study, with many practical insights about self-supervised learning (SimCLR here)
Trying to answer important questions about e.g. the effect of dataset size, domain etc on SSL
Scaling is important, but I’ll refer everyone who believes scaling is _all_ you need to Chen’s beautiful work, as an example of how far we can go with genuine ideas, without scaling.
(1/n) Hello Friends! I wanted to share my paper with a
#tweeprint
. We made an RL module we call contrastive introspection (ConSpec). It enables learning when rewards are sparse and multiple key steps are required for success, as often occurs in real life
@NicoleCRust
The optimist inside me would argue that building brains will lead to understanding brains (à la Feynman) and eventually fixing them, so the current fast rate of progress in building brains is actually part of the progress toward fixing them.