Lots of people asking me about my journey into Machine Learning. I gave two interviews about my journey few months back (links in the bottom). Key takeaways:
• Learning Mandarin taught me consistency and patience. Success is a natural consequence.
• Quit my manager job at the
A complete explanation of Kolmogorov-Arnold Networks, including a review of multilayer perceptron (MLPs), Bézier Curves, B-Splines, the Universal Approximation Theorem and how KANs compare to MLPs:
#kan
#machinelearning
#deeplearning
#mlp
A full explanation of the mathematical foundation of Reinforcement Learning from Human Feedback (RLHF) and PPO, starting from first principles. Includes PyTorch implementation explained line-by-line:
#rlhf
#ppo
#math
#tutorial
#pytorch
#aialignment
Great article by NVIDIA demystifying chunked prefill of a prompt and other techniques for optimizing inference of LLMs. Someone (😇😇😇) is very interested in chunked prefill of the KV-Cache and related optimizations, so expect to hear more from me about it.
Article:
Chunked Prefill generates the KV-Cache by splitting the prompt into chunks. We prove that while prefilling, we can exploit the partial KV-Cache to generate extractive summaries and then append them at the end of the prompt to improve the ability of the model to extract
Tutorial on distributed training with PyTorch, with code walk-through and explanation of Gradient Accumulation, Collective Communication (Broadcast, Reduce, All-Reduce), DistributedDataParallel, Bucketing:
#pytorch
#tutorial
#distributeddataparallel
In this video I will explain quantization: asymmetric and symmetric quantization, dynamic and static Quantization, Post-Training Quantization and Quantization-Aware Training.
Including PyTorch code!
#deeplearning
#quantization
#pytorch
#tutorial
@9LkVSi8E3NWf19q
Study the code (by running it), comment it with your personal annotations. Try to re-create the project by using your commented code. Takes lots of patience, lots of research.
Start with very small projects, they help you build the confidence for pursuing bigger ones.
@detention361757
Eat light at dinner (no later than 18.30), no electronic device (except e-ink reader) after 20.30, sleeping by 21.30 most of the time.
@iScienceLuvr
@tri_dao
I made a video in which I derive the mathematical foundation of State Space Models, including Mamba, from first principles. I also teach differential equations in 5 minutes for those who lack this background.
Check it out:
@atgorans_k
Give the money to the poor, they need it more than me. I have a very long list of video I'd like to make, stay tuned and I guarantee you will keep getting amazed. Have a nice day!
@detention361757
You can sleep even 9 hours. Just be consistent with whatever schedule works for you.
People underestimate the power of studying consistently even 1 hour every single day.
@dchaplot
I made a video about the Mixture of Experts Sliding Window Attention, Rolling Buffer Cache, Pre-Fill and Chunking, Model Sharing. I provide mathematically why the sliding window attention can capture information outside the window size.
Check it out!
Little known fact we exploited: you can always delete something from the end of the KV-Cache and replace it with something completely different. Do you know why? Each token in the KV-Cache is a contextualized embedding about all the "past" tokens, so you cannot delete anything
@ohhbatu
Because we also have the batch dimension (when you train a model, you have a batch of sentences), and the "h" dimension, which depends on the number of heads of the multi-head attention; d_k = d_model / h.