“Training data-efficient image transformers & distillation through attention” paper explained!
How does the DeiT transformer for image recognition by @facebookai train with around 100x less training data than ViT?
➡️ AI Coffee Break Merch! 🛍️ https://aicoffeebreak.creator-spring.com/
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
🔥 Optionally, pay us a coffee to boost our Coffee Bean production! ☕
Patreon: https://www.patreon.com/AICoffeeBreak
Ko-fi: https://ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
📺 ViT Transformer: https://youtu.be/DVoHvmww2lQ
📺 Transformer architecture explained: https://youtu.be/FWFA4DGuzSc
📺 Visual Chirality: https://youtu.be/rbg1Mdo2LZM
Outline:
* 00:00 Facebook’s DeiT
* 01:34 Why is DeiT cool?
* 03:03 How does it work?
* 07:10 What does this mean?
📄 DeiT paper: https://arxiv.org/pdf/2012.12877.pdf
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou (2020) “Training data-efficient image transformers & distillation through attention”
💻 DeiT code: https://github.com/facebookresearch/deit
📚 For an in-depth understanding of how it works, check out this wonderful post by @JacobGildenblat https://jacobgil.github.io/deeplearning/vision-transformer-explainability
📚 On-point blog post by Andrei-Cristian Rad: https://radandreicristian.medium.com/what-to-do-if-training-on-jft-300m-is-not-an-option-convnet-teachers-to-the-rescue-104a7bf7dccb
News music 🎵 source:
News Theme 2 by Audionautix is licensed under a Creative Commons Attribution 4.0 licence. https://creativecommons.org/licenses/by/4.0/
Artist: http://audionautix.com/
🔗 Links:
YouTube: https://www.youtube.com/AICoffeeBreak
Twitter: https://twitter.com/AICoffeeBreak
Reddit: https://www.reddit.com/r/AICoffeeBreak/
#AICoffeeBreak #MsCoffeeBean #DeiT #MachineLearning #AI #research
Add comment