What is tokenization? Ms. Coffee Bean explains tokenization in general, explains why flexible tokenization is important and then moves onto explaining the “Charformer: Fast Character Transformers via Gradient-based Subword Tokenization” paper (explained and visualized).
➡️ AI Coffee Break Merch! 🛍️ https://aicoffeebreak.creator-spring.com/
Paper 📄: Tay, Yi, Vinh Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu and Donald Metzler. “Charformer: Fast Character Transformers via Gradient-based Subword Tokenization.” (2021). https://arxiv.org/abs/2106.12672
📺 Replacing self-attention with the Fourier Transform: https://youtu.be/j7pWPdGEfMA
📺 Convolutions instead of self-attention. When is a Transformer not a Transformer anymore? : https://youtu.be/xchDU2VMR4M
📺 Transformer explained: https://youtu.be/FWFA4DGuzSc
Outline:
00:00 What are tokenizers good for?
02:49 Where does rigid tokenization fail?
03:51 Charformer: end-to-end tokenization
08:33 Again, but in summary.
09:57 Reducing the sequence length
10:37 Meta-comments on token mixing
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕
Patreon: https://www.patreon.com/AICoffeeBreak
Ko-fi: https://ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
🔗 Links:
YouTube: https://www.youtube.com/AICoffeeBreak
Twitter: https://twitter.com/AICoffeeBreak
Reddit: https://www.reddit.com/r/AICoffeeBreak/
#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research
Add comment