What happens when we call a tokenizer on some texts, and how does it compute the numbers it outputs?
This video is part of the Hugging Face course: http://huggingface.co/course
Open in colab to run the code samples:
https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/videos/tokenizer_pipeline.ipynb
Related videos:
– The pipeline function: https://youtu.be/tiZFewofSLM
– Learn more about the tokenizer algorithms: https://youtu.be/VFp38yj8h3A
– Learn more about attention masks: https://youtu.be/M6adb1j2jPI
– Learn more about token type IDs: https://youtu.be/0u3ioSwev3s
Have a question? Checkout the forums: https://discuss.huggingface.co/c/course/20
Subscribe to our newsletter: https://huggingface.curated.co/
Add comment