A general introduction to the different types of tokenizers.
This video is part of the Hugging Face course: http://huggingface.co/course
Related videos :
– Word-based tokenizers: https://youtu.be/nhJxYji1aho
– Character-based tokenizers: https://youtu.be/ssLq_EK2jLE
– Subword-based tokenizers : https://youtu.be/zHvTiHr506c
Have a question? Check out the Hugging Face forum: https://discuss.huggingface.co/c/course/20
Don’t have a Hugging Face account? Register now: http://huggingface.co/join
Add comment