What is a character-based tokenizer, and what are the strengths and weaknesses of those tokenizers.
This video is part of the Hugging Face course: http://huggingface.co/course
Related videos:
– Tokenizers overview: https://youtu.be/VFp38yj8h3A
– Character-based tokenizers: https://youtu.be/ssLq_EK2jLE
– Subword-based tokenizers: https://youtu.be/zHvTiHr506c
Have a question? Checkout the forums: https://discuss.huggingface.co/c/course/20
Subscribe to our newsletter: https://huggingface.curated.co/
Add comment