The 🤗 Datasets library lets you use and process datasets that don’t fit in RAM. Learn how it can do this with memory mapping and how to use the streaming feature.
This video is part of the Hugging Face course: http://huggingface.co/course
Open in colab to run the code samples:
https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/videos/memory_mapping_streaming.ipynb
Related videos:
– Loading a custom dataset — https://youtu.be/HyQgpJTkRdE
– Slide and dice a dataset 🔪 — https://youtu.be/tqfSFcPMgOI
Don’t have a Hugging Face account? Join now: http://huggingface.co/join
Have a question? Checkout the forums: https://discuss.huggingface.co/c/course/20
Subscribe to our newsletter: https://huggingface.curated.co/
Add comment