Ahmad Al-Dahle is Meta’s VP of generative AI. He joins Big Tech War Stories to disucss how the company built its Llama 3 model, releasing today. Training Llama 3 took ten times more data, one hundred times more computing resources, and personality tweaks to make it more willing to answer questions it otherwise would’ve refused.
Takeaways
Lama 3 includes an updated 8 billion parameter model and a 70 billion parameter model, which are state-of-the-art and high-performing.
The models are trained in two phases: pre-training, where the model consumes general knowledge, and post-training, where human supervision is involved.
Scalability, infrastructure, and data work are crucial in building these models.
Weights in the models represent knowledge and require hardware to run.
The models’ personality and behavior are carefully engineered to balance usefulness and safety.
The process of deploying the models in products involves close collaboration with application teams and rigorous quality checks.
Open sourcing the models is a consideration, but safety and cybersecurity are important factors to address.
MetaAI is being made more prominent in Meta’s products, with integration into messaging platforms and search.
The long-term goal is to achieve artificial general intelligence, but the timeline and specifics are uncertain.
Chapters
00:00 Ahmad’s Background and Tech Journey
02:50 Training AI Models: Data and Compute Resources
09:22 Scaling from Llama 2 to Llama 3
26:03 Advancing Reasoning Capabilities in AI
You can get the full show as a paid subscriber on https://www.bigtechnology.com
Add comment