1st Multilingual Model Workshop – Overview of Japanese Efforts to Train LLMs​

Like other countries, companies and government agencies in Japan have been investing heavily in LLMs. This talk will give an overview of the recent efforts to pre-train LLMs in Japan in both academia and industry. Details of the corpus, models, and pre-training configuration will be highlighted for some of the largest runs with up to 175B parameters. The first half of the talk will focus on the overview of the institutions involved, funding, and computational resources. The latter half will focus more on the technical details of the pre-training, and issues being faced.