Presentation
In the last few years, radical increase in the scale of deep neural language models (both in terms of the size of the training data and the size of the models themselves) has led to impressive achievements in various natural language processing tasks. "Celebrity" models, like ChatGPT, LLaMa, BLOOM or PaLM are already sometimes described to as "approaching artificial intelligence", although the reality can differ from over-hyped media coverage.
In this talk, Kutuzov will describe the foundations of the technology behind large-scale language models. Two most important components behind their success are 1) state-of-the-art deep learning architectures (in particular, Transformer) and 2) the availability of tremendous amount of textual data used to train such models. The interaction of these two poses intricate theoretical and practical questions, also linked to issues with unequal distribution of computing resources. Do we have enough good-quality training data for languages other than English? Is "data poisoning" with automatically generated texts is a real danger? Why is it important to open source both training data and model weights?
Speaker
Andrey Kutuzov (PhD, UiO, 2020) is an Associate Professor in the Language Technology Group at the University of Oslo. He currently serves as the Norwegian on-site manager of the High-Performance Language Technology (HPLT) project. His academic interests include Computational linguistics and natural language processing, semantic change detection and diachronically aware language models, distributional semantics, machine learning and large-scale language models. In 2022, Kutuzov received the Norwegian Artificial Intelligence Research Consortium (NORA) award as a Distinguished Early Career Researcher.
Program
11:45 – Doors open and lunch is served
12:00 – "Large language models under the hood: artificial neural networks and who they belong to" by Andrey Kutuzov (Associate Professor, Research Group for Language Technology)
13:00 – Mingling (and goodbye)
Recording of the seminar
Download Kutuzov's slides here (PDF).
About the seminar series
Once a month, dScience will invite you to join us for lunch and professional talks at the Science Library. In addition to these, we will serve lunch in our lounge in Kristine Bonnevies house every Thursday. Due to limited space (40 people), this will be first come, first served. See how to find us here.
Our lounge can also be booked by PhDs and Postdocs on a regular basis, whether it is for a meeting or just to hang out – we have fresh coffee all day long!