BCCN PhD Lecture Series
Location: BCCN Berlin, Haus 6 - lecture hall 9
BCCN PhD Lecture Series
Abstract:
Large language models (LLMs) are a major focus of current research in natural language processing (NLP). To enable the study and development of such models, we require reliably ways of evaluating their knowledge. This becomes especially important in a continual learning scenario, in which we take an existing LLM and further train it on new data. Here, we need to know if new knowledge is successfully learned and if any existing knowledge is (catastrophically) forgotten. To address this, I introduce a novel probing approach called "BEAR" and apply it to rank a number of prominent open source LLMs. I connect this knowledge probing to our current efforts in continual learning of LLMs. Our approach is open sourced as the "LM Pub Quiz", allowing any other LLM to be easily added to the benchmark. Additionally, I give an overview of other related research and open source projects in my group.
Further links:
- LM Pub Quiz: https://lm-pub-quiz.github.io/
- BEAR paper: https://arxiv.org/abs/2404.04113
Large language models (LLMs) are a major focus of current research in natural language processing (NLP). To enable the study and development of such models, we require reliably ways of evaluating their knowledge. This becomes especially important in a continual learning scenario, in which we take an existing LLM and further train it on new data. Here, we need to know if new knowledge is successfully learned and if any existing knowledge is (catastrophically) forgotten. To address this, I introduce a novel probing approach called "BEAR" and apply it to rank a number of prominent open source LLMs. I connect this knowledge probing to our current efforts in continual learning of LLMs. Our approach is open sourced as the "LM Pub Quiz", allowing any other LLM to be easily added to the benchmark. Additionally, I give an overview of other related research and open source projects in my group.
Further links:
- LM Pub Quiz: https://lm-pub-quiz.github.io/
- BEAR paper: https://arxiv.org/abs/2404.04113
>>> For BCCN PhD students: don't forget to sign the attendance sheet!
>>> For BCCN master students: the lectures may provide ideas for lab rotations &/ thesis topics!
>>> For BCCN master students: the lectures may provide ideas for lab rotations &/ thesis topics!