This seminar covers recent topics in Natural Language Processing. This term´s seminar is for BSc and MSc students and focuses on Parameter-Efficient Models and Model Training in Modern Natural Language Processing.

Description

In this seminar, you will:

• Select one of the offered subtopics (see Subtopics).
• Read, understand and explore scientific literature (with listed papers as starting points for your analysis)
• Organize the collected knowledge for a meaningful presentation about your topic (15 minutes + 5 minutes Q&A)
• Summarize your topic in a concise report (6 - 8 pages + references)

Subtopics 

Each student explores and presents one of the below subtopics. For each topic, prominent papers are provided as a starting point for the seminar exploration. Your presentation and report are based on the provided papers and  other work, which you are expected to discover on your own, starting from the given papers. Your goal is to clearly summarize the body of work on the selected subtopic, it's relevance and impact, and draw your own conclusions about the limitations and/or future efforts in this area.

• Sparse Fine-Tuning
• Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer.
• LoRA
• Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.
• Adapters
• Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP.
• Soft-Prompting
• Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.
• Hypernetworks
• Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. Contextual parameter generation for universal neural machine translation.

Additional Material

• Survey on modular deep learning:
  • Pfeiffer, J., Ruder, S., Vulić, I., & Ponti, E. M. (2023). Modular deep learning.

Schedule 

• Register in the WueCampus2 course
• Attend the kickoff meeting on 29.10.24, 16:30 in Seminarraum 1 in the CAIDAS Building (No. 50)
• Work individually on your topic and get feedback from your supervisor(s)
• Give a meaningful presentation including a Q&A session, beginning of February 2025. 
• Hand in your final report by the end of February 2025.