Abschnittsübersicht

• Abschnitt Allgemeines auswählen

  Allgemeines

  Alles einklappen Alles aufklappen

  This seminar covers recent topics in Natural Language Processing. This term´s seminar is for BSc and MSc students and focuses on Parameter-Efficient Models and Model Training in Modern Natural Language Processing.

  Description

  In this seminar, you will:

  • Select one of the offered subtopics (see Subtopics).
  • Read, understand and explore scientific literature (with listed papers as starting points for your analysis)
  • Organize the collected knowledge for a meaningful presentation about your topic (15 minutes + 5 minutes Q&A)
  • Summarize your topic in a concise report (6 - 8 pages + references)

  Subtopics 

  Each student explores and presents one of the below subtopics. For each topic, prominent papers are provided as a starting point for the seminar exploration. Your presentation and report are based on the provided papers and  other work, which you are expected to discover on your own, starting from the given papers. Your goal is to clearly summarize the body of work on the selected subtopic, it's relevance and impact, and draw your own conclusions about the limitations and/or future efforts in this area.

  • Sparse Fine-Tuning
  • Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer.
  • LoRA
  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.
  • Adapters
  • Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP.
  • Soft-Prompting
  • Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.
  • Hypernetworks
  • Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. Contextual parameter generation for universal neural machine translation.

  Additional Material

  • Survey on modular deep learning:
    • Pfeiffer, J., Ruder, S., Vulić, I., & Ponti, E. M. (2023). Modular deep learning.

  Schedule 

  • Register in the WueCampus2 course
  • Attend the kickoff meeting on 24.04.24, 16:30 in Seminarraum 1 in the CAIDAS Building (No. 50)
  • Work individually on your topic and get feedback from your supervisor(s)
  • Give a meaningful presentation including a Q&A session, mid-July 2024, 
  • Hand in your final report by the end of July 2024. 

  • Aktivität Ankündigungen auswählen

    

    Ankündigungen Forum
  • Aktivität Kick-Off-Slides auswählen

    

    Kick-Off-Slides Datei