- Titel
- Finetuning and Benchmark on Abstract Generation for Long Documents
- Zitierfähige Url:
- https://nbn-resolving.org/urn:nbn:de:bsz:l189-qucosa2-981547
- Datum der Einreichung
- 02.06.2025
- Abstract (EN)
- This research evaluates and aims to improve Large Language Models' ability to generate high-quality scientific abstracts from long dissertations, in collaboration with the Deutsche Nationalbibliothek (DNB). The primary contribution is DissBench, a new benchmark for long-context abstract generation using 102 dissertations of up to 125K tokens. Performance is measured using an LLM-as-a-Judge framework with structured criteria for content (Objectives, Methods, Results, Conclusions) and presentation quality (Coherence, Academic Standards). We also conducted parameter-efficient supervised fine-tuning (SFT) on Llama−3.1−8B−Instruct using Low-Rank Adaptation (LoRA) to enhance its performance. Eleven state-of-the-art models were evaluated. Qwen2.5-72B-Instruct achieved the best overall score (81%), while Gemma-3-27B-it excelled in presentation. Most models handled long contexts robustly, though some showed significant degradation with increased document length. In contrast, all SFT experiments decreased performance compared to the base model, often leading to answer deterioration. This work contributes a robust evaluation framework for long-context scientific text processing, identifying reliable models for practical deployment. However, the results highlight the challenges of adapting models for complex tasks with parameter-efficient methods, suggesting that full fine-tuning or different dataset strategies are necessary.
- Freie Schlagwörter (EN)
- Large Language Models, Long Context, Benchmark, Fine-Tuning
- Den akademischen Grad verleihende / prüfende Institution
- Hochschule für Technik, Wirtschaft und Kultur Leipzig, Leipzig
- Förder- / Projektangaben
- Deutsche Nationalbibliothek (DNB)
DH Stipendium
Inhaltliche Erschließung von wissenschaftlichen Publikationen durch Machine Learning - Version / Begutachtungsstatus
- angenommene Version / Postprint / Autorenversion
- URN Qucosa
- urn:nbn:de:bsz:l189-qucosa2-981547
- Veröffentlichungsdatum Qucosa
- 04.08.2025
- Dokumenttyp
- Masterarbeit / Staatsexamensarbeit
- Sprache des Dokumentes
- Englisch
- Lizenz / Rechtehinweis
CC BY-SA 4.0- Inhaltsverzeichnis
Abstract III List of Figures VII List of Tables VIII 1 Introduction 1 1.1 Background and Motivation 1 1.2 Problem Statement 1 1.3 Research Questions and Objectives 2 1.4 Thesis Structure Overview 2 2 Theoretical Background 3 2.1 Large Language Models 3 2.2 Benchmarking in LLM Research 6 2.3 Scientific Abstracts: Structure and Disciplinary Variation 16 2.4 Language Model Finetuning 19 3 Material and Methods 28 3.1 Data Description 28 3.2 Initial Dataset Creation 29 3.3 Qualitative Processing 31 3.4 Benchmark Development 35 3.5 Finetuning for Abstract Generation 43 3.6 Environment and Hardware 46 4 Results 48 4.1 Benchmark Results 48 4.2 Finetuning Results 56 5 Discussion 62 5.1 Benchmark 62 5.2 Finetuning 65 5.3 Future Work 67 6 Conclusion 69 Literaturverzeichnis 71 Appendix 82