A comparative study of prompting and fine-tuning for binary text classification of sustainable development goals

Date

2024-8

Author

Atay, Mert

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

94
views

0
downloads

Cite This

The announcement of the 17 Sustainable Development Goals (SDGs) in 2015 by the United Nations has higlighted the need for automated document classification aligned with these global goals, as the volume of documents continues to increase. Current research focuses on traditional methods for automatic SDG classification, with Fine-Tuning being the latest approach, but neglects recent advancements in Large Language Models (LLMs) which excel in diverse tasks through Prompting or Prompt Engineering. This research compares the performance of binary text classification for SDG relevancy using state-of-the-art LLMs ChatGPT and Gemini via Prompting techniques, including Zero-Shot and Few-Shot Prompting with varying context levels, against traditional Fine-Tuning applied to Transformer-based models BERT and its resource-efficient variant DistilBERT, which serve as benchmarks. Based on comparative analysis, Fine-Tuning Transformer-based models outperforms Prompting with LLMs in SDG binary text classification, which aligns with existing research findings. Zero-Shot Prompting shows superior results for both models, with performance improving as the context level of the Prompt increases. In contrast, both models exhibit lower performance with Few-Shot Prompting, with ChatGPT notably experiencing a significant decline. Increasing the number of examples in Few-Shot Prompts does not lead to improved performance for either model. Among Fine-Tuned models, DistilBERT and BERT perform similarly in SDG binary text classification, with DistilBERT being particularly advantageous due to its computational efficiency. However, LLMs have the potential to demonstrate improved text classification performance with the development of more capable models and effective Prompting techniques.

Subject Keywords

Large language models, Fine-tuning, Prompting, Prompt engineering, Binary text classification, Sustainable development goals, Natural language processing

URI

https://hdl.handle.net/11511/111537

Collections

Graduate School of Natural and Applied Sciences, Thesis