, 2026
AZFAKENEWS: A BENCHMARK DATASET FOR FAKE NEWS DETECTION IN THE AZERBAIJANI LANGUAGE
Disinformation on digital platforms is a very important problem for public trust and political debate. Even so, most research on automated fake news detection has stayed on a small number of high-resource languages. This paper presents AzFakeNews, the first large-scale benchmark dataset for fake news detection in Azerbaijani. Azerbaijani is a low-resource Turkic language with more than 30 million speakers. We built the dataset from two sources: scraping of news articles from five major Azerbaijani media websites, and generation of synthetic fake articles via Meta's LLaMA language model. The final dataset has 3,782 articles (3,000 authentic and 782 synthetic) across 34 topic categories. For the baseline we fine-tuned the Azerbaijani aLLMA model on this corpus. It reached 91.03% accuracy and 91.11% macro-F1 on the test split, because of which we consider it a strong baseline compared to mBERT, XLM-RoBERTa and the base LLaMA on the same data. We hope the dataset helps close a gap in Azerbaijani NLP and supports cross-language research on disinformation (39-44).
Keywords: Fake news detection, Azerbaijani language, Large language model, BERT, Natural language Processing
References
- Samadi, M., Mousavian, M., & Momtazi, S. (2021). Persian Fake News Detection: Neural Representation and Classification at Word and Text Levels. Transactions on Asian and Low-Resource Language Information Processing, 21, 1 - 11.
https://doi.org/10.1145/3472620 - Koru, G. K., & Uluyol, Ç. (2022). Fake news detection in Turkish using machine learning algorithms and fastText with word embedding. In 2022 7th International Conference on Computer Science and Engineering (UBMK) (pp. 410–415). IEEE. https://doi.org/10.1109/UBMK55850.2022.9919518
- Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the ACL (pp. 8440–8451). https://doi.org/10.18653/v1/2020.acl-main.747
- Cruz, J. C. B., Tan, J. A., & Cheng, C. (2020). Localization of fake news detection via multitask transfer learning. In Proceedings of the 12th Language Resources and Evaluation Conference (pp. 2596–2604). European Language Resources Association. https://aclanthology.org/2020.lrec-1.316/
- Dai, H., Liu, Z., Liao, W., Huang, X., Cao, Y., Wu, Z., Zhao, L., Xu, S., Liu, W., Liu, N., Li, S., Zhu, D., Cai, H., Sun, L., Li, Q., Shen, D., Liu, T., & Li, X. (2023). AugGPT: Leveraging ChatGPT for Text Data Augmentation. IEEE Transactions on Big Data, 11, 907-918. https://doi.org/10.48550/arXiv.2302.13007
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
- Gereme, F., Zhu, W., Ayall, T., & Alemu, D. (2021). Combating Fake News in “Low-Resource” Languages: Amharic Fake News Detection Accompanied by Resource Crafting. Information, 12(1), 20. https://doi.org/10.3390/info12010020
- Hossain, M. Z., Rahman, M. A., Islam, M. S., & Kar, S. (2020). BanFakeNews: A dataset for detecting fake news in Bangla. In Proceedings of the 12th Language Resources and Evaluation Conference (pp. 2862–2871). European Language Resources Association. https://doi.org/10.48550/arxiv.2004.08789
- Isbarov, J., Huseynova, K., Mammadov, E., Hajili, M., & Ataman, D. (2024). Open foundation models for Azerbaijani language. In Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024) (pp. 18–28). Association for Computational Linguistics. https://doi.org/10.48550/arXiv.2407.02337
- Schütz, M., Pisoiu, D., Liakhovets, D., Schindler, A., & Siegel, M. (2024). GerDISDETECT: A German Multilabel Dataset for Disinformation Detection. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, & N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 7683–7695) https://doi.org/10.63317/3srv3uamd5em
- Kaddour, J., & Liu, Q. (2023). Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models. https://doi.org/10.48550/arXiv.2310.01119
- Kesgin, H. T., Yuce, M. K., & Amasyali, M. F. (2023). Developing and evaluating tiny to medium-sized Turkish BERT models. arXiv preprint arXiv:2307.14134. https://doi.org/10.48550/arxiv.2307.14134
- Lee, N., Wattanawong, T., Kim, S., Mangalam, K., Shen, S., Anumanchipalli, G., Mahoney, M., Keutzer, K., & Gholami, A. (2024). LLM2LLM: Boosting LLMs with novel iterative data enhancement. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 6498–6526). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.388
- Li, Z., Zhu, H., Lu, Z., & Yin, M. (2023). Synthetic data generation with large language models for text classification: Potential and limitations. In Proceedings of the 2023 Conference on EMNLP (pp. 10443–10461). https://doi.org/10.18653/v1/2023.emnlp-main.647
- Nakamura, K., Levy, S., & Wang, W. Y. (2020). Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In Proceedings of the 12th Language Resources and Evaluation Conference (pp. 6149–6157). European Language Resources Association. https://doi.org/10.48550/arXiv.1911.03854
- Sharma, D. K., & Garg, S. (2022). IFND: A benchmark dataset for fake news detection. Complex & Intelligent Systems, 8, 1123–1140. https://doi.org/10.1007/s40747-021-00552-1
- Shibu, H. M., Datta, S., Miah, M. S., Sami, N., Chowdhury, M. S., & Islam, M. S. (2025). From scarcity to capability: Empowering fake news detection in low-resource languages with LLMs. In Proceedings of the First Workshop on Natural Language Processing for Indo-Aryan and Dravidian Languages (pp. 100–107). Association for Computational Linguistics. https://doi.org/10.48550/arXiv.2501.09604
- Shu, K., Mahudeswaran, D., Wang, S., Lee, D., & Liu, H. (2020). FakeNewsNet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big Data, 8(3), 171–188. https://doi.org/10.1089/big.2020.0062
- Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and Efficient Foundation Language Models. ArXiv, abs/2302.13971. https://doi.org/10.48550/arXiv.2302.13971
- Wang, W. Y. (2017). “Liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the ACL (Volume 2: Short Papers) (pp. 422–426).
https://doi.org/10.18653/v1/P17-2067 - Wardle, C., & Derakhshan, H. (2017). Information disorder: Toward an interdisciplinary framework for research and policy making. Council of Europe Report DGI(2017)09. https://rm.coe.int/information-disorder/168076277c
- Ziyaden, A., Yelenov, A., Hajiyev, F., Rustamov, S., & Pak, A. (2024). Text data augmentation and pre-trained language model for enhancing text classification of low-resource languages. PeerJ Computer Science, 10, e1974. https://doi.org/10.7717/peerj-cs.1974
- Mehdiyev, J., & Shahbazov, V. (2026). AzFakeNews: Azerbaijani fake news dataset. GitHub. https://github.com/mehdiyevcs/AzFakeNews
