Natural Language Processing (NLP) has seen rapid developments in the past few years with the introduction of the transformers model architecture and Large Language Models (LLMs). These developments have led to improved results for a multitude of NLP tasks, including text classification. A complex subproblem in text classification is Multi-Label Text Classification (MLTC), which aims to assign multiple labels (e.g., topics or emotions) to a single text. In this post, we introduce Multiscope, a Python tool that aims to facilitate the process of preparing multi-label data, in addition to training and evaluating multi-label classification models. The code and detailed user guidelines can be found on GitHub.
Loading Data
The tool accepts JSON, CSV and Excel files. The user can also opt to load in an existing dataset from the HuggingFace hub. Then, the user can choose to create custom train, validation and test splits from the data. The tool then provides statistics and visualizations from the data.


Training
After loading the data, the user can choose to train either a BERT-like transformer (Devlin et al. 2019) or a Support Vector Machine (SVM). For both options, the user can specify the models’ hyperparameters. For this, the tool leverages the Transformers library (Wolf et al. 2020) and the Sci-kit learn library (Pedregosa et al. 2011). The tool also integrates Weights & Biases, which tracks the training and evaluation process of the models.

Evaluation
After training a model, the tool provides multiple automatic evaluation methods. The tool calculates several classification metrics adapted to MLTC, provides a classification report and visualizes a multi-label confusion matrix (Heydarian et al. 2022).

How to Cite
Jens Van Nooten and Walter Daelemans. 2024. Multiscope: A User-Friendly Multi-Label Text Classification Dashboard. CLiPS Technical Report Series 11 (CTRS 11). ISSN 2033-3544. Computational Linguistics, Psycholinguistics, and Sociolinguistics Research Center.
References
Devlin, Jacob, Chang, Ming-Wei, Lee, Kenton, & Toutanova, Kristina. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, & Thamar Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). Minneapolis, Minnesota: Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
Mohammadreza Heydarian, Thomas E. Doyle and Reza Samavi. 2022. MLCM: Multi-label confusion matrix. IEEE Access, 10, 19083–19095. https://doi.org/10.1109/ACCESS.2022.3151048
Pedregosa, Fabian, Varoquaux, Gaël, Gramfort, Alexandre, Michel, Vincent, Thirion, Bertrand, Grisel, Olivier, Blondel, Mathieu, Prettenhofer, Peter, Weiss, Ron, Dubourg, Vincent, Vanderplas, Jake, Passos, Alexandre, Cournapeau, David, Brucher, Matthieu, Perrot, Matthieu, & Duchesnay, Édouard. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85), 2825–2830. Retrieved from http://jmlr.org/papers/v12/pedregosa11a.html
Wolf, Thomas, Debut, Lysandre, Sanh, Victor, Chaumond, Julien, Delangue, Clement, Moi, Anthony, Cistac, Pierric, Rault, Tim, Louf, Rémi, Funtowicz, Morgan, Davison, Joe, Shleifer, Sam, von Platen, Patrick, Ma, Clara, Jernite, Yacine, Plu, Julien, Xu, Canwen, Le Scao, Teven, Gugger, Sylvain, Drame, Mariama, Lhoest, Quentin, & Rush, Alexander M. (2020). HuggingFace’s Transformers: State-of-the-art natural language processing. arXiv. https://arxiv.org/abs/1910.03771


















