Evaluating robustness of hierarchical and multi-label requirements classification: Effects of data partitioning and supervision design

Context: Natural language processing techniques for software requirements classification have achieved strong performance in binary Functional vs. Non-Functional Requirement (FR vs. NFR) tasks. However, performance degrades substantially when addressing deeper taxonomies, multi-label requirements, and evaluation protocols that better reflect real-world conditions. Existing studies largely rely on single datasets, flat or shallow taxonomies, and random cross-validation, so the effects of data partitioning and supervision design are insufficiently understood. Objective: This study investigates how data splitting strategies and supervision design jointly influence the robustness and reliability of hierarchical software requirements classification across multiple taxonomy levels. Methods: We define a three-level FR/NFR taxonomy and propose a hierarchical transformer-based classification model with two supervision schemes: (i) a joint end-to-end model trained across all hierarchy levels, and (ii) a staged supervision approach that propagates predictions sequentially from parent to child levels. The models are evaluated under multiple data partitioning strategies to assess stability and performance degradation across hierarchy depths. Results: Under the best-performing configuration, the proposed model achieves Macro-F1 scores of approximately 0.90, 0.72, and 0.55 at Levels 1–3, respectively, with Micro-F1 exceeding 0.93 at Level 1 and 0.69 at Level 3. The staged supervision strategy consistently improves performance at deeper levels and demonstrates greater robustness under challenging split conditions compared to joint optimization. Conclusion: The results demonstrate that both data partitioning strategy and supervision design play a critical role in hierarchical requirements classification. Staged supervision mitigates hierarchical performance degradation and yields more reliable evaluation outcomes, which shows the need for evaluation protocols and model designs that explicitly account for taxonomy depth and data characteristics.