Exploring the Black-Box: Testing Image Synthesis Systems through Metamorphi

The increasing complexity of deep learning models, especially in black-box scenarios, presents significant challenges to traditional software testing methods. Due to the lack of transparency in neural networks’ decision-making processes and the non-deterministic nature of model outputs, traditional test oracle approaches become inadequate. To address this problem, Metamorphic Testing (MT) and its extended approach, Metamorphic Exploration (ME), provide new ideas for validating deep learning systems by defining Metamorphic Relations (MR) between inputs and outputs. However, existing image transformation-based MR faces new challenges in image synthesis scenarios, as these operations may destroy the contextual information and affect the model’s performance. This paper proposes a novel ME design for deep learning image synthesis networks and demonstrates its effectiveness using a visible-infrared image fusion network as the case study. The result identifies the performance degradation problem due to the tensor dimension manipulation error, which indicates that the ME not only detects defects but also helps developers deeply understand the internal mechanisms of complex systems through the Hypothesized Metamorphic Relation (HMR), thus providing unique value for software quality assurance (SQA) of AI-driven software.