Factorized generative models, which typically aim to regularize a latent style variable \( z_s \) by aligning its marginal distribution with a Gaussian prior, may not achieve the intended independence from class information. Recent findings indicate that simply matching the marginal distribution does not constrain class-conditional distributions, allowing the latent style to retain significant predictive power over class labels, even when the aggregated appearance is Gaussian.
In an analytical decomposition, the study highlights that the mismatch between marginal and class-conditional distributions represents one of four conditions essential for effective factorized sampling. The model's empirical evaluation reveals that while achieving a near-zero global Maximum Mean Discrepancy (MMD), it still enables a linear probe to recover class labels with an accuracy range of 74% to 100%. Specifically, the model presented boasts a clustering accuracy of 99.15%, whereas the generation of class-conditional samples remains limited, succeeding only 16% of the time. This consistency in leakage persists across six variations in model capacity, curriculum, prior geometry, and supervision within two datasets.
Four different mitigation strategies were tested to decrease probe accuracy, achieving reductions to 21%–46%, though they did not effectively alter within-class dependence. Additionally, a post-hoc conditional prior improved class generation performance to 0.97 on MNIST and 0.41 on CIFAR-10. An empirical style bank produced a score of 0.88 on CIFAR-10. Overall, the results underscore that evaluating divergences solely based on marginal distributions does not certify independence from class labels, challenging the validity of claims often associated with factorized generative models.