Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidance
Saeid Naderiparizi, Xiaoxuan Liang, Setareh Cohan, Berend Zwartsenberg, Frank Wood
TL;DR
This paper introduces Gen-neG, a novel score-based generative modeling method that uses oracle guidance to improve generation within constrained domains, demonstrated in safety-critical applications like self-driving and human motion.
Contribution
We develop Gen-neG, a diffusion model that incorporates oracle side-information to effectively generate data within support constraints, advancing constrained domain modeling.
Findings
Gen-neG effectively guides generation within support regions.
Application to collision avoidance in self-driving improves safety.
Enhances human motion generation with safety constraints.
Abstract
Score-based diffusion models are a powerful class of generative models, widely utilized across diverse domains. Despite significant advancements in large-scale tasks such as text-to-image generation, their application to constrained domains has received considerably less attention. This work addresses model learning in a setting where, in addition to the training dataset, there further exists side-information in the form of an oracle that can label samples as being outside the support of the true data generating distribution. Specifically we develop a new denoising diffusion probabilistic modeling methodology, Gen-neG, that leverages this additional side-information. Gen-neG builds on classifier guidance in diffusion models to guide the generation process towards the positive support region indicated by the oracle. We empirically establish the utility of Gen-neG in applications…
Peer Reviews
Decision·ICML 2024 Poster
- The problem that the authors study is interesting. In many applications, it is critical to ensure that the generated samples do not violate certain rules. - The proposed method is intuitive. - The theoretical results seem correct. - The presentation of the paper is clear. - Experimentally, the method seems to achieve its objective in the settings where it is tested.
- There are no comparisons with baseline methods. It would be nice to include numerical comparisons with other methods that have tried to solve the same problem, e.g. some of the methods mentioned in the Prior Work section. - The theoretical results are not exactly new. A lot of the proofs use standard results.
The paper has good figures to support the claim and is fairly well structured. The proofs seem to correctly support the theoretical claims made. A wide number of experimental settings were explored making the paper and its proposed method more compelling.
If the experiments could have more comparison with other discriminator-guided methods that would be helpful to understand the work in a better context. A couple of the proofs in the supplementary seem rather elementary such as the delta method and the relationship between kl divergence and cross entropy. It might be better to just cite textbooks for them.
The paper provides a solution to enforce constraints in the generative process for diffusion models, by reusing standard training of diffusion models and classifier guidance. The concept is relatively simple and straightforward, and the empirical results show effectiveness in reducing the generation of data outside of the support of the data distribution on some simple datasets.
Some parts of the paper are poorly written and have errors and skipped words. Some sections of the appendix are not referenced in the paper. In my opinion, a reference to works such as [1,2] and a paragraph explaining differences and similarities could be beneficial. The proposed procedure, while improving on reducing the percentage of "invalid" samples, tends to reduce the ELBO, making it an appealing solution only when generating valid samples is crucial even at the expense of the quality of t
The major strength of this paper is that the proposed method is simple yet natural and effective. This makes it easy to understand and readers would tend to believe it will wrok. It leverages the advantages of several exsiting techniques: training a classifier from the oracle, using guidance to suppress negative samples, and distillation for better efficiency. Another strength is the two downstream tasks do face the realistic negative-sample concerns, which further emphasizes the importance of
One weakness is that there is no theoretical guarantee of the effectiveness of the proposed method. In practice the classifier is not perfect, leading to a bias in the guidance term. It is important to show that infraction can be upper bounded by a small constant if the learned classifier is near optimal. It is also vital to characterize differences for binary classifiers with different Lipschitz constants (i.e. whether the output changes slowerly vs drastically) near the decision boundary. One
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Gaussian Processes and Bayesian Inference
MethodsDiffusion