Our paper, "A Kernelised Stein Statistic for Assessing Implicit Generative Models," was published at NeurIPS 2022 (the Thirty-Sixth Conference on Neural Information Processing Systems). It's co-authored by Sentinel Assurance co-founder Wenkai Xu, together with Gesine D. Reinert (University of Oxford).
What the paper does
- Introduces a Kernelised Stein Discrepancy-type test, built on a non-parametric Stein operator, designed specifically for implicit generative models: ones you can sample from but that have no tractable density.
- Works with a flexible number of samples drawn from the generator while holding the observed dataset fixed, which matters for models that are cheap to sample from but expensive to compare against real data.
- Shows improved performance over existing quality-assessment methods, on both synthetic distributions and trained generative models.
Our take
Generative models are increasingly components that other systems, agents included, build on and implicitly trust. If you can't write down what a model's output distribution actually is, you still need a rigorous way to check whether it's behaving the way you think it is. That's the same standard worth holding any component to before it gets to sit upstream of a decision an agent makes.