Fast Bayesian Record Linkage for Streaming Data – Andee Kaplan

On July 21, 2022, Andee Kaplan presented work on record linkage for streaming data. In streaming record linkage, files arrive sequentially in time and estimates of links are updated after the arrival of each file. This problem arises in settings such as longitudinal surveys, electronic health records, and online events databases, among others. The challenge in streaming record linkage is to efficiently update parameter estimates as a new data file arrives. In this talk, we will approach the problem from a Bayesian perspective and present methods for updating link estimates after the arrival of a new file that is faster than fitting a joint model with each new data file. To accomplish this aim, we generalize a two-file Bayesian Fellegi-Sunter model to the multi-file case and present two methods to perform streaming updates. We examine the effect of prior distribution on the resulting linkage accuracy as well as the computational trade-offs between the methods when compared to a Gibbs sampler through simulated and real-world survey panel data. With these streaming updates, we are able to achieve near-equivalent posterior inference at a small fraction of the compute time.