CDS&E: Two- and Three-point Correlations for Large Data Sets with TreeCorr

University Of Pennsylvania, Philadelphia PA

Investigators

Abstract

This researcher has developed the leading software for measuring two-point correlation functions on large data sets in astronomy and elsewhere. However, for very large datasets the current version must be split across multiple computer nodes, which can be complicated to do correctly and efficiently. This project will enhance the software to use the Message Passing Interface (MPI) to split a job efficiently over a given number of nodes. Other software changes will further increase efficiency, add capabilities for three-point cross correlations, and include a more accurate shear estimation method. All of these modifications have been explicitly requested by current users of the software, and will make it especially useful for future experiments with much larger data volumes and tighter accuracy requirements. These improvements will help many scientists to carry out their research more efficiently and effectively. Throughout the project, the principal investigator will continue his campaign to improve the coding skills of other astronomers, helping them with code design, algorithm choices, good development workflow, and other software development issues. Some of his lessons are already freely available through YouTube. The software, TreeCorr, uses a very efficient ball-tree algorithm to compute correlations in O(N log N) time, making it significantly faster than other available options. TreeCorr uses OpenMP within a single computer node, and adding MPI will provide optimum overall efficiency. Other improvements include changing internal code for how TreeCorr decides when to stop traversing its tree, allowing arbitrary binning and arbitrarily small bin slop. Three-point auto-correlations are already supported at O(N log N) complexity, but the three-point cross-correlations that enable studies such as count-count-shear have been unavailable due to the complex bookkeeping required: this limitation will be removed. The new shear method, MetaCalibration, requires additional calculations that again can be somewhat complicated to do correctly. Correct and efficient code for MetaCalibration will be built into TreeCorr. This software has been downloaded via pip over 40,000 times and is used by many research teams working on current and forthcoming surveys: they will all benefit greatly from the algorithmic advances to be made by this study. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.