The original Brookhaven National Laboratory feature article can be read here.
Light source facilities are tackling some of today’s biggest scientific challenges, from designing new quantum materials to revealing protein structures. But as these facilities continue to become more technologically advanced, processing the wealth of data they produce has become a challenge of its own. By 2028, the five U.S. Department of Energy (DOE) Office of Science light sources, will produce data at the exabyte scale, or on the order of billions of gigabytes, each year. Now, scientists have come together to develop synergistic software to solve that challenge.
With funding from the DOE for a two-year pilot program, scientists from the five light sources have formed a Data Solution Task Force that will demonstrate, build, and implement software, cyberinfrastructure, and algorithms that address universal needs between all five facilities. These needs range from real-time data analysis capabilities to data storage and archival resources.
“It is exciting to see the progress that is being made by all the light sources working together to produce solutions that will be deployed across the whole DOE complex,” said Stuart Campbell, leader of the data acquisition, management and analysis group at the National Synchrotron Light Source II (NSLS-II), a DOE Office of Science user facility at DOE’s Brookhaven National Laboratory.
In addition, the new software will be designed to facilitate multimodal research—studies that combine data collected from multiple experimental stations, called beamlines. Typically, each beamline at a light source uses custom-built data acquisition software that is incompatible with another beamline’s, making it difficult for scientists to collect and compare data from multiple experimental stations. The task force aims to develop flexible software that can be deployed at multiple beamlines across all five facilities, expanding the possibilities for scientific collaboration
To develop the new software, the task force will start by building up existing solutions that can already be found at the five light sources. Two of the key components are Bluesky, an open source software that was created at NSLS-II, and Xi-CAM, which was developed at the Advanced Light Source (ALS) and the Center for Advanced Mathematics for Energy Research Applications—both at DOE’s Lawrence Berkeley National Laboratory. Together, Bluesky and Xi-Cam will provide capabilities like live visualization and interactivity, data processing tools, and the ability to export data in real time into nearly any file format.
Each of the five light sources in the task force is bringing unique tools and skillsets to help develop a more robust and scalable solution to extract scientific knowledge from data for the nation’s light sources. Task Force representatives from the Argonne National Laboratory Advanced Photon Source (APS) are Eric Dufresne, Pete Jemian, Faisal Khan, Jeff Kirchman, Suresh Narayanan, Alec Sandy, Nicholas Schwarz, and Qingteng Zhang, all with the Argonne X-ray Science Division (XSD) of the APS, which is a DOE Office of Science user facility at the DOE’s Argonne National Laboratory.
“There is tremendous enthusiasm at the light sources for solving the data challenge,” said Alexander Hexemer, senior scientist and computing program lead at ALS. “We strongly believe this will be the path forward for light sources to work together in the future.”
Schwarz, Group Leader of the XSD Scientific Software Engineering & Data Management Group said, “The potential to impact the development of state-of-the-art data processing and experiment control by working together across all of the light sources is enormous. This is the first time all of the DOE light sources have worked together to develop common solutions for experiment control and data processing at this scale. The APS is excited to work with the other x-ray light sources that are supported by the Basic Energy Sciences program to develop solutions that will be applied across the complex. These will help ensure the success of the APS Upgrade.”
With the task force in its early stages, researchers have begun running test experiments on beamlines at NSLS-II and installing Bluesky and Xi-CAM at the APS.
By the end of the two-year pilot project, “we plan to deliver a set of tools that will provide an end-to-end software solution for the targeted scientific areas that can be deployed and used on different beamlines across all the DOE light sources,” Campbell said.
Alongside the task force pilot, the five light sources are working with DOE to develop data systems solutions that will scale to the unprecedented data rates that will be produced in the near future, using the new generation of “exascale” computers being built by DOE.
Brookhaven National Laboratory is supported by the U.S. Department of Energy’s Office of Science. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
The Advanced Photon Source is a U.S. Department of Energy (DOE) Office of Science User Facility operated for the DOE Office of Science by Argonne National Laboratory under Contract No. DE-AC02-06CH11357.
Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation's first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America's scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC, for the U.S. DOE Office of Science.
The U.S. Department of Energy's Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit the Office of Science website.