Whether studying the fundamental behavior of metals or the particles that compose them, designing more functional materials or reconstructing the evolution of the universe, researchers are finding the resources to accomplish their goals at one or a combination of U.S. Department of Energy (DOE) Office of Science User Facilities. These often-massive scientific facilities enable science at scale, providing researchers the most advanced tools of modern science.

Each year, DOE’s 27 user facilities welcome more than 30,000 researchers, called users, from across the nation and around the world. Many leading innovative efforts in their fields, they come to access the array of advanced technological and leadership-class computational resources not readily available to many companies or other research institutions.

The competitive nature of the applications required to use the facilities beckons the best and brightest researchers, users motivated to make the kinds of pivotal discoveries that can radically advance whole fields of science, provide industries with a competitive edge, or ensure the safety and resiliency of our nation.

DOE Office of Science User Facilities are pioneering leaders in the areas of materials, physics, supercomputing, energy science and the environment, each providing the best tools for the job. For example, at its three Advanced Scientific Computer Research (ASCR) computing facilities, DOE offers its users access to some of the fastest and most powerful supercomputers in the world.

While certain facilities serve as standalone, highly focused entities, others are highly collaborative, like the Atmospheric Radiation Measurement (ARM) facility, which involves nine DOE laboratories facilitating climate science research at stationary and mobile sites around the world.

DOE’s Argonne National Laboratory is home to four world-class DOE Science User Facilities and operates a fifth, ARM’s Southern Great Plains atmospheric observatory, in Lamont, Oklahoma. From the Advanced Photon Source, among the DOE’s busiest user facilities, to the Argonne Leadership Computing Facility, preparing to unveil the nation’s first exascale supercomputer, Argonne provides a combination of user facilities designed to address the most complex scientific, technical and manufacturing issues.

Moreover, there is a synergy and collaboration among the user facilities, where multidisciplinary teams frequently take advantage of multiple facilities and world-class resources to solve complex problems, squarely defining the Argonne directive of producing science at scale.

For example, research into developing a solid material to conduct lithium ions more quickly through electrical systems not only crossed the domains of materials science and condensed matter physics, but involved the resources and cooperation of three research institutions and three Argonne user facilities.

“The Office of Science User Facilities have played a pivotal role in advancing scientific knowledge through unfaltering support of its user communities and the nation,” says Juan de Pablo, Vice President for National Laboratories at the University of Chicago. “We are proud to continue that legacy by helping to provide the types of exceptional resources users have come to expect from DOE, Argonne and the University.”

The following highlight just a few of the users and the research they’re conducting at DOE Office of Science User Facilities at Argonne.

“The Office of Science User Facilities have played a pivotal role in advancing scientific knowledge through unfaltering support of its user communities and the nation. We are proud to continue that legacy by helping to provide the types of exceptional resources users have come to expect from the DOE, Argonne and the University.”  — Juan de Pablo, Vice President for National Laboratories at the University of Chicago.

Advanced Photon Source

Despite our complex biology, the human immune system is easily defeated by deadly viruses, some made up of half a dozen or fewer moving parts. Discovering how to defeat them is the work of Erica Ollmann Saphire, and just one measure of the broad scope of research users are addressing at the Advanced Photon Source (APS) at Argonne.

“How does a virus with only four genes defeat human cells made up of over 20,000 genes?” asks Saphire, a professor at the La Jolla Institute for Immunology, in La Jolla, California, and a user of the Eli Lilly protein crystallography beamline, among those operated by APS collaborative access teams. “If we can see the structures of the viral molecules, understand their functions and learn where they are vulnerable, we can design vaccines and antibodies to target and defeat these emerging viruses.”

The ability to “see” infinitely small structures and understand hidden processes in unprecedented detail is what attracts users like Saphire to the facility. Providing the brightest source of high-energy X-rays in the Western Hemisphere, the APS attracts more than 5,700 users in a typical year.

Hailing from academia, industry and other research institutions, these users — including remote and mail-in users — conduct studies on subjects as disparate as condensed matter physics and dinosaur physiology.

“We’ve just been really happy at APS. We get the beam time we need and working with specialty beamlines, like Eli Lilly’s, has helped us get some things discovered, crystallized, solved and out during the course of a disease outbreak.” — Erica Ollmann Saphire, Professor, La Jolla Institute for Immunology.

Where high-brightness beamlines help Saphire’s team illuminate the critical structures of a virus’s surface proteins, Anthony Rollett is using X-ray imaging to understand laser melting in additive manufacturing.

In a recent Science article, Rollett and his team describe the use of APS X-rays to study the formation of keyholes, or vapor depressions — a phenomenon inherent in 3D metal printing that can cause fatigue in printed metal parts.

“We use a mock-up of a (part) that we can put in front of the X-rays and then use those X-rays to interrogate exactly what is happening at the very high speeds that might occur during the actual laser melting process,” explains Rollett, a professor in the Department of Materials Science and Engineering at Carnegie Mellon University.

Despite the APS’ expansive suite of beamlines, it certainly isn’t the only game in town — DOE’s Office of Science supports a total of five X-ray light source user facilities in the United States. But users have additional reasons for coming.

“SLAC has excellent beamlines as well, and we’ve just been really happy at APS,” notes Saphire. “We get the beam time we need and working with specialty beamlines, like Eli Lilly’s, has helped us get some things discovered, crystallized, solved and out during the course of a disease outbreak.”

Rollett argues that the APS has “the hottest and the mostest X-rays” of any other rings available and that they will only get better with the completion of the APS Upgrade Project.

“The upgrade will offer better focus and higher intensity of X-rays, which is always useful for imaging experiments,” he says. 

Argonne Leadership Computing Facility

“We are Argonne people,” exclaims Aiichiro Nakano, a computer scientist with the University of Southern California (USC). Nakano’s history as a former Argonne employee, coupled with the intensity of his research that requires world-class supercomputing resources, makes him a dedicated regular at the Argonne Leadership Computing Facility (ALCF).

One of three DOE leadership computing facilities in the nation dedicated to open science — the others located at Oak Ridge and Lawrence Berkeley National Laboratories — the ALCF played host to nearly 1,000 national and international users, last year.

Since its inception in 2004, the ALCF has pushed the tech envelope, consistently housing some of the most powerful supercomputers in the world while advancing new codes and software to drive those computers and the research conducted on them.

And with more than 400 projects currently on the books, ALCF resources, such as the Theta and Mira supercomputers, are driving discovery in a wide range of science, from the acoustics of jet engines to the dynamics of cosmic structures.

Sean Couch, a long-time ALCF user and current member of its User Advisory Council, attributes his 10-year affiliation with the ALCF to this balance of hardware, software and science.

Using large-scale 3D simulations to investigate the way in which large stars collapse, his work requires large portions of ALCF supercomputers.

“Without the ALCF’s resources, we couldn’t push the leading edge of what’s computationally possible,” says Couch, assistant professor of astronomy and astrophysics at Michigan State University. “And the scale of computing continues to increase, allowing us to add complexity and scale to our work.”

Nakano is using the ALCF to study how light can change the structures and properties of atomically thin materials to create better, cheaper semiconductors. His team pushes the boundaries of Mira’s capabilities to fully reveal how individual electrons behave to result in the observed atomic motion.

Next steps in Nakano’s research involve folding these ultra-thin sheets, which could create altogether-new materials. But to study that will take more supercomputing power than is currently available.

“We want to revolutionize the way we make devices. We want to do simulations that are a 100 times bigger, and the ALCF and Aurora will get us there.” — Aiichiro Nakano, computer scientist, University of Southern California.

The future for the DOE and the ALCF is the introduction of the exascale computer, Aurora, in 2021. Capable of operating at a quintillion calculations per second, it will help researchers like Nakano and Couch take their work to the next level and beyond.

“We want to revolutionize the way we make devices,” says Nakano, who is leading an Aurora Early Science Program project to prepare for the future exascale system. “We want to do simulations that are a 100 times bigger, and the ALCF and Aurora will get us there.”

Argonne Tandem Linac Accelerator System

From fundamental research that tests the Standard Model of particle physics to applications such as nuclear forensics, the Argonne Tandem Linac Accelerator System (ATLAS), a pioneering superconducting linear accelerator, plays an integral role in exploring the complexities of nuclear physics.

Each year, the facility hosts 200–300 users who are attracted to ATLAS’s innovative combination of experimental equipment and flexibility in accelerating ions of all elements, both stable and radioactive, to explore the structure, dynamics and decay properties of nuclei.

For Nick Scielzo, a physicist at Lawrence Livermore National Laboratory and Chair of the ATLAS User Executive Committee, one of the draws to ATLAS is the CARIBU (Californium Rare Isotope Breeder Upgrade) facility, where he studies the beta decay of atomic nuclei. This type of radioactive disintegration plays an important role in the production of heavy elements in the cosmos and the operation of nuclear reactors, and provides key information for nuclear-security applications.

A postdoc at Argonne in the early 2000s, Scielzo continues his relationship with ATLAS staff, in part, through his work with CARIBU, which can deliver pure beams of individual isotopes produced by nuclear fission at the intensities required for precise measurements of beta decay, he notes.

“Argonne and Livermore have worked together to pioneer several new approaches to collect and confine radioactive ions, and to study all the radiation emitted from beta decay,” says Scielzo. “Now we are tackling a variety of long-standing problems by combining these approaches with the high-quality beams available at ATLAS.”

Steven Pain comes to ATLAS equipped with his own state-of-the-art particle detector. Called GODDESS (Gammasphere ORRUBA: Dual Detectors for Experimental Structure Studies), it works in concert with world-class gamma-ray spectrometers GRETINA (Gamma-Ray Energy Tracking In-Beam Nuclear Array) and Gammasphere located at ATLAS to capture a glimpse of both the charged particles and gamma rays emitted from reactions caused by energetic beams of specific nuclei.

“Our collaboration of more than 50 scientists brought GODDESS to ATLAS for this and a number of other experiments probing nuclear structure and astrophysics. ATLAS is currently the only place in the world with the marriage of equipment and beams we need to effectively study these reactions.” — Steven Pain, research scientist, Oak Ridge National Laboratory.

A research scientist in astrophysics at Oak Ridge National Laboratory, Pain’s work focuses on the role of nucleosynthesis in stellar explosions, like novae, supernovae and neutron-star mergers. One of the nuclei he is interested in is phosphorus-30, which is important for understanding thermonuclear runaway in novae, occurring on the surface of white dwarf stars.

“This experiment requires a pure and intense beam of phosphorus-30 at the right energy, combined with particle and gamma-ray detectors with very high resolution and efficiency,” says Pain. “Our collaboration of more than 50 scientists brought GODDESS to ATLAS for this and a number of other experiments probing nuclear structure and astrophysics. ATLAS is currently the only place in the world with the marriage of equipment and beams we need to effectively study these reactions.”

Center for Nanoscale Materials

Advanced techniques and state-of-the-art tools make Argonne’s Center for Nanoscale Materials (CNM) an invaluable commodity to users exploring materials and phenomena at the nanoscale.

While such amenities contribute to science at scale and faster time to solution, long-time CNM user and former Argonne researcher Goran Karapetrov finds access to knowledgeable staff just as important to his work.

“I think the CNM’s gold mine is its people,” says Karapetrov, professor of physics at Drexel University in Philadelphia. “The interactions with staff members may be the most valuable resource a user facility can provide.”

Karapetrov is using CNM’s ultrafast lasers to study transient optical reflectivity of quantum materials at very fast time scales — information that could lead to the development of novel computational devices, some with potential for use in quantum computers.

In addition to lasers, he takes advantage of other CNM resources, including cleanroom-based nanofabrication techniques, like lithography. And while he is also familiar with many of CNM’s 100-plus tools and capabilities, he maintains that the staff is the essential element on the path to pivotal discoveries.

“Whenever you need to use new techniques or tools — some of them quite sophisticated — it’s important that you have people you can talk to who are experts in the areas you are studying,” says Karapetrov.

Launched in 2007, the CNM rounded out an already impressive roster of user facilities at Argonne and is one of five DOE Nanoscale Science Research Centers (NSRC), important contributors to the federal National Nanotechnology Initiative, currently celebrating its 15^th anniversary.

“I think the CNM’s gold mine is its people. The interactions with staff members may be the most valuable resource a user facility can provide.” — Goran Karapetrov, professor of physics, Drexel University.

The CNM’s 600-plus annual users come from a broad range of institutions, industries and geographies to take advantage of the center’s unique array of cutting-edge tools and capabilities and to work with CNM’s scientific and user support staff.

 “It’s really amazing to have that kind of access to state-of-the-art facilities,” says Diana Berman, assistant professor in the Department of Materials Science and Engineering at the University of North Texas. “CNM allows us to do detailed synthesis, characterization and analysis of the new materials and structures all in one place and at the same time.”

Berman’s team studies the potential for porous materials to act as anti-reflective coatings that can significantly reduce light reflection from different surfaces. While the material has many applications, one of them is to increase energy absorption and efficiency in solar panels.

Recognizing CNM’s importance in providing academia and industry with access to its state-of-the-art facilities, she concurs with Karapetrov’s nod to the expertise CNM provides.

“There are other user facilities, but they don’t have the experience we need for the work we’re doing. We highly appreciate that at CNM we can talk to experts in the field who can help with our measurements and guide our analysis process,” says Berman.

“That’s what CNM provides. It really helps us get to the heart of a problem.”

Funding

ALCF Funding

Sean Couch’s work is supported by a DOE Office of Science Early Career Award and the DOE Office of Science SciDAC.

Aiichiro Nakano’s work was supported as part of the Computational Materials Sciences Program funded by the DOE Office of Science, Basic Energy Sciences.

APS Funding

Anthony Rollett’s work is supported by the U.S. Department of Defense, Office of Economic Adjustment; Laboratory Directed Research and Development (LDRD) funding from Argonne National Laboratory, provided by the Director of DOE’s Office of Science; the DOE National Nuclear Security Administration; and the National Aeronautics and Space Administration.

Erica Ollmann Saphire’s work is funded by the National Institute of Allergy and Infectious Diseases.

ATLAS Funding

Steven Pain’s work is supported by the DOE Office of Science, Nuclear Physics, and the DOE National Nuclear Security Administration’s Stewardship Science Academic Alliances (SSAA) program.

Nick Scielzo’s work is supported by the DOE Office of Science, Nuclear Physics; Lawrence Livermore National Laboratory’s LDRD program, and the DOE National Nuclear Security Administration.

CNM Funding

Diana Berman’s work at CNM is currently supported through University of North Texas start-up funding.

Goran Karapetrov’s project is supported by the National Science Foundation.