DOE's cloud computing project wins "Best Use of HPC in the Cloud" award

By Eleanor TaylorDecember 1, 2010

The U.S. Department of Energy’s cloud computing testbed project, known as “Magellan”, has been recognized in the annual HPCwire Readers’ and Editors’ Choice Awards. Representatives of Argonne and Lawrence Berkeley national laboratories, where Magellan testbeds are located, received the award at SC10, the international conference for high-performance computing, networking, storage and analysis, in New Orleans.

The Magellan project, which is exploring the suitability of cloud computing to help meet the computation science requirements of DOE researchers, was honored with the Readers’ Choice Award for “Best Use of High Performance Computing (HPC) in the Cloud.” Magellan systems, funded under the American Reinvestment and Recovery Act, are located at the Argonne Leadership Computing Facility (ALCF) in Illinois and the National Energy Research Scientific Computing (NERSC) Center at Berkeley Lab in California.

The HPCwire Readers’ Choice Awards are determined through online polling of the global HPCwire audience, along with a rigorous selection process involving HPCwire editors and industry luminaries. The awards are an annual feature of the publication and constitute prestigious recognition from the HPC community. These awards are revealed each year to kick off the SC conference, which showcases high performance computing, networking, storage, and data analysis.

"Magellan is a research project that involves a hardware testbed at both sites and experiments with a wide range of scientific applications running on different types of cloud software. The studies have already revealed a number of interesting opportunities and challenges of using a cloud model for science, and it’s rewarding to see these contributions recognized by the HPCwire audience," said NERSC director Kathy Yelick.

"We are excited to see the impact Magellan results are having in the cloud research community, especially in such a short time," said Susan Coghlan, who leads Argonne's Magellan project. "Argonne and NERSC are looking forward to continuing to explore all of the opportunities and impacts of the cloud on midrange scientific computing."

SC10 attendees learned more about how Magellan is being used for evaluating cloud computing in science in two presentations on Wednesday, Nov. 17. Argonne’s Susan Coghlan presented "DOE ARRA Magellan Cloud Testbed Update", and NERSC’s Shane Canon, the research lead for the NERSC Magellan effort, presented "Magellan at NERSC: A Testbed to Explore Cloud Computing for Science." Canon’s presentation featured live demos of Magellan running real science workloads. Earlier in the week, a workshop on petascale data in the cloud also featured talks by Coghlan, Canon and Yelick.

One of the first key users of NERSC's Magellan cloud computing system was DOE’s Joint Genome Institute (JGI). In March, the facility had a sudden need for increased computing resources, and in less than three days, NERSC and JGI staff provisioned and configured an experiment involving hundreds of processor cores on the Magellan system to match the computing environment available on JGI's local compute clusters. At the same time, staff at both centers collaborated with network engineers from DOE’s Energy Sciences Network (ESnet) to deploy a dedicated 9Gbps virtual circuit between NERSC's Magellan system and JGI over ESnet's Science Data Network within 24 hours. This strategy demonstrated that JGI researchers could access increased computational capacity without any changes to their software or workflow, or without having to move large datasets. JGI users were able to submit scientific computing jobs locally to queues that are managed by hardware located at JGI in Walnut Creek, Calif., but the jobs execute remotely at NERSC.

Projects taking place at Argonne include investigating Special Priority and Urgent Computing Environment, or SPRUCE, in a cloud environment and working with the Open Science Grid to allow condorG jobs to be run seamlessly on Magellan. Argonne is also investigating multiple open source software stacks for use in scientific computing. In the application space, members of the Deep Soil Consortium were able to analyze the microbial populations of two distinct locales, using software that was created for Argonne's MG-RAST metagenomics analysis server. Another group is using a cloud environment to support decision makers in their response to natural and man-made incidents. The first example of this will be in support of Pandemic Influenza Modeling using the EpiSimdemics and EpiFast simulations developed by Virginia Tech's Network Dynamics and Simulation Science Laboratory.

More information on the HPCwire awards can be found at the HPCwire website.