Many of us recall with not-so-fond memories the ubiquitous multiple-choice exams we faced in school. So why did a team of researchers at the U.S. Department of Energy’s Argonne National Laboratory decide to hold a multiple-choice questions hackathon for a project called AuroraGPT?
“Well, in this case we reversed the situation: the participants got to write the questions,” said Robert Underwood, a postdoctoral appointee in Argonne’s Mathematics and Computer Science (MCS) Division.
He explained that the goal of the hackathon, held July 8, was to have students and postdocs from across the lab help prepare evaluation questions that could be used to assess the scientific capabilities of AuroraGPT across different science domains.
The AuroraGPT project is leveraging Argonne’s Aurora exascale supercomputer to build a science-focused large language model (LLM). The aim is to provide Argonne scientists with an artificial intelligence research assistant with unprecedented capabilities for analyzing scientific problems, formulating hypotheses, designing experiments, and carrying out related tasks.
But essential to realizing this goal is that researchers understand what LLMs can do in their domains. The AuroraGPT team therefore developed an interface for researchers to ask questions and receive immediate responses.
“The question-authoring interface is integrated with state-of-the-art LLMs, allowing authors to create questions that would be challenging for LLMs to answer,” said Sandeep Madireddy, a computer scientist in Argonne’s MCS Division. “The inference team at AuroraGPT provides real-time feedback on the questions as they are written.”
The hackathon was a huge success, attracting about 70 attendees from across the lab. The participants created 275 questions in fields ranging from computer science and physics to materials science, chemistry, and climate (see Fig. 1). Approximately 90% of participants found it easy to contribute a question, and 85% found it easy to review questions.
“[The] experience is great. I feel excited about it! I want to contribute more,” said one attendee. And indeed, about 85% of the attendees said they plan to continue contributing to the effort.
“The questions, coupled with more than 400 reviews, are critical to ensuring that AuroraGPT can perform the kinds of tasks we do every day at the lab,” said Franck Cappello, a senior computer scientist in Argonne’s MCS Division. “We are really thankful for the support of the Argonne community.”