Nexus Services

To enable integration of computing at the ALCF with experiments at DOE facilities, Nexus has deployed a collection of services to meet the needs of scientists conducting experiments, including simplified authentication, on-demand queues for time-sensitive analyses (e.g. at experiment time), management of workflows for moving data and submitting and monitoring jobs, and shared access to collected results. 

Service Accounts
DOE user facilities typically grant time competitively to users who visit the facility to run their experiment and return to their home institution with their data. Experiments at the 68 beamlines at Argonne’s Advanced Photon Source can run 24 hours a day for three days straight, requiring an experimental team to work in shifts throughout the entire time. Beamlines enabled for remote analysis at the ALCF require an ALCF account, but APS users typically don’t have ALCF accounts and requiring them would create a delay. 

Nexus overcomes this limitation by issuing service accounts for the beamlines. When a dataset is collected at the beamline, a Globus Flow is instantiated to transfer the data to the ALCF and submit a job using this service account. This happens automatically; the beamline scientist does not need to access or create an ALCF account.

On-demand Queueing
Experiment-time analysis is by its nature time sensitive, requiring immediate access to computing during experiment cycles. Unlike modeling and simulation workloads which compute continuously, experimental computing can occur with a cadence peculiar to the work being done. For example, in fusion experiments at the DIII-D National Fusion Facility experiments repeat on a 20-minute cycle that requires data analysis at this same pace. Using the ​“demand” queue, Nexus enables experiments to obtain computing cycles with zero-to-minimal wait time. Between analysis cycles, the compute nodes can be idle; during these idle periods, ​“preemptable” jobs run; users who submit these jobs have prepared their code to be productive within short windows, even if they are terminated to allow on-demand jobs to run.

Workflows
Workflows are at the core of integrating experimental facilities with remote computing facilities, encompassing all aspects of managing experiment-time computing. Nexus relies on Globus Compute and Globus Flows for translating work back and forth between facilities. The Globus Compute model allows users to define functions that wrap analysis codes which can then be securely scheduled for execution on a target resource where a Globus Compute ​“endpoint” has been started by the user. This is similar to the endpoint model used by Globus Transfer and leverages the same authentication infrastructure. The Globus site has full documentation and examples for Globus Compute.

Globus Flows enables users to compose complex workflows from a collection of Compute functions and Transfers. Beyond the simple workflow case of TransferIn-Execute-TransferOut, users can construct graphs of operations that include many transfer and execution steps, targeting multiple different storage and execution endpoints (e.g. Polaris at the ALCF and a local computing cluster), with conditional steps and branching to adapt to workflow dependencies or machine availability. The Globus site has documentation and examples of Globus Flows.

While Nexus has chosen to deploy Globus tools, we recognize that a wide range of tools are used across experimental communities, and are willing to work with teams to deploy them. 

Data Sharing
The ALCF has deployed Globus Sharing to enable users to share data with collaborators directly from ALCF filesystems via the Globus web application or command-line applications. This capability allows users to share their project data via a scalable security model, granting access to specific directories to individuals who have an ALCF account, a Globus account, or publicly over a network via https. For more information on how to enable sharing for your project data, see the ALCF documentation.