Data Science and Learning

SciStream

Architecture and toolkit for data streaming between federated science instruments
SciStream architecture for enabling federated streaming between scientific facilities.

Today, we have more data than ever before. And tomorrow? We’ll have even more. However, data is often processed much slower than it’s collected. Sometimes, data is processed at speeds slower than the instruments collecting the data are capable of. This is because of challenges like slow network connections, security mismatches, and a lack of tools to handle large data volumes in real time. This means some powerful scientific instruments operate below their potential and/or discard significant portions of data without processing. 

SciStream aims to address data reporting and processing issues by designing and implementing efficient and secure memory-to-memory data streaming between reporting instruments and their receiving counterparts.

Plans to achieve this mission can be split into three key steps: 

  1. Develop the necessary components for smoother data streaming through the creation of a set of gateway nodes located at instruments and receiving facilities.  
  2. Create protocols to ensure data streaming between instruments and facilities is authenticated and transparent. 
  3. Build a prototype and develop additional tools and APIs necessary to create the node gateway in operational, real-world environments. 

As a crucial piece for making near-real-time analysis of high-speed data possible, SciStream has the potential to transform current scientific workflows. By removing bottlenecks associated with traditional systems, SciStream can enhance both the speed and efficiency of scientific data processing for all machines and applications that require memory-to-memory data streaming.