A version of this story, written by Jane Palmer, originally appeared on the .
Each day hundreds, if not thousands, of sensors record climate and environmental data at various points in Nevada for the Solar Nexus Project, which is focusing on the linkage between solar energy generation and Nevada's limited water resources. It鈥檚 information that will help researchers maximize the efficiency of solar energy while minimizing the impact on the environment. But this vast amount of data needs to be transported, stored, and represented in a way that will help the scientists interpret it.
鈥淣exus research is based on the observations we make in the data that we see, and the conclusions that we draw,鈥 says NEXUS researcher , a professor in the department of electrical and computer engineering at 性视界传媒. 鈥淪o by being able to process data efficiently, we, as scientists, begin to get a better idea of what works and what doesn鈥檛.鈥
The mission of NEXUS cyberinfrastructure (CI) is to support scientific discovery by providing data management, communication, and processing through the Nevada Research Data Center (NRDC). The challenge CI faces is that the volume of data is large, and continues to grow.
鈥淐onsidering the fact that most of these data are picked up on a 24/7 basis constantly, year around, you can imagine that there鈥檚 going to be a lot of data that is collected,鈥 says Latifi. 鈥淪o taming this explosion of data calls for efficient mechanisms to transfer this voluminous data and store it.鈥
Predicting the Future
Perhaps the most fundamental aspect of data management is communicating and banking the recorded observations so that they can be accessed, analyzed, and interpreted by scientists. But to do this for every recorded variable would be computationally unsustainable. To address this challenge, Latifi and his team have investigated data compression techniques: algorithms to reduce the amount of data that needs to be transferred and stored.
In particular the team has used neural network models, which, when fed large quantities of data, begin to recognize inherent patterns and trends in the information. These models effectively accumulate knowledge about how a particular system behaves. Using that stored knowledge, the model can then make accurate predictions about what type of data it might see in the future in certain circumstances. 鈥淚t is like the adage that says the past is a good indication of the future,鈥 Latifi says.
To 鈥渢rain鈥 the model, Latifi and his team fed it large amounts of climate data, variables such as humidity and temperature observed at specific locations in Nevada. Then, after the network was fully trained, instead of transferring the new observations, the scientists simply programmed the model to predict what it believed those variables would be, based on past experience.
鈥淭he way the compression works is you guess what the data is going to be and then you send only the difference between the predicted and actual values,鈥 Latifi says. 鈥淚f you鈥檙e right on, the difference would be zero and you have nothing to send.鈥 If the prediction doesn鈥檛 match the observations all that needs to be stored is the difference between the predicted and observed result, not the entire new recorded data set. 鈥淪o by making a sophisticated network model, we tried to come to what the next sequence of data is, you know, as closely as possible, and then encode the error that we made,鈥 Latifi says.
Using such a technique no information is lost in the compression. For such 鈥渓ossless compression鈥 typically the amount of data transferred is only reduced by a factor of two or three. But using the neural network model, the team has been able to achieve compression factors of eight or nine. 鈥淭hese compression ratios are phenomenal and should be able to save us lots of space in terms of storing the data and a lot of bandwidth in terms of communicating the data,鈥 Latifi says.
Speedy and Secure Solutions
The team also focuses on facilitating quick data transmission from the field and between scientists. 鈥淏ecause these sensors pick up the data constantly you want to be able to transmit them on the fly,鈥 Latifi says. 鈥淎nd you don鈥檛 want scientist A to wait forever to analyze the data sent by scientist B on another campus.鈥
To facilitate such speedy data transfer requires better network protocols, a focus of Latifi鈥檚 research. But Latifi鈥檚 research also ensures that such high-speed communication would be as secure as possible. To achieve this aim the team uses algorithms to identify security breaches and block potential threats and improved network protocols that allow different layers of security for data transmission.
Investigating such techniques for fast and secure data transmission proves a passion for Latifi. 鈥淲hen you can send gigabytes of information in a blink of an eye or when you can compress data by a factor of two or three better than it is supposed to be it鈥檚 exciting,鈥 Latifi says. 鈥淭he fact that you can save so much storage and communication time.鈥
Ultimately, however, he derives great satisfaction from being able to serve the mission of the NEXUS project. 鈥淚t鈥檚 just so exciting how we are able to connect different projects and bring people together and facilitate communication,鈥 Latifi says.
The Solar Nexus Project is a five-year research project funded by the National Science Foundation鈥檚 Experimental Program to Stimulate Competitive Research 鈥淓PSCoR鈥 focusing on the nexus of (or linkage between) solar energy generation and Nevada鈥檚 limited water resources and fragile environment.
