Data analysis is a challenging task. Traditional data analysis tools, regardless of their effectiveness, still have a few downsides.
Topological data analysis (TDA) involves extracting information from clouds of data points and using the information to analyze data, predict trends or recognize patterns.
Using Topological data analysis (TDA), A team of scientists from EPFL’s Lab for Topology and Neuroscience, L2F, and HEIG-VD has developed a model called “Giotto-tda” that can predict when a system is about to undergo a major shift.
Their model is available as an open-source library and can help scientists identify when events like a stock-market crash, earthquake, traffic jam, coup d’etat, or train-engine malfunction are about to occur.
The research team, therefore, drew on methods from TDA to come up with a new approach based on the fact that when a system reaches a critical state, such as — when water is about to solidify into ice, the data points representing the system begin to form shapes that change its overall structure. By closely monitoring a system’s data point clouds, scientists can identify the system’s normal state and, thus, when an abrupt change is imminent.
Another benefit of TDA is that it’s resilient to noise, meaning the signals don’t get distorted by irrelevant information.
Noise and muddled signals
The scientists tested giotto-tda on the stock-market crashes in 2000 and 2008. They looked at daily price data from the S&P 500 – an index commonly used to benchmark the state of the financial market – from 1980 to the present day and compared them with the forecasts generated by their model. The price-based graph showed numerous peaks that exceeded the warning level in the run-up to the 2 crashes.
“Conventional forecasting models contain so much noise and give so many signals that something is about to go awry, that you don’t really know which signals to follow,” says Matteo Caorsi, head of the project team at L2F. “If you listen to them all you’ll end up never investing because there are very few times when the signals are truly clear.”
But the signals were very clear with giotto-tda, as the peaks indicating the upcoming crashes were well above the warning level. That means TDA is a more robust technique for making sense of volatile movements that may indicate a crash is looming. but, the scientists’ findings concern only one specific market and cover a short period of time, so the team plans to conduct further research with the help of another Innosuisse grant.
“The next step will be to apply TDA to deep-learning methods. That will give us valuable information about our model, how interpretable its results are, and how robust it is,” says Caorsi.