Introduction
It all started in 2017 with an article titled “Attention is all you need” in which Vaswani et.al. proposed now famous the Transformer architecture. After that, its history in the making. Many aspects of our life are changing, the way we solve problems, take decisions, write codes, write articles, do business and many more such aspects either already changed or changing fast. All these transformations are currently driven by this new workhorse of AI with an worthy name The Transformer. After showing its great prospect in solving the problems in language understanding and language generation Transformers are now being deployed to solve the age old time series forecasting problems.
Time series forecasting is evolving towards foundation models due to their success in other artificial intelligence (AI) areas. The revolutionary Transformer architecture is showing promise for unlocking patterns in the often chaotic world of financial markets.
Why Transformers for Finance?
Traditional forecasting methods like ARIMA and LSTM often struggle with the complexities of financial time series. Stock prices, trading volumes, and economic indicators exhibit long-range dependencies, shifting volatility regimes, and intricate cyclical patterns. Transformers, with their attention mechanisms and encoder-decoder structures, are uniquely equipped to address these challenges. Thats why more and more researchers coming up with different variants of the Transformer model to solve the timeseries puzzle.
The New Era: Time Series Foundation Models
Inspired by the success of large language models, researchers are developing time series foundation models. Here we are trying to identify the advantages of different forecasting models proposed so far which are based on the Transformer architecture:
- Hidden Regimes: Key feature of Patch-TST and similar models (e.g. iTransformer) is their ability to identify distinct periods called regimes within time series data. This ability is important for financial timeseries forecasting for quick detection and respond to abrupt changes in volatility and trend patterns can lead to more accurate predictions and better-informed decision-making.
- Few-Shot Forecasting: TimeGPT’s capability in zero-shot inference make it a good candidate which can perform well with limited historical data. It can leverage meta- learning or fine-tuning on a small number of samples to provide good forecasting model.
- Frequency Patterns: TimesFM has ability to capture and model seasonality and cyclical trends in time series data. This capability, demonstrated by empirical performance on financial datasets, makes TimesFM a suitable for financial forecasting.
- Changing Dynamics: Moirai’s hierarchical attention mechanism can potentially capture dependencies between variables in multivariate financial data. So, Moirai should capture the shifting dynamics among each univariate series in a financial time series setup.
- Longterm Relationship: The iTransformer’s capabilities are similar to Moirai but it should be able to capture much longer term relation among the timeseries variables. At the time of writing this article iTransformer has achieved state-of-the-art results on benchmark datasets.
- Recent Trends: Lag-LLama’s focus on recent past data might lead to accurate forecasts.
So here’s a general guide:
1. Regime Shifts and Volatility: iTransformer, Patch-TST
2. Few-shot Inference**:** TimeGPT
3. Strong recent trends: Lag-LLama
4. Seasonality: TimesFM
5. Temporal Relationships: iTransformer
Final Thoughts
Training and finetuning Transformers can be computationally expensive and require large datasets. Moreover, Transformers are black-box by nature and this lack of interpretability can be a challenge for financial domain where real money at stake.
However, The world of time series foundation models is rapidly evolving. This Transformer-driven revolution holds the potential to transform the nature of financial forecasting just like LLMs have changed the way we interact with text data.
So, our observation is that the models which rely on time-patches in their feature processing are more likely to succeed in forecasting financial time series because they are able to capture the regimes within the timeseries. Considering historical patches as input to the models allow the models to get rid of the heteroscedastic nature of the timeseries data. As an organization specialized in handling financial timeseries data and forecasting, this new league of Time Series Foundation Models is really a must watch game. Expecting this elucidating world of Forecasting to be as exciting as the rise of Language Models.
Reference:
https://towardsdatascience.com/itransformer-the-latest-breakthrough-in-time-series-
forecasting-d538ddc6c5d1
https://towardsdatascience.com/chronos-the-rise-of-foundation-models-for-time-series-
forecasting-aaeba62d9da3