Profitability of Machine Learning Models in Forecasting the S&P 500 Index

This study addresses whether machine learning models can use technical analysis data to forecast one-day-ahead movements of the S&P 500 stock index. The input data are used to train fifteen machine learning models involving linear regression, logit, random forests, XGBoost, support vector machines, elastic nets, nonparametric regression, and k-means nearest neighbors to make one-day-ahead forecasts of expected returns. In absence of transaction costs, the best models generate abnormal returns of up to 24% annually for 1970–1987, but only 9% annually for 1988–2024. This decline in profitability over time is consistent with the predictions of the adaptive market hypothesis. The results are promising for systems design because they show that machine learning techniques can adapt and remain profitable over a 55-year period. Nevertheless, there is considerable variability in the annual abnormal returns and model performance could be different in real time applications.