VoxEU Column Frontiers of economic research Macroeconomic policy

Macroeconomic model comparisons and forecast competitions

Where were economists when the global recession hit? Or rather, where were their forecasts in the years before? This column argues that clearly some of the models were at fault. To correct this, it proposes a ‘comparative approach’ to macroeconomic analysis where models compete for the right to be taken seriously.

The failure of economists to predict the Great Recession of 2008–09 has rightly come under attack. The areas receiving most criticism have been economic forecasting and macroeconomic modelling. Distinguished economists – among them Nobel Prize winner Paul Krugman – have blamed developments in macroeconomic modelling over the last 30 years and particularly the use of dynamic stochastic general equilibrium (DSGE) models for this failure.

Key policymakers take a more pragmatic view, namely that there is no alternative to the use of simplified models, but that the development of complementary tools to improve the robustness of policy decisions is required. For example, former ECB President Jean-Claude Trichet said in late 2010:

The key lesson I would draw from our experience is the danger of relying on a single tool, methodology or paradigm. Policymakers need to have input from various theoretical perspectives and from a range of empirical approaches... We do not need to throw out our DSGE and asset-pricing models: rather we need to develop complementary tools to improve the robustness of our overall framework (Trichet 2010).

Against this backdrop, we present  a new paper (Wieland et al 2012) in which we propose a comparative approach to macroeconomic policy analysis that is open to competing modelling paradigms. We have developed a database of macroeconomic models that enables a systematic comparative approach to macroeconomic modelling with the objective of identifying policy recommendations that are robust to model uncertainty. This comparative approach enables individual researchers to conduct model comparisons easily, frequently, at low cost, and on a large scale.

The macroeconomic model database is available to download from www.macromodelbase.com and includes over 50 models. We have included models that are used at policy institutions like the IMF, the ECB, the Fed, and in academia. The database includes models of the US economy, the Eurozone, and several multi-country models. Some of the models are fairly small and focus on explaining output, inflation, and interest-rate dynamics. Many others are of medium scale and cover many key macroeconomic aggregates.

This database can be used to compare the implications of specific economic policies across models, but it can also serve as a testing ground for new models. New modelling approaches may offer more sophisticated explanations of the sources of the financial crisis and carry the promise of improved forecasting performance. This promise should be put to a test rather than presumed (see Wieland and Wolters 2011 for details).

In recent years, researchers such as Smets and Wouters (2004), Adolfson et al (2007) and Edge et al (2010) have reported on the strong forecasting performance of DSGE models. However, the existing papers are based on samples with long periods of average volatility and therefore can not address specifically how well DSGE model-based forecasts perform during recessions and recoveries. With this in mind, we analyse the forecasting performance of models and experts around the five most recent NBER-defined recessions. Turning points pose the greatest challenge for economic forecasters, are of most importance for policymakers, and can help us to understand current limitations of economic forecasting, especially with respect to the recent financial crisis.

We use two small micro-founded New Keynesian models, two medium-size state-of-the-art New Keynesian business-cycle models – often referred to as DSGE models – and for comparison purposes an earlier-generation New Keynesian model (also with rational expectations and nominal rigidities but less strict microeconomic foundations) and a Bayesian VAR model. For each forecast we re-estimate all five models using exactly the data as they were available for professional forecasters when they submitted their forecasts to the SPF. Using these historical data vintages is crucial to ensure comparability to historical forecasts by professionals. We compute successive quarter-by-quarters forecasts up to five quarters ahead for all models.

Predicting the recession of 2008–09

Figure 1 shows forecasts for annualised quarterly real output growth for the recent financial crisis. The black line shows real-time data until the forecast starting point and revised data afterwards. The grey lines show forecasts collected in the SPF and the green line shows their mean. Model forecasts are shown in red. While data for real GDP become available with a lag of one quarter, professional forecasters can use within-quarter information from data series with a higher frequency. In contrast the models can process only quarterly data. To put the models on an equal footing in terms of information with the forecasts of experts, we condition their forecasts on the mean estimate of the current state of the economy from the SPF.

Figure 1.


Notes: Solid black line shows annualised quarterly output growth (real-time data vintage until forecast starting point and revised data afterwards), grey lines show forecasts from the SPF, green line shows mean forecast from the SPF, red lines show model forecasts conditional on the mean nowcast from the SPF.

The forecasts shown in the left graph start in the third quarter 2008 and have been computed before the collapse of Lehman brothers. It is apparent that all professional forecasters failed to foresee the downturn. The mean SPF forecast indicates a slowdown of growth in the fourth quarter of 2008 followed by a return to higher growth in the first quarter of 2009. The model-based forecasts would not have performed any better and predict even higher growth rates than most professional forecasters. The graph on the right shows that in the fourth quarter of 2008, following the Lehman debacle, professional forecasters drastically revised their assessments of the current state of the economy downwards. Still, growth turned out to be even much lower than estimated. Professional forecasters as well as model forecasts wrongly predicted that the trough had already been reached. While the models predict positive growth rates one quarter ahead, some of the professional forecasters were somewhat more pessimistic. The model-based predictions and the professional forecasters are, however, far from predicting an extreme downturn of as much as 6% output growth.

Given this failure to predict the recession and its length and depth, the widespread criticism of the state of economic forecasting before and during the financial crisis applies to business forecasting experts as well as modern and older macroeconomic models. Professional forecasters, who are able to use information from hundreds of data series including information about financial market conditions and all kinds of different forecasting tools and thus have clear advantage over purely model-based forecasts, were not able to predict the Great Recession either. Thus, there is no reason to single out DSGE models, and favour more traditional Keynesian-style models that may still be more popular among business experts. In particular, Paul Krugman’s proposal to rely on such models for policy analysis in the financial crisis and disregard three decades of economic research is misplaced.

Is there any hope left for economic forecasting and the use of modern structural models in this endeavour?

Figure 2 shows professional and model-based forecasts starting in the first and the second quarter of 2009. Professional forecasters continued to revise their estimated nowcast downwards for the first quarter of 2009 and predict an increase of growth rates afterwards. Interestingly, from the first quarter of 2009 onwards the model-based forecasts perform quite well in predicting the recovery of the US economy. Three-quarters-ahead model-based forecasts dominate expert forecasts in several cases.

Figure 2.

Comparing the forecasting accuracy of professional and model-based forecasts

The model forecasts are on average less accurate than the mean SPF forecasts (see Wieland and Wolters 2011 for detailed results). Of course, taking the mean of all forecasts collected in the SPF can increase the forecasting accuracy compared to individual forecasts. Looking at individual forecasts from the SPF we observe that the precision of the different model forecasts is well in line with the precision range of forecasts from professionals.

Computing the mean forecast of all models we obtain a robust forecast that is close to the accuracy of the forecast from the best model. Conditioning the model forecasts on the nowcast of professional forecasters (reported in the paper) can further increase the accuracy of model-based forecasts. Overall, model-based forecasts still exhibit somewhat greater errors than expert forecasts, but this difference is surprisingly small considering that the models only take into account few economic variables and incorporate theoretical restrictions that are essential for evaluations of the impact of alternative policies but often considered a hindrance for effective forecasting.

Conclusion

Both model forecasts and professional forecasts failed to predict the financial crisis. At the current state of knowledge about macroeconomics and the limitations to use all this knowledge in simplified models, large recessions might just be difficult to forecast.

By comparing the forecasts from different models we can hedge against outliers and find predictions that are robust across several models. Our macroeconomic model database provides a testing ground for macroeconomists to compare new models to a large range of existing benchmarks. We thus provide the tools for a comparison with established benchmarks and current forecasting practice as documented in the SPF. It is important to base discussions about competing modelling approaches on a solid basis. In our research we show how such a comparison of different models can be pursued.

References

Adolfson, M, MK Andersson, J Linde, M Villani, A Vredin (2005), “Modern  forecasting models in action: Improving macroeconomic analyses at central banks”, Sveriges Riksbank Working Paper No 190.

Del Negro, M and F Schorfheide (2004), “Priors from general equilibrium models for VARs”, International Economic Review, 45(2),643–73.

Edge, RM, MT Kiley, and JP Laforte (2008), “Natural rate measures in an estimated DSGE model of the US economy”, Journal of Economic Dynamics and Control, 32:2512–35.

Edge, RM, MT Kiley, and J-P Laforte (2010), “A comparison of forecast performance between Federal Reserve staff forecasts, simple reduced form models, and a DSGE model”, Journal of Applied Econometrics, 25(4):720–54.

Smets, F and R Wouters (2004), “Forecasting with a Bayesian DSGE model: An application to the euro area”, Journal of Common Market Studies, 42(4):841–67.

Smets F and R Wouters (2007), “Shocks and frictions in US business cycles: A Bayesian DSGE approach”, The American Economic Review, 97(3):586–606.

Taylor, J and V Wieland (2011), “Surprising Comparative Properties of Monetary Models: Results from a New Monetary Model Base”, Review of Economics and Statistics, forthcoming.

Trichet, Jean-Claude (2010), “Reflections on the nature of monetary policy non-standard measures and finance theory”, speech given on the occasion of the ECB Central Banking Conference Frankfurt, 18 November.

Wieland, V, T Cwik, GJ Müller, S Schmidt, MH Wolters (2012), “A new comparative approach to macroeconomic modelling and policy analysis”, Manuscript, Center for Financial Studies, Frankfurt.

Wieland, V and MH Wolters (2011), “The Diversity of Forecasts from Macroeconomic Models of the US Economy”, Economic Theory,47(2-3):247-292. Also available as CEPR Discussion Paper 7870.

10,184 Reads