Macroeconomic forecasting in Poland: The role of forecasting competitions

Macroeconomic forecasters are often believed to idealistically work on improving the accuracy of their estimates based on for example the Root Mean Squared Error (RMSE). Unfortunately, reality is far more complex. First of all, forecasters are not awarded equally for each of their estimates. They have their targets of acquiring publicity or to earn prestige. Therefore, they may put greater attention to improve only some predictions, while sacrificing accuracy in the case of others. To illustrate the problem, Hann, Ogneva, and Sapriza (2012) compared the prediction of aggregate earnings, which was made by equity analysts and macroeconomists. While the first group makes frequent revisions based on companies’ news, the latter is much more sluggish in incorporating information as macroeconomics models rely mainly on GDP data, which is released quarterly.

Furthermore, preferences of public opinion regarding optimal forecasting and preferences of forecasters may significantly differ. For example, professional forecasters tend to make more frequent revisions during a period of recession comparing to normal times (Loungani, Stekler, & Tamirisa, 2013), affectively react to economic surprises (Eroglu & Croxton 2010). Multiple articles are trying to solve whether analysts are fully rational or if they are using the most up-to-date information. But research on forecasters’ incentives is relatively scarce.

This article aims to study the results of Parkiet's competitions of macroeconomic forecasting during 2015–2019. The announcement of winners has a strong influence on economic debate in the Polish media—winners give many interviews about their economic views, advertising expertise of their institutions. Furthermore, the influence of such media comments or research reports on financial instruments’ valuations and actions of the public institutions is frequently greater compared to academic research papers. Therefore, such competitions affect the work of commercial economic research time, for example, set priorities in forecasting.

Studies regarding analysts’ performance and competition between them are not publicly available. From this perspective, Poland is an interesting example—the majority of commercial research is published to a wide audience without additional charges.

Based on a logit model, we analyse whether more accurate forecasting of some selected macroeconomic variables (e.g. inflation) increases the chance of winning the competition by a greater degree comparing to the others. Second, we verify if there exists some persistence of scoring one of the best five places among the forecasters in the respective categories.

Our research shows that among macroeconomic variables, three groups have a significant impact on the final score: inflation (CPI and core inflation), labour market (employment in the enterprise sector and unemployment rate) and financial market indicators (EUR/PLN and 10-year government bond yields). Each group is characterised by a low disagreement between forecasters. Logit models suggest that top forecasters are capable to persistently be in the lead in case of the inflation block. The competition gives little benefit for correctly forecasting activity and leading sentiment surveys.

This section presents a review of the literature regarding the motivations of professional economists to regularly present their macroeconomic forecasts to the public. The frequent illusion is that financial analysts or commercial economists tend to optimise their forecast based on some widely renowned metrics, for example, RMSE. Subject literature shows that this is not the truth.

First of all, forecasts are not produced in a vacuum. There is strong evidence that forecasters are influenced by other decisions (e.g. Scharfstein & Stein 1990, Pons-Novell 2004, Ottaviani & Sørensen 2006), which results in two opposite phenomena. Individual analysts may strategically approach their jobs. One solution is to self-censor his or her forecasts and closely follow the market consensus to avoid major mistakes. This phenomenon is known as herding behaviour. The opposite strategy suggests exaggerating the model finding to crowd-out from the others and attract attention. Both strategies have acquired the mathematical and game theory foundations (e.g. Elliott, Komunjer, & Timmermann, 2008; Marinovic, Ottaviani, & Sorensen, 2013; Pierdzioch, Rülke, & Stadtmann, 2012).

A second strain of literature reports that forecasters are prone to behavioural biases—we will focus on this directly related to motivation and credibility. Lamont (2002) showed that older and more recognised forecasters tend to more frequently stand out of the crowd and produce non-consensus ideas (not exactly more accurate). Ashiya (2009) reported the existence of affiliation biases—there is a tendency that forecasters working in the same sector, for example, commercial banks are likely to generate similar mistakes. The author suggests that this problem may be related to the wishes of the forecasters’ employers.

A common feature of the analysts is publishing their estimates in the competition, for example, from Consensus economics, Focus economics, Bloomberg or Reuters ranking. The subject literature is relatively silent regarding the impact of the rules of such competition on economic forecasting. The results of such contests frequently have a significant impact on the assessment of commercial economists’ work and their financial payoff. Furthermore, the mechanisms evaluating forecasts are prepared by non-professionals and may strongly deviate from academic state-of-the-art practices. To fill this gap, we evaluate the impact of the most prestigious competition in Poland from Parkiet daily.

This section aims to present the Parkiet competition for macroeconomic forecasting and review the statistical properties of the formula. The competition is the most prestigious contest in Poland—it gathers the greatest number of forecasters participating in the panel, and results are published in the media. The influence of the Parkiet's ranking is visible, for example, in the analysts’ interactions on Twitter. According to the author's knowledge, some economists in the research teams have directly linked the financial bonuses to performance in the competition.

The competition requires monthly submission of Nowcasts for 14 macroeconomic variables and 5 financial indicators. Macroeconomic variables consist of: PMI index, CA balance, exports and imports value, CPI, core inflation, employment and wage growth in the enterprise section, PPI, growth of industrial production, construction output, retail sales and M3, and unemployment rate. Financial variables consist of two exchange rates (EUR/PLN and USD/PLN), two interest rates (on 2-year and 10-year government bonds) and NBP policy rate. Once per quarter there is also a submission of three additional indicators: growth of gross domestic product, private consumption expenditures and gross fixed capital formation.

Participants represent mainly the financial sector. In the most recent edition, there is also a public sector institution (Polish Institute of Economics) and an independent think-tank (prognozy-gospodarki.pl). The panel composition is unbalanced due to mergers in the banking sector or new entries. The dataset spans over the years 2015–2019.

Forecasts sent by analysts are evaluated based on a function described in Eq. (1). (1)pointscat,t,i=e−|εcat,t,i|σcat,t{\rm points}_{cat,t,i} = {e^{- {{\left| {{\varepsilon _{cat,t,i}}} \right|} \over {{\sigma _{cat,t}}}}}} where: ε_{cat, t, i} denotes forecast error of i-th analysts for the economic variable cat in the period t. σ_{cat, t, t} describes the standard deviation between forecasts submitted by all competition's participants for the variable cat . points_cat,t,i is the final amount of scores earned by i-th analyst. The final value lies between zero and one.

The final score is calculated as a simple sum of points in all categories, without weighting. The calculation is presented in Eq. (2). n denotes the number of variables in the competition.

From a perspective of the competition's participants, an important feature of this function is strong penalisation of relatively small forecast errors, when disagreement is low. This property is important especially in the case of categories with low forecast disagreement (e.g. employment, unemployment rate) where differences related to the rounding of numbers can halve the final score. The distribution of points depending on disagreement and the forecast error is presented in Figure 1.

Fig. 1

Second, the function is strongly penalising big forecast errors in a situation when final realisation missed every individual estimate. The exact amount of points related to standardised values of error is presented in Figure 2.

Fig. 2

In case of a big surprise, when the error exceeds 2.5 standard deviations of disagreement between forecasts, the number of earned points is not greater than 0.1 in any case. Therefore, even if there is a statistically significant difference in the accuracy (e.g. best forecaster has an error of 2.5*σ_t and market consensus equal 5*σ_t), the difference between his/her reward and other participant scores is not meaningful. Therefore, this contest may discourage greater scrutiny of more volatile categories or during the period of greater uncertainty.

The mentioned problems have a strong impact on the final scores. First of all, there are strong discrepancies in the expected number of scores between the macroeconomic variables. The median score in the unemployment rate is frequently three times higher than the PMI index. The numbers are presented in Table 1.

Second, the dispersion between scores of top performers and median forecasters also differs strongly. Standard deviations for categories are presented in Table 2. The differences between scores for PMI or Industrial production can be relatively small. Furthermore, they can vary over time. In case of variables where strong surprises are more often, no one is capable to ex-ante predicting the total amount of scores but can expect the discrepancies between forecasters should be small (see Table 2).

Those facts suggest that the competition gives greater motivation towards accurate forecasting of categories with low disagreement. We attempt to verify whether successful projections of these variables result in a greater chance of winning the contest.

This section presents the methodology of our research. First of all, we will present a logit model that aims to answer whether successful forecasting in some categories has greater importance for final success in the competition than the others. Success is defined as a place in the top 5—only these participants are announced in the newspaper. Second, we will look at whether there exists evidence that winners in the previous years in some categories are more likely to triumph also in the next edition. If so, such teams are more likely to utilise greater resources compared to others either by developing more complex models, by analysing a greater amount of information or by purchasing non-public data.

Let's define a binary variable that describes whether a competitor achieved a top five performance score. The representation is described in Eq. (3). (3)successi,t={1positiont,i≤50positiont,i>5{\rm success}_{i,t} = \left\{{\matrix{1 \hfill & {{\rm{position}}_{t,i} \le 5} \hfill \cr 0 \hfill & {{\rm{position}}_{t,i} > 5} \hfill \cr}} \right. where index i identifies forecaster and index t denotes a time period.

The probability of finishing the contest among five top performers is described by a logit regression. Such model classification is described by an unobservable latent variable y_i,t. This latent variable takes a positive value when participants are among top performers, and negative or equal to zero otherwise (see Eq. 4).

To describe the value of those latent variables, we use information about the top five performers in specific macroeconomic or financial variables. The formula is presented in Eq. (5). (5)yi,t=β0+β1*successcat1,i,t+…+βn*successcatn,i,t+εi,t{y_{i,t}} = {\beta _0} + {\beta _1}*{\rm{success}}_{cat1,i,t} + \ldots + {\beta _n}*{\rm{success}}_{catn,i,t} + {\varepsilon _{i,t}} where cat1 denotes the first macroeconomic variable (category) i.e. PMI index, cat2 the 2^nd one—CPI, and so on. Catn describes the last n-th category—Polish government bond 10-year yield. The following enumeration was used only in methodological sections to shorten formulas—each table presenting the outcome of the model has a full description of variable instead.

We hypothesize that some of the β parameters have positive and statistically significant values. In such circumstances, some variables have a greater influence on the final result than others.

Second, we will analyse if forecasters are exploiting these opportunities. We aim to verify if some analysts are capable to persistently achieve the top five places in case of variables, which brings a higher score. We propose a simple logit model where the binary variable will be positive in case when forecasters finished the contest with positions 1–5 in the selected category and zero otherwise. The latent variables will be explained by last year's performance.

Positive values of β₁ suggest that the best forecaster can maintain pole positions and earn a higher score (on average). This implies that they dedicate greater effort to score in the category.

This section aims to present the results of the models presented in the previous chapter. The model describing the probability of victory in the final competition is presented in Table 3. Statistically non-significant variables were removed.

First of all, our models suggest that good performance in the current account (CA) category to the greatest extent increases odds of the final score, but the corresponding components (exports/imports) do not guarantee any advantage. This phenomenon is puzzling—differences between analysts’ scores in this category typically are not strong. One of explanations is that analysts who have a greater motivation to win the competition makes an effort to correctly predict the annual flow of the EU funds typically occurring in January (reading is published in March). The information on what amount of money was located on a CA is provided by the ministry of finance, but less-motivated analysts may not use this report.

The problems described in Section 3 are visible in case of the employment component—this is the second category where a good score increases the odds of winning the overall competition in a greater manner. In this case, the score in January is dependent on GUS statistical procedure. The statistical office annually rebalances its panel of enterprises, which creates unpredictable distortions. Furthermore, in other months, small errors may result in significant losses of scores. This problem applies also to forecasting the unemployment rate where a good score also increases the odds of victory.

The labour market variables are rather not the most forward-looking predictors of the business cycle. At the same time, there is no incentive to improve accuracy in case of monthly activity indicators (e.g. construction output, industrial production) or sentiment survey.

Finally, there is consistent evidence that focuses on inflation and financial markets forecasting pays off. The importance of inflation forecasting is obvious—these figures influence central banks’ decisions regarding interest rate policy. But there are several obstacles that make financial variables at least controversial. First of all, there is a rich literature confirming that short-term foreign exchange movements are an example of a random walk process (e.g. Kilian & Taylor 2003, Bacchetta & Van Wincoop 2007). Second, in case of stronger volatility on FX/FI market and bigger shifts in valuations, the function is unlikely to grant scores to analysts.

After identifying the shortcomings and advantages in the scoring of Parkiet's competition, we analysed whether research teams are utilising these flaws to perform better (Table 4).

Three values in columns β₁ and β₀ represent respectively parameter estimate, its standard deviation and p-value for Z-statistics. Positive values of β₁ denote that the forecaster who achieved a top five position in the previous edition of the contest is more likely to repeat this achievement next year, than others.

We found some evidence that top forecasters are capable to more persistently achieve a top five position in the categories of CA and inflation. On the other hand, we do not see a bigger persistence in case of employment and the unemployment rate. These categories usually have the biggest number of scores and limited relevance—the number of instant comments after these releases is much lower compared to CPI or industrial production. Therefore, there is no evidence that participants are trying to somehow exploit the contest by achieving greater scores only in this group.

In this article, we have also analysed the statistical properties of the function that evaluates forecaster estimates in Parkiet's competition. Our analysis showed that the scoring mechanism is not rewarding analysts, when surprises are exceeding two standard deviations of the disagreement. Therefore, this function discourages analysts to make a greater effort in case of periods of elevated uncertainty. Ultimately, it is not supporting more accurate forecasting when the potential error is greater during a depression or structural changes. At the same time, accurate forecasting is usually most valuable during such periods. Therefore, instead of promoting accurate forecasting generally, this function promotes forecasting in stable conditions only.

Our model highlighted that Parkiet's forecasting competition generates incentives to focus on predictions of inflation and labour market data rather than economic activity. Each of the mentioned categories is characterised by low forecast disagreement and usually small forecast errors. The motivation to accurately forecast these variables may result in utilising a greater amount of time for forecast where potential errors are not that important (e.g. employment and unemployment rates). But, accurate nowcasting in these two categories does not lead to greater scrutiny of long-term forecasts—commercial economists rarely publish their estimates of labour market for the horizons greater than 1 month.

Finally, we analysed whether some persistence in victories of top-performing analysts occurs. Our models showed that last year's winners are statistically more likely to triumph in the next year. Still the advantage is not that great—proposed models do not generate a greater gain in predictability comparing to a constant probability model. Therefore, we found no evidence that participants are trying to somehow exploit the contest by achieving greater scores only in most scoring groups.