Thursday, December 20, 2012

Observations on economic forecasting

I decided to build a macroeconomic forecasting model. This is a strictly statistical model--it doesn't impose a lot of economic theory on the data. For readers who know a bit of statistics, it's a Bayesian VAR model with a Normal/inverse Wishart prior, basically patterned after this one.* (Many thanks to Saeed Zaman for a lot of helpful guidance on this--he explained some of his own work in this area, including a working paper on which my model is based, and he pointed me to a bunch of good BVAR resources. This is very similar to the exercise I described here, which was also done by Zaman.)

I'm going to talk about the model a bit then offer some observations about the problem of economic forecasting generally. Each section of this blog post is independent, so skip the ones that look boring to you (yes, I know that will probably mean you skip them all). I think that the section of the most general interest is the last one.

The model

For those unfamiliar with this kind of modeling, the setup is basically this: choose a handful of important economic variables (as dictated by forecasting needs and theory), collect a bunch of data on them (in my case, quarterly data from 1964 to the present), then use a statistical method to figure out the (linear) relationships that exist (a) between different variables and (b) within each variable over time. That basically describes VAR (vector autoregression) models.

It turns out that standard VAR models are dismally poor forecasters because they are too complicated (too many parameters); they can provide a really good fit of existing data, but they catch too much random movement in variables and aren't able to predict averages out of the data sample (they have an "overfitting" problem). Modelers have been able to improve them by using Bayesian methods. Specifically, Bayesian methods allow the modeler to, essentially, bias the model towards parsimony and simplicity (using the word "bias" loosely). Then the data have to be really informative to move the model away from that simplicity.

In my model, I start with the prior view that each variable is a "random walk" (with drift)--that is, each variable just moves randomly, maybe with some trend, and none of them influence each other. I also impose priors that set the bar progressively higher on how much past values of variables can influence present values. (for those who know, the prior mean is a random walk for each variable, and the prior variance for each coefficient shrinks as the lag increases--this is a shrinkage estimator, which is common in the BVAR literature).

To put it simply, my model considers movement and relationships among variables to be random unless convincingly shown otherwise. This kind of parsimony has been found to significantly improve forecasts.

Some model results

How good is my model? Imagine Woody Harrelson's voice asking, "compared to what?"

This may sound sad to some non-economists, but a common benchmark for evaluating economic forecast models is whether they can forecast better than a random walk--a model that literally specifies all variables as moving randomly. The standard way of doing this is to estimate your model on just part of your data, then use it to forecast the other parts of your data, and see how well the forecasts do by getting the average error (squared). Then compare that error to the error of a random walk with drift. I did this and looked at results for four variables: GDP, unemployment rate, inflation (core PCE), and the Fed's policy interest rate. I made some tables, but the one person still reading this will leave if I post them, so I'll just say this: my model beats a random walk model for GDP forecasts up to 6 quarters ahead, unemployment forecasts at least 8 quarters ahead, and inflation at least 8 quarters ahead. It fails miserably at forecasting the Fed's policy rate; this is pretty standard in models like this (for understandable reasons I won't go into).

By the way, the random walk with drift approach isn't as silly as it sounds. If I had been living in a cave for a few years and someone demanded that I forecast U.S. real GDP for next year, I'd probably say "2.5 to 3 percent higher than it was this year." (The variables are logged, so the drift term is the average growth rate).

I can also compare my model to a standard VAR--that is, a very similar model that does not use any kind of Bayesian approach. My model handily beats the standard VAR for all variables at all forecast horizons. Imposing some parsimony on such models has huge payoffs. Force the data to really tell you something.

I think it's ok to say that on average, across lots of time, the model is a decent forecaster.** But...

Could it predict our huge recession?

No.

No models I know of really predicted the Great Recession with any reasonable amount of warning (and yes, a few people seem to have seen it coming, but a lot of them have since been shown to be stopped-clock predictors). I have some thoughts on this problem later in this post.

Here are a few charts of the model trying to predict the Great Recession. Here's what the model forecasts when estimated on data through the 3rd quarter of 2008 (click for larger image):

So here's what's going on in the chart above. The solid black line plots the actual data for GDP. The black dashed line with + (plus) signs is what a random walk model (the benchmark!) predicted for the path of GDP. The rest of the lines show the path for different simulations of my model; I simulated the model 100,000 times, and the red lines surround 80 percent of the simulated paths.*** The blue lines surround 40 percent of the simulated paths. The black dashed line in the middle is the mean forecast. See how poorly it does? Let's see if the next quarter's forecast is any better.

Amazing! Forecasting from the 4th quarter of 2008 goes really well. But...

The model overshoots (actually undershoots), failing to catch this upward turning point just as it failed to catch the downward turning point (I don't show the unemployment forecasts, which are slightly better). This bad forecasting doesn't surprise me too much; I don't think many, if any, models are great for forecasting turning points. But there are other excuses people could make, probably along the lines of things not included in the model (like ad hoc policies).

For those who didn't read the previous section, this doesn't mean the model is an epic failure. It actually forecasts ok on average. But just when good forecasts are most needed, it doesn't perform (though it still beats some other models, and likely beats a lot of more informal forecasting methods).

The problem of forecasting

I've written before that I don't think the purpose of the study of economics is forecasting. The value of economics as a field comes primarily from its ability to provide understanding of economic phenomena and do good quantitative policy analysis. The majority of economists spend little or no time trying to forecast things. We don't have a crystal ball or tea leaves. I did this project to satisfy a time series class and because I thought it'd be fun to think a little harder about forecasting, but this is not my main research agenda.

But it would be nice if we could predict recessions and other economic crises, right? Of course. But there are a lot of things that make that kind of predicting hard to do.

One difficulty is the most obvious: the economy is complex. It has a lot of moving parts. It is subject to all sorts of shocks and vulnerabilities (and they don't follow nice 2-parameter distributions, by the way). It's just really hard to predict the future of complex systems that involve a lot of randomness. Luckily there seems to be some inertia to the macroeconomy, allowing for ok prediction during "normal times," but when we really need good predictions they're not so good. It is somewhat easier to look at the past or think with models to come up with an idea of how things work or how policies might affect economic outcomes or counterfactuals (my BVAR model can do some really cool ones), and that's why we spend most of our time doing that instead of trying to tell fortunes.

Another difficulty is the Efficient Market Hypothesis (actually this isn't only a difficulty; it can help too). Some versions of this are really controversial, but a weak version just says that prices already reflect the information that is available to the public, so you can't predict their movements unless you know something the markets don't. This idea has huge implications for investing and financial punditry--entire careers exist that wouldn't if people understood the EMH. It also has big implications for forecasters, namely, that what you see in markets is sometimes all there is. There isn't much other information you can grab to say a lot about where markets will be tomorrow. The upside of this is that markets can do some of our forecasting for us, but the downside is that forecasters who think they know more than the markets are likely to be proven wrong.

Finally, I think the biggest reason we can't very well predict recessions in particular is that we don't experience them often enough. This is a good thing, of course; but it makes recessions difficult to study. Basically, we've had about a dozen recessions since we started collecting halfway decent data. We've had about half that many since we started collecting good data. We've had five recessions since the Fed started really doing its job in the early 1980s. And during the time of good data, how many recessions were as deep as this last one?

It's really hard to study something that rarely happens. It's really hard to find reliable predictors of something that rarely happens. It's a small sample of data. And it's extra difficult since by now most economists think that not all recessions are caused by the same thing. If there are several things that can cause recessions, and we've only had several recessions during the period of good data, it should not surprise anyone that our predictions are not very reliable. Some people say that economics is a young science, and that's part of it. But I think the bigger problem is that good data collection is a relatively new practice, and we need a much bigger sample. The good news is that the sample is constantly being expanded, both over time and in terms of richer "drill-down data" that allow us to study macroeconomic questions with micro data (see here).

The economy grows most of the time, so it's hard to make a model reliably predict economic contraction. The good news is that we're reasonably good at knowing how the economy responds to recessionary forces. But still, we're not great at quantitative prediction at any long horizon.

Throwing our hands up and saying that forecasting is a waste of time is not the solution. Forecasting is absolutely necessary for economic decisionmaking; we can do it systematically and transparently, or we can rely heavily on our cognitive biases or idiot pundits or people trying to sell us gold. Over time, my money is on the systematic approach. The important thing is understanding the limits of our knowledge and fostering realistic expectations in the consumers of forecasts.


So I think this is a mixed bag. In "normal times", a model like my BVAR is likely to do a better forecasting job than most or all other approaches. This is just a first pass at it, so its performance can be improved further with a bit of work (which I may or may not do). But I don't think it's going to predict the next recession, and it won't predict the one after that either.


*This model started out as a replication of this working paper; I even received some very helpful guidance from Saeed Zaman (an author on that working paper). This is a 17-variable BVAR (almost the same 16 variables as the working paper model, plus I add real residential investment); the variables are motivated by the Fed's forecasting needs and medium-scale New Keynesian models (which, after all, are typically solved by linearizing into a VAR format). I employ a number of order-selection criteria and settle on a 2-lag model. The prior is similar to the famous Minnesota prior with a few modifications.

**To be transparent, though, a better test of useful forecasting ability requires the use of real-time data--that is, data as they were when first released, rather than the more accurate data that have been revised. Real forecasters have to rely on real-time data, not knowing revisions until after they had to make forecasts.

***The posterior distribution of the coefficient vector is multivariate t (that's the beauty of the Normal/inverse Wishart prior--it is a natural conjugate). There are many ways to get the predictive density, such as Gibbs sampling or numerical integration (since all needed distributions have closed forms); I go with numerical simulation instead. The simulation exercise requires getting 100,000 draws of the coefficient vector from the MVT and 100,000 draws of the model's disturbance term (actually 100,000 for each forecast horizon), which I specify as multivariate normal with covariance matrix chosen empirically. So a careful modeler can improve forecasts if they have an idea of what sort of shocks may be coming.

Wednesday, December 12, 2012

Should I short your startup?

I've been looking at some data on startups. For today, I'm just looking at one table that somewhat illustrates the uphill battle faced by startups (once again, I just call any new firm a startup; I've heard that's not the hip buzzword lingo the kids are using. Whatever.).

This forthcoming paper by Manju Puri and Rebecca Zarutskie is a fairly comprehensive analysis of startups and venture capital. Whatever one thinks of their research design, the paper provides a wealth of stylized facts about startups and venture capital. Its main advantage is that it builds on the Longitudinal Business Database, which includes every private nonfarm business establishment in the US. Take a look at Table VI; click for a larger image and spend a few minutes (I only include Panel A).

Click for larger image


I notice a few things.
  • Startups that don't get VC assistance often fail; about half will have failed by age 5 and two-thirds by age 10.
  • Startups that don't get VC assistance are unlikely to be acquired. Of course, for most startups getting acquired isn't the goal. But your Facebook friend who keeps talking about his Boston/Silicon Valley/Silicon Slopes/NYC startup is probably hoping for acquisition or something similar.
  • Startups that get VC assistance are much less likely to fail than those that don't. But even for VC-assisted startups, by age 3 the probability of failure is higher than the combined probability of getting acquired or having an IPO.
  • Almost one-third of VC-assisted firms fail within 10 years.

Nothing on this table is causal, of course. The table doesn't show that VC assistance causes lower failure rates. You'd expect VC-assisted firms to do better if only because venture capitalists are actively trying to invest in firms with good survival prospects. Read the paper if you want to think harder about causality.

Also, there is a useful distinction to be made between the guy who starts an oil change shop with no intentions of further growth and the guy who starts a tech company with hopes of conquering social media. That distinction is lost in this table, but other parts of the paper look at things like industry and firm growth.

The broader point this table suggests, though, is that lots of startups fail, particularly those that don't get assistance from venture capitalists.

Saturday, December 8, 2012

My five favorite books of 2012

Actually, I'm allowing myself to choose from the books I read in 2012 that were published in either 2011 or 2012. In order of preference:

1. Turing's Cathedral (George Dyson, 2012). I wrote a little review of it here; this is a book about the invention and early development of computers, particularly the one built at the Institute for Advanced Studies. This one easily takes the top of the list; anyone who is not fascinated by John Von Neumann is crazy (not to mention Nicholas Metropolis, Julian Bigelow, and Alan Turing). There is plenty for everyone here; it appealed to me as an economist but also due to my (embarrassingly amateur) interest in math, computer programming, and history (lots of great WWII and Cold War stuff). It also hits a lot of other topics ranging from biology to weather forecasting. Read this book.

2. The President's Club (Nancy Gibbs and Michael Duffy, 2012). This is a book about relationships between presidents--past, present, and future. For example, I did not know about the interesting relationship between Herbert Hoover and Harry Truman. Some readers may be surprised to learn that Nixon was a key adviser to not only Reagan but also Clinton, who said that the death of Nixon was like the death of his mother to him (being among the Nixon-obsessed, I liked this vignette). Like everything I read about him, this lowered my opinion of Eisenhower. The book is full of interesting stories and reminded me that these are actually decent human beings when they're not trying to get elected to something--the post-Presidents are often likable, reliable, and useful to current presidents, regardless of past fights or the partisan divide.

3. Bismarck: A Life (Jonathan Steinberg, 2011). See Henry Kissinger's review here. I thoroughly enjoyed this book. The narrative is engaging, and the characters are fascinating--Bismarck most of all. If you like European history, diplomatic history, biographies of incredible people, etc., you will like this book.

4. The Signal and the Noise (Nate Silver, 2012). Silver is probably overhyped by now, but it was nice watching him embarrass the pundit establishment. The book is very good, providing perspectives on forecasting from many fields including weather, sports (of course), economics, and many others. It is a very readable, nontechnical discussion, though it does ask readers to learn Bayes Rule (as everyone should). I will add the caveat that while his discussion of economic forecasting seems mostly fair to me, he did leave out a newer class of models (Bayesian DSGE) that seem to be improving forecasts. This book is a quick read, and I think most people with interest in even a few of the fields he covers would enjoy it.

5. Unintended Consequences (Edward Conard, 2012). Ignore the ridiculous subtitle; I reviewed this book here. I liked this book. It helped me get better economic intuition, though his discussion of monetary economics is wanting (and a few other topics can be sketchy, like his weird theory about inequality). Conard combines a lot of high-level business experience with a pretty strong academic economics bibliography. I wouldn't recommend it as the only book you ever read on economics, but I would recommend it as one important component of a popular press approach to the subject. And before you assume he's another conservative hack (my initial assumption), be aware that this is heavy on analysis and hits both American political factions pretty hard on some of their more ridiculous stances (e.g., progressives on taxation and conservatives on immigration).

Honorable mentions:

  • Grand Pursuit (Silvia Nasar, 2011). This may have made my top five if I could remember it better.
  • Hypocrites and Half-Wits (Donald Boudreaux, 2012). This one probably deserves a spot in the top five, but it's a very different kind of book. You must buy it. I reviewed it here.
  • Fairness and Freedom (David Hackett Fischer, 2012). I enjoyed this one because I have ties to New Zealand and read a lot of NZ history, but I'm puzzled to see it on so many other top books lists. The approach was too contrived--NZ and the US just don't make for great comparisons. The US is too big and diverse in terms not only of institutions but also people (including indigenous people). I would recommend this book to my friends with ties to NZ but probably not many others.

Sunday, December 2, 2012

Startups and the Great Recession

...and by "startups" I just mean new firms (I know some people prefer a narrower definition). Anyhow, here are some charts I made from BDS data (click for larger images).

Figure 1
Figure 1 plots the number of new firms by year since 1980; these are administrative data observed in March, so the 2010 observation tells you how many firms existed in March 2010 that did not exist in March 2009. As you can see, there was a staggering decline in startups prior to and during the Great Recession.


Figure 2
Figure 2 plots the number of startups as a percent of the total number of firms in the (private nonfarm) economy, showing that startups didn't just take a beating in absolute terms. That declining secular trend is also noteworthy.


Figure 3
Figure 3 plots jobs created by startups. Again, a pretty epic decline prior to the Great Recession.


Figure 4
Figure 4 plots the component of the overall job creation rate (using the DHS definition) accounted for by startups [edit: the figure is mislabeled]. So the overall story is that new firm creation and job flows therefrom were decimated in the years leading up to the Great Recession, both in absolute and relative terms. Some of this may be explained by the Moscarini model, but I think there may be other things going on as well (such as housing collateral).