The (Not So Clear) Advantages of Monte Carlo Simulation

PFAU ON MC

In this post by Wade Pfau, he stakes out a net positive position on Monte Carlo Simulations.  Despite the many pros and cons of this type of tool he comes down firmly on the positive side after weighing the core issues he sees as an academic researcher. These include, by my reading and in super digested form:

• MC has a wider variety of scenarios than historical simulations. HistSims have a finite number of periods since 1926 especially using 30 year planning horizons.  [even worse for an early retiree if you happen to have a 40-50 year planning horizon]

•  HistSims have too many overlapping datapoints. Between 1926 and 2016, those two years will show up in the data series once each but 1971 will show up a lot. Per Pfau, 1955-86 show up thirty times  [ then again, maybe over-representing the 70s is not bad as a conservatising assumption with a bear market for bonds, high inflation twice, and crummy stock returns after 1970]

• Sims with large numbers of runs can show "a wider variety of return sequences that support a deeper perspective about possible retirement outcomes."

• MC might, since datapoints are independent, have less of a negative bias on bonds compared to the 1970s heavy HistSims.

• MC will show scenarios where a strategies fail where they might have "accidentally" succeeded with the luck of the draw in historical series.  (again Pfau points towards bond strategies showing better in MC)

Pfau does add a few "cons" to the analysis to round out and temper the joys of simulating.  For example:

• MC is bad at simulating dependent returns, trending, and mean reversion despite what we see in historical data. 

• MC sims "that do not include “fat tails” in the distribution of returns may not create extreme low or high returns as frequently as seen in reality." [note, this is relatively easy to model if one knows how to do it]

• MC sims results are only as good as the input assumptions [ and the robustness of the model, too, by the way. Just look at the raging back and forth on climate models.  Of course a perfect model would be reality, in which case who needs a model…except that one can't model and run reality 10,000 times.  Or can we?]

"Overall, the advantages of Monte Carlo simulations likely more than make up for any deficiencies when compared to the results we obtain using historical simulations." - Pfau 

ME ON MC

Now, let me add a few points as a non-academic -- and as a retiree and an early retiree at that -- on simulators.  I like MC sims but I feel like I have grown into a place where I have a more reasonable, but not perfect, understanding of where they fit and where they don't.  Here are some miscellaneous thoughts on simulators.  This is a first cut; I'll add to this as I think about it some more. 

1. Cost.  It may seem strange in 2016 to mention cost when one speaks of MC simulation given the vast and growing array of free internet MC simulators out there. On the other hand, it is not so long ago that my advisor with a large well known US bank/brokerage offered to run a financial planning analysis -- that, unknown to me before the fact was nothing more than a very dressed up MC sim -- for around $3-4k.  I balked because that was and is insane. So then they offered the first taste for free. Fine, I did that but I realized that, like I said, it was in the end only a dressed up MC sim.  Any additional one run was going cost me another 3-4k and take a week or two for results.  Are you kidding me? I was so irritated that just to prove a point I built a simulator from scratch in less than a week. It wasn't that hard and in fact it had features and assumptions that were demonstrably superior to what was being offered by the bank…for free…whenever I wanted.  The downside is that it is slow but certainly not slower than asking for a $4,000 crappy sim to show up next month.  My advice is not to build your own but there are clearly a large number of capable tools out there that do a great job for free or at a small cost. Don't get duped by an adviser to pay for this stuff.

2. Opacity.  This is mostly, but not always, true for either free or pay Sims.  It is not the input assumptions so much, it is the black-box-ness of it.  How exactly are return, or inflation for that matter, assumptions pulled into the model? Historically bootstrapped or a simulated distribution? What kind of distribution? How do you calculate your bond returns? Is it yield or total return and what kind of bonds? Corporate? Treasury? Intermediate? What?  And what about the failed runs? Can I see exactly a decomposition of what happened? (I know it was bad returns and high inflation and stubborn spending but show me).  What…is…inside…the…black…box?

3. The Illusion of Prediction.  Yes, I know this is much commented on.  MC is a great research tool but it is either misunderstood by -- or, worse, misrepresented to -- the unwary as a predictive tool.  The future is and will remain unknown.  A MC run every once in a while to gauge generalized risk in the current year if assumptions in the plan are not able to bend a bit is probably a healthy thing. Locking in on a 10% fail rate as some kind of apodictic "truth" is not.

4. Many Simulators, including mine, do not model the exact details well.  Some tool builders are finally addressing this but not all tools are good enough at factoring in things like the effect of marginal tax rates (vs average tax rates or sometimes no taxes), robust asset allocation and rebalancing, multiple account types, different account withdrawal strategies, etc. There are some other details like this but I can't remember them as I write…

5. Duration.  You usually have to pick a fixed retirement duration for the simulation. Is it 30 or 40 or 2 years? Who knows? That's the thing about longevity, it's random.  Or, rather, for a population as a whole it follows a generally recognizable probability distribution (look at a Gompertz equation or a SS table for example), but for any one person it's pretty random. Most MC sims don't vary terminal dates. You have to pick. Right there I think it's off to a bad start. Me, that was the first thing I modeled in my own simulator.  It's not that hard.  Use a Gompertz formula or a SS table to create a distribution of possible terminal dates for the given start age.  Some runs will run short and some long.  It's certainly not the same as current tools I see, and it will under and over estimate risk in different places, but gives a different sensibility to success and fail rates that I like.

6. Auto-correlation. Pfau touches on this a bit.  There is generally no provision for any type of auto-correlation or auto-regression. This is the idea that if inflation was 12% last year it might be 11 or 13 or 10 next year but probably not -7%. Most sims will make inflation or returns (or whatever else) vary to whatever value the model randomness can pick rather than turn it into a series or a trend.  I'm not totally sure it could be done but who knows?  The brute force of a lot of runs might accidentally create a series that looks auto-correlated but that's pretty weak.

7. Spending Shocks. Most people in real life have spending shocks. Most simulators don’t simulate that (as far as I know...except for mine, of course). Ask anyone you know that is retired and they likely have more than enough stories about divorces, health care nightmares, new roofs, unexpected dental work, or something. Sims don’t. Or maybe you were a Russian with money in Cypriot bank and were exposed to a capital haircut in 2011. I'll bet that wasn't in your plan or your last simulation. Most people have small scale variable spending "noise," too (and it is not normally distributed which will be a results-drag over a long enough timeframe). Sims usually don't.   Dirk Cotton takes this idea a little further than I do. See item 11 below.

8. Distribution of Returns. (Wade touched on this above) Some, but not all, simulators do not allow for non-normal distributions of returns and if we know anything now it is that market returns are not normal. Also "modern" returns look like they might be lower than past history would imply and that is not often reflected in the software I've seen. For example, in the 1910s thru 1930s, the depression notwithstanding, there were quite a few higher "max" returns than we've seen over the last 30-40 years. Also, we are in an unprecedented era of high valuations in both stocks and bonds. If a software solution uses either historical or normal distributions to randomly serve up returns, it might pull up unrealistic or irrelevant returns that are not likely to match current or future conditions…in my opinion. That's a hard problem to solve. Also, except through the brute force and luck-of-the-draw nature of a large numbers of simulations, the simulators typically do a bad job of modeling sequence of returns risk which is known to be a retirement killer. 

9. The "So What?" Factor. There really are, in my mind, no real, crisp policy prescriptions that come out of any MC result. That means I think this is a research tool more than a retail service. The best that can be said is that if one runs a really big "fail" one will tend to look at the obvious: spending reductions, asset allocation changes (maybe), additional work years, blah blah blah.  But you don't need a high cost MC sim to tell you that except maybe in the worst case results to get someone's attention. 

10. Life is Complex. In real life there are way too many variables. No software on earth can really capture the specific circumstances of an individual person and all the variability they might encounter. Also, I personally have some specific retirement and spending plans that I expect to execute.  I don't really want you to force your software's parameters over the top of what I really think I will do.  I want perfect customization of course.  For the most part, you can't do that. For the various reasons above I think that most Monte Carlo simulations probably under-report long term retirement risk by quite a bit and even when they don't I'm not sure they are all that helpful.

11. Retirement, especially spending, may be subject more to "chaos" than randomness.  Dirk Cotton at theretirementcafe.com in a post on retirement and chaos theory suggests that retirement plans are probably -- especially when it comes to spending -- subject more to chaotic processes than they are the randomness of stochastic systems. That means, among many other things, that they will generally have a prediction horizon.  After running a lot of MC style simulations he concluded that "the prediction horizon for retirement portfolio balances is a less than a year, beyond which the outcomes diverge dramatically…" He goes on to say as well that: "Retirement income studies tend to use probabilities to focus on long-term sustainability of savings as a function of market volatility alone. This approach won't catch many quickly developing expense-related crises, especially since the studies tend to ignore expense uncertainty altogether. When we say a retiree has a 5% risk of outliving her savings, we mean a 5% risk of outliving savings due solely to market volatility. But, there are other risks to those savings that should also be considered… These studies explain long, slow declines in standard of living, not catastrophic failures, in a world where market returns are normally distributed and mean-reverting and no one ever needs to spend more than their 'sustainable withdrawal.' Their recommendations – diversification and spending adjustments – provide little help in a spending crisis."