Fail Rates: Is there Any Simplicity To Be Found On The Far Side Of Complexity?

SIMPLICITY vs COMPLEXITY

Larry Rosenberger, a former CEO of Fair Isaac Corp -- a company where if you open the hood carefully you realize it isn't, or at least wasn't at the time, all that much more than a whole bunch of really, really smart quant PhDs creating and selling complex risk models -- once told me that there is almost always simplicity on the far side of complexity.  This came to mind recently because I have been processing through an article I read earlier this year by Moshe Milevsky about the idea of avoiding ruin rate analysis and simulators in retirement planning earlier than is necessary or useful (It's Time To Retire Ruin Probabilities). In that article he gives complexity a proper thrashing especially for advisors and their clients (with an elbow nudge towards the client) that have not taken baby steps in common sense before they jump into the abyss of deeper, stochastic retirement analysis.  In general and up to a point I totally agree with that point of view but I also wanted to personally see if there might be elements of simplicity on both sides of this ruin complexity problem and whether we can redeem anything from simulation tools by using some simple questions for ourselves.

Here are some bullet points from the article that capture the main points he is trying to make, a list that runs the risk of recapitulating the whole article:

• sometime in the last decade or so, the concept of ruin probability transitioned from being a mathematical construct used by actuaries, mathematicians, and risk management specialists to being a very trendy metric for wealth management.  

• my point here is not to argue with the absurdity of assuming that people will continue driving blindly at the same speed until they run out of gas in the middle of the desert. People adapt to changing financial circumstances[9]. Rather, my quarrel is with the paradigm of retirement ruin—…probabilities as the guiding risk metric for retirement income planning…this approach can get out of hand and is subject to abuse.  

• Some argue that a 51% chance of success is all that is needed for a good financial plan… At the other extreme, would you get on an airplane with a 0.1% probability of failure? Why should a retirement plan be any different? [comment: Exactly!  why would one, unless steeped in the science of simulation, accept any risk rate above zero in the first place if it could be avoided. Trick question, of course.  In addition, what exactly is the difference between a fail rate of 5 or 10 or 15%? ]  

• All 10% shortfall probabilities are equal, but some are more equal than others. In less Orwellian but slightly more technical terms, different statistical distributions can share the same tail probability but have distinct risk and return profiles…Shortfall probabilities are not like LDL (i.e., bad) cholesterol; lower is not necessarily better  

• There are subtle assumptions embedded in these forecasts that have a nontrivial impact on the outcome. Can you justify these assumptions…? Do you even know these assumptions? …The tool’s user is given the illusion of control with a handful of statistical levers, but the head office wizards control 20. [comment: having built a fairly robust simulator from scratch I can attest to this.  There are a ton of assumptions and biases embedded in the construction of a tool and even more assumptions and biases in the interpretation of the results] 

• The highly technical and subtle stochastic 2.0 lecture makes no sense until the deterministic 1.0 lecture is crystal clear  

• Here is the main problem. How do we explain to people that they (1) are spending too much, (2) are retiring too early, or (3) have not saved enough? More importantly, how do we get them to act on this knowledge?

This is great commentary.   I am especially sympathetic to points 2, 5 and 7 and nothing I have to add here is likely to ever contradict those points. In fact, as we'll see much of my own commentary below is already implicit, if not explicit, in these bullets so I may, in the end, be adding nothing at all.   

On the other hand I also don't think that probability or simulation based tools are completely unhelpful or should be entirely avoided, even in an early-and-simple stage of planning, because I don't think the problem is necessarily in the tool or the complexity and I don't think one needs to be an actuary or risk management specialist to derive some important value.  The problem, in my mind, is that the right questions are not being asked and also, and maybe more importantly, that some retirees have been known to hand over ownership of decision making to both tools and advisors (hence the mentioned risk of abuse) because of the intimidation factor involved in what can feel like magical tools dealing with a complex if not unsolvable underlying problem area. And to be fair, it really can be complex. Here, for example, is Richard Thaler, behavioral economist at the University of Chicago, on retirement: 

"For many people, being asked to solve their own retirement savings problems is like being asked to build their own cars."

Here is William Sharpe, Nobel Laureate, Stanford :  

"[Retirement income planning] is a really hard problem. It’s the hardest problem I’ve ever looked at."

Here, finally, is Harry Markowitz, the "father" of Modern Portfolio Theory, on the personal finance problem vs. the breakthroughs he had with MPT for institutional investors:

"...the game [of life, i.e., allocating scarce resources over an uncertain lifetime] is complex; most likely beyond analytic techniques."

So, no, I don’t think we should ditch complex tools yet -- or tools in general; the more the merrier I say -- just because they can be abused or can confuse. Let's put some or all of it back on the retiree where it properly belongs.  I might be wrong; let's see where this goes.   

FOUR SIMPLE (AND SOMETIMES HARD) QUESTIONS

In order to hang onto our complex tools and put some of the necessary effort back in the retiree's lap let me pitch the idea of using more tools rather than less but at the same time "up the game" for the tool user by making them (and us) ask at least 4 questions[1], the asking of which I contend might make simple and complex tools "equal" or at least make them play by the same rules so that there is some degree of simplicity on both sides of a complexity divide:    

1. How much time have you (we) spent thinking about and understanding the assumptions and do you (we) "own" them? Take responsibility for these, I say!
   
   
2. Is there any palpable, really really good and solid reason to worry, stand up and take action? 
   To flesh this point out a bit, I mean the following: does the feedback from any analysis do anything to get our attention and why? Are we on the edge of our seat or are we kind of chill? Is there anything in any analysis that should catalyze hard action? Are we above/below some kind of academic or maybe personally defined threshold? What threshold, if any, makes sense and where does it come from? If action is indicated, are we really prepared and able to take action and if so what action and by how much and when?   

3. Do we have anything resembling broad consensus from multiple, independent[2] analytical sources and methods to confirm our edge-of-seat status or need for action? What are the most conservative and aggressive tools saying to us?  How confident are we in what the numbers say and mean and whether they are all saying the same thing? Does common sense have anything to tell us while we are at it? Do we trust?  

4. How is this analysis changing or trending over time and does that tell us anything or change our action plan? Should we ask, perhaps, how we "feel" about it? 

AN INITIAL (FAKE) TEST

For a complex tool, let's say we run a hypothetical Monte Carlo Simulation (MC) on a hypothetical recent early retiree.  For the simple tool we'll use the formula for portfolio longevity that Milevsky suggests using as an alternative, or prelude, to MC simulation[3].  For the moment, we have allowed the advisor or the tool to use some vaguely generic or default assumptions.  Ignore what they are for now, I'm making all of this up.

The fake results:  85% success rate from the MC and 30 years from the formula for portfolio longevity from Milevsky.  

Before we ask "the questions" from above I'd have to say I give Milevsky's simple formula the nod.  That's because the 85% MC success rate, especially given all the complexity and buried assumptions, is more or less meaningless to me; it could be scary or reassuring depending on how you look at it.  What exactly does one do with a 15% fail rate?  "30 years of portfolio life," on the other hand, is at least a number I can work with -- depending on age -- if I accept it.  The assumptions are mostly transparent and I get the concept.  Now let's ask "the questions."  In my opinion both tools fail at this point.  Walk them through the questions and they can't be answered.

The case might have been clearer, of course, and one or more of the tools more helpful if: a) we had done more homework on the assumptions, and b) the fail rate were to have been more like 50% and the portfolio longevity, say, 20 if one were age 55.  But I also think that this still does not go far enough nor does it satisfy questions 3 and 4 for either tool (which may only be relevant to me, I guess, but let's keep playing the game).

A CASE STUDY

Now let's do a case study for an unnamed early retiree but now with "the questions" in the back of our mind as we proceed: 1) what are the assumptions…and own them, 2) know when there is a reason to worry/act and why, 3) look for confirmation and context, 4) look at time series context. Then integrate everything and stir.  Since we are looking for confirmation and context, a whole bunch of tools will be used at the same time including the simple formula and Monte Carlo as well as other methods. In the charts below:

K         Kolmogorov Differential Equations for Lifetime Ruin probability

MC      Custom Monte Carlo Simulator

Fire      FireCalc.com - Free Historical Simulator [8]

cFire    cFireSim.com - Free Historical Simulator

B          Regression Formula as Monte Carlo proxy

FRP     FlexibleRetirementPlanner.com - Monte Carlo Simulator

EL        Expected Portfolio Longevity formula

Since time and trends matter now, we will track this from 2010 to 2016 and where we can we will integrate methods into one view (portfolio longevity is shown separately since it does not measure success/fail rates).  Assumptions vary depending on tool but there was some attempt to make sure they were consistent.  Also some assumptions changed over time, for example: forward looking return rates are getting more conservative in 2015 an 16 in some tools. Simulators and success formulas are graphed as fail rates rather than success rates. 

This is what we come up with: Portfolio Longevity vs Time (simple method) on the top and Fail Rates for misc tools vs Time (more complex stuff) on the bottom:    

If we happen to look at these before we ask "the questions," I still think Milevsky is more right than not because I still have problems with the meaning of fail rates and I have no confirmation from anything else if I haven't even looked for it. I have also not pre-thought about assumptions or thresholds or time series (I wouldn’t even have these charts).

After the questions, though, it gets more interesting.  Let's walk it through.  

1. Do we know and own the assumptions?  Because someone, hopefully the retiree, has gone way, way overboard to look at and integrate a bunch of both simple and complex tools, it is more than likely that the assumptions have been closely looked at and analyzed if for no other reason than to make sure they are consistent across many tools.  It's hard to do that without knowing and owning them.  This is not "simple" but it is the case study we have.

2. Is there any palpable, really really good and solid reason to worry, stand up and take action?  Let's start in 2010 and 2011.  This is a little bit of sleight-of-hand because I did not set up something this extreme in the initial test.  The answer: well…hell yes.  By any stretch of anyone's imagination an 80-100% fail rate estimate would get attention and demand action, especially when combined with question 3 on confirmation.  In addition, because the retiree has read widely and come to his or her own conclusions, he/she decided that anything over 30% fail estimate consistently presented over more than a 6-12 months time frame might demand action.  For the Portfolio Longevity "simple" formula: 20 years of expected portfolio longevity for a 50 year old -- after a policy of "must last past age 95" has been implemented before the analysis[4] -- is also something that requires urgent action here...for at least a couple of the growth rate assumptions.  In 2012, on the other hand, all of the analysis is in a more nuanced situation. There is a wide range of results (see question 3) but we are below threshold for the most part and at least trending better (question 4) while there is an "average" consensus of results that feels like it might give more confidence than not.  In 2013+ this whole thing we are discussing looks like it is nothing more than a continuous process under control, like an industrial manufacturing process.  The experience of fear in 2010 and 2011 (to say nothing of 2008-9) might keep the retiree on the edge of a seat poised to act[5] as needed to keep the process within it's control limits (see question 4 on how trends and emotions might interact) but steady as she goes for now. 

3. Do we have anything resembling broad consensus from multiple, independent analytical sources and methods to confirm our edge-of-seat status or need for action?  Yes.  And by independent we mean here not only different analytical methodologies (historical sims vs MC sims vs formulas vs common sense vs a spreadsheet, etc) but also that the retiree has taken intellectual control from the advisors or software that may or may not have conflicts or biases that are either hidden or visible.  In the case study there is a range of results, of course, but the consistency of results over time might breed some minimum level of confidence about the general sense of things.

4.  How is this analysis changing or trending over time and does that tell us anything or change our action plan or maybe how we feel about it?  Some of this was answered in #2 above but let's look at 2012 in the case study.  In isolation, without past (or future) data points, it might have been a confusing or unnerving year.  The presence of a trend might, however, have informed the process.  At the least it would have said to the retiree: "keep doing what you did last year and the year before because it worked then and it'll probably work now."  On the other hand this topic of time and trend is a little tricky.  Notice that the case study time frame happens to be part of a giant bull market.  Just as a mind experiment: if one were to make an "all else equal" assumption, and then look at something like 2008 and 09...fail rates would have skyrocketed for no other reason than the change in nest egg[6].  I wrote about this effect here: What Would A4% Fixed Withdrawal Actually Have Felt Like In 2009?  It depends on how much you spend of course but in the 4% constant spend case I profiled there the fail rates not only would have gone up steeply in 2008, they would have stayed near or above 30% for something like five years (even longer in the 1970s) and then come down again.  That begs the question of when thresholds should be respected and when they should have an adaptive flex.  The retiree in this study chose 6-12 months maximum before decisive action but there is no hard answer to this I think. That might be another post…  

ANY CONCLUSIONS?

So where does all of this leave us?  Looking back on all of the mess above it looks like I have made things more complex. It looks like simplicity is nowhere to be seen on the other side of anything;  I'm not so sure that I proved anything about the "far side of complexity" being simple.  The only thing I can conclude, I guess, is that while some of the tools themselves may be complex the answers to the "questions" using those tools can be very simple -- on both sides of complexity, when they are looked at the right way.  Both simple and complex tools, in my opinion, give the same simple answers to the same questions and I think that having more answers than less doesn't hurt in tough situations. A bad situation is bad whether using simple or complex tools and a good situation is the same in both as well.  For ambiguous situations, I think the four questions above still help and in that sense I think that complex tools, rather than being shunned, should more likely be embraced as an additive and constructive context-building exercise.  The weakness was never in the tool or its abuse or it's ambiguity but in the need for decisive action at the right time by the right person with the right questions...but this is something already pointed out in Milevsky's article. 

I'll finish with a few casual take-aways from the case study. No science or exhaustive lists here, just some personal thoughts:

- Own your own assumptions and know what they mean

- Simple tools can be as useful as complex ones and vice versa

- Don't retire early unless you know what you are doing

- Have a plan and thresholds and think about when to act

- Have multiple and independent analytical points of view that you trust

- Use context and extreme results to catalyze action

- Track data over time

- Have ready some big levers to pull especially with respect to spending and income

- Don't fear complexity or complex tools. Use it to inform decision making

- Don't abandon common sense

The last word, you may be surprised to hear, I will give to the Financial Samurai at www.financialsamurai.com who makes the point about ruin probabilities way better than either Milevsky or I did. His post is: The Fear Of Running Out Of Money In Retirement Is Overblown.  While I'm trying to point out here that complexity and probabilities and fail rates (which is nothing other than thinking about the risk of running out of money) are probably useful in some extreme situations and that they might add a bit to "simplicity" in a perverse way that no one but me may understand (and maybe that they can also confuse and be abused in less extreme situations), and while Milevsky is saying that analytical baby steps are probably a good thing before starting to run with complex models, FS points out, on the other hand, that they -- fail rates or the fear running out of money -- are a wee, tiny bit of a red herring altogether.  Read the article; I think it's worth the time. His opinion and rationale, in summary, include these points:

1) You will need less than you think.

2) You don’t need to save for retirement once you are retired.

3) You will adapt to different income levels.

4) You will be in a lower income tax bracket.

5) You will find many ways to make money.

6) You have more to offer than you think.

7) Your existing assets have upside.

8) You can always tap your pre-tax retirement accounts early.

9) You can create your own destiny.

10) Plenty of safety nets.

and...You’re Stronger Than You Think 

These are the real reasons that fail rate analyses are sometimes foolish, not that they are complex or stochastic or too subtle (a point that Milevsky acknowledged in the 2nd bullet I quoted above).  But with respect to the arguments made by both Milevsky and FinancialSamurai, I have to say that I think that sometimes an unaware retiree might need a nasty little catalyst to wake up to real, destructive, and imminent risk[7].  In my opinion, a super-high (and complex) ruin probability won't hurt in achieving that goal … if it doesn't eat you alive before you wake up, that is. 

--------------------------Notes--------------------------------------

[1] I made these questions up just to blog. I am sure there is a longer, more rigorous list.  

[2] This is important.  By independent I mean both tools and people.  For tools, I want to say use as many methodologies as possible coming from as many directions as possible.  Use formulas, Use simulators, use both montecarlo and historical rolling simulators.  Create a spreadsheet. Whatever.  With respect to people, it is all about trust: do they have visible (or hidden) biases, assumptions, or conflicts of interest.  You should know or at least think about it. 

[3] Milevsky/Fibonacci formula for portfolio longevity:

EL        estimated portfolio longevity

M         nest egg (money)

w         withdrawal rate in dollars per year

g          annual real growth rate of portfolio as a %. It should also be

            net of fees, taxes, and maybe market risk. Much of the constructive

           discussion, in an advisory conversation, will revolve around g

[4] As mentioned, to-age-95 is a policy decision.  It comes from knowing things like personal health, family history, and the general shape of the longevity probability density function for a age something-year-old (say 58) where the mean is something (say 83), the mode is another (say 88) and the longest recorded human live is something like 122.  That means age 95 is a policy choice rather than a statistic.  Conservative but well founded perhaps and vaguely supported by no small number of well known advisors and researchers.   

[5] The whole concept of an action plan: how, when, how much, etc. is not covered here.  If it were, I'd maybe point to an income statement if one is kept.  Knowing exactly what spending can be cut is helpful and a simple spreadsheet can help with sensitivity analysis.  New income is a whole other topic but not foreign to younger retirees.  If I had a list I'd point to other bloggers like Darrow Kirkpatrick or Financial Samurai or Mr Money Mustache or Dirk Cotton or...

[6] this would be a good segue to a discussion on the sequence of returns risk.  For this I'd recommend Wade Pfau at retirementresearcher.com.  Just do a search. 

[7] Check out Dirk Cotton at www.theretirementcafe.com.  His posts on chaos theory and bankruptcy spirals in retirement are another type of wakeup call.  His contention is that fail rates and sequence of returns risk are dwarfed by the risk of rapidly spiraling multi-cause retirement bankruptcy.  Sweat that idea out for a while.

[8] There are some important differences between historical and MC sims that goes untouched here.  In general, and by their nature for an early retiree, I think that historical rolling models -- the long time frames for ER mean that historical rolling models can only examine a small number of historical spans given the data, spans that tend to bias towards the middle of the 20th century -- tend to be more optimistic than I'd prefer.  This has been covered by others better elsewhere. This, I think, is a case for integrating multiple points of view, though.

[9] Here is Darrow Kirkpatrick on this concept of adaptation thwarting ruin:

“…complete financial failure is unlikely if the analysis shows a relatively low probability. Why? Because most sensible people are going to modify their lifestyle if the numbers start heading in the wrong direction. So the failure rate associated with a probability-based analysis is more a metric for the likelihood and severity of having to adjust your lifestyle, than it is a prediction for the odds of eating cat food later in retirement.” 

----------------Addendum--------------------------------------

Just for fun, this is what the Milevsky-Fibonacci formula looks like if you blow it out for different withdrawal rates and different real growth rate assumptions.  This might be a little bit of a downer to someone that wants to retire young, has high spending-rate thoughts, but also happens to have a more realistic than not set of assumptions of future growth rates.  Maybe a simulator would help?