How much faith do I have in the forthcoming equity risk premium for planning purposes?

To ask the title question is to probably have an opinion in hand which I do. I've read a lot of finance over the years. The academic papers and the advanced practitioner papers that focus on markets and instruments and factors -- as opposed to those that deal with retirement, consumption and decumulation -- will casually mention the equity risk premium as if it is a Newtonian force or a guarantee. Take risk, must get paid. Heh. I mean, yes there is something to it and certainly in productive and relatively unfettered economies, capital used for growth gets compensated one way or another or at least it has historically been comped in the US. And yes, there is of course risk that is usually measured in standard deviations. But the wise will also consider shocks, chaos and fat tails. But in addition to all that there are also macroeconomic and policy forces along with crud that someone once referred to as "opulence, corruption, extravagance and waste" that might force one to eventually bend a knee in abject submission, forces that could make one doubt the whole enterprise of planning using historical data for conjuring forthcoming return expectations over our human planning horizons.

Case A for me is Japan over the last 70 years. I don't personally have a bead on the underlying reasons for what happened but economists can probably explain it all I'm sure.  Also, if I understand it correctly, the Nikkei does not report total returns (div) so in general my analysis will probably be pretty sketchy and/or naïve. But let's take the index at face value for a second just for fun. Anyone is free to comment below their thoughts on Japanese markets because I know so little about this except at a surface level. 

First, here is a chart of the daily index since 1949:

Figure 1. Nikkei

Let's pretend this is a total return index (maybe I'm wrong, idk, too lazy to research) to make a point. Imagine being an about-to-be Japanese retiree in late 1989 who plans to allocate a lot to the Japanese market represented by the Nikkei. If I were to look backwards there would have been ~14% compound growth since 1949. The linear antecedent is maybe ~15% if I got it right. So, that number might be (might. I mean, it probably would have been reasonable to be a little circumspect about 15%, even in 1989) my input to MVO or other planning because "history," duh, and because a bunch of awesome, genius models had been tightly fit to historical data and the "real world" and they also had all those super duper PhD level autoregressive features and fancy stochastic inflation and cool return distributions with fitted asymmetry and that realistic stochastic vol thing, etc. So, one would, of course, have had a ton of confidence, over the forthcoming next 20 years of retirement because, well, because the equity risk premium is a "guarantee," right? and all the models were really really smart and they told me it'd all work out fine and that I could spend a constant 4% for at least 30 years if not longer.

How did it work out? For our 1989 retiree, the world gave way and gave way in a way that even a smart model would have never predicted and probably for longer than it would have predicted if it had predicted it. Late 1989 to the lowest point of the trough (~20 years) was something like a compound -8%. Measured to 2021 (~32 years) it was about a compound -1%. These are reasonable retirement horizons, btw. Plug -1% (or it's asset allocated linear-antecedent equivalent) into your retirement calculator and see what happens. I mean, in real life I'd perhaps be as circumspect about -1% as I was about +15% but the point is, can I trust my forthcoming return expectations for any kind of structural stability. Sure, based on history but history says nothing really about anything except history. The future is an entirely different game. In the US, what return expectations should I use if I don't trust. And if I do use those expectations are they going to be stable "enough" over the very important next 20 or 30 years? Are we about to "give way?" No idea. My point, in its essence, whether I have understood the Nikkei correctly or not, is that one never knows whether one is at a decisive and meaningful inflection point or is merely about to experience some ordinary and/or prolonged volatility. Even modeling fat tails or a chaotic pulse (say 1987) will not really show you the risk of real strategy decay unless you explicitly model that scenario or isolate only the dark paths within a simulation. 

But then I suppose it all depends on the forces that one thinks might be at play coming up. The productivity, energy, innovation and enterprise of Americans engaged in the American economy? I certainly hope so; I trust that more than the more academic "equity risk premium." Will there be volatility? of course. Shocks? probably. What about macro-economic forces and national policy choices, revolution, social decay, repression,  opulence, corruption, extravagance and waste? I hope not.  The reality is that I can and do hope that my 3-4% real return assumptions will work, even with vol and shocks, over the next 20-30 years. But I don't really know and I don't yet have a plan other than expectations adaptation and spending adjustments. I'll bet at every tick down in the Nikkei I would have thought that that was the turning point. For 20 years.