In the stock market, stuff happens you didn’t see coming. It could be something huge, like a war breaking out (some call this a “black swan” event), or relatively small, like a company using a new technology to build products. These events can cause prices to go up or down, but how do you incorporate such possibilities into your decision to buy or not to buy? That is the question.
Strangely, economists’ models typically don’t account for these types of unforeseeable changes. They tend to ignore what’s called “Knightian uncertainty,” named for the influential University of Chicago economist Frank Knight, who noted the importance of recognizing future risks that you can’t precisely measure. Today, most economists and economic forecasters work with simplified mathematical models that assume this uncertainty away. Which means that they are often deeply, spectacularly wrong. See: Financial Crisis, 2008.
Economist Roman Frydman, who teaches at New York University, has been thinking for a long time about how people form their outlooks on the future and was an early critic of the “Rational Expectations Hypothesis,” a theory that swept through academic economics in the 1970s. Since then, Frydman witnessed more and more of his colleagues embrace the hypothesis as the way to model precisely how market participants think about the future. But he remained deeply skeptical and devoted his academic career to honing his critique and developing an alternative.
Talking to Frydman is a bit like entering a fascinating meditation on the limits of knowledge and the nature of truth. Having lived through the failure of centralized planning in communist Poland, he is particularly sensitive to the hubris of believing that the future can be exactly mapped. Although the models that he has worked on are mathematical, Frydman’s approach to economics has drawn on the narrative insights of Friedrich Hayek, John Maynard Keynes, and Frank Knight. But beyond theoretical concerns about the role or efficiency of financial markets, his work has practical import for questions as big as how governments should interact with the economy, as technical as how financial services firms should design risk-management systems or use econometric models, and as personal as how to weigh advice from your financial guru.
As Frydman sees it, economists so yearned for a theory that would somehow predict the future nearly perfectly that they had to sail far from the shores of reality. With Michael D. Goldberg, he has written two books over the last decade that call for a change of course: Imperfect Knowledge Economics (2007) and Beyond Mechanical Markets (2011), both published by Princeton University Press. He also chairs the program on Imperfect Knowledge Economics (a title that itself is a challenge to the mainstream’s pretense to exact knowledge) at the Institute for New Economic Thinking.
Frydman’s research with Goldberg has led him to develop an alternative, which he calls the “New Rational Expectations Hypothesis.” His radical reframing of how economists should understand and represent rationality incorporates the most penetrating insights of the giants of twentieth-century economics into a formal mathematical approach that can better capture the contours of our ever-shifting world.
Lynn Parramore: The 2013 Nobel Prize in Economics awarded to Robert Shiller, Eugene Fama and Lars Hansen surprised many people because Shiller and Fama are widely seen as espousing opposite theories. Economist Paul de Grauwe tweeted: “Nobel Prize for Fama who led millions to believe financial markets are efficient and for Shiller who showed opposite. What a contradiction.” Was the Nobel committee saying something significant in awarding the prize to seemingly clashing positions?
Roman Frydman: The prize usually goes to long-accepted achievements. Not so this time. Fama was a pioneer. He focused on the importance of fundamentals – things like a company’s sales or dividends – in how stock prices move. Although his theory fueled the rational expectations “revolution” in finance, it had serious empirical shortcomings.
Shiller was working on those problems, noticing that equity prices move around too much to be justified by Fama’s theory. So something else had to be going on. Shiller’s answer was psychology. Who could object? We all have emotions: we become scared or irrationally exuberant. So-called behavioral-finance models, which focused on such considerations, became the leading competitor to the rational expectations approach.
The two theories were not only seemingly contradictory. They had sharply differing implications for how we should think about markets. For Fama, markets are nearly perfectly efficient. For Shiller, not so much – they sometimes resemble casinos with prices that don’t reflect the real value of assets. So who was right?
That issue has remained unresolved. I imagine that when the Nobel committee members wanted to give a prize related to market theories, they couldn’t give it for a settled notion because there wasn’t such a thing. Maybe they were communicating that both of these scholars had emphasized something very important about markets, but there was not yet a theory that could account for the importance of both fundamentals and psychology.
LP: It seems obvious that both fundamentals and psychology matter. Why haven’t economists developed an approach to modeling stock-price movements that incorporates both?
RF: It took a while to realize that the reason is relatively straightforward. Economists have relied on models that assume away unforeseeable change. As different as they are, rational expectations and behavioral-finance models represent the market with what mathematicians call a probability distribution – a rule that specifies in advance the chances of absolutely everything that will ever happen.
In a world in which nothing unforeseen ever happened, rational individuals could compute precisely whatever they had to know about the future to make profit-maximizing decisions. Presuming that they do not fully rely on such computations and resort to psychology would mean that they forego profit opportunities.
LP: So this is why I often hear that supporters of the Rational Expectations Hypothesis imagine people as autonomous agents that mechanically make decisions in order to maximize profits?
Yes! What has been misunderstood is that this purely computational notion of economic rationality is an artifact of assuming away unforeseeable change.
Imagine that I have a probabilistic model for stock prices and dividends, and I hypothesize that my model shows how prices and dividends actually unfold. Now I have to suppose that rational people will have exactly the same interpretation as I do — after all, I’m right and I have accounted for all possibilities. As economist Bob Lucas rightly put it, why wouldn’t they? They don’t want to lose their money, so I can represent how they forecast with my probabilistic model. This is essentially the idea underpinning the Rational Expectations Hypothesis.
LP: So being irrational means forecasting differently from the economist’s probabilistic model?
RF: Strangely, yes. In the hypothetical world of economists’ models, irrationality means inconsistency with an economist’s exact probabilistic hypothesis about how the world might unfold over time. This is very different from the dictionary definition of rationality, which is that people think for themselves, have their own objectives, and make decisions that advance those objectives. In a standard economic model, the economist assumes what the right way to think about the world is – and that the right way is to assume that unforeseeable change will not happen. I remember vividly how uneasy I felt when I first heard this Orwellian twist.
But, beyond language, this refashioning of rationality has had profound implications for the discipline. If you stick to probability, the rational expectations hypothesis does represent rationality. But that leaves you with only two options: Fama or Shiller. That’s why De Grauwe thought that these two positions were contradictory. But they’re only contradictory if you assume that the world can be represented without unforeseeable change, that Knightian uncertainty does not matter in real-world markets.
After all, in the real world, people do recognize the importance of unforeseeable change – indeed, they obsess about it. That’s precisely why they are so often forced to resort to their understanding of market sentiment and other psychological factors in deciding how to forecast the effect of fundamentals on future prices.
LP: Let’s talk a bit about how the media represents the stock market. The Financial Times runs a story with a typical headline: “Equities face worst quarter since 2011 over fears for global economy.” Other than suggesting that I need a stiff drink before looking at my retirement account, what does that headline tell me about the way stock forecasts get made?
RF: It tells you that both Shiller and Fama had some part of that story. You have the prospects of the global economy, which is Fama’s part, the fundamentals. Then you have fear, which would certainly be Shiller, the psychology. The headline puts these two things together. Yet economic theory is unable to put them together! That is the puzzle. What is the theory that could do that? It would have to be one that would open the model to unforeseeable change and Knightian uncertainty.
LP: You like to toss students a brainteaser: “If markets were perfect, we wouldn’t need them.” What does that mean, and what does it say about the need for a new understanding of the role of markets?
RF: For Fama and Lucas and their followers, if an economist gives you a rule that describes how outcomes will actually unfold, then there’s only one rational way to think about the future. But markets exist to enable society to take advantage of many different rational evaluations of the future. If there is only one rational forecast, the main reason for having markets disappears!
In the real world, there’s never just one correct interpretation or way to interpret. That’s why Friedrich Hayek argued – rightly – that central planning is impossible: it’s a pretense of exact knowledge. The planning committee sits down and figures out how the world will unfold and then allocates resources accordingly. What happens? In communist Hungary, people couldn’t find size 10 shoes.
Whenever humans – whether they are planners or economists – pretend that unforeseeable change doesn’t matter, they screw up. The Chinese are just now finding out that their weird combination of markets and centralized communist rule doesn’t work very well. After all, markets go down as well as up. George Soros understood this decades ago.
LP: What did Soros discover and how did it relate your work?
RF: I met George Soros in the early 1980s. The timing could not have been more serendipitous. I had just published a paper arguing that reliance on the Rational Expectations hypothesis in macroeconomics and financial modeling presumes that market participants and the economist must have already discovered a “true” model. Soros was completing his book arguing that participants’ understanding of the world never perfectly corresponds to the way it works.
Soros’s concept of reflexivity introduced the key reason for unforeseeable change in financial markets. Fallible market participants revise how they think about the future at times and in ways that they themselves cannot fully foresee, and this revision alters how outcomes unfold over time. So there can be no “true” probabilistic model of the market, as the last four decades of futile searching should tell us. Every model that fits the data for a period of time sooner or later becomes irrelevant. The only truth about quantitative economic relationships is that they are temporary. There are no universal laws. That’s why Soros titled his book “The Alchemy of Finance.”
Academic economists and finance practitioners ignored Soros’ argument that their models were alchemy, not science. In the absence of an alternative, many continued to search for rational expectations models that would account for longer-term economic regularities. And while others turned against the rational expectations approach and embraced behavioral finance, they continued to search for probabilistic models of irrational behavior.
I agreed that Soros’ arguments pointed to a fatal flaw in mainstream macroeconomics and finance. Once you focus on unforeseeable change as an inherent source of fallibility, it becomes immediately clear why there are no universal laws. For example, the consequences of historical events, such as the appointment of a Fed chair, are, at least in part, unique. These events change how market outcomes will unfold in ways that cannot be fully foreseen, let alone with a probabilistic rule.
Unlike Soros, I held out the possibility that economics was not just alchemy. But we did not see for quite some time how we could replace the probabilistic models on which mainstream macroeconomics and finance rested.
A simple idea was that unforeseeable change could have relatively moderate effects on the economy for periods of time. In such periods, you could imagine approximating outcomes with an econometric model. Although such a model could not capture the economy exactly, it could be quite useful for uncovering and testing qualitative regularities.
This pointed to a way forward for economics. But it was a sobering experience. If our models are to be useful for understanding real-world markets and rationality, we need to stop looking for a crystal ball.
LP: Your new theory embodies that stance. And yet you call it the “New Rational Expectations Hypothesis.” How does it work, and why does it contain the name of the old theory used by Fama and Lucas?
RF: When John Muth formulated the Rational Expectations Hypothesis back in 1961, it was truly brilliant. He said that market participants’ forecasts should be consistent with the relevant economic theory. Economists picked – and this is the key –probabilistic models as the relevant theory. But consistency just means that when you model, you represent how the market forecasts in light of your own hypothesis about how the future will unfold. It doesn’t mean that your hypothesis is correct!
The problem lies in deciding what will be relevant for the future. If my assumption about the future is that unforeseeable change is unimportant, then I’m going to get rational expectations. I’m going to get no systematic forecast errors. I will get perfectly efficient markets. But once I admit the existence and importance of unforeseeable change, I can show that rational expectations models represent irrationality in the most egregious sense. They represent people who forgo profits, because they ignore change and adhere to unchanging probabilistic rules.
The New Rational Expectations Hypothesis (NREH) solves this problem. You keep the consistency, but you base your representation of market forecasting on a model that is open to Knightian uncertainty. Now your model allows for more than one way to forecast the future, and thus better reflects the fact that rational profit-seeking participants in real-world markets do respond to unforeseeable change.
LP: What happens to psychology?
RF: NREH takes it into account. On the most elementary level, it influences how we choose to think about the future. There are many ways to think about it, and in order to choose among them, something has to drive you toward one alternative or another. And it isn’t simple calculation. Keynes called it “animal spirits.” A gut instinct. But it’s not pure psychology or irrational herd behavior. It’s connected to how you’re judging the fundamentals. Your psychological instincts, combined with your interpretation of fundamentals, add up to something quite rational.
Pure psychology may exacerbate market swings, but our research shows that psychological considerations alone drive the market only extremely rarely. We studied Bloomberg News “market wrap” reports from 1993 to 2009, and found that psychological factors are usually mentioned in connection with some movement of some fundamentals. It’s neither pure psychology nor pure fundamentals. It’s both.
Psychology is an essential component of rational forecasting. What NREH does is to capture this.
LP: If I’m a risk manager in a bank, how does this new understanding help me do my job better?
RF: The problem for risk managers is that standard models relate risk to the volatility of prices. By this measure, there was not much risk prior to the collapse of 2008! It’s well known that as the market keeps going up, prices become less variable. Yet Standard & Poor’s, relying on standard models, actually pronounced Lehman Brothers to be an A-rated firm just before it collapsed and nearly brought down the entire financial system.
You need something better. First, you need to be able to assess risk properly if the stock price moves away from something perceived to be a normal, fundamental value. It’s an idea that goes back to economist Jim Tobin and, before him, to Keynes: risk doesn’t depend on price variability, but on how far away prices are from the norm – which is not easy to determine, by the way. That’s the fundamentals part. But you need something from behavioral theorists, too. Two other Nobel laureates, Daniel Kahneman and Amos Tversky, showed that loss aversion is more important than variability to how people make decisions about uncertainty. Keynes knew that, too: the higher the risk of loss, the greater the fear of it. That’s the psychology part. You need a model – Goldberg and I have developed one – that combines both elements.
Then, your risk model needs to distinguish between large and small unforeseeable change. If you know that the world is buffeted by unforeseeable change, you figure that eventually there will be a big change, and that your model will become irrelevant. To do your job well, you must be on guard, test frequently for such change, and be prepared to alter your model – even though it may have worked well for quite some time. A fixed model simply won’t do the job.
LP: Does your new theory help economics become more scientific?
RF: Yes. Economists have long worried about being able to test their predictions empirically. If they couldn’t test them, they were philosophers, not scientists, as Soros put it.
As it turns out — and this is an important part of what I’m working on with pioneering colleagues at the University of Copenhagen — you may well be able to test the predictions of an NREH model. But you have to give up one thing: exact knowledge. You can get qualitative predictions: for example, you can say that when a company’s earnings rise, its stock price tends to rise as well. But you can’t get quantitative predictions: you can’t say exactly how much.
LP: Seems like a big thing to give up. How will economists and economic forecasters stay in business?
RF: It’s a big cost. It changes the status of economists. They are no longer fortune-tellers.
But, we gain something big, too. If we don’t become wedded to quantitative predictions, and we admit that both psychology and fundamentals matter, we can find out a lot about the world. We begin to understand that it is not a world of near-equilibrium; financial markets fluctuate. And we understand that these fluctuations are not necessarily a result of market inefficiency. They may be part of the way that markets try to assess what the actual values are. We begin to appreciate what markets really do.
And, when we understand better what markets do, it has an effect on government regulations, because we understand that markets sometimes go too high or too low. This awareness of the provisional nature of our knowledge has a very practical effect, actually, in the financial industry. We can adjust models as needed. In a broader sense, we gain the openness of the world and give play to creativity. We say that no one has the final answer. We get away from either/or thinking and imagine the multiplicity of possibilities.
LP: So the only truth is the non-existence of the one true model?
RF: It’s the genuine openness that makes our ideas – and education – more exciting. Students can think about things in an open, yet structured way. We don’t lose the structure; we just renounce the pretense of exact knowledge.
Economics is not mechanistic. It requires understanding of history, politics, and psychology. Some say that economics is an art, but NREH is actually rigorous economics. It simply recognizes that there’s a limit to what we can know.
Economists may fear that acknowledging this limit would make economic analysis unscientific. But that fear is rooted in a misconception of what the social scientific enterprise should be. Scientific knowledge generates empirically relevant regularities that are likely to be durable. In economics, that knowledge can only be qualitative, and grasping this insight is essential to its scientific status. Until now, we have been wasting time looking for a model that would tell us exactly how the market works.
LP: Chasing the Holy Grail?
RF: Yes. It’s an illusion. We’ve trained generation after generation in this fruitless task, and it leads to extreme thinking. Fama and Shiller need not see themselves in irreconcilable opposition. There is no one truth. They both have had critical insights, and NREH acknowledges that and builds on their work.
