The Power of Bias in Economics Research - Politics Forum.org | PoFo

Wandering the information superhighway, he came upon the last refuge of civilization, PoFo, the only forum on the internet ...

"It's the economy, stupid!"

Moderator: PoFo Economics & Capitalism Mods

Forum rules: No one line posts please.
#14857022
Berkeley Initiative for Transparency in the Social Sciences wrote:
The Power of Bias in Economics Research

Tom Stanley, friend of BITSS and one of the leaders of the Meta-Analysis of Economics Research Network (MAER-Net), recently published a paper in The Economic Journal with John Ioannidis and Chris Doucouliagos on statistical power in economics research that should be of interest to the BITSS network.

From Tom:

“The power of bias in economics research” is the first large-scale study of the bias, statistical power, and hence the scientific credibility of economics. A survey of 64,076 economic estimates from 159 areas of research and 6,700 empirical studies finds that the median statistical power is 18%, or less. That is, the probability that an empirical economic investigation is able to identify what it seeks is usually 18%, or less. Impotence begets bias. This survey also identifies widespread bias. Typically, reported economic effects are inflated by 100% with one-third inflated by a factor of four or more. In other words, over half of economic research results are reported to be twice as large as they actually are, and one-third are exaggerated to be four times too big. Lastly, 90% of economics findings are under-powered, relative to the widely accepted convention that 80% defines ‘adequate’ power, for half of these areas of economics research.

Download the paper here.


I find myself hoping that this rather devastating assessment turns out to be an exaggeration. My view of the social sciences has been quite dim for a while now, and economics was one of the few fields that moved the balanced in a more positive direction.
#14857025
Kaiserschmarrn wrote:economics was one of the few fields that moved the balanced in a more positive direction.


The dismal science, really? I take all social science with a very large grain of salt, the systems under study are vast and complex and academics are all too human. Even with the hard sciences realism is a stretch, scientism is rampant within Western culture.
#14857030
Sivad wrote:
The dismal science, really?

Yes, I know the term, but economics has the reputation of being very open to criticism and robust debate. Their data is also much more likely to be publicly available or shared if requested and their quantitative education is much better than in other social sciences, as far as I know.

I assumed that this would significantly improve research in the field compared to others.

Sivad wrote:
I take all social science with a very large grain of salt, the systems under study are vast and complex and academics are all too human. Even with the hard sciences realism is a stretch, scientism is rampant within Western culture.

One of the problems is that today everything in politics has to be "evidence-based", so we are encouraging confirmation bias even more in that people produce data and research in order to increase the chances for their preferred policy.
#14857474
Looks like a weird interpretation of empirical research to me. If the null hypothesis cannot be rejected that means just that, namely "no effect" cannot be rejected, meaning it would be wrong to claim an effect. Obviously it is possible for there to be an effect regardless and rejecting "no effect" is impossible due to a variety of reasons (such as small sample size etc.). That goes without saying.

The idea that statistical power can actually be measured post-hoc seams rather shady to me (I haven't read all of the paper).

Edit2: Here's a good short summary:

http://www.dokeefe.net/pub/okeefe07cmm-posthoc.pdf
#14859469
Kaiserschmarrn wrote:Their data is also much more likely to be publicly available or shared if requested and their quantitative education is much better than in other social sciences, as far as I know.

The latter is quite true, the empirical methods employed in economics are some distance ahead of those in the other social sciences - the seminar culture is also, I feel, much more robust versus the other disciplines, and much more capable when it comes to weeding out bad results.

The former claim I don't believe to be true outside of macroeconomics - and most research is in microeconomics - which is a significant issue which, at least, the discipline is starting to address. Though a bigger issue is that there is a very limited incentive to run replication studies in economics.

---

Perhaps an unpopular opinion, but I am less concerned about inflated effect sizes. Given how noisy a lot of these observations are, persistent qualitative agreement across studies seems of much greater substance.
#14863581
Kaiserschmarrn wrote:Yes, I know the term, but economics has the reputation of being very open to criticism and robust debate.

Are you serious? Modern mainstream neoclassical economics is the least open to criticism and debate of any discipline I am aware of, including theology.
Their data is also much more likely to be publicly available or shared if requested and their quantitative education is much better than in other social sciences, as far as I know.

But it is all based on invalid definitions, so the data are misclassified, wrongly aggregated, and meaningless.
I assumed that this would significantly improve research in the field compared to others.

At best, economics is like the discipline of mechanics before Newton defined the difference between "weight" and "mass." At worst -- i.e., in modern mainstream neoclassical economics -- its principal function is to provide fallacious rationalizations for unjust and harmful public policies.
One of the problems is that today everything in politics has to be "evidence-based", so we are encouraging confirmation bias even more in that people produce data and research in order to increase the chances for their preferred policy.

And that is easy to do when the empirical data are rendered meaningless by incorrect definitions and classification. Imagine the state of mechanics if its practitioners had no way to measure mass other than weight.
#14863701
Truth To Power wrote:At worst -- i.e., in modern mainstream neoclassical economics -- its principal function is to provide fallacious rationalizations for unjust and harmful public policies.


That's all it is, it's a captured discipline that should just be disregarded at this point. I find it surprising that any dispassionate thinking people take anything coming out of the social sciences seriously. It's unfortunate but almost everyone and every institution in Western civilization is almost entirely full of shit. People are assholes and the world is a joke.
#14863807
Sturgeon's law states that ninety percent of everything is crap. In my experience Sturgeon's law isn't so much that ten percent of the output of any given discipline is unadulterated truth, but that the conclusions of even the best research only contain a degree of truth. A good corollary to Sturgeon would be something like 'the best output of any field is only mostly crap'.
#14863840
Young (2017)'s recent paper which suggests that the use of Instrumental Variables doesn't seem to be very often justified - huge deal within the field. Potential candidate for what's driving the issue the OP highlighted since their use has exploded since the 1990s.

---

I also hear that economics is captured a lot but usually with little reference to the actual literature. For example, here's the unpaywalled version of an NBER working paper which released this week. It came up on the top of the list of new papers which gets sent to me at the start of each week - the order of which is randomised (since about last year following a paper which suggested that papers which appeared at the top of the NBER email list got higher levels of citations).

You can also look at, this paper or this paper, if you wish. Both of which were also published by NBER this week.

I'm just, genuinely, curious as to where exactly you guys feel this capture makes itself obvious.
#14864262
Vlerchan wrote:I also hear that economics is captured a lot but usually with little reference to the actual literature.

See Gaffney, "The Corruption of Economics." It is extensively referenced.
For example, here's the unpaywalled version of an NBER working paper which released this week.

I'm just, genuinely, curious as to where exactly you guys feel this capture makes itself obvious.

Notice it is taken as a given in this paper that people's perceptions of their economic security are determined by how well they are doing as land speculators.
#14864304
Fed Accused of Academic Capture–Is FASB next?

Priceless: How The Federal Reserve Bought The Economics Profession

“The Fed has a lock on the economics world,” says Joshua Rosner, a Wall Street analyst who correctly called the meltdown. “There is no room for other views, which I guess is why economists got it so wrong.”

One critical way the Fed exerts control on academic economists is through its relationships with the field’s gatekeepers. For instance, at the Journal of Monetary Economics, a must-publish venue for rising economists, more than half of the editorial board members are currently on the Fed payroll — and the rest have been in the past.

[...]

The Fed has been dominating the profession for about three decades. “For the economics profession that came out of the [second world] war, the Federal Reserve was not a very important place as far as they were concerned, and their views on monetary policy were not framed by a working relationship with the Federal Reserve. So I would date it to maybe the mid-1970s,” says University of Texas economics professor — and Fed critic — James Galbraith. “The generation that I grew up under, which included both Milton Friedman on the right and Jim Tobin on the left, were independent of the Fed. They sent students to the Fed and they influenced the Fed, but there wasn’t a culture of consulting, and it wasn’t the same vast network of professional economists working there.”

But by 1993, when former Fed Chairman Greenspan provided the House banking committee with a breakdown of the number of economists on contract or employed by the Fed, he reported that 189 worked for the board itself and another 171 for the various regional banks. Adding in statisticians, support staff and “officers” — who are generally also economists — the total number came to 730. And then there were the contracts. Over a three-year period ending in October 1994, the Fed awarded 305 contracts to 209 professors worth a total of $3 million.

[...]

Just how dominant is the Fed today?

The Federal Reserve’s Board of Governors employs 220 PhD economists and a host of researchers and support staff, according to a Fed spokeswoman. The 12 regional banks employ scores more. (HuffPost placed calls to them but was unable to get exact numbers.) The Fed also doles out millions of dollars in contracts to economists for consulting assignments, papers, presentations, workshops, and that plum gig known as a “visiting scholarship.” A Fed spokeswoman says that exact figures for the number of economists contracted with weren’t available. But, she says, the Federal Reserve spent $389.2 million in 2008 on “monetary and economic policy,” money spent on analysis, research, data gathering, and studies on market structure; $433 million is budgeted for 2009.

That’s a lot of money for a relatively small number of economists. According to the American Economic Association, a total of only 487 economists list “monetary policy, central banking, and the supply of money and credit,” as either their primary or secondary specialty; 310 list “money and interest rates”; and 244 list “macroeconomic policy formation [and] aspects of public finance and general policy.” The National Association of Business Economists tells HuffPost that 611 of its roughly 2,400 members are part of their “Financial Roundtable,” the closest way they can approximate a focus on monetary policy and central banking.

[...]

The Fed keeps many of the influential editors of prominent academic journals on its payroll. It is common for a journal editor to review submissions dealing with Fed policy while also taking the bank’s money. A HuffPost review of seven top journals found that 84 of the 190 editorial board members were affiliated with the Federal Reserve in one way or another.

“Try to publish an article critical of the Fed with an editor who works for the Fed,” says Galbraith. And the journals, in turn, determine which economists get tenure and what ideas are considered respectable.

[...]

Being on the Fed payroll isn’t just about the money, either. A relationship with the Fed carries prestige; invitations to Fed conferences and offers of visiting scholarships with the bank signal a rising star or an economist who has arrived.

Affiliations with the Fed have become the oxygen of academic life for monetary economists. “It’s very important, if you are tenure track and don’t have tenure, to show that you are valued by the Federal Reserve,” says Jane D’Arista, a Fed critic and an economist with the Political Economy Research Institute at the University of Massachusetts, Amherst.
https://www.huffingtonpost.com/2009/09/07/priceless-how-the-federal_n_278805.html
#14864365
I'm just, genuinely, curious as to where exactly you guys feel this capture makes itself obvious.


Wherever the literature dissagrees with my values and philosophical presuppositions about how reality works of course. :)
#14864460
Vlerchan wrote:Perhaps an unpopular opinion, but I am less concerned about inflated effect sizes. Given how noisy a lot of these observations are, persistent qualitative agreement across studies seems of much greater substance.


Not sure what you mean by inflated effect size. You mean that work that shows significant effects is more likely to get published, thus a "literature review" will overestimate the effect? Funnily enough, if that is an issue, the fact that so many underpowered studies are being published is actually a good thing (not that I ever thought of it as being bad).
#14864573
Economics peaked with the classical era of political economics. Smith, Marx, Ricardo, etc relied on making cogent arguments accessible to educated intelligent persons. The modern tendency towards mathiness obscures the fact that economic models don't really model anything - they are demonstration toys with little or no predictive power.

The basic problem is that economic systems are artificial constructs. They are created by humans for human purposes. Rule-based systems whose rules can be (and often are) changed to fit circumstances aren't really the best candidates for constructing empirically verifiable models.
#14864579
The modern tendency towards mathiness obscures the fact that economic models don't really model anything - they are demonstration toys with little or no predictive power.


What makes you say this? There is no particular reason you couldn't mathematically model behaviors if you understand the basic behaviors as behavioral economics and sociology try too. That we dont have a perfect model underlines that we don't have perfect understanding. If smith, marx, ricardo, etc's. arguments got at that real understanding then our models would work now based on their work using their assumptions about human behavior.

And yet they don't work to model the economy, which suggests that saying they were the peak of economics is a bit foolhardy. We should never assume any field is or was solved, that bias will make us incapable of ever changing our minds or progressing in our understanding.

The basic problem is that economic systems are artificial constructs. They are created by humans for human purposes. Rule-based systems whose rules can be (and often are) changed to fit circumstances aren't really the best candidates for constructing empirically verifiable models.


Except this is not so significantly different than biology. It's an emergent system of chemistry and it's interactions which adapts in it's functions with ever changing rules and adaptations and environments. Yet we've come to a very good understanding of many of it's principles and has rendered itself quite well to understanding through empirically verifiable models.
#14864592
mikema63 wrote:What makes you say this? There is no particular reason you couldn't mathematically model behaviors if you understand the basic behaviors as behavioral economics and sociology try too.
[...]
Yet we've come to a very good understanding of many of it's principles and has rendered itself quite well to understanding through empirically verifiable models.


I. MISTAKING BEAUTY FOR TRUTH

It’s hard to believe now, but not long ago economists were congratulating themselves over the success of their field. Those successes — or so they believed — were both theoretical and practical, leading to a golden era for the profession. On the theoretical side, they thought that they had resolved their internal disputes. Thus, in a 2008 paper titled “The State of Macro” (that is, macroeconomics, the study of big-picture issues like recessions), Olivier Blanchard of M.I.T., now the chief economist at the International Monetary Fund, declared that “the state of macro is good.” The battles of yesteryear, he said, were over, and there had been a “broad convergence of vision.” And in the real world, economists believed they had things under control: the “central problem of depression-prevention has been solved,” declared Robert Lucas of the University of Chicago in his 2003 presidential address to the American Economic Association. In 2004, Ben Bernanke, a former Princeton professor who is now the chairman of the Federal Reserve Board, celebrated the Great Moderation in economic performance over the previous two decades, which he attributed in part to improved economic policy making.

Last year, everything came apart.

Few economists saw our current crisis coming, but this predictive failure was the least of the field’s problems. More important was the profession’s blindness to the very possibility of catastrophic failures in a market economy. During the golden years, financial economists came to believe that markets were inherently stable — indeed, that stocks and other assets were always priced just right. There was nothing in the prevailing models suggesting the possibility of the kind of collapse that happened last year
http://www.nytimes.com/2009/09/06/magazine/06Economic-t.html?_r=1&pagewanted=all
#14864596
Models have failures, we don't understand economics or indeed biology or physics to explain a great many phenomenon.

That our understanding is limited and our models incomplete does not mean they have no predictive power at all. Newtonian physics is a great way to model motion and orbits and a great many things, it's a terrible way to understand quantum mechanics.

It's unreasonable to expect perfection from economics and not physics and declare economics unmodelable by mathematics.

It is implausible that the IDF could not or would[…]

Moving on to the next misuse of language that sho[…]

@JohnRawls What if your assumption is wrong??? […]

There is no reason to have a state at all unless w[…]