The Free Dictionary  
mailing list For webmasters
Welcome Guest Forum Search | Active Topics | Members

Reactions to the Reproducibility Project, at First Baffling Options
Epiphileon
Posted: Saturday, August 29, 2015 4:19:19 AM

Rank: Advanced Member

Joined: 3/22/2009
Posts: 4,161
Neurons: 151,478
I suppose I had a background suspicion for some time that things weren't quite right. There were even a number of times I had encouraged caution concerning some of the claims we've seen in headlines concerning new findings in psychology. The news this week has been unsettling in this regard; however, as more information has come to light I am somewhat encouraged. The first headline was...
"Many Psychology Findings Not as Strong as Claimed, Study Says"
I had days to ponder this article, and left it open in a tab, as I felt I really wanted to address this; however, although I thought I could see between the lines of the first article, I was afraid it would be too depressing and was dragging my heels.

My biggest fear was that there had been a widespread failure of rigor in social science research. I should qualify that though, I never felt there was sufficient rigor in most social science research to begin with, but this article seemed to be painting a severely more dismal picture. I still think there is a big problem, but what I find very encouraging is that this project was undertaken at all.

A very big worry I still have though concerns the dynamics of how this problem came to be in the first place, largely the funding process. Who funds, who gets funding, expectations of the funders biasing, or outright influencing the research, and so forth. Many of these problems are discernible by people who understand what rigor means in science, unfortunately this is a relatively small percentage of the population, and even worse a very, very small, percentage of elected officials who control public funded research.

A follow up article was published today that was quite a relief to read...
"Psychologists Welcome Analysis Casting Doubt on Their Work"

(Caution both of the above articles are from the NY Times, so if you think you may want to refer back to them, open them in new tabs and leave them open, the NYT has limits on how often you can view pages without a subscription.)

What is extremely cool, exceedingly groovy, far out, and even neato, is the entire study, its subject papers, and results are available at...
The Open Science Framework page, "Estimating the Reproducibility of Psychological Science"

Question authority. How do you know, that you know, what you know?
Ni un voto al PRIAN PVEM
Posted: Saturday, August 29, 2015 10:35:04 AM

Rank: Advanced Member

Joined: 12/16/2013
Posts: 460
Neurons: 54,261
Location: Mexico City, The Federal District, Mexico
Very often scientific studies are published or retracted to fit the political agenda of those in power. In 1910-1920 those ruling in the United States had the idea of creating a superior race, long before Hitler. So the biologist Charles B. Davenport, published several papers to justify selective breeding and massive sterilization of those mentally unfit, jail inmates, poor and sick people. Thousands of people were forcibly sterilized by the government based on his papers.

Right now there's a strong movement for gay rights, and the scientific literature is leaning in that way accordingly. An eye catching conclusion published in a psychology journal I read : 'homophobia is linked to homosexual tendencies', or removing homosexuality from the DSM. Papers showing a link between vaccines-autism, and mobile phones-brain cancer have been quickly dismissed.
Any researcher claiming otherwise will put his career in peril and ostracism from his/her peers. Silver or lead, most people take the silver
Rajinder Tyagi
Posted: Saturday, August 29, 2015 12:00:36 PM

Rank: Newbie

Joined: 8/13/2015
Posts: 190
Neurons: 2,254
Location: New Delhi, NCT, India
I agree with all that Epiphileon has conveyed. There are numerous cases where the industry has not permitted the invention to succeed as it would jeopardize their product.An example is the radiation from mobile phones and linkage with cancer. Similarly their are studies and counter researches to justify the use of coffee and chocolate. It is the high and mighty who decide the agenda. The rest, including manipulation of research results, follows.
Epiphileon
Posted: Sunday, August 30, 2015 6:42:39 AM

Rank: Advanced Member

Joined: 3/22/2009
Posts: 4,161
Neurons: 151,478
While I agree that there can, and at times has been inappropriate political and corporate/industrial influence on research and that these are issues of concern, that is not the point of the project nor of my posting on it here. The point here, as I see it, is that apart those influences there is a problem with a lack of studies being replicated.

This project also, it seems to me, brings to light another huge problem. It seems like, (by the way all these hedging qualifiers are intentional), it seems like people have come to think that the results of a paper are authoritative!?!? That is most decidedly not how it is supposed to work. I've written a couple of papers on research in psychology, and worked on the analysis of uncountable others, and never, absolutely never did we state in the discussion section of those papers anything as conclusive, on the basis of only those studies, or on the basis of only our own work. That would be abandoning rigor, and opening the investigation to all sorts of biasing and invalidating variables. Apparently though this has happened to a degree, and I do not think it is due to either political or corporate conspiracies.

I see a number of critical issues raised by the results of the Reproducibility Project.
1. A possibility that the results of a single paper are coming to be seen as authoritative.
2. A lack of motivation to do replication studies.
3. Related to number 2, a hyper concentration on original work by professional journals.
4. The manner in which scientific papers are covered by the media.
5. The willingness of the public to accept sensationalism rather than applying critical thinking to what they read. (ok so this one is really just a pet peeve, not actually an issue illumed by this research)

I find it very encouraging that this project is being conducted, and will look for ways to support its continuance, even if that is only writing to the projects authors and expressing my appreciation.


Question authority. How do you know, that you know, what you know?
FounDit
Posted: Sunday, August 30, 2015 9:45:51 AM

Rank: Advanced Member

Joined: 9/19/2011
Posts: 11,217
Neurons: 56,952
I must say I was not surprised to see these findings.
I think your points below are the most salient.

1. A possibility that the results of a single paper are coming to be seen as authoritative.
As a society, we've been conditioned to trust "experts" on nearly everything without applying the critical thinking you mention next.

5. The willingness of the public to accept sensationalism rather than applying critical thinking to what they read. (ok so this one is really just a pet peeve, not actually an issue illumed by this research)
You highlight what I think is an important point here in that sensationalism too often becomes a public meme, especially in today's world with social media. This wouldn't be such a problem but for the fact that the meme can be totally wrong, but no one would know that unless they apply some critical thinking.




We should look to the past to learn from it, not destroy our future because of it — FounDit
Ni un voto al PRIAN PVEM
Posted: Sunday, August 30, 2015 6:47:03 PM

Rank: Advanced Member

Joined: 12/16/2013
Posts: 460
Neurons: 54,261
Location: Mexico City, The Federal District, Mexico
Quote:
it seems like people have come to think that the results of a paper are authoritative!?!?

Where did that bizarre idea come from?


Quote:
5. The willingness of the public to accept sensationalism rather than applying critical thinking to what they read. (ok so this one is really just a pet peeve, not actually an issue illumed by this research)

..and a very capitalist way of thinking 'let's privatize the gains and socialize the losses'. If something goes wrong is everyone's fault for not reading the small print

Quote:
I see a number of critical issues raised by the results of the Reproducibility Project.
1. A possibility that the results of a single paper are coming to be seen as authoritative.
2. A lack of motivation to do replication studies.
3. Related to number 2, a hyper concentration on original work by professional journals.
4. The manner in which scientific papers are covered by the media.

Nothing of this exclude the possibility of a underground political agenda. Just have to ask more: Why studies are not replicated? Why media overhype in the news some studies and not others? Just coincidences?
'From error to error, one discovers the entire truth'. Sigmund Freud

Hope123
Posted: Saturday, September 5, 2015 12:22:15 PM

Rank: Advanced Member

Joined: 3/23/2015
Posts: 8,710
Neurons: 49,922
Location: Burlington, Ontario, Canada
Epi,

I agree with your points. Replication of results is always critical. I realize you are talking specifically about psychology but it seems to me that any thinking person always questions what the media spouts about one study in any field and wants more proof. If results are being over emphasized, that should be stopped.

How much responsibility do the Journals that publish articles have to take to see these are replicated?

In other fields I have heard people who don't usually ruminate about issues, say things like "Forget it. First coffee is good for you. Now it is bad. Soon there won't be anything to eat". Or "Why did they waste money doing such a stupid study?" So I am not sure exactly how much influence these have on people who just want to get on with their daily lives or who decide the experts are wrong anyhow, that they know better.

I hope you don't mind that I have generalized from the field of psychology to other fields in Science.

One also has to be careful not to take all allegations as gospel and publish them until it has been investigated thoroughly. It can damage not only the individual scientist, but also the whole field. Quote is from "The Lancet".

"Last week, allegations of research misconduct against Paolo Macchiarini, a professor at the Karolinska Institute in Stockholm, Sweden, were finally dismissed. The university issued its final ruling, concluding that there was “nothing to support suspicions of scientific misconduct”. Macchiarini's work on the transplantation of tissue-engineered tracheas was published in The Lancet in 2011. This article, and several further research and review papers, were the targets of criticism by some of Macchiarini's former colleagues. But Karolinska judged that he was emphatically “not guilty”.
...The allegations made against Paolo Macchiarini were not only harmful to one individual. They also raised questions about the quality of research into regenerative medicine itself. Rebuilding confidence in the field of tissue engineering may take some time to achieve."

All I am saying here is that there needs to be caution in both directions. Make sure the studies are sound, but do so without damaging legitimate studies. Replication studies need to be replicated/investigated to make sure they are not just 'sour grapes', an attack on a scientist rather than on the study. This scientist had his reputation under a cloud for four years - and many will remember only the allegations and not the acquittal.

It is good that these points are being investigated, that the scientists themselves are welcoming the accountability demanded. It should be done in medicine too. How many people realize that there have been no studies in many areas and operations and drugs are GRAS when no real data exists?

"Scientific studies can be structured so as to produce a certain outcome. This is a known fact and is an art in itself. The structure of the study often decides its outcome." There are studies on both sides about the advisability of using stents for heart disease.

Quote is from -

http://heartattacknew.com/faq/do-heart-catheters-and-the-insertion-of-stents-really-help/

As for the Cancer Biology trying to replicate results or find the most effective treatment, from personal experience it seems to me they are doing the replication. I have three friends being treated for cancer who are/were in studies re treatment. Two are actively in studies. One finished the study but unfortunately it did not help her - she did not make it.

I was just in a study re risk for MVP - mitral valve prolapse. (No risk,) My doctor's whole practice is doing a study on colon cancer.
I got a letter in the mail yesterday asking me to do a cholesterol study re statins but I don't need them.
I am glad that they are doing such studies.




"The louder he talked of his honor, the faster we counted our spoons." Ralph Waldo Emerson (1803-1882)
Users browsing this topic
Guest


Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Main Forum RSS : RSS
Forum Terms and Guidelines | Privacy policy | Copyright © 2008-2019 Farlex, Inc. All rights reserved.