Bad Science Sleuthing

[BMWTurboDzl]

For lunch time reading: https://retractionwatch.com/

Came across this while reading an article in the WSH about bad science and individuals who actually dive deep into the data and the resulting conclusions. This is a searchable database. Oh and something to keep in mind. Thousands of papers are published every year.

 

[Vil_Riker]

He seems to be ostracized for not following the doctrine.

 

[bunnspecial]

Retraction watch was regular reading for me when I was in graduate school.

Unfortunately, in general, things like data fabrication are all too common. A classmate of mine was dismissed late into his PhD when his advisor did some basic sleuthing and caught edited elemental analyses. The advisor already had suspicions, but lab mates saw him using a PDF editor and the advisor just requested copies directly from the lab.

With the pressure involved, the temptation is very real especially if your experiments aren’t going the way you expect. It happened to me, but rather than massaging the data, I just ran with investigation the seeming anomalies and was able to show something beyond a shadow of a doubt that contradicted what my advisor thought should happen actually didn’t, and my seemingly anomalous data was actually correct. That quickly became the focus of my thesis

Of course too there are still the odd things that can frustrate you. I remember looking at .csv files of GC-MS data that were thousands of rows long and thinking how easy it would be to make a peak disappear or add a peak. Of course I would have been caught in about 10 seconds if anyone had pulled the easily-accessible data off the instrument, but the frustration and temptation was certainly there.

 

[loneryder]

A BIL's daughter was working on her PHD at Columbia in Bio Chem or something like that. The professor she was working under wanted her to change some of her data which falsified the premise. She refused and he refused her the PHD. She went all the way to the President of Columbia and got her PHD. She was so disgusted with the whole deal she left the profession. You can't believe any published science these days.

 

[Owen Lucas]

It happened to me, but rather than massaging the data, I just ran with investigation the seeming anomalies and was able to show something beyond a shadow of a doubt that contradicted what my advisor thought should happen actually didn’t, and my seemingly anomalous data was actually correct. That quickly became the focus of my thesis

That was the easy solution for your mate. You can make a bug a feature!

 

[Pablo]

No matter what, follow the money.

There will ALWAYS be bias. No it wasn't pure in the past.

The only way for more clarity is absolute transparency and even then. Some people want more .gov control, but as we have seen, that just makes more .gov controlled out comes.

Even scientists have human wants and needs. Yes some checks and balances by peer review and oversight - but peers are in on the game too.

 

[PandaBear]

Scientific research requires reproduction by other people outside your organization. It would be a matter of time before someone else catch a fraud.

The biggest lie actually comes from social studies, performing arts, economics, religion, finance, etc. Those fields often have unquantifiable datas and personal beliefs involved, and human cultures are always changing.

 

[Pablo]

Scientific research requires reproduction by other people outside your organization. It would be a matter of time before someone else catch a fraud.

The biggest lie actually comes from social studies, performing arts, economics, religion, finance, etc. Those fields often have unquantifiable datas and personal beliefs involved, and human cultures are always changing.

Outside sometimes never comes and yet decisions are made

True on the rest to a point, but mixing some of these with so called science is even more dangerous.

 

[BMWTurboDzl]

Retraction watch was regular reading for me when I was in graduate school.

Unfortunately, in general, things like data fabrication are all too common. A classmate of mine was dismissed late into his PhD when his advisor did some basic sleuthing and caught edited elemental analyses. The advisor already had suspicions, but lab mates saw him using a PDF editor and the advisor just requested copies directly from the lab.

With the pressure involved, the temptation is very real especially if your experiments aren’t going the way you expect. It happened to me, but rather than massaging the data, I just ran with investigation the seeming anomalies and was able to show something beyond a shadow of a doubt that contradicted what my advisor thought should happen actually didn’t, and my seemingly anomalous data was actually correct. That quickly became the focus of my thesis

Of course too there are still the odd things that can frustrate you. I remember looking at .csv files of GC-MS data that were thousands of rows long and thinking how easy it would be to make a peak disappear or add a peak. Of course I would have been caught in about 10 seconds if anyone had pulled the easily-accessible data off the instrument, but the frustration and temptation was certainly there.

IMO the layperson conflates the publishing of a study with a peer-reviewed study. They then run around saying "but this XZY study supports my ascertain".

 

[PandaBear]

Outside sometimes never comes and yet decisions are made

True on the rest to a point, but mixing some of these with so called science is even more dangerous.

Well, sometimes you just have to make a decision so you can move forward, that's a fact of life. In the end over a long period of time things have to add up or the equations will balance out in a way you don't anticipate, for both sides.

 

[Pablo]

Well, sometimes you just have to make a decision so you can move forward, that's a fact of life. In the end over a long period of time things have to add up or the equations will balance out in a way you don't anticipate, for both sides.

No sometimes we don't need to act.

Man is a curious creature.

 

[bunnspecial]

Scientific research requires reproduction by other people outside your organization. It would be a matter of time before someone else catch a fraud.

Part of the issue here too is that a lot of research published is low impact enough that no one notices it when published and bothers to repeat it. It could be 10, 20, 30 years before someone digs it up, tries to repeat the results, and then reports the issue. By that point the PI may be retired or even dead and the grad students/postdocs who worked on it may be who knows where. Good journals will still retract(provided they are still around...) but a lot of people may well have finished out their careers based on fraud.

There's one chemistry journal I know of-Organic Syntheses-where, as part of the peer review process, every published synthesis is sent to two different labs that will replicate the results following only the published procedure. It's a a nice CV builder for the grad students/post docs who do the replication, and it gives the journal a lot of prestige since you know it was independently replicated prior to publication.

During my(less than one year) stint thinking I wanted to be a synthetic organic chemist, my advisor had me working on a synthesis that he wanted to publish there. Unfortunately it ended up being desk rejected because they didn't like one of the reagents we used(mercuric chloride, and they want to see more green chemistry) and we ended up with it in a less prestigious journal, but it was a good attempt and still good chemistry...

 

[bunnspecial]

IMO the layperson conflates the publishing of a study with a peer-reviewed study. They then run around saying "but this XZY study supports my ascertain".

There's also the whole issue of most people not knowing how to properly read research. I don't mean that in a patronizing way-it's just that it's definitely a taught/acquired skill. A good graduate program should have you spending a year or two in a seminar class or equivalent primarily to learn how to read and critique papers.

Many publications may be perfectly valid, sound science with the authors applying their conclusions only to a very specific of circumstances. They may extrapolate how the results could be generalized, but that portion shouldn't be considered a true conclusion of their research. Still, though, many people will read a very nuanced but perfectly valid conclusion based on limited experimental data and try to make it universally applicable.

 

[Shel_B]

No sometimes we don't need to act.

Man is a curious creature.

Creative indifference ...

 

[Pablo]

Creative indifference ...

Acting on improper or partial science is very often worse than the status quo. Even when we do know something unintended consequences bite us.

 

[SC Maintenance]

Trust the science.

The check is in the mail.

The government is here to help you.

And so on.

 

[PandaBear]

Part of the issue here too is that a lot of research published is low impact enough that no one notices it when published and bothers to repeat it. It could be 10, 20, 30 years before someone digs it up, tries to repeat the results, and then reports the issue. By that point the PI may be retired or even dead and the grad students/postdocs who worked on it may be who knows where. Good journals will still retract(provided they are still around...) but a lot of people may well have finished out their careers based on fraud.

There's one chemistry journal I know of-Organic Syntheses-where, as part of the peer review process, every published synthesis is sent to two different labs that will replicate the results following only the published procedure. It's a a nice CV builder for the grad students/post docs who do the replication, and it gives the journal a lot of prestige since you know it was independently replicated prior to publication.

During my(less than one year) stint thinking I wanted to be a synthetic organic chemist, my advisor had me working on a synthesis that he wanted to publish there. Unfortunately it ended up being desk rejected because they didn't like one of the reagents we used(mercuric chloride, and they want to see more green chemistry) and we ended up with it in a less prestigious journal, but it was a good attempt and still good chemistry...

Agree. To me the biggest "problem" with research is there aren't that many high impact work. Oftentimes we have too many researchers fighting for projects and funding. I'm sure in the science and engineering communities we are already fortunate enough to have the backing of industries, to fund researches that supposed to lead to products and profits. Low impact publications seems to be just a right of passage to help people graduate and get funding, and frauds not discovered till much later or ever speaks more of the need for these works than the quality of these works.

I really wish we as a society fund more useful work to our society instead of the quantities of researches, researchers, etc. Many of these researchers would have been teachers and lecturers instead of researchers and professors and be just as happy, if they are compensated better / the same and our society respect them as such. It is nice that we have people working on their passions but do we really need that many social studies graduates when they can't find works that will pay back their student loans?

 

[PandaBear]

Trust the science.

The check is in the mail.

The government is here to help you.

And so on.

We all based our decision on something, something we trust as simple as 1+1 = 2 or traffic lights is not trying to kill us with green on all 4 ways. Science is building future knowledge based on past knowledge, so it is more important to build it right instead of discard it completely. What are you going to build your decision on without science? Your engine oil is based on science and you don't put caster oil in your engine in 2023 because Castrol used to do it.

There are plenty of societies back in times and today with no functioning government. You do not want to live in one (Somalia).

 

[SC Maintenance]

We all based our decision on something, something we trust as simple as 1+1 = 2 or traffic lights is not trying to kill us with green on all 4 ways. Science is building future knowledge based on past knowledge, so it is more important to build it right instead of discard it completely. What are you going to build your decision on without science? Your engine oil is based on science and you don't put caster oil in your engine in 2023 because Castrol used to do it.

There are plenty of societies back in times and today with no functioning government. You do not want to live in one (Somalia).

Except "science" has become political, or business where a specific outcome is wanted - not search for the truth for the purpose of finding the truth. They have another objective. Yes, I would love to believe in science. I have an engineering degree which is based on a little science at least. However in the last few years my faith has been curtailed.

As for government - there job is supposed to be to deal with foreign affairs and enforce some minimal set of laws - though shall not kill, though shall not steal, etc. Except there no longer enforcing very many laws although they continually write new ones to further restrict the law abiding, and there not doing very well with foreign affairs either as the world seems to be one big brush fire. They have plenty of social workers and public housing administers and welfare processors though.

 

[PandaBear]

Except "science" has become political, or business where a specific outcome is wanted - not search for the truth for the purpose of finding the truth. They have another objective. Yes, I would love to believe in science. I have an engineering degree which is based on a little science at least. However in the last few years my faith has been curtailed.

As for government - there job is supposed to be to deal with foreign affairs and enforce some minimal set of laws - though shall not kill, though shall not steal, etc. Except there no longer enforcing very many laws although they continually write new ones to further restrict the law abiding, and there not doing very well with foreign affairs either as the world seems to be one big brush fire. They have plenty of social workers and public housing administers and welfare processors though.

That's what I said, hard science where results can be reproduced. You can't say a cancer drug doesn't work if the test result show, or a new ceramic is political, because the result shows it does.

Regarding to who funds what research, they are always funded with intention in mind. Nobody does science for the greater good alone because nobody is charity. This is the reason we still tolerate patents despite it is not "fair" to gouge people with a monopoly. Without profit nobody would fund future research and advance our society. You can't argue with results when science is there to prove it. You can choose to ignore the results and take the consequences (i.e. blood transfusion decision, taking certain medicine or not, using cell phone or not, eating GMO food or not), but that's not science, that's personal decision and politics. This is why science and engineering advances when there's wars and competitions, because you have to win regardless of what you believe, only those with competitive advantages will win.

I don't think scientists (not social scientists, they aren't real science) have anything to do with laws, that's up to the local laws and whether people can vote them in or out, and take their own consequences.