Lewandowsky “Moon Landing” paper … down for the Count?

29 Mar

Geoff Chambers posted some of the comments from the invite only members section at Skeptical Science (that the admins there apparently left public for a period of time). One post from John Cook is highly relevant to this discussion (emph. mine):

2010–10-8 ” …a while ago, I added a bias field to the user database and a bit of code so as comments came in, I could specify whether the user was skeptics or warmest/proAGW/mainstream (still haven’t found a satisfactory term for our side). I only assign bias if its obvious from the comment. I haven’t done anything with that data yet, I’m not even sure why I’m doing it other than my obsessive compulsion to collect data. The other day, Steve Lewandowsky (cognitive scientist) asked if I had any numbers on the ratio of skeptics to warmists so I dove into the database and counted up around 100 assigned skeptics and around 400 assigned warmists.

If I did my math correctly 100 out of 500 total = 20% … exactly the same number as the alleged “content analysis” done on “1067 comments” from allegedly “unique visitors” at Skeptical Science (SKS).

It is simply far too great a coincidence that an analysis of 1067 comments, provided exactly the same ratio as Lewandowsky co-author John Cook said his internal database showed was the ratio of skeptic vs non-skeptic participants.

Which also almost exactly matched the 18% Lewandowsky reports another survey in the US found:

“… survey of the U. S. public in June 2010 pegged the proportion of “skeptics” in the population at 18% (Leiserowitz, Maibach, Roser-Renouf, & Smith, 2011) found.

Imagine that.

That all is bad enough on its own – however, there is another gigantic hole in their claims.

Their “content analysis” of 1067 comments, and their resultant claimed ‘skeptic share’ at Skeptical Science has effectively no real life relationship with the number and share of skeptic participants at SKS. This is especially true considering SKS’s highly antagonistic attitude and actions towards skeptics who dare venture to their little club – from regular non-skeptic participants and moderators alike.

Comparing comment counts offers no legitimate insight into the skeptic vs non-skeptic participant ratio, which is the metric supposedly being measured.

A simple review of one post at Skeptical Science is illustrative of the almost complete lack of association between skeptic comments and skeptic participant shares.

Lets choose the recent cross post from Shaping Tomorrows World, of John Cook’s excuse-making, err … ‘explanation’ of the Recursive paper posted here.

This post has generated, to date, a total of 106 responses. Lets break down the responses:

  • Barry Woods – skeptic – posts 9,13,14,15,22
  • Geoff Chambers – skeptic – posts 24,27,28,38,40,60,78 (heavily moderated and strong mod warning), 89 (moderated again), 101 (moderated – final warning)
  • Brad Keyes – skeptic – posts 30, 31 (deleted), 32, 34, 49, 51, 52, 55, 57, 58, 63, 66, 69, 70, 73, 92, 93, 96, 98 (moderated and banned – Brad’s foray is a typical skeptic experience at SKS)

There are a couple other posts that could be definied as skeptical of the SKS position in this thread, but cannot be defined as AGW skeptic’s – which is the subject of this study.

While we’re reviewing lets look at a few of the typical comments from non-skeptics to see how skeptics are treated at SKS:

  • Post 47 definitely NOT a skeptic – but a perfect demonstration of the attitude towards skeptics at the site (you’ll note this clear ad hominem attack was ignored by mods):

    Although a recent occurrence came close, no thread was better deserving of the following metaphorical advice: “don’t wrestle with a pig. You’ll both be covered in mud and the pig loves it.” I urge all that are able to think rationally to not waste their time.

  • Post 68 – from a site moderator:

    I’m not going to engage you, because this is silly. John Cook’s paper says it all, and I have no intention of spending my time arguing about “sides” that have been entirely fabricated by a small community of “victims” who have identified themselves as separate and special.

  • Post 74 – another site moderator – John Hartz:

    @John Cook: Do you now have enough raw material from this comment thread for another paper in your series?

  • Post 102 – post by site moderator – “John Hartz” another site moderator tells him not to “feed the troll” referring to geoff chambers.

So – what’s the count:

  • Skeptic comment count: N=35 comments (out of 106 total) … 33.02% of all comments
  • Non-Skeptic comment count: 71 comments (of 106 total) … 65.98% of all comments
  • Skeptic individual participants count: N=3 (out of 42 total participants) … 7.14% of all participants are SKEPTIC’s
  • Non-Skeptic individual participants count: N=38 (out of 42 total participants) … 90.48% of all participants are NON-SKEPTIC’s

An interesting aside – regarding participation by moderators and/or SKS staff:

  • SKS moderators/staff commenting individually count: N=40 comments (out of 106 total) … 37.74% of all comments were made by SKS staff/moderators
  • SKS moderators/staff individual participant count: N=10 (of 38 non-skeptic posters) … SKS moderators/staff comprised 26.32 of all participants

The data – at least in this post from SKS is clear … contrary to the authors conclusions, while 33% of the comments here were from skeptics, just 7% of the participants were the same.

This shows the SKS formula provided in the Lewandowsky Supplemental Information – which uses total skeptic comments vs non-sceptic comments as support for and proof of the alleged diversity of the SKS, provides little or no meaningful information on the share of skeptic’s who participate at SKS.

Additionally, the paper’s “comment analysis” is, to be charitable, highly suspect, considering this supposed standalone comment analysis finds EXACTLY THE SAME conclusion as author John Cook found in his wholly separate and unrelated forum post – using his own internal participant “bias” information

Worse, the Lewandosky authors extend this seriously flawed “comment analysis” conclusion as definitively representative of all 7 other non-skeptic sites that offered the survey.

A more detailed review would likely find somewhat different results, however the basic premise here – that a “comments” review does not provide meaningful insight into the share of “participants” who are skeptics, is not likely to significantly change.

[Authors Note: please consider this a draft – quick online – version. Am in process of more detailed fact and error checking – thanks]

8 Mar

JOSH_lewpaper_bustedAs the authors have not responded to any of the questions submitted regarding the “Recursive fury: Conspiracist ideation in the blogosphere in response to research on conspiracist ideation” and the underlying “NASA faked the moon landing | Therefore (Climate) Science is a Hoax: An Anatomy of the Motivated Rejection of Science”  (known as LOG12) paper …

I formally submit the following direct questions for Mr. Lewandowsky (and/or any of the authors of either paper).

1. You have repeatedly claimed, since approx. July 2012, that the “LOG12/Moon Landing is a Hoax” paper has been peer reviewed, accepted for publication by Psychological Science, and has been published. You repeat that claim in the current “Recursive” paper and again in your comments above.  You have also cited the LOG12 paper as a reference in multiple subsequent papers, including the current “Recursive” paper.

Yet to the best of my knowledge the paper has never been published, online or in print, nor has Psychological Science ever acknowledged acceptance of the paper for publishing.

Please provide:

(a.) … information that proves your claim the LOG12 paper has been published, and;

(b.) … information which shows, or provides acknowledgement, that the LOG12 paper actually has been peer reviewed and accepted for publication by Psychological Science. 

2. You have repeatedly claimed, including in this blog, and again several times in the “Recursive” paper, that the online supplementary material for LOG12 contains the raw data, information and detailed methods necessary to review and validate the LOG12 findings.

Please provide:

(a.) … evidence this supplementary material has been peer reviewed, accepted for publication, and published, and;

(b.) … information on where this supplementary material may be accessed online as you have repeatedly claimed, and;

(c.) … how/where the supplementary material may be obtained from the authors?

3. Legitimate questions have been raised regarding the current “Recursive” paper as well. Here too you have claimed the paper was proper peer reviewed, accepted for publication, and has been published. In this current case, we at least know the paper has been published online by the Frontiers journal. However, there are a number questionable activities associated with the publishing of this paper as well.

Please provide a response:

(a.) It is well established that peer review must be completed prior to acceptance, and certainly prior to publication. Yet here the list of peer reviewers has undergone 3 significant changes in the weeks after publication – there have been 4 different iterations of listed peer reviewers in the course of a few weeks post publication.

The current listed peer reviewers include the Associate Editor of the paper, which seems highly suspect and of questionable ethics and professionalism – when we know other qualified reviewers were available.

The sole remaining independent reviewer is a graduate student in journalism, not Psychological Science, with a pretty clearly identifiable, highly sympathetic bias towards the authors positions, and a business relationship with the authors institution.

Whether true or not, this creates a clear perception of “pal” not “peer” review of this work.

Question: How and why have the listed peer reviewers changed multiple times after publication, and why were apparently qualified reviewers removed, after being listed, and replaced by the Associate Editor in charge of the paper?

(b.) Upon publication a PDF was provided of the full paper. This PDF was included on the Frontiers page for the paper, and is referenced by Mr. Lewandowsky in the blog post above. In the days after publication, this PDF was removed from the Frontiers page. And several days later the link in the authors blog post above – which clearly states it is to the PDF of the paper – has been changed to point only to the abstract.

Question: Why was the PDF removed and when will it be replaced? 

(c.) The authors of “Recursive” several times in the paper reference the supplementary material, and that it contains the additional detail information to support the claims and findings of the paper. Yet, as with the prior LOG12 paper, no supplementary has, by all appearances, been made available.

Question: How/where can the supplementary material be obtained from the authors?

4. In the Recursive paper the term “LOG12″ appears 120 times.  The LOG12 paper is cited by the authors as a reference. And in fact the LOG12 paper is the entire basis for the Recursive paper. Yet the LOG12 paper as we know has not appeared in print or online, has not been published in any form to date, and no independent public acknowledgement has been made that the work has been reviewed and accepted by a scholarly journal for publication.

Question: Please explain the professional, ethical and scholarly reasoning behind citing prior work as a reference in subsequent papers when this work; has drawn significant criticism (including from supporters of the authors), has not been published, and where the authors actions in withholding supplementary material – including raw data, methods and information which they note is critical to the papers findings – have resulted in the inability of outside reviewers to validate this work. 

5. In the “Recursive” paper the authors note as one of their “recursive hypothesis” the conspiracy theory “Control data suppressed (6)” … regarding the authors failed attempt to collect a “control” sample for the LOG12 papers findings from the campus population. The authors attributed this conspiracy theory to “the pervasive belief that something must be wrong (NI, MbW)”.

In footnote “5” regarding the above comments, the authors note note they ”subsequently obtained a control sample via a professional survey firm in the U.S” and that “This representative sample of 1,000 respondents replicated the results involving conspiracist ideation reported by LOG12.”

The authors own actions regarding this concern, that the criticism prompted the authors to admit their campus survey had failed (a fact they did not disclose in the LOG12 paper), and that they subsequently obtained a professional “control” survey, show that the criticism was valid and well founded.

Question: When a criticism is found to be accurate, please explain how you find it to be a “conspiracy theory” – an example of conspiracist ideation? Please explain how this is not demonstrably proven, by the authors own response, to be pure and simply legitimate criticism? 

These are legitimate, reasoned, questions of the authors. They deserve a professional and honest response.

Replication of Lewandowsky Survey

6 Sep

There has been considerable discussion about the methodology and data regarding the recent paper “Lewandowsky, S., Oberauer, K., & Gignac, C. E. (in press). NASA faked the moon landing – therefore (climate) science is a hoax: An anatomy of the motivated rejection of science” (copy here)

This allegedly peer reviewed paper claims their survey data show climate skeptic’s are supporters of wild conspiracy theories, such as “NASA faked the moon landing.” The author admits, however, no climate skeptic sites were involved in the survey, that essentially all survey results were obtained thru posting the survey on pro-global warming sites.

Due to the serious and legitimate questions raised, I have recreated the Lewendowsky Survey in an attempt to replicate and create a more robust set of replies, including from skeptic users.

Please click on the Lewandosky Survey Page above and take the survey survey. This survey re-creates the original survey … replicates the questions (as identified in the Lewandowsky paper and several other sites) in the original survey,  including those questions deleted from the survey results.

The only change was to use a 1 to 5 ranking vs. Lewandowsky’s 1 to 4, which several people with experience have noted should improve the overall responses.

Each visit to the survey is tracked. Access is password protected for an additional layer of tracking.

THE PASSWORD FOR THE SURVEY IS “REPLICATE” (case sensitive)

Please only complete and submit once. Also, please respond to each question with the answer that best reflects your position, even though the question may not be perfectly worded.

This survey is built on the Google Doc’s open access platform. Results are collected automatically. As no significant randomization or counterbalancing was performed on the original survey none is applied here. Data collected will be provided upon request.

A. Scott

Yet Another New Blog …

6 Sep

… as if we needed a few more.

Seems I have figured out how to do a blog. Not sure what I’ll do with it yet … time will tell.

;-)

Follow

Get every new post delivered to your Inbox.