Trading Ethos for Logos

Up until very recently (the last 10 years) it has been uncommon for social science researchers to share their data even when the sharing would neither compromise the private information of the subjects nor the validity of the study. [1]

Even more uncommon is it for researchers to willingly share the code they used to transform their data from its raw state to the state required to produce their analysis.

Researchers instead refer to what is often an incomplete “write up” included in their published work as reference for inquiries.

As a matter of course, they by and large do not publicly recognize the possibility of coding inconsistencies unintentionally or intentionally created as existing except if it were the case in which an error was so large or gross that a reviewer could detect it through superficial examination.

Of course, as every programmer knows, errors (or bugs) inevitably pop up and are only removed through a combination of careful examination of code, experience at error detection, and luck.  For researchers to not share both their code and their data is therefore an act of unscientific hubris at best, deceit and publication chasing at worst.

Even when code is shared, the code which is shared is often defaced and unintelligible without comments or instructions on how to connect the data to the code.  I have experienced this situation first hand, the result being that unsurprisingly it is usually more energy efficient to recode the entire analysis from scratch rather than attempting to interpret the baffling and intractable code of another.

Furthermore social science research is generally ranked by the number of citations it inspires, a measure more related to its ability to create a splash, than its truthfulness.  Thus it might be profitable for a researcher to exaggerate or selectively pick results that further the researcher’s position at the expense of identifying true effects or results.

In this paradigm, even when it became widely known that a paper employed faulty methods, the critique of that research needs to undergo the hurdle of being presented or published in a secondary source at some later point in time resulting in a large scale failure to disseminate corrections to analysis.

It is ironic, that in a profession (economics) that takes as its base assumption that humans are utility maximizing and only moral when it is cost effective, that such a lax incentive structure would have developed.

The result of this sordid academic publishing incentive system is a lemons market for academic research (See Recent Economist’s Article) in which it is believed that there is good research “out there”, but it is uncertain how that research is being done and by whom.

So what is the solution?
Enter the Open Access Revolution. This revolution is characterized by a new structure of analytical work heralded by a rise in open access journals, open data providers, requirements for the sharing of data by mainstream journals, and the general proliferation of shared research methods through online communities.  Open access journals in particular are exceedingly helpful in improving the ethics of scientific research by providing standards in which research, data, analysis, as well as critiques are required to be shared.

The hope is that by transitioning from black box research methods to a system in which information is shared, the lemons market will be squeezed to death as unethical and sloppy research is crushed out of the market.  Not to say that all “bad” research is the results of unethical behavior or lack of carefulness in coding or analysis.  It is simply impossible to know what the quality of the research being done is until it can be assessed externally.

Open Source Statistical Programming Languages (R, Julia, etc.)
In order to accomplish the full dissemination of research and research methods not only is the sharing of data and code ideal but the sharing of the means of running the analysis.  To this end I believe open source programming languages are a highly effective tool.  Analysis done in these languages and shared through publications is highly accessible to any researcher, even researchers who do not have a background in that language, since anybody can install these languages for free, run the code, and get help from their highly active communities.

This however may not be the case with other statistical programming languages which present large barriers to analysis by requiring the purchase of expensive or difficult to find software.  In addition, other software options may not be as well supported as these programming languages, resulting in a situation in which even if you have someone else’s code, there might not be a support network available to help debug and service that code.

It is therefore my belief that using an open source statistical programming languages should become the standard in statistical analysis and the setting of standards for scientific research, especially in the highly subjective fields known as the “social sciences”.

Data Aggregators: (Quandl, the Social Science Research Network, etc.)
In order to further the goal of open access to data, data aggregators and in particular Quandl have emerged.  These independent organizations have taken on the mammoth responsibility of gathering, organizing, and redistributing publicly available data for the purpose of advancing scientific research. 

Combining code from an open source statistical programming language and the data from Quandl, a new possible data analysis paradigm has emerged which is as different from the old paradigm as mountain spring water is from muddy puddles.  In this new paradigm, it is possible to have every piece of an analysis including the access and downloading of data, coded, and openly available for review and duplication.  Students and fellow researchers will not have to trudge through the difficult and often impossible work of trying to rebuild the analysis of others based on their published works.

Disclosure
For the sake of full disclosure, there is the possibility that I will be a guest blogger on the Quandl blog in the near future.  Thus my favoring of Quandl in this post, might be influenced by that anticipated opportunity.  That said, I find Quandl a truly remarkable undertaking and hopefully a game changer for the standards in future research.

[1] Purely personal observation.  I do not have data to support this claim.