CHAPTER 8 of A Judge's Deskbook on the Basic Philosopies and Methods of Science, by Shirley A. Dobbin, Ph.D, and Sophia I. Gatowski, Ph.D Peer Review, Publication, and General Acceptance
In determining whether theory or technique is scientific knowledge that will assist the trier of fact, and, thus, whether expert testimony is admissible, is whether the theory or technique has been subject to peer review and publication. Fact of publication of theory or technique, or lack thereof, in a peer-reviewed journal will be relevant, though not dispositive, consideration in assessing scientific validity of a particular technique or methodology on which expert opinion is premised; submission to scrutiny of scientific community is a component of "good science," in part because it increases the likelihood that substantive flaws in methodology will be detected.(1) 'General acceptance' of scientific theory or technique can have bearing in determining admissibility of expert testimony ... Widespread acceptance of scientific theory or technique can be an important factor in ruling particular evidence admissible, a known technique that has been able to draw only minimal support within the community may properly be viewed with skepticism.(2)	Learning Objectives for Chapter 8 Upon completion of this chapter, the reader should be able to: Provide an overview of the peer review process; Identify and discuss critiques of the peer review process; and Discuss the differences between referred and unreferred journals and books, and presentations at professional conferences.
The Peer Review Process Though pre-publication review is important, it does not by itself establish validity. The review by other scientists that comes after publication is far more significant. Valid scientific knowledge results from mutual criticism and intellectual cooperation. This process does not merely reflect or accompany the scientific method, it is the scientific method. Sharing new discoveries with other scientists, and establishing priority, lies at the heart of science. Peer review is the process by which scientific papers submitted to journals for publication are reviewed by experts in the field. These experts, or peer reviewers, are familiar with the topic area from their own work and thus can be considered peers of the authors. Reviewers are asked to evaluate the importance and usefulness of the research and judge whether it was performed carefully and accurately. Journal editors consider the reviewers' opinions in deciding whether to publish the submissions. It is important to note that in most cases reviewers do not have access to the raw data. Rather, the reviewers must rely on the narrative description of the research design, including how the data were collected, and from what source, and how the data were interpreted.	Institutional Review Mechanisms Because the process of formulating and testing hypotheses is far from simple and trivial, a probing and careful review is an indispensable part of the scientific enterprise. The review of scientific ideas takes place in a variety of contexts. Informal review can occur when scientists discuss their work with one another at the laboratory bench, during seminars, and at scientific meetings. Formal peer review is generally an integral part of the scientific publication process and the process by which funds are allocated for the conduct of research. Any claim that would significantly add to or change a body of scientific knowledge must be regarded skeptically if it has not been subjected to some form of peer scrutiny, preferably by submission to a reputable journal.
To what extent do judges around the country find the concept of peer review a useful criterion for critically evaluating scientific evidence? All of the judges surveyed (N=400) were asked how useful the concept of "peer review" is for determining the admissibility of scientific evidence. The majority of judges surveyed felt that it was a useful concept. Just over half (52%) felt that it was "very useful" and 40% felt that it was "somewhat useful." Only 6% reported that peer review was "not at all useful" when determining the admissibility of scientific evidence. When the responses of judges in FRE/ Daubert states (n=205) were examined, ratings were the same, with half (53%) reporting that peer review was "very useful" and 40% reporting that it was "somewhat useful." 7% of FRE/ Daubert judges felt that peer review was "not at all useful" to determinations of admissibility. When asked a question about how they might apply the concept of peer review to a determination of the admissibility of proffered evidence, 71% of all of the judges surveyed (N=400) provided responses that demonstrated a clear understanding of the scientific peer review process. In 26% of responses, it was not clear that the judge fully understood the concept. In only 3% of the responses, however, was it clear that the judge did not understand the concept of peer review. When the responses of judges in FRE/ Daubert states (n=205) were examined, the results were almost identical. Some sample comments related to the peer review process included: "You have to look at the motivations behind publication -- generally, you have to assume that articles are published in good faith and not just for the purposes of litigation -- you have to examine why the study was done" "I would give greater weight to a criticism of the technique or procedure if it appeared in a significant number of high status journals -- or if the prestige of the criticizer was high -- nevertheless, practically speaking, it would be difficult for me to evaluate the prestige of the critic" "Peer review gives you an idea of whether this is a scientific idea that has been debated in the field -- it would be important to have the experts describe and debate the peer-reviewed literature to determine the acceptability of the evidence"
The Evaluation of Research Papers Scientific evaluation is made on the basis of criteria established by the journal to which a research paper is submitted. Typically, the more prestigious the journal, the more rigorous the evaluation, and the more difficult it is to get a paper published in that journal. While some journals may accept almost all manuscripts they receive, others may reject 90% or more. Especially high rejection rates develop as a consequence of a journal's high prestige. That is, the most prestigious journals receive many more papers than they can publish, thus allowing the journal to set even higher standards for acceptance.	Critiques of the Peer Review Process -- Commentators have recently criticized the peer review process on several grounds: it lacks explicit standards for review; it lacks objectivity and fairness in the selection of manuscripts for review; it favors conservative, traditional research and excludes innovative work by independent researchers; editors are biased in their selection of reviewers, sending manuscripts only to those individuals who share the opinions of the editorial board; and reviewers may not spend enough time critically reviewing the methods and procedures of the research.
Selection of Referees Managing editors initially evaluate submissions and select the individuals who will act as referees or peer reviewers. Peer reviewers are usually selected on the basis of their own research experience in the field and their expertise with the methodologies or statistical procedures used. Any given manuscript is typically reviewed by more than one referee. Depending on the scientific discipline and the journal, referees may or may not know the identity and professional affiliation of the authors. When the identity of the author or authors is not known to the reviewer, the process is referred to as a blind review. The identity of the referees, however, is not disclosed to the authors. After the referees return their reviews of the manuscripts, the managing editor or an editorial board decides whether the comments are reasonable. A preliminary decision regarding suitability for publication is sent to the authors, accompanied by the referees' comments. The author(s) may or may not respond to the comments. Depending on the nature and scope of the referees' comments, the author(s) may address the reviewers comments and resubmit the paper for a second review.
Refereed vs. Unrefereed Journals Articles published in refereed journals are typically given more weight and afforded more status than articles published in unrefereed journals -- that is, in journals that do not send out submitted articles for independent peer review.	Refereed Journals: articles submitted for publication are sent for independent peer review Unrefereed Journals: articles submitted for publication are not independently peer reviewed
Determining the Status of a Journal High status journals are generally considered to be those journals that have a high rejection rate for submitted manuscripts (i.e., the journal only accepts for publication a very small percentage of the number of manuscripts submitted for review). Other factors taken into account in determining the prestige of a journal include: its prominence in the relevant field; its circulation; who subscribes to the journal (e.g., researchers, practitioners, legal professionals); the target audience; and the editorial board membership. It is important to recognize that while authors may count a journal's reputation more than any other factor in deciding where to submit an article, the appearance of an article in a lower-status journal does not necessarily mean that the article was rejected by a higher-status journal. There are a variety of legitimate reasons why an author may choose to submit an article to a lower-status journal. For example, the author may have decided to submit an article to a journal with higher acceptance rates and quicker response times in an effort to ensure that the research reaches the relevant community in a relatively timely manner. In addition, the higher-status journal may have recently published an article on a similar topic and would be likely to reject a similar submission on that ground alone. Also, many journals are becoming increasingly specialized and target increasingly specialized audiences. Thus, while a journal may not be defined as prestigious in terms of a high circulation rate, it many be considered a high status, prestigious journal within its specialized, narrow field. The prestige of books is often measured by who published them. Typically, books published by "prestigious" university presses are counted more heavily than books published by other publishing companies. However, many of the criticisms leveled at journal publications also apply to status determinations about books.	Typical Questions Addressed by Referees When Evaluating a Manuscript for Publication Is the subject matter appropriate for readers of the journal? Different journals cover different subjects and target different audiences. The referee must determine whether the work described in the paper is consistent with the stated intent of the journal. Are the methods used suitable for the questions under study? Are the data analyses convincing? Do the statistical analyses support the conclusions drawn? Are there other possible explanations for the findings? The referee must determine whether the research design was appropriate, whether appropriate controls were used, whether the data were presented adequately, and whether the interpretation of the results was reasonable. The referee must consider whether the methods and analysis plan were presented clearly enough, and in enough detail, and whether or not the research could be replicated. Does the paper make an important contribution to the field? Will the work be of broad-reaching working in the field? Is the paper in the top 1%, 5%, 10%, etc. of papers written in this field or for this journal? The referee's evaluation may go beyond the adequacy of the work and also consider the potential impact of the research on the relevant field or discipline. More prestigious journals want to publish research that is most likely to have a significant impact on the field. The referee must evaluate the impact of the research within the context of the relevant field -- the current problems and questions of interest, currently accepted methodologies, and so forth.
Informal Peer Review The process of publishing a research paper can often take six months to a year between submission of a paper and its appearance in a journal, although there is great variation among different scientific disciplines. If investigators want to make their research known to the relevant scientific community, they must find a means of reporting the work in a more preliminary form. Preliminary research reports are often presented at meetings of professional societies or associations. In most cases, researchers request the opportunity to make a presentation at a meeting. By communicating the major conclusions of the study, the work becomes known by the relevant community, and the researcher establishes a claim of ownership and priority over the research. When a researcher is invited by the organizers of a professional society to present her current work at a meeting or conference, it is an indication that the researcher's work is expected to be of interest to the professional audience and that the researcher is an authority whose recent work is probably reflective of the current thinking in the relevant field. Unlike formal research papers, oral presentations are reviewed minimally or not at all and the details of the methods and materials are generally not presented. Other researchers can learn the thought style behind the work and the general methodologies utilized, but not the precise methods by which the research was accomplished. [W]hile peer review improves the overall quality of research, as well as the likely integrity of individual publications, it does not mechanically guarantee reliability. Courts therefore should neither be wholly dismissive of peer review nor accord it more respect than it deserves.(3)
Peer Review in the Computer Age The underlying assumptions and procedures of peer review are receiving increasing attention within scientific disciplines. One development giving urgency to this particular topic is the revolution taking place in scholarly communication. As technology advances it becomes possible to make every work available electronically to all scholars, obviating the need for printed volumes and potentially eliminating the peer review process. This eliminates practical constraints that until now have made it impossible for journals to publish huge amounts of material and for libraries to acquire and store them. As a consequence of advances in technology, much more can be "published" without long delays and backlogs. This increased capacity for communication is also accompanied by relaxed standards of peer review (most electronic journals are not refereed).
General Acceptance [G]eneral acceptance can yet have a bearing on the inquiry. A reliability assessment does not require, although it does permit, explicit identification of a relevant scientific community and an express determination of a particular degree of acceptance within that community. Widespread acceptance can be an important factor in ruling particular evidence admissible and a known technique which has been able to attract only minimal support within the community may properly be viewed with skepticism. ... The focus, of course, must be solely on principles and methodology, not on the conclusions they generate.(4) Just when a scientific principle or discovery crosses the line between the experimental and demonstrable tests is difficult to define. Somewhere in this twilight zone the evidential force of the principle must be recognized, and while courts go along way in admitting expert testimony deduced from a well-recognized scientific principle or discovery, the thing from which the deduction is made must be sufficiently established to have gained general acceptance in the particular field in which it belongs.(5)	Things to consider when determining the status of a scientific journal: whether or not articles published in the journal are peer-reviewed; rejection rate of submitted manuscripts level of prominence in the relevant field; circulation record; subscribers target audience; and editorial board membership Questions to consider when evaluating scientific evidence ... In what ways has the research been peer-reviewed? Was the research published in a referred or unreferred journal? If the research was published in a referred journal, was it a blind review? What is the relative status and reputation of the journal in which the research was presented? Was the disciplinary focus of the journal appropriate for the research?
CRITICAL QUESTIONS REVIEWED In what ways has the research been peer-reviewed? Was the research published in a referred or unreferred journal? If the research was published in a referred journal, was it a blind review? What is the relative status and reputation of the journal in which the research was presented? Was the disciplinary focus of the journal appropriate for the research? Are the theories or principles underlying the research generally accepted in the relevant scientific field? If the theory or principles underlying the research are not generally accepted, how are they defended? What is the relevant scientific field? Are the methods employed in the research generally accepted in the relevant scientific field? Are there relevant but unidentified scientific disciplines that may have a different perspective on the issue? If the methods employed are not generally accepted, how is their use defended?
To what extent do judges around the country find the concept of general acceptance a useful criterion for critically evaluating scientific evidence? All of the judges (N=400) were asked how useful the concept of general acceptance is when determining the admissibility of scientific evidence. The vast majority of judges indicated that general acceptance was a useful criterion, with 63% rating it as "very useful" and 33% rating it as "somewhat useful." When judges in FRE/ Daubert states (n=205) were examined, the results were much the same, with 61% reporting that general acceptance was "very useful," 36% reporting that it was "somewhat useful," and 2% reporting that general acceptance was "not at all useful" to admissibility decision-making. When asked a question about how they might apply the concept of general acceptance to a determination of the admissibility of proffered evidence, the majority (82%) of all of the judges surveyed (N=400) provided responses that demonstrated a clear understanding of the concept of general acceptance. In 17% of responses, understanding of general acceptance was questionable. It was clear that the judge did not understand the scientific concept of general acceptance in only 1% of responses.
Endnotes 1. Daubert vs. Merrell Dow Pharmaceuticals 509 U.S. 579, 113 S.Ct., at 2786. 2. Ibid. 3. Sheila Jasanoff (1997). Judging Science: Issues, Assumptions, Models. 4. Supra note 1, at 2797. 5. Frye, at 1014.	Stop and reflect ... How might new or novel theories or methods defend themselves against a lack of general acceptance? Does a lack of general acceptance in the relevant scientific community necessarily draw into question the validity and reliability of the research principles and methods? Why or why not?
Glossary blind review -- when identity of the author is not known by the reviewer peer review -- the process by which scientific papers are submitted to journals for publication are reviewed by peers in the field; peer reviewers typically evaluate the importance and usefulness of the research and assess whether the research was carried out in a methodologically appropriate manner and supports the conclusions drawn referred journals -- articles submitted for publication are sent for independent peer review unreferred journals -- articles submitted for publication are not independently peer reviewed
Suggested Readings Bell, R. (1992). "Examining the Peer Review Process." In Impure Science: Fraud, Compromise and Political Influence in Scientific Research. New York: Wiley. DuRant, R.H. (1994). "Checklist for Evaluation of Research Articles." Journal of Adolescent Health, Vol. 15, pgs. 4-8. Imwinkelried, E.J. (1997). "Frye's General Acceptance Test v. Daubert's Empirical Validation Standard - 'Either ... Or' or 'Both ... And?" Criminal Law Bulletin, Vol. 33, pg. 72. Lock, S. and Smith, D. (1990). "What do Peer Reviewers Do?" JAMA, Vol. 263, No. 10 (9 March), pgs. 1341-1343.
--- CHAPTER 7 --- FRONT PAGE --- CHAPTER 9 ---