Judge's Deskbook -- Chapter 7

CHAPTER 7
of A Judge's Deskbook on the Basic Philosopies and Methods of Science,
by Shirley A. Dobbin, Ph.D, and Sophia I. Gatowski, Ph.D
An Introduction to Qualitative Methods

Research based on qualitative methods may appear in a variety of court cases as proffered evidence. For example, a sociologist might offer testimony about the experiences of members of new religious movements (often referred to as "cults") or gang members based upon his participant observations of these groups. A social worker or social anthropologist might offer testimony in a child protection case based on field studies he conducted with different cultural groups about child-rearing practices (e.g., different culturally derived definitions of appropriate discipline or differences in the acceptable ages for children to contribute to the family income). The list of possible examples of how qualitative research might appear in court is endless.
Qualitative research is grounded in a philosophical tradition that is broadly interpretivist; the focus is on how the social world is interpreted, understood, experienced, or produced. It is based upon methods of data generation and collection which are flexible and sensitive to the social context within which the data are produced and on methods of analysis and explanation building which involve understandings of complexity, detail, and context.⁽¹⁾
Qualitative research ... is about "ways of seeing." The goal of the research, whatever the methodology, is understanding gained through a process of discovery. What is expressed in qualitative research is a process of discovery that asserts particular assumptions of how knowledge is perceived and acquired. Qualitative research generally focuses on complex human social interactions.⁽²⁾

Learning Objectives for Chapter 7

Upon completion of this chapter, the reader should be able to:

Understand the differences between quantitative and qualitative research;

Discuss when qualitative methods might be the most appropriate methods and why;

Understand the general method of qualitative research;

Understand both the limits and power of qualitative methods; and

Critically evaluate research utilizing qualitative methodologies.

Developing a Plan

A well developed plan for a qualitative study addresses data collection, analysis, and report-writing. A plan includes: a well articulated definition of the research design; the unit of analysis used in the design (i.e., what will be studied -- A specific individual? A specific group of people? A particular situation?); specification of the various data sources; an outline of data analysis procedures; formats for reporting the information; and any other special characteristics of the design.

Qualitative research should be:

systematically and rigorously conducted;

strategically conducted, yet flexible and contextual;

involve self-scrutiny by the researcher; active reflexivity;

produce social explanations that are generalizable in some form.

Acknowledging and Identifying the Role of the Researcher

Qualitative research is interpretative research. As such, the biases, values, and judgment of the researcher must be explicitly acknowledged. A discussion of the researcher's role should include statements about past experiences of the researcher that provide familiarity with the topic, setting, or the informants; his attitudes toward the research problem; why he is interested in the problem; and the like. The steps taken by the researcher to gain access to the setting and to secure permission to study the informants or situation must also be specified.

Qualitative research should not be seen as necessarily in opposition to, and uncomplementary with, quantitative research.

Qualitative Data Collection
Observations (as an observer or as a participant)
Unobtrusive observer: the researcher tries to avoid responding in any way to the subject who is under observation
Participant observer: the researcher becomes part of the situation and, to a greater or lesser degree, contributes to the situation
Advantages: researcher has first-hand experience with informant; researcher can record information as it is observed; unusual aspects can be noticed and recorded immediately; useful in exploring topics that may be uncomfortable for informants
Disadvantages: may be seen as intrusive; private information may be observed that researcher cannot record; researcher may not have good observation skills; certain informants (e.g., children) may present problems with building rapport

Interviews (face-to-face, telephone, mail, or focus (delphi) groups)
Advantages: useful when informants cannot be directly observed; informants can provide historical information; allows researchers a degree of "control" over the line of questioning
Disadvantages: provides "indirect" information filtered through the view of those interviewed; information is removed from the immediate natural setting

Documents (public or private)
Advantages: researcher can obtain the language of informant; more time convenient; represents data that are thoughtful to the extent that informants have given attention to compiling the information (e.g., personal journal)
Disadvantages: may be protected information unavailable to the researcher; materials may be incomplete; documents may not be authentic or accurate

Visual Materials (photographs, videotapes, art objects, film)
Advantages: may be unobtrusive; provides an opportunity for informant to share directly his or her "reality"

Disadvantages: may be difficult to interpret; may not be accessible publicly or privately

Specification of Data Collection Procedures

Data collection steps involve:

Setting the Boundaries for the Study;

Collecting Data; and

Establishing the Protocol for Recording Data.

Reflecting on the Assumptions of Qualitative Research

Qualitative researchers

are concerned primarily with process, rather than outcomes or products;

are interested in meaning - how people make sense of their lives

and experiences;

qualitative research involves fieldwork - the researcher physically goes to the people, setting, site, or institute to observe or record behavior in natural settings;

qualitative research is descriptive

qualitative research is inductive - the researcher builds abstractions, concepts, hypotheses, and theories from details

Adapted from Creswell (1994) . Research Design: Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage Publications.

i. Setting the Boundaries
Four general study parameters should be considered:

the setting: where did the research take place?

the actors: who will be observed or interviewed?

the events: what were the actors interviewed about or observed doing?

the process: what was the evolving nature of events undertaken by the actors within the setting?

ii. Collecting Data
Data collection procedures differ considerably in how the behaviors or observations of interest are recorded and coded. At one end of the continuum are methods that are relatively unstructured and open-ended. On the other end of the continuum are more structured and pre-defined methods, which itemize, count, and categorize behavior.
iii. Establishing Protocols for Recording Data
Standard forms are often used for observing, collecting, and organizing information. Data protocols can vary in the extent to which they provide structured events to watch for, specific questions to ask, and particular information to uncover.

Data Analysis

In qualitative research, data analysis often occurs simultaneously with data collection, data interpretation, and narrative report-writing. In this respect, qualitative analysis clearly differs from the quantitative approach of engaging in the separate activities of data collection, analysis, and report-writing. In qualitative analysis several simultaneous activities engage the attention of the researcher: collecting information from the field; sorting information into categories; formatting the information into a story or picture of the event; and actually writing the qualitative narrative report.
In qualitative research, the researcher takes a voluminous amount of information and reduces it into meaningful categories, patterns, or themes and then interprets the information. Flexible rules govern how one goes about sorting through interview transcripts, observational notes, documents, and visual material. However, the researcher generally forms categories of information and attaches codes to these categories. These categories and codes form the basis of the emerging story to be told by the qualitative researcher. Although the rules of this process are flexible, the process should be systematic and well articulated as part of the qualitative plan and final report. While much of the work of analysis is taking apart and organizing information, the final goal is the emergence of a larger, consolidated picture of some social reality.

Reliability and Validity

The internal validity of qualitative results can be assessed in a number of ways. For example:

through triangulation - that is, by assessing the amount of convergence (or agreement) among different sources of information, different investigators, or different methods of data collection;

through feedback from informants (subjects) to assess whether interpretations by the researcher, and conclusions drawn, accurately reflect the perspectives of the informants; and

through involvement of key informants in the interpretation and development of results.

In Chapter 5, the importance of a well articulated statement of the problem and research design was discussed. Although problem statements and their ultimate development into one or more research hypotheses are most formalized and best developed in experimental research, the articulation of a well developed research problem is important in all types of research, both quantitative and qualitative.

It is important to recognize that researchers conducting quantitative, experimental, or quasi-experimental research are subject to many of the same biases as qualitative researchers. However, in qualitative research, personal biases, values, attitudes, and so forth are explicitly acknowledged and, to a greater or lesser degree, reflected in the research.

Combining Quantitative and Qualitative Approaches
Often researchers will combine methodologies from both the quantitative and qualitative research approaches. For example, qualitative research is often done in the initial exploratory phases of research. Once tentative hypotheses emerge from the initial qualitative research, they are tested through experimental or quasi-experimental designs. Qualitative research may also be conducted at the end of a research project in an attempt to determine how well experimental findings generalize to field settings. When quantitative and qualitative approaches are combined, each phase of the research should be evaluated according to the standard practices and agreed upon principles of the particular approach.

The intent of qualitative research is not to generalize findings, but rather to form a unique interpretation of events for a given group of individuals or institutions, within a given context, at a particular point in time. Like the issue of generalizability, the uniqueness of the study within a particular context mitigates against replicating the research exactly in another context. However, the inclusion of statements about the researcher's position, the central assumptions of the study, the method by which informants are selected, the biases and values of the researcher, all enhance the ability to replicate the research in other settings.

Questions to consider when evaluating qualitative evidence ...

Was the basic type of qualitative design adequately specified in enough detail?

Was the research design consistent with the underlying assumptions of qualitative research?

Was the researcher explicit about her background, values, experiences, and so forth, and why this particular research issue and research method were selected?

Was a detailed description provided of the steps taken to gain entry and approval for data collection at the research site?

Were the viewpoints of all participants considered?

Were the procedures for collecting data adequately discussed?

Was the analysis plan adequately discussed?

Was there an appropriate level of inference regarding the generalizability of the results?

CRITICAL QUESTIONS REVIEWED

Was the basic type of qualitative design adequately specified in enough detail?

Was the research design consistent with the underlying assumptions of qualitative research?

Was the researcher explicit about his background, values, experiences, and so forth, and why this particular research issue and research method were selected?

Was a detailed description provided of the steps taken to gain entry and approval for data collection at the research site?

Were the viewpoints of all participants considered?

Were the procedures for collecting data adequately discussed?

Was the analysis plan adequately discussed?

Was there an appropriate level of inference regarding the generalizability of the results?

ENDNOTES
1. Mason, J. (1996). Qualitative Researching. Sage Publications, pg. 4
2. McCombs, G.M.Y., Maylone, T.M. (1998). "Qualitative Research." Library Trends, Vol. 46(4), Introduction.

GLOSSARY
data protocols standard forms developed to assist in observations and help to organize data systematically
participant observation the researcher becomes part of the situation being observed and, to a greater or lesser extent, contributes to the situation
qualitative research research grounded in a philosophical tradition that is broadly interpretivist; the focus is on how the social world is interpreted, understood, experienced, or produced; context sensitive methods of data collection are flexible
triangulation the convergence of difference sources of information, different investigators, or different modes of data collection; type of validity check
unobtrusive observation the researcher avoids becoming involved in the situation or event being observed; the researcher tries to avoid responding in any way to the subject who is under observation

SUGGESTED READINGS
Creswell, J.W. (1994). Research Design: Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage Publications.
Denzin, N.K. and Lincoln, Y.S. (Eds.) (1994). Handbook of Qualitative Research. Thousand Oaks: Sage Publications.
Kirk, J. and Miller, M.L. (1986). Reliability and Validity in Qualitative Research. Beverly Hills, CA: Sage Publications.
Mason, J. (1996). Qualitative Research. Thousand Oaks: Sage Publications.
Robinson, C. (1993). Real World Research: A Resource for Social Scientists and Practitioner-Researchers. Oxford: Blackwell.

--- CHAPTER 6 --- FRONT PAGE --- CHAPTER 8 ---

CHAPTER 7 of A Judge's Deskbook on the Basic Philosopies and Methods of Science, by Shirley A. Dobbin, Ph.D, and Sophia I. Gatowski, Ph.D An Introduction to Qualitative Methods
Research based on qualitative methods may appear in a variety of court cases as proffered evidence. For example, a sociologist might offer testimony about the experiences of members of new religious movements (often referred to as "cults") or gang members based upon his participant observations of these groups. A social worker or social anthropologist might offer testimony in a child protection case based on field studies he conducted with different cultural groups about child-rearing practices (e.g., different culturally derived definitions of appropriate discipline or differences in the acceptable ages for children to contribute to the family income). The list of possible examples of how qualitative research might appear in court is endless. Qualitative research is grounded in a philosophical tradition that is broadly interpretivist; the focus is on how the social world is interpreted, understood, experienced, or produced. It is based upon methods of data generation and collection which are flexible and sensitive to the social context within which the data are produced and on methods of analysis and explanation building which involve understandings of complexity, detail, and context.⁽¹⁾ Qualitative research ... is about "ways of seeing." The goal of the research, whatever the methodology, is understanding gained through a process of discovery. What is expressed in qualitative research is a process of discovery that asserts particular assumptions of how knowledge is perceived and acquired. Qualitative research generally focuses on complex human social interactions.⁽²⁾	Learning Objectives for Chapter 7 Upon completion of this chapter, the reader should be able to: Understand the differences between quantitative and qualitative research; Discuss when qualitative methods might be the most appropriate methods and why; Understand the general method of qualitative research; Understand both the limits and power of qualitative methods; and Critically evaluate research utilizing qualitative methodologies.
Developing a Plan A well developed plan for a qualitative study addresses data collection, analysis, and report-writing. A plan includes: a well articulated definition of the research design; the unit of analysis used in the design (i.e., what will be studied -- A specific individual? A specific group of people? A particular situation?); specification of the various data sources; an outline of data analysis procedures; formats for reporting the information; and any other special characteristics of the design.	Qualitative research should be: systematically and rigorously conducted; strategically conducted, yet flexible and contextual; involve self-scrutiny by the researcher; active reflexivity; produce social explanations that are generalizable in some form.
Acknowledging and Identifying the Role of the Researcher Qualitative research is interpretative research. As such, the biases, values, and judgment of the researcher must be explicitly acknowledged. A discussion of the researcher's role should include statements about past experiences of the researcher that provide familiarity with the topic, setting, or the informants; his attitudes toward the research problem; why he is interested in the problem; and the like. The steps taken by the researcher to gain access to the setting and to secure permission to study the informants or situation must also be specified.	Qualitative research should not be seen as necessarily in opposition to, and uncomplementary with, quantitative research.

Qualitative Data Collection Observations (as an observer or as a participant) Unobtrusive observer: the researcher tries to avoid responding in any way to the subject who is under observation Participant observer: the researcher becomes part of the situation and, to a greater or lesser degree, contributes to the situation Advantages: researcher has first-hand experience with informant; researcher can record information as it is observed; unusual aspects can be noticed and recorded immediately; useful in exploring topics that may be uncomfortable for informants Disadvantages: may be seen as intrusive; private information may be observed that researcher cannot record; researcher may not have good observation skills; certain informants (e.g., children) may present problems with building rapport Interviews (face-to-face, telephone, mail, or focus (delphi) groups) Advantages: useful when informants cannot be directly observed; informants can provide historical information; allows researchers a degree of "control" over the line of questioning Disadvantages: provides "indirect" information filtered through the view of those interviewed; information is removed from the immediate natural setting Documents (public or private) Advantages: researcher can obtain the language of informant; more time convenient; represents data that are thoughtful to the extent that informants have given attention to compiling the information (e.g., personal journal) Disadvantages: may be protected information unavailable to the researcher; materials may be incomplete; documents may not be authentic or accurate Visual Materials (photographs, videotapes, art objects, film) Advantages: may be unobtrusive; provides an opportunity for informant to share directly his or her "reality" Disadvantages: may be difficult to interpret; may not be accessible publicly or privately

Specification of Data Collection Procedures Data collection steps involve: Setting the Boundaries for the Study; Collecting Data; and Establishing the Protocol for Recording Data.	Reflecting on the Assumptions of Qualitative Research Qualitative researchers are concerned primarily with process, rather than outcomes or products; are interested in meaning - how people make sense of their lives and experiences; qualitative research involves fieldwork - the researcher physically goes to the people, setting, site, or institute to observe or record behavior in natural settings; qualitative research is descriptive qualitative research is inductive - the researcher builds abstractions, concepts, hypotheses, and theories from details Adapted from Creswell (1994) . Research Design: Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage Publications.
i. Setting the Boundaries Four general study parameters should be considered: the setting: where did the research take place? the actors: who will be observed or interviewed? the events: what were the actors interviewed about or observed doing? the process: what was the evolving nature of events undertaken by the actors within the setting? ii. Collecting Data Data collection procedures differ considerably in how the behaviors or observations of interest are recorded and coded. At one end of the continuum are methods that are relatively unstructured and open-ended. On the other end of the continuum are more structured and pre-defined methods, which itemize, count, and categorize behavior. iii. Establishing Protocols for Recording Data Standard forms are often used for observing, collecting, and organizing information. Data protocols can vary in the extent to which they provide structured events to watch for, specific questions to ask, and particular information to uncover.
Data Analysis In qualitative research, data analysis often occurs simultaneously with data collection, data interpretation, and narrative report-writing. In this respect, qualitative analysis clearly differs from the quantitative approach of engaging in the separate activities of data collection, analysis, and report-writing. In qualitative analysis several simultaneous activities engage the attention of the researcher: collecting information from the field; sorting information into categories; formatting the information into a story or picture of the event; and actually writing the qualitative narrative report. In qualitative research, the researcher takes a voluminous amount of information and reduces it into meaningful categories, patterns, or themes and then interprets the information. Flexible rules govern how one goes about sorting through interview transcripts, observational notes, documents, and visual material. However, the researcher generally forms categories of information and attaches codes to these categories. These categories and codes form the basis of the emerging story to be told by the qualitative researcher. Although the rules of this process are flexible, the process should be systematic and well articulated as part of the qualitative plan and final report. While much of the work of analysis is taking apart and organizing information, the final goal is the emergence of a larger, consolidated picture of some social reality. Reliability and Validity The internal validity of qualitative results can be assessed in a number of ways. For example: through triangulation - that is, by assessing the amount of convergence (or agreement) among different sources of information, different investigators, or different methods of data collection; through feedback from informants (subjects) to assess whether interpretations by the researcher, and conclusions drawn, accurately reflect the perspectives of the informants; and through involvement of key informants in the interpretation and development of results.	In Chapter 5, the importance of a well articulated statement of the problem and research design was discussed. Although problem statements and their ultimate development into one or more research hypotheses are most formalized and best developed in experimental research, the articulation of a well developed research problem is important in all types of research, both quantitative and qualitative. It is important to recognize that researchers conducting quantitative, experimental, or quasi-experimental research are subject to many of the same biases as qualitative researchers. However, in qualitative research, personal biases, values, attitudes, and so forth are explicitly acknowledged and, to a greater or lesser degree, reflected in the research. Combining Quantitative and Qualitative Approaches Often researchers will combine methodologies from both the quantitative and qualitative research approaches. For example, qualitative research is often done in the initial exploratory phases of research. Once tentative hypotheses emerge from the initial qualitative research, they are tested through experimental or quasi-experimental designs. Qualitative research may also be conducted at the end of a research project in an attempt to determine how well experimental findings generalize to field settings. When quantitative and qualitative approaches are combined, each phase of the research should be evaluated according to the standard practices and agreed upon principles of the particular approach.
The intent of qualitative research is not to generalize findings, but rather to form a unique interpretation of events for a given group of individuals or institutions, within a given context, at a particular point in time. Like the issue of generalizability, the uniqueness of the study within a particular context mitigates against replicating the research exactly in another context. However, the inclusion of statements about the researcher's position, the central assumptions of the study, the method by which informants are selected, the biases and values of the researcher, all enhance the ability to replicate the research in other settings.

Questions to consider when evaluating qualitative evidence ... Was the basic type of qualitative design adequately specified in enough detail? Was the research design consistent with the underlying assumptions of qualitative research? Was the researcher explicit about her background, values, experiences, and so forth, and why this particular research issue and research method were selected? Was a detailed description provided of the steps taken to gain entry and approval for data collection at the research site? Were the viewpoints of all participants considered? Were the procedures for collecting data adequately discussed? Was the analysis plan adequately discussed? Was there an appropriate level of inference regarding the generalizability of the results?

CRITICAL QUESTIONS REVIEWED Was the basic type of qualitative design adequately specified in enough detail? Was the research design consistent with the underlying assumptions of qualitative research? Was the researcher explicit about his background, values, experiences, and so forth, and why this particular research issue and research method were selected? Was a detailed description provided of the steps taken to gain entry and approval for data collection at the research site? Were the viewpoints of all participants considered? Were the procedures for collecting data adequately discussed? Was the analysis plan adequately discussed? Was there an appropriate level of inference regarding the generalizability of the results?
ENDNOTES 1. Mason, J. (1996). Qualitative Researching. Sage Publications, pg. 4 2. McCombs, G.M.Y., Maylone, T.M. (1998). "Qualitative Research." Library Trends, Vol. 46(4), Introduction.
GLOSSARY data protocols standard forms developed to assist in observations and help to organize data systematically participant observation the researcher becomes part of the situation being observed and, to a greater or lesser extent, contributes to the situation qualitative research research grounded in a philosophical tradition that is broadly interpretivist; the focus is on how the social world is interpreted, understood, experienced, or produced; context sensitive methods of data collection are flexible triangulation the convergence of difference sources of information, different investigators, or different modes of data collection; type of validity check unobtrusive observation the researcher avoids becoming involved in the situation or event being observed; the researcher tries to avoid responding in any way to the subject who is under observation
SUGGESTED READINGS Creswell, J.W. (1994). Research Design: Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage Publications. Denzin, N.K. and Lincoln, Y.S. (Eds.) (1994). Handbook of Qualitative Research. Thousand Oaks: Sage Publications. Kirk, J. and Miller, M.L. (1986). Reliability and Validity in Qualitative Research. Beverly Hills, CA: Sage Publications. Mason, J. (1996). Qualitative Research. Thousand Oaks: Sage Publications. Robinson, C. (1993). Real World Research: A Resource for Social Scientists and Practitioner-Researchers. Oxford: Blackwell.

--- CHAPTER 6 --- FRONT PAGE --- CHAPTER 8 ---