DESIGNING VIABLE ASSESSMENT PLANS

Evaluating Assessment Strategies

In this section we examine and evaluate the value of each of the major assessment categories. We conclude the discussion of each strategy with specific recommendations to maximize the utility of that approach.

CLASSROOM/COURSE DATA


NATURE OF CATEGORY:

This collection of assessment strategies involve methods that instructors have traditionally used to judge classroom performance (e.g., essay and objective testing) as well as approaches that reflect more recent attention to assessment-driven teaching-learning processes. These include embedded assessment strategies in which departments identify specify classes in which to embed assessments that are endorsed and designed by the department as well as classroom assessment techniques articulated by Cross and Angelo (1993).

OVERALL ANALYSIS

Advantages:
+maximizes faculty autonomy and investment in student learning
+facilitates prompt feedback
+can provide immediate feedback to faculty about teaching effectiveness

Disadvantages:
-limited by pedagogical constraints of instructor
-can produce unreliable evaluation results
-results affected by instructor/departmental evaluation bias
-generally can promote disconnected course experiences

Recommendations:
Faculty who are new to accountability mandates often protest that other kinds of assessment activity are unnecessary. They advocate course grades as a meaningful index of student learning. Grades that reflect classroom performance do constitute one important source of data about student learning. However, most accrediting agencies recognize that solely relying on grades is not adequate evidence of learning quality. Responsible assessment plans will include strategies that make developing evidence of quality dependent on measures of particular target behaviors, rather than on more global measures such as grades.

OBJECTIVE TESTS
(multiple choice, true-false, fill-in-the-blank items)

Advantages:
+ displays good psychometric properties
+ facilitates rapid feedback through ease of scoring
+ develops norms
+ inexpensive
+ comprehensive
+ improves test validity through item analysis
+ facilitates differential group scoring

Disadvantages:
- usually involves testing low level knowledge
- constructing high quality test questions difficult
- question banks are often of poor quality
- can be compromised by student test banks that may foster differential access

Recommendations:
Although constructing solid objective tests that tap deeper levels is not impossible, it is challenging. Instructors need to help students understand how objective testing can be designed to go after different levels of knowledge. Some find that teaching students Bloom's taxonomy as an organizer that faculty might intuitively use to create more targeted challenges will help students understand questions as challenging rather than picky.

ESSAY TESTS

Advantages:
+ showcases deeper learning, higher order thought processes
+ requires transfer, integration of learning from other sources
+ can include applications or problem-based learning
+ develops writing skills and critical thinking
+ cheap and easy to administer
+ faster to construct than objective tests

Disadvantages:
- questionable psychometric properties
- may disadvantage ESL, students with poor writing or thinking skills
- takes longer to grade and provide feedback
- produces narrower sample of content knowledge

Recommendations:
Despite the labor intensiveness of essay evaluation, this kind of performance effectively addresses many aspects of what we want students to learn. Critical to defensible evaluation of essays is a well-designed rubric. Instructors can benefit from training to produce reliable feedback for student performance. Careful consideration should also be given to the instructions to clarify performance expectations. Some faculty provide an array of potential essay questions as a study guide, selecting a select number of those questions to comprise the actual exam.


EMBEDDED QUESTIONS AND ASSIGNMENTS

Advantages:
+ saves time since assignments will already be required for the course
+ overcomes faculty resistance due to reduced intrusion of external assessment activity
+ encourages faculty to discuss common course outcomes, goals, & objectives + promotes shared responsibility for agreeing where embedding should occur
+ assessment phobic faculty exhibit greater comfort with embedded designs
+ obligates faculty to have public discussion about their pedagogy
+ limits demand characteristics

DisAdvantages:
- can be time-consuming to coordinate effort
- may be taxing to isolate key aspects of performance
- limits faculty autonomy within the course

Recommendations:
Embedding departmental assessment measures in existing coursework will emphasize a strong relationship between course content and assessment content. Individual faculty autonomy is essentially preserved; however, the faculty must collaborate within the department and be responsible for reporting to department colleagues. That level of obligation may not be standard procedure. The department must also control, store, and protect data, including protection from misinterpretation and misuse by outside sources.


CLASSROOM ASSESSMENT TECHNIQUES
e.g., 1-minute papers, course focus groups, free-writing, etc.

Advantages:
+ promotes experimental attitude in faculty about course design
+ convenience
+ provides immediate feedback to faculty about success
+ vividly demonstrates faculty commitment to student satisfaction

Disadvantages:
- focus on teacher performance
- should be combined with other methods for full picture of student learning
- perceived to sacrifice content coverage for time required to assess
- demand characteristics may compromise validity of results

Recommendations:
Enthusiasts of classroom assessment advocate these techniques as a way of implementing continuous improvement efforts. Careful context-setting will avoid or minimize students making unfavorable judgments that the activities are potentially time-wasting, particularly when faculty share the conclusions drawn from the assessment data with the students and make efforts to address concerns, where appropriate.

INDIVIDUAL PROJECTS/PERFORMANCE ASSESSMENT


NATURE OF CATEGORY:
Individual projects have historically provided students the opportunity to apply their learning in projects that make optimal use of their potential intrinsic interest in the subject matter. The category includes individual writing, speaking, and graphic and poster production. Performance assessment strategies, sometimes also referred to as authentic assessment, are also evaluated in this section.

OVERALL ANALYSIS

Advantages:
+ student-centered design promotes investment, motivation
+ promotes transfer of skills and integration of content
+ clear expression of knowledge base
+ engages active learning
+ encourages time outside of class
+ promotes library use
+ can provide study in depth not possible during allotted class time
+ student benefits directly from experience
+ provides venue for creativity

Disadvantages:
- time consuming and labor intensive to design and execute both for instructor and students
-may use materials wastefully (e.g., making transparencies for one speech)
-narrows content range for which student is responsible
-student variability (ability, motivation) challenges reliability and value of performance
-labor intensive for student
-cost may be prohibitive

Recommendations

The types of projects faculty choose as assessment vehicles will depend, in part, on the expertise the faculty have in evaluating works in various modes. The clear articulation of expectations will be critical to success. Specifying student creativity as a criterion will facilitate efforts that go beyond minimum achievement of criteria. Some products may involve decisions about storage space. For example, student videos may have a limited shelf-life.

WRITTEN PRODUCTS (term papers, lab reports, critiques)

Advantages:
+ facilitates student command of specific area
+ provides practice in critical skill area of writing
Disadvantages:
- challenging to writing-compromised students
- labor-intensive to score and return with timely feedback
- can be plagiarized created time-consuming/strategic confrontation with serious consequences for students who are caught
- instructors can be plagued with consequences of student procrastination

Recommendations:
Many professors design writing projects in stages that promote multiple drafts. Getting feedback in stages may be easier for students to incorporate and easier for faculty to see the impact of their feedback work. Learning disabled, ESL, and other writing challenged students may require additional support. Efficient feedback can be facilitated using rubrics or style sheets. Writing projects should be tailored to the developmental level of the student. For example, beginning courses can employ letters to friends to explain a concept. Formal term papers typically work best in advanced courses. Departments may adopt a style sheet based on APA writing conventions that can help students practice consistent format strategies.

ORAL PRESENTATIONSe.g., debate, role play

Advantages:
+ builds expertise in important communication area of oral expression
+ promotes importance of sharing knowledge
+ enhances oral skills
+ Q & A promotes Òthinking on your feet
+ assists professor to cover course content

Disadvantages:
-may burden students with ESL, speech and language difficulties, speaking anxiety
-time consuming and time-wasting when work quality is bad or boring
-may be hard to grade

Recommendations:
Students understandably resist assignments that require them to speak in classes since public speaking remains one of our most pervasive social phobias. Success in oral presentations will depend on several elements:
=>providing lots of guidance and structure beforehand
=>normalizing speaking discomfort and pointing out that overcoming those fears can happen only through practice
=>specifying and sticking to assigned time limits
=>circumscribing topic areas or requiring topic approval
=>coaching regarding use of support technologies
=>developing appropriate performance criteria

GRAPHIC TEST AND DISPLAYS
(e.g., concept maps, outlines)
Advantages:
+ provides experience in applying and organizing course concepts
+ assists in thinking through organization of information
+ additional grappling with the material enhances recall
+ appeals to visual learners

Disadvantages:
- students have limited practice with displaying graphic skills
- students may not have sufficient experience in interpreting graphics
- technological sophistication will influence production quality
- may waste resources

Recommendations:
Faculty have found some success in asking students to translate lecture input into graphic displays, such as a concept map. These strategies appeal to visual learners who may be able to encode and remember more course content by adopting this strategy.


POSTERS

Advantages:
+ hold students accountable for independent project
+ reduces grading burden compared to writing projects
+ provides opportunity to integrate communication skills (e.g., writing,
graphics, oral defense)
+ can incorporate team effort
+ expert judgment, peer review can be facilitated with criteria
+ simulates typical debut venue for most psychology scholars

Disadvantages:
-may need to make special arrangements for space
- students may invest money in project for one-shot exposure
- lack of aesthetic sense may handicap poster effectiveness
- stronger social interaction skills may produce halo effect in judging quality
- numbers of posters to be judged can create quality pressures on grading
- may not motivate best effort

Recommendations:
Providing models or performance criteria will facilitate better productions. Poster sessions can be scheduled within classes or across classes as a departmental event. Awarding best of show maybe a helpful strategy to enhance motivation among the best students. All-department events can become a public relations resource as well as an opportunity to work with local high school psychology teachers to recruit future students.


STRUCTURAL/SITUATIONAL ASSESSMENTS
(e.g., guided learning, in-baskets, critical situations, etc.)

Advantages:
+ provides realistic testing circumstance
+ reality engages and motivates students
+ promotes transfer of information, application
+ taps complex skills

Disadvantages:
- difficult to construct and measure
- locating designed instruments is challenging
- prone to history/context/age cohort effects
- students may rely on common sense under pressure rather than their knowledge from the course

Recommendations:
The situation should correspond closely to the learning conditions to promote the best transfer of knowledge. Evaluating performance will be facilitated by clearly developed criteria. The quality of the rubric and the training of the evaluators will influence validity. If inter-rater reliability is not high, the results will be of limited value. Rubrics will sometimes not provide for unexpected, creative responses.

SUMMATIVE PERFORMANCE ASSESSMENT


NATURE OF CATEGORY:

Summative assessment strategies tend to be employed for purposes of evaluating program quality rather than primarily to provide developmental feedback to students. This collection of assessment strategies include methods that involve a single episode of data collection (e.g., nationally or locally normed tests) as well as those that incorporate tracking student performance over time (e.g., portfolio, case studies, longitudinal studies. Capstone courses and internships can also be appropriate contexts for summative evaluation.

OVERALL ANALYSIS:

Advantages:
+ promotes coherence in curriculum planning
+ provides feedback loop to improve quality
+ some strategies can be adapted to student interests
+ supports to earlier curriculum recommendations (e.g., St. Mary's conference to provide vehicle for integrating learning)

Disadvantages:
- some options are labor and/or cost intensive
- students may not receive direct feedback regarding their performances, thus limiting their own gains from effort expended
- departments may ignore available data in their planning

Recommendations:
Summative procedures can be invaluable in making the case for the overall quality of programs. Although all of the methods have advantages and drawbacks, the most benefit can be gained to all constituents when students receive direct feedback regarding their summative performance. Finding out relative scores on comprehensive exams or receiving feedback regarding performance over time can assist students with career and life planning in some instances.


STANDARDIZED TESTS

Advantages
+ typically one shot assessment
+ facilitates comparisons over time
+ convenient

Disadvantages
- may not reflect gains or growth across time
- exiting students may not benefit from feedback
- existing instruments may not match to the mission and goals of departments
- expensive
- students may not be motivated to due their best work
- when test occurs may not maximize true learning
- administration may not be flexible
- not student-centered
- limited faculty ownership
- verifying bad performance can be threatening to motivation
- scores may be delayed in return, reducing the impact of feedback
- there may not be a standardized test for the identified content
- can facilitate problematic comparisons to other programs (e.g., comparisons may not take into account differential resources, student characteristics, etc.)

Recommendations:
The disadvantages of the use of standardized tests can be minimized with some additional planning. Embedding the capstone test in an existing course will enhance student motivation since the student may take the experience more seriously. When student performance can also be tied to course grading, maximum motivation to do well is likely. Describing how well the existing test matched the required curriculum will encourage faculty support and student cooperation.


LOCALLY DEVELOPED EXAMS

Advantages:
+ can be tailored to match curricular and program goals
+ standardizes local use
+ relatively inexpensive
+ provides opportunity to develop meaningful local norms
+ avoids specious comparison with other colleges
+ foster coherence in department about their objectives
+ speedy feedback
+ cheaper than national products
+ after initial investment, saves time in the long run
+ may be embedded in specific standard courses

Disadvantages:
- complex, time-consuming to develop
- may impede curricular change since test would need retooling after reforms
- reliance on test bank may not inadequate due to test bank quality
- vulnerable to student theft and distribution
- can be misused by comparing faculty member's areas

Recommendations:
Comprehensive local exams are very time-intensive on the front end; however, the pay-off for this activity is multiple. This strategy encourages strong collaboration across department members and will help department members learn about the academic goals of their colleagues. Security will be an important issue to keep the department test safe from test files that may exist across campus.


CAPSTONE EXPERIENCES

Advantages:
+ fosters aura of importance that may motivate students throughout the curriculum
+ encourages departmental endorsement of culminating experience
+ promotes student responsibility for engaged course
+ supports program coherence for faculty and students
+ course content can be flexible
+ topical design of capstone can engage faculty in planning (e.g., seminar topics can be taught in special interest areas as long as the performance goals meet department expectations)

Disadvantages:
- high stakes performance can be impaired by performance anxiety
- typically low enrollment course is expensive to provide seats for all seniors
- faculty can generate territorial concerns over right to teach capstone
- graduation may depend on successful completion of capstone which can generate some anxiety for faculty and students when performance wobbles late in the course

Recommendations:
Departments can use capstone courses as a unique way to express special interests of the faculty. Departments should secure the support of administration for this expensive option before broad implementation. Typically, capstones tend to have small enrollments to maximize faculty-student interaction. Capstones provide a great opportunity to have the student reflect meaningfully over the course of the curriculum. Putting in place some checkpoints on the process may prevent last-minute difficulties in the capstone that can compromise graduation plans.


INTERNSHIPS/PROFESSIONAL APPLICATIONS

Advantages:
+ popular choice for students
+ provides opportunity to sample future career
+ positive public relations vehicle related to well-prepared students

Disadvantages:
- time intensive for faculty mentors to connect with on-site mentors and coordinate opportunities
- challenging to foster learning experiences across multiple sites
- poorly prepared students create public relations problems

Recommendations:
Departments may reduce the pubic relations strain by screening students for their readiness to represent the program in public contexts. Qualifying criteria that stress quality and quantity of course experience as well as professional expectations in the intern role can set a positive, appropriate tone. Maintaining close contact with on-site mentors can also reduce unsuccessful student performance.


PORTFOLIOS

Advantages:
+ shows sophistication in student performance
+ illustrates longitudinal trends
+ highlight student strengths
+ identify student weaknesses for remediation, if timed properly

Disadvantages:
- collection will be no better than the quality of collected instruments
- time consuming and challenging to evaluate
- space and ownership challenges making evaluation difficult
- content will vary widely with students
- students fail to remember to collect items
- transfer students may not be in position to provide complete portfolio
- time intensive to convert to meaningful data

Recommendations:
Clear expectations about the purpose and collection responsibilities will help students succeed in using the portfolio method. The works that student select will be more satisfying if the students can compare to established criteria. If the faculty want student portfolios to represent student development over time, they will need to be scrupulous about setting forth the performance demand of collecting and examining works throughout the student's career. The success of the portfolio may be enhanced when students reflect on how all the pieces work together to express their learning or meet department criteria.


ASSESSMENT CENTER METHODS
e.g, in-baskets, guided problem-solving

Advantages:
+ complex tasks can enhance student motivation
+ designing relevant authentic assessment practices challenging
+ facilitates integration of diverse skills and content areas

Disadvantages:
- expensive in material preparation and time
- students may not always perceive relevance of assessment to their studies

Recommendation:
Not all disciplines may lend themselves as readily to problem solving situations that seem to be at the center of those challenges.


CASE AND LONGITUDINAL STUDIES

Advantages:
+ can provide rich detail
+ level of attention can build esteem
+ builds allegiance

Disadvantages:
- transfer students may be omitted
- expensive and time-consuming
- small sample limits generalization
- attribution of historical or cohort effects may taint participant reports
- selection for tracking may influence outcome and change student experience

Recommendations:
Departments need to clarify selection criteria if only a sample of students will be tracked. The results simply may not be representative of the group as a whole. Special care must be taken to have a satisfying instrument if results will be compared across cohorts. A department member may need to coordinate this activity if the department commits to this strategy.

SELF-ASSESSMENT


Advantages:
+ multiple modes and variable sophistication possible
+ quality of self-assessment related to quality of content knowledge
+ flexible in format; prompts provided or not
+ might ask about change over time
+ empowers students to practice self-evaluation
+ promotes transfer of accountability to other situations

Disadvantages:
- student judgment may not be accurate
- self-assessment are prone to evaluative biases (e.g., Lake Woebegone Effect, underestimation due to self-esteem issues)
-students have limited experience being held accountable to judge their own work
- students may define assessment as job of teacher
- faculty may perceive this practice to set up more grade conflicts

Recommendations:
Students should receive feedback on the accuracy of their self-evaluations. Early assignments might fare best with more global criteria. For example, "what aspects of your performance were effective?" and "What would you do differently if you had more time?" may engage the student in being reflective. Over time, students should be able to apply more discrete criteria to their own performance, and eventually they should be able to help formulate criteria by which performances should be judge. The quality of self-assessment may be very dependent on the careful construction of the self-assessment prompts.

COLLABORATION


RESEARCH TEAMS & GROUP PROJECTS

Advantages:
+ student-centered designs promote engagement
+ provides opportunity to practice group skills, time management
+ promotes independent work at deeper level
+ breadth of assignments can address content coverage issue
+ simulates how professional activities/achievement transpires
+ produces synergy and excitement around project completion
+ creates a venue to synthesize content bases from multiple courses

Disadvantages:
- students have limited training in group dynamics
- social loafers can tax equitable judgments about grading
- erroneous ideas that are not caught and corrected spread across group members
- challenging to faculty to judge when to redirect or rescue student groups in trouble
- time-consuming

Recommendations:
Selection of the group members will influence group outcomes. For example, some projects will work best when the groups are heterogeneous with regard to student characteristics. Other projects might be most efficient when groups are homogeneous. Students may need assistance in understanding how groups work. Their work will improve with some prompts to pay attention to the process of the group in addition to solving the problem at hand or creating the product. Students will fare best in research teams where they clearly understand group norms and expectations. For example, what are the penalties for nonparticipation? Whenever possible, students should be given feedback on the quality of their participation.


ONLINE ACTIVITIES
(e.g., maintaining print record of interactions in chat room or other internet-based contact)

Advantages:
+ the data already exist as part of regular course
+ records trends in collaborative skill
+ tracks process
+ cheap and convenient
+ demand characteristics may be reduced
+ students have equal opportunity to participate
+ faculty monitoring can be unobtrusive
+ appeals to some students who may have greater difficulty in oral expression
+ provides archive through automatic recording
+ documents feedback for instructor on what has been covered or what is still unclear

Disadvantage:
- content analysis is time-consuming
- privacy issues can be compromised
- students may be handicapped by computer savvy and tech patterns
- faculty need to be computer savvy

Recommendations:
Instructors using online strategies may need to overcome individual differences in using this mode by requiring participation. Circumscribing the content may help to avoid some ethical challenges that result in chat room participation. Students should be informed that their discussions are being monitored for assessment purposes from the outset. This strategy may entail IRB review to confer the best protection. Faculty also need to assess ease of web access for students before making on-line participation a requirement.

INTERVIEWS AND SURVEYS


ATTITUDE MEASUREMENT (General Analysis)

Advantages:
+ easy to administer
+ cheap
+ easy to score
+ quick feedback
+ can be reliable but not valid

Disadvantages:
- validity hinges on good design
- may not be valid
- demand characteristics may distort results
- participants may not have good knowledge about their own attitudes
- participants may demonstrate response bias or dishonesty
- labor intensive to interpret

Recommendations:
Valid attitude measures depend on quality of design and implementation. For example, the participants must be motivated, careful, and candid to generate data that will be meaningful. Care should be exercised to produce to design appropriate measures for intended purposes that minimize sources of error (e.g., selection bias, demand characteristics, literacy challenges, etc.).


SATISFACTION SURVEYS
(Alumni, Employers, Grad School Advisors, Parents, etc.)

Advantages:
+fosters positive public relations because activity signals faculty concern for quality
+targets of survey may be prompted to other positive actions (e.g, donations, hiring, recruitment of new students)
+external judges may be more objective in their appraisal of student abilities, achievements
+recurring insights may point to some problems that need remediation
+provides important perspective on relevance of program to various occupations

Disadvantage
- tracking down and engaging targets may be problematic
- low return rates compromise validity
- some respondents may be motivated not to tell the truth (e.g., don't want to bear bad news, demand characteristics)

Recommendations
Long surveys will influence completion rate. The return rate also provides some indication of how robust the results are. For example, in alumni surveys, the students who are most successful will be more motivated to complete the surveys and may produce an overestimate. When appropriate, a lie scale or some other strategy to verify truthfulness in response will also increase validity. In designing satisfaction instruments, instructors need to think through the quality of education from the perspective of the interview subject. Well-designed surveys are difficult to create so some pilot data may help identify trouble spots in proposed instruments.


PERFORMANCE REVIEWS
(Alumni, Employers, Grad School Advisors)

Advantages:
+ promotes evaluation based on objective appraisal of behavior
+ builds positive public relations
+ external judges may be more objective in their appraisal of student abilities, achievements
+recurring insights may point to some problems that need remediation
+provides important perspective on relevance of program to various occupations

Disadvantage
- tracking down and engaging targets may be problematic
- low return rates compromise validity
-some respondents may be motivated not to tell the truth (e.g., don't want to bear bad news, demand characteristics)

Recommendations:
Departments committed to evaluating their graduate's performance from interested stakeholders are likely to find the time invested to be worthwhile, both in terms of data gathered as well as public relations impact.


EXIT INTERVIEWS

Advantages:
+ provides realistic picture
+ provides catharsis
+ provides in-depth, personal perspective on experience of major
+ can be embedded in existing courses to capture broad range of student experience
+ demonstrates overt department commitment to high quality
+ may promote long-term allegiance among graduating students
+ can generate reinforcing feedback to help departments sustain effectiveness

Disadvantages:
-volunteers may have a negative or a positive agenda that may not be representative, producing a selection bias
-time-consuming to coordinate and evaluate the results
-students may not show up for discussion
- negative discussion may influence formerly neutral students to redefine their experience negatively
- completion challenge
- participants may paint too rosy a picture partially due to timing
- expensive
- results can be influenced by the quality of the interviewer and protocol

Recommendations:
Departments will need to decide on the scale and format of focus exit interviews. These activities can be conducted individually or in small groups. Departments can commit to interviewing every graduating seniors or elect to sample from the group. Instructors need to determine how much credence to place on the results of group discussions with students based on sample size and representation. Questions should target the data that the department wishes to gather. The department should also determine how to interpret the results of the interview. Collaborative design of the interview protocol will promote greater enthusiasm by department members to deal with the consequences of the interview. Conducting the interviews with department faculty may influence student participation since they may be more candid with an external reviewer.


FOCUS GROUPS

Advantages:
+small discussion groups promote engagement
+ can be employed to provide feedback on a class, course, or program
+ participants can benefit directly from changes that result from their feedback
+ demonstrates overt department commitment to high quality
+ can generate reinforcing feedback to help departments sustain effectiveness
+ development of protocol can be involving for faculty
+ may tap unforeseen areas of concern

Disadvantages:
- current students may feel some pressure not be completely candid for fear of retribution
-volunteers may have a negative or a positive agenda that may not be representative
- time-consuming to coordinate and evaluate the results
- students may not show up for discussion

Recommendations:
Departments should develop a good rationale for selecting students for focus group linked to the purpose for which the group is being convened. The discussion protocol can produce both quantitative and qualitative data that can be beneficial to the department. However, student commentary in a focus group may not be representative of the typical student's experience.


FOLLOW-UP ALUMNI INTERVIEWS
(This method involves telephone follow-up to graduates to assess information other than satisfaction with the major. Graduates can be contacted and interviewed on various outcome measures, including knowledge of the major, civic practices, or other indices of interest to the department. Demand characteristics are strong in this strategy.)

Advantages:
+ facilitates spontaneous assessment of studentÕs application of knowledge & skill
+ measures enduring learning and skill transfer
+ scope can be broad-ranging

Disadvantages:
- could be construed as deceptive practice
- might require IRB oversight

Recommendations:
Avoiding demand characteristics is a significant problem with this approach. Alumni may feel compelled to help out by inflating their accomplishments or satisfactions in response to a phone interview.


EXTERNAL EXAMINERS
(Exit interviews conducted by objective, external expert)

Advantages:
+ promotes objective reports where students are assured of anonymity
+ data summary and interpretation conducted external to regular department activities
+ improves face validity of assessment activities
+ supports department courage regarding willingness to expose their practices to outsider

Disadvantages:
- expensive to employ qualified consultant
- sensitive information is at some risk for getting beyond control of department

Recommendations:
Departments may want to involve the external examiner in the construction of the interview protocol to avoid problems of drift toward the examiner's own interests and values in the interview. Qualified external examiners can be identified through the Psychology Department Consulting Bureau operated by the Society for the Teaching of Psychology.


ARCHIVAL MEASURES

TRANSCRIPT ANALYSIS/ANALYSIS OF TRANSFER PATTERNS
(can answer questions about prerequisites, transfer patterns)

Advantages:
+ existing data
+ provides overall picture
+ trends of targeted students at particular times
+ exposes problematic trends for transfer, including drop out rates, time to degree completion, course articulation success, subsequent courses performance

Disadvantages:
- time-consuming
- potentially boring in level of detail required
- may require cooperation to gain access to data

Recommendation:
The analysis of course patterns by itself may not address directly the questions regarding quality. Transcript analysis can answer narrowly focused questions that should be well thought through to justify the time required.


SYLLABUS AUDIT

Advantages:
+ promotes coherence within the department
+ can identify areas of neglect or overemphasis
+ facilitates adoption of similar writing standards and other expectations
+ promotes student understanding of cognitive goals

Disadvantages:
- time-consuming
- may be difficult to engage all department members fully in review/consensus
- students may pay little attention to the syllabus as overall learning guide

Recommendations:
Although this practice is time-consuming, many departments find a syllabus audit is fundamental to answering all kinds of questions about the manner in which the faculty implement the curriculum.


DEMOGRAPHIC DATA ANALYSIS/ALUMNI DATABASE

Advantages:
+ facilitates thorough understanding of student body
+ prepares department for unusual trends that might affect course scheduling
+ predicts where recruitment efforts will pay off
+ points to specific remediation needs
+ identifies potential donors for ongoing program needs

Disadvantages:
- time-consuming
- possible too have too much data

Recommendation:
With careful planning, departments can execute well crafted strategies to collect data that will be useful for their planning in recruitment, retention, and fund-raising.

LIBRARY USE/WEB HITS

Advantages:
+ provides input about how seriously students take assignments
+ allows analysis of trends in use
+ presents overall picture of value

Disadvantages:
- contaminated with faculty use
- interpretation is difficult, boring, and time-consuming
- students may get sources from other than the library

Recommendations
This measure may be most helpful feedback from the library to assist in future ordering. Combining library use and web hit statistics with other measures may provide more meaningful measure.


Webpage Source:

© 2006 American Psychological Association
Education Directorate
750 First Street, NE • Washington, DC • 20002-4242
Phone: 202-336-5970 • TDD/TTY: 202-336-6123
Fax: 202-336-5962•
http://www.apa.org/ed/eval_strategies.html