Metamorphosis

VOLUME 14, NO. 1 FEBRUARY 2000

BEMIDJI STATE UNIVERSITY

CENTER FOR PROFESSIONAL DEVELOPMENT

BEMIDJI, MN 56601-2699

(218) 755-3984

http://www.bemidjistate.edu/cpd

IN THIS ISSUE:

• Using Teaching Evaluation Techniques
• Assessment: Going Beyond What We Need to do for NCA

UPCOMING EVENTS

• Wednesday, March 1 Tips for Documenting Teaching Effectiveness
• Thursday, March 23 Sports for Individuals with Disabilities

1999-2000 CENTER STAFF

• Russ Lee, Director, CPD, Professor & Chair of Psychology
• Pat Riley, Professor of Music
• Kay Robinson, Professor of Speech and Theatre
• Eileen Walsh, Assistant Professor of History
• Rose Weaver, Assistant Professor of English
• Marty Wolf, Associate Professor of Computer Science
• Holly Marie Bethune, Graduate Assistant
• Marti Lundin, Office Specialist

Using Teaching Evaluation Techniques

Russell Lee

Professor & Chair of Psychology, and Director, Center for Professional Development

In thinking about how to document their teaching effectiveness, many professors think only of conducting end of semester formal teaching and course evaluations using a standardized instrument. I think, however, that such an instrument is appropriate only under some condition. This article will discuss the teaching evaluation techniques that I feel are most appropriate to use in different circumstances. You should know the purpose for an evaluation before you will evaluate.

Using Teaching Evaluation Techniques

You wish to evaluate and try to improve your teaching, but are under no external pressure to do so. A tenured full professor are you? I suggest that you use multiple sources of data. Many writers (Centra, 1979; Miller, 1987; Seldin, 1980; Weimer, 1991) support this point. Weimer (1992) suggests that for this purpose you seek a blend of positive and negative feedback. If you try a formal instructor and course evaluation instrument alone and your scores are very high, the instrument will not help you improve very much. Whether you use an instructor and course evaluation instrument or not, try to get specific information rather that general information. Many teachers find it helpful to pass out blank pieces of paper to the class, asking the class to comment on specific areas where the instructor could improve. You might want to do a formal evaluation one semester, the Center for Professional Development's midterm evaluation another semester, and blank pieces of paper another semester. The confidential services of the Center are available to you as well. We can sit in on you classes, talk to your students, or whatever you wish to gather information for you.

You are teaching a new course, or using a new technique, and want specific feedback about it, not for retention or promotion purposes. You might want to use the Center's midterm evaluation for that course, or you might want to design a midterm form of your own that focuses on your particular concerns, including items such as "Did you think that the short story I assigned added to your understanding of depression as a clinical diagnosis?" or "have you found the study guides I prepared useful to you?" Midterm evaluations might suggest possible changes to the course or your teaching of it while there is still time left in the semester to make the changes. Consider using Small Group Instructional Diagnosis, a process whereby a member of the Center staff interviews your class, and summarizes for you what to say. Angelo and Cross (1993) suggest passing out index cards following a new technique. You would ask the students to write a few evaluative words about it, then collect them in an envelope to be analyzed later. They also suggest preparing a small preprinted evaluation form to pass out to evaluate new techniques.

You want to give evidence of your teaching strengths and weaknesses for the purposes of retention or promotion. Again, use multiple sources of data, but in this case I suggest that you avoid the blank pieces of paper. Cashin (1990) strongly states that open-ended comments are useful for improvement, but not for promotion and tenure decisions. Definitely use a formal instructor and course evaluation instrument as one of your sources if you can find one you like. Reviews of the literature on such instruments (i.e., Angelo & Cross, 1993; Centra, 1993; Cashin, 1988) usually conclude that student ratings tend to be quite reliable, fairly valid, relatively unbiased, and, actually, quite useful. I would certainly not recommend that they be used as the sole criterion for a performance decision, but they can allow you to demonstrate your teaching performance in a relatively objective fashion. Make sure that you sample across courses and across time (Gilmore, Kane, & Naccarato, 1978). I mentioned earlier that for purposes of improvement you want specificity in the terms; you use, but for evaluation use a few bread or summative items, such as items 14 (overall instructor) and 15 (overall course) on the Center's instructor and course with student learning than specific items are. Consider creating a teaching portfolio, in which you submit not only particularly clever assignments, original material; you created for a course, and examples of discussion prompts and class activities that you use.

If you choose to use a formal instructor and course evaluation instrument, I suggest that you:

1. Choose an instrument that seems to meet your particular needs, and, ideally, one that can be customized to your particular requirements (Fernald, 1990). Some instruments measure pedagogical techniques that you may not value highly. Of course, some departments encourage the use of particular forms. A common departmental from allows comparison across the department. Although these comparisons can be useful, try not to rank order within your department. Obviously half of your department has to be in the bottom half, even if all are doing very well.
2. Try to get most of your class to evaluate you. If your class is small, try to get all of your students to evaluate you. Otherwise any one student has too much poser in determining your rating. If your class is smaller than 10, formal student evaluations may not be appropriate (Cashin, 1990).
3. Use specific items (1-13 on the Center's form) for improvement, summative items (14 and 15 on the Center's form) for evaluative purposes.
4. If you are using the instrument for evaluative purposes, administer it towards the end of the semester so that the students will have had time to accurately evaluate the class.
5. Try to administer the form on a regular basis so that you have a broad baseline of how you are doing, and so that you can see the impact of changes you make.
6. Consider making some changes in your teaching to bring up lower scales. Boice 1992 found that new faculty defensively attribute lower ratings to their students' inability to handle challenging material, and that they rarely attempt new ways of changing teaching to try to raise lower evaluations.

References

Angelo, T.A., & Cross, K.P. (1993). Classroom assessment techniques: A Handbook for college teachers. San

Francisco, CA: Jossey-Bass, R. (1992). The new faculty member. San Francisco, CA: Jossey-Bass.

Fernald, P.S. (1990). Students' ratings of instruction: Standardized and customized. Teaching of Psychology, 17 (2),

105-109.

Gilmore, G.M., Kane, M.T., & Naccarato, R.W. (1978). The generalizability of student ratings of instruction: Estimation of

the teacher and course components. Journal of Educational Measurement, 15, 1-13.

Cashin, W.E. (1988). Student ratings of teaching: A summary of the research. IDEA Paper No. 20. Manhattan: Kansas

State University. Center for Faculty Evaluation and Development.

Cashin, W.E. (1990). Student ratings of teaching: Recommendations for use. IDEA Paper No. 20. Manhattan: Kansas

State University. Center for Evaluation and Development.

Centra, J.A. (1993). Reflective faculty evaluation: Enhancing teaching and determining faculty effectiveness. San

Francisco, CA: Jossey-Bass

Centra, J.A. (1979). Determining faculty effectiveness: Assessing teaching, research, and service for personnel

Decisions and improvement. San Francisco, CA: Jossey-Bass

Cohen, P.A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection of validity

Studies. Review of Educational Research, 51, 281-309

Miller, R.I. (1987). Evaluating faculty for promotion and tenure. San Francisco, CA: Jossey-Bass

Seldin, P. (1980). Successful faculty evaluation programs: A practical guide to improve faculty performance and

Promotion/tenure decisions. Crugers, NY: Coventry Press.

Weimer, M. (1991). Improving college teaching. San Francisco, CA: Jossey-Bass

What can the Center for Professional Development do

To help you evaluate your teaching?

While we do not write letters of support for faculty, or even keep written records of consultations, you can use consultations with us to improve your teaching. Formally, to support your PDP, you can mention that you have had the Center evaluate your teaching, but yo7u should not disclose the findings of that evaluation. If what you want to do is improve, you need to have your worst stuff looked at: those areas are the ones where you can best improve. If you know that your evaluations will be made public, you are more likely to have only your best stuff looked at, even though you are less likely to improve as much from feedback about your very best teaching. We offer several ongoing services.

Individual Teaching Consultation

Confidential consultations may include our observing and analyzing your classroom teaching, assisting you in identifying skills and competencies expected of students in a particular course or discipline, or helping you identify techniques and approaches appropriate for a discipline. Videotaping is optional.

Course Evaluations

Every quarter the Center makes available both midterm and end of term evaluation forms designed to provide confidential feedback to faculty.

Student Group Instructional Diagnosis

A member of the Center staff can make a 15-minute visit to a regular class session to facilitate the expression of student opinions and suggestions about the course. Consensus views are written down and discussed privately with the instructor.

Help in Documenting Teaching Effectiveness

Help the Center is not involved in the administrative evaluation of faculty and will not provide letters of recommendation for faculty, staff members are available to assist instructors in developing appropriate ways to demonstrate or document teaching effectiveness.

Assessment: Going Beyond What We Need to do for NCA

Laurie Desiderato

Associate Professor of Psychology

Ever get the "What do we have to know for the test?" question from students? For me this question ranks only slightly higher on irritation scale than the other commonly asked question, "did we do anything in class on Friday?" My latest response to this latter question is "Well, in fact we discovered the meaning of life (or the fountain of youth or whatever) and unfortunately you missed it."

What makes both of these questions from students bothersome is their implication that getting a good grade in a class is the paramount goal of education. To be fair, many students do share their faculty members' enthusiasm for discovery, enjoy stimulating discussions, and take pleasure in the challenge of exploring or understanding a complex issue. Too often, however, those goals are eclipsed by the pressures of other demanding classes, competing responsibilities, and too little time, all of which leaves students more grade-oriented that learning-focused.

Similarly, faculty are chronically pressed for time and are not immune to this tendency to focus on short term contingencies at the expense of attending to more meaningful, but less easily acquired long range goals. For instance, in responding to the BSU University Assessment Plan which required departments and programs to develop their own assessment plans related to the dimensions of student learning, many departments initially submitted plans that clearly focused narrowly and exclusively on satisfying North Central Accrediting Association's requirements.

It is true that the demonstration of BSU's participation in a thorough assessment of student learning at the department and program level will be a critical component in the success of our 10-year accreditation review. However, the real potential of assessment rests in our looking beyond this immediate goal of "passing the NCA test." The more important and more difficult challenge of assessment lies in evaluating the strengths and weaknesses of our curriculum reform. To accomplish this more ambitious goal, we must first invest time in identifying the critical skills, attitudes, and information we hope our students will acquire throughout the major, and then develop creative strategies to assess students in these areas. Finally, we need to be willing to listen to and learn from the results of these investigations and use the information to guide improvement.

Over the past three years the University Assessment Committee has reviewed assessment plans submitted by every major and program (except one). Many departments and programs have taken up the challenge and are benefiting from what they have learned through doing assessment and are making changes in their curriculum and classes based on what they learned. The remainder of this article will highlight some of the ways assessment has been used to enhance leaning at BSU.

Improving New Courses Through Assessment

One of the most important, yet difficult, programs to assess is our Liberal Education program. It is not only comprehensive and multi-disciplinary but, while each focus area has very clear objectives, the methods by which those objectives are assessed vary across each course in the area. To address these challenges the Liberal Education Committee has developed an ambitious plan, which assesses liberal education as a whole as well as highlighting a particular Area each year. This year Area I - Written and Oral Communication is highlighted.

Last year, the faculty who developed the People and the Environment course for Area VII - Focus on the Environment participated in an ongoing assessment of this new course by developing a pre-test and post-test with 28 items drawn from the General Sociological Survey which they felt assessed key objectives of the course. Additional information was gathered through student evaluations.

The results of their assessment allowed the faculty to determine which components of the course needed to be strengthened, supported their decision to select an alternate textbook which better matched the goals of the course, and provided ideas about how to enhance the interdisciplinary and participative aspects of the class. In particular, according to Pat Welle, one problem they identified was that after taking the course students indicated a heightened awareness of environmental problems, but retained a pessimistic attitude aobu8t solutions to these problems. After redesigning the course to increase exposure to possible solutions to environmental problems and better integrate the interdisciplinary focus into the course, post test scores and evaluations improved in these areas.

Listening to Students Through Assessment

A number of departments who have established programs in assessment stretching back several years took the opportunity during semester conversion to revise their programs. In particular, the Economics Department has been listening to student input using senior exit interviews and focus groups since 1993. The theme that emerged consistently was that students saw their program as emphasizing graduate school rather than career preparation. While the relatively minor changes in course content that they had attempted seemed to have minimal impact, they did a major overhaul of the curriculum with the switch to semesters that was designed to emphasize preparation for the job market and to make that emphasis more about job opportunities and how to pursue them. One strategy to address this gap in information has been to helping student's transition from school to work.

Strengthening Programs Through Assessment

In Psychology we have used the Major Fields test in Psychology, a standardized test for seniors that allows us to compare our students' performance against other seniors in Psychology majors generally score near the 80^th percentile. More importantly, the results of the test allowed us to compare our Psychology majors and Applied Psychology: Human Services majors in core areas of the discipline which lead to our making a variety of curricular changes, such as requiring all students in both programs to take a more rigorous statistics course.

We also formed an Applied Psychology Advisory Committee composed of representatives from area human services agencies. These valuable resources have given us feedback about proposed changes in our curriculum, advised us about emerging trends in human services, commented on the qualities and skills they would like our graduates to have, and helped us revise our internship.

Evaluating Curricular Decisions Through Assessment

Other departments have used assessment to evaluate the consequences of making curricular changes during semester conversion. For instance the Visual Arts Department combined their two entry level courses, Introduction to Design and Introduction to Drawing, into a more integrated course, Visual Foundations I. Their assessment plan allowed them to determine the impact of that decision on each component of the course using pre and post testing. In fact, while a written test indicated that students are learning the fundamentals of theory, design, and associated terminology, the drawing tests suggested that students are lagging behind expectations in this area. Now they must decide whether to return to the previous format or revise the existing course, but at least they have a good idea of how the change has impacted the learning of their students.

Integrating Assessment with Accreditation

Some of the most active departments in assessment participate in a national accreditation proce4ss that requires rigorous, ongoing documentation of goals and standards in the discipline. For instance the Nursing Program, reviewed on a ten year cycle by the National League of Nursing in the past, has been assessing students' progress towards acquiring the skills of critical thinking, communication, moral reasoning as well as self-directed learning, therapeutic interventions, and employment and program satisfaction well before as self-directed learning, therapeutic interventions, and employment and program satisfaction well before NCA made assessment an important component of university accreditation. Assessment of a formative nature is integrated into particular classes that all students take as they progress through the program and yields useful information for ongoing curricular change. For instance, strategies used to assess how students acquire critical and reflective thinking includes, but is not limited to, reviewing journal entries, analyzing cases and course projects, and scrutinizing evaluations of students by practicum advisors. While the results generally confirm that students are heightening these skills as expected, the process permits early detection of those students who may be falling behind.

Courses in the Nursing program are also rigorously evaluated on an ongoing basis. Each semester the Curriculum Committee reviews information about each course (student evaluations of courses and professors, evaluations of the course by teachers, course objectives, syllabi, textbooks, assignments) to ensue consistency with program goals and philosophy. Summative assessment which examines the program overall by the performance of graduating seniors includes conducting focus groups and using standardized testing with measures of critical thinking and moral reasoning. It is fair to say that assessment is integrated into every facet of the nursing program tightly linking outcomes of the educational experience to curricular decisions on daily basis.

Another example of a program that involved in assessment for external review is Professional Education. Formally assessed by the National Council for the Accreditation of Teacher Education (NCATE) Board of Examiners every five years, Professional Education programs also conform to the requirements of the Minnesota Board of Teaching (BOT) which has adopted 10 standards and 120 indicators of effective teaching practices. For accreditation purposes, Professional Education must be able to show that their graduates can model each of the indicators. Collecting information on where each of the 120 indicators is taught and assessed in the new (proposed) core elementary and secondary courses will allow the department to look for missing indicators and assign them to a core course. In addition, the department is moving towards a system of establishing an electronic portfolio for each education student that contains work (e.g. videos of teaching performance, examples of lesson plans) specific to each of the indicators.

Also, for several years Education students have been asked to describe in writing what they understand about each of the teacher education standards at three different points of their education: during their first class in education, just before student teaching, and after student teaching. The department reviews their responses and resulting changes to the program have included several revisions of the standards and their indicators, and changes in the course assigned to address individual standards.

In Sum

This sampling is not an exhaustive list of what we are learning across the campus using assessment, since all departments are involved in assessment of student learning. Yet, these examples provide an idea of how assessment has been a valuable tool for faculty in making important decisions about revising courses, converting to semesters, evaluating changes made during semester conversion, meeting students' vocational needs, ascertaining that programs are meeting their educational goals for each and every student in the major, demonstrating adherence to disciplinary standards through accreditation - and yes, getting ready for NCA!

ÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖ

Center for Professional Development

Spring 200 Calendar of Events

March

1^st -Newer Faculty Discussion Series, Tips for Documenting Teaching Effectiveness, 4:p.m., David Park House

7^th-Showcasing Teaching with Technology, Power Point and the World Wide Web, 3:30 p.m., Sattgast 242

23^rd-Faculty Forum, Sports for Individuals with Disabilities, 3:p.m., location to be announced

24^th-Undergraduate Teaching Associates Program, 10^th Anniversary Celebration, to be announced

29^th-Newer Faculty Discussion Series, Tips on academic Advising, 4:p.m., David Park House

April

6^th-Park House Dialogue, to be announced, 3:30 p.m., David Park House

20^th-Showcasing Teaching with Technology, Interactive Syllabi, 3:p.m., Education Arts 420

25^th-Newer Faculty Discussion Series, Grant Writing Tips, 4:p.m., David Park House

26^th-Faculty Forum, Lasers Unlimited, 4:p.m., Satgast

May

10^th-Park House Dialogue, to be announced, David Park House

12^th-GRADUATION