Empowering Instructors | Supporting Excellence
Choosing and collecting data to measure course effectiveness.
When instructors teach, whether intentionally or not, they evaluate their courses. Evaluation may occur informally, based on their opinions of how the course went, reflections on what worked well, and which aspects needed improvements. Data collection may also be more systematic, such as analyzing student evaluations or grades. Regardless of the data used, or how it is used, it is best to develop a cohesive plan to ensure that you make purposeful and meaningful choices.
The following are types of data grouped by who is making the evaluation (students, instructors or faculty peers) and what is measured (e.g., student achievement, experience with the course and experience with instruction).
What: Self-perception of achievement, such as content understanding and skills (e.g., competence; confidence; knowledge)
Data Collection: Poll, survey, in-person questioning
Questions: Do you understand X? What do you find confusing? How confident do you feel about doing X? Is this too difficult?
What: Self-perception of course experience, such as what contributed to or detracted from learning
Data Collection: Poll, survey, in-person questioning, student evaluations of the course
Questions: Was there too much content? Anything too easy? Which areas did you need more time? Why?
What: Self-perception of instructional experience (experience of how the instructor contributed to and detracted from learning)
Data Collection: Poll, survey, in-person questioning, student evaluations of teaching
Questions: Was the instructor talking too fast, confusing, approachable, helpful, gives clear examples, available for office hours, culturally responsive, etc.?
What: Achievement measures
Data Collection: Exams or quizzes; assignments; projects; portfolios; clinical field work; observation of work; DFRW rates
Questions: Specific to assessing learning outcomes
What: Design rubrics and self-assessment of experience
Data Collection: Observation of interactions with students; Inference from student work and achievement; Learning management system (LMS) user indicators such as student time spent in the system
Questions: What do I think is going well? What are students struggling with? What are they confused by? When did students seem engaged? Which assignments were done well? With which assignments did students struggle? When did students seem to have enough time/not enough time? When did students work well together?
What: Perception of teaching experience and direct observation (e.g., recording of teaching)
Data Collection: Observation of interactions with students; analysis of teaching using an instructional rubric
Questions: Did I feel students asked questions? Were students congenial with me? Did I see students were taking chances? Did students demonstrate trust? Did students give compliments or positive affirmation about my teaching? What might have contributed to the positive/negative interactions?
What: Observation of evidence of student achievement
Data Collection: Observations of student work and performances
Questions: Specific to assessing learning outcomes.
What: Assessment of course design; observation of course
Data Collection: Observation of syllabus, course plan, course content and materials
Questions: Same as instructor questions. Specific to observation instrument (e.g., active learning criteria, etc.)
What: Observation of instruction
Data Collection: Informal teaching observation; observation using observation instrument; separate analysis of student survey responses
Questions: Same as instructor questions; What did students like/dislike about the instructor from survey questions?
It is important to note that different people will have varying levels of expertise to inform your course evaluation and suggest improvements. For example, students may be experts in their experiences (e.g., whether they can understand you, whether they feel they understand the material) but not experts in other areas (e.g., are you providing the latest research or relevant content?). Further, a peer faculty member might be a good source for scope of content, but not for whether you are communicating well with your students. To elicit this type of information, you could ask a mentor to observe your class formally or informally.
The design of your course may be reviewed or evaluated before the course begins, as well as during instruction. It may also be difficult to separate the effects of instruction from course design. For example, if you have designed too much content in a lecture, and students inform you that you have talked too quickly in the lecture, is this a design or instructional issue? There are, however, factors that are not part of your design. Do you speak loudly or clearly enough? Do students feel they can approach you for help? These are mainly instructional effects connecting back to how you interact with students including verbal and non-verbal communication. Part of your later analyses may require you to determine which factor or combination of factors are responsible.
The following sections elaborate on the types of data you might collect. These may be used for the course as a whole or for an individual unit. There are two major considerations for what data to use:
These questions are often linked in that how the data will be used often determines when you need to collect it. For example, observations of teaching will likely occur in real time but could be recorded for later analysis. Or if you plan to give formative feedback to students, or receive feedback about your instruction from students, this will need to occur while changes can still be made. Consider these questions as you make choices about your data collection.
Questioning students about their perception of understanding or skill development can take a variety of forms. Formally by using mid-semester surveys or polls, or informally by asking students comprehension questions during class to assess their understanding. For several examples of this formative data collection see the following:
The Student Assessment of Learning Gains (SALG) survey directly asks students about their perceptions of what they improved on and what helped. It can be tailored for each course.
When test items and rubric steps are carefully aligned to course learning outcomes, data from summative assessments can be aggregated for each learning outcome to provide an indication of overall student success in achieving them. While it is hoped that all students achieve the desired outcomes, it is likely that not all will. Patterns of achievement can be used to determine which course learning outcomes are most problematic for students. For these outcomes, it may be necessary to revisit instructional delivery and student support.
Comparing improved exam and course grades to previous courses is not necessarily straightforward. Some key issues to consider are:
You may need to use the new assessment and improve it several times before it is calibrated and reliable for comparing courses.
Having a peer faculty member or instructional designer review student achievement can give insights about the success of student outcomes (i.e., are these outcomes expected or satisfactory) as well as why patterns in grades or responses are occurring. These are the same measures used by the instructor but may also include observation of teaching and learning.
Examining DFRW (drop, fail, resign and withdrawal) rates compared to rates from past offerings of the course, or to other sections of the course, can provide important information about course and instructor accessibility. While the goal is to increase the number of students passing the course, patterns may not always be clear. For example, there may be an increase in the withdrawal rate from a revised course if the syllabus and what is expected of students is now clearer causing some students to realize that the course is not a good fit for them. On the other hand, increased DFRW rates can signal content or assignments that are misaligned to student needs, or to instructor differences in providing adequate student support, as opposed to course design differences. Interpreting DFRW rates should always occur in the context of what occurred in the course and should be used in conjunction with other indicators, within and beyond that particular course.
Like surveys and polls for students to self-assess their achievement (see above), you can ask about specific design questions such as the difficulty of certain topics or the amount of time given for projects.
The Teaching Practices Inventory is a faculty self-assessment instrument to determine how many best practices for course design are followed by the instructor. This instrument gives an overall score for design and can be used as a guide to improve several aspects of your course.
While the OSCQR is a non-evaluative assessment, it does provide categories for reviewing courses for quality and effectiveness and provides support for improvements.
Like surveys and polls for students to assess course design or self-assess achievement (see above) you can ask about specific instructional questions such as the clarity of instruction, approachability of the instructor, or whether the instructor provided enough support.
TDOP is an instrument that can be used to code observations of your course for what students and faculty are doing, the types of technology used, how often students are interacting and asking questions, amongst a variety of other important in class actions.
COPUS is an instrument that allows observers of your course to code how students and faculty spend their time in the classroom.
Use the following steps to plan your data collection:
Once you have finished planning data collection, the next steps are to consider how you will combine and analyze data to make improvements.