Taking the suggestion from previous students seriously, we will be working with a common term-long example that we will work through as a class. This approach does two important things for you as a budding scholar-teacher:
First, it will help us work through together some of the key parts of research study design. From considering potential questions or queries around a common issue. Second, it also operationalizes or actualizes programmatically based research since a number of you will likely be an administrator where learning how to do program evaluation alongside research is a necessary skill. Following is an overview of a project that we will discuss as a common example throughout the term.
But before you get to that, you may want to
- skim through this analysis of pedagogical and programmatic research in the field of TPC.(opens in a new window.). While specific to research in TPC this should help you see the different types and kinds of research to better contextualize how you might do this type of research yourself. Keep in mind that you definitely don’t have to agree with the categories or the findings or the implications. But it should give you information to transfer into your own programmatic or pedagogical work.
- skim this forthcoming work using USF Writes data (https://doi.org/10.1177/00472816221124605 will get you to find it in the library database) This piece started just like I’m asking y’all to think about. It was also a mentoring sort of process where the student group learned (hopefully) how to move from general query all the way through the publication.
- skim taxonomy of research questions (this is in an progress work; do not circulate or cite) Much like the first piece here, this one is something to simply consider as ways to approach programmatic or pedagogical work in a more systematic way.
- skim continuous improvement models.
These pieces will give you insights into my own approach (and biases) when doing pedagogical and programmatic research. They also provide ways to consider for yourselves the type of research that could result from USF Writes data or other programmatic or pedagogical data. The thing for you to consider when it comes to programmatic research is that you have to consider the context of the problem. For example, in the forthcoming article above, we knew a particular assignment was not working so we wanted to figure out why. That’s a problem that led to a question that led to us systematically looking at data in hopes of figuring out some ways to improve the assignment for student learning.
When I use data here, I use it to me that we analyze things to help us make actionable decisions to improve the program. As administrators (and faculty), we have the power to change things within a program that aren’t working. using data helps us figure out why something isn’t working and gives insights into ways that we can change it.
Following is a PTC example on feedback. The FYC example we discussed in class on 2-2-23 is here.
Collective Feedback File
The TPC Program takes formative feedback to mean a mechanism that “supports teachers’ and students’ inferences about strengths, weaknesses, and opportunities for improvements in learning,” and it “includes both general principles, and discipline-specific elements that comprise the formal and informal materials, collaborative processes, ways of knowing, and habits of mind particular to a content domain” (Cizek et al., 2019, p. 14). Seen in this way, formative feedback is instrumental to connecting student learning outcomes (SLOs) to expectations within assignments.
The TPC program takes a unique approach in that we ask instructors to use a CFF rather individual feedback. This approach aligns with research around effective formative feedback (Doan, 2019; King & King, 2020; Still & Koerber, 2009; Taylor, 2011), and my own research and data gathered since we shifted to this model shows that CFF does not negatively impact student learning and may actually improve it.
When integrated effectively in the classroom, CFF deliberately ties revision to the goals of the assignment and the SLOs. To make the connection between SLOs and the assignment, the PTC Programs uses collective feedback file (CFF). The CFF contains three main elements:
- a compilation of excerpted anonymous examples from student drafts that illustrate the most common areas that need improvement
- an explanation of why these examples can be improved by relating the example back to course content
- an example of how to improve the document
When instructors create a CFF in USF Writes, they tag their comment to students to a SLO keyword. Take a moment to review the document series and information design assignments and their outcomes. (Scroll down the page to the “our current curriculum)
Google folder with 4 spreadsheets:
- 21 Fall and 22 Spring doc series CFF data
- 21 Fall and 22 Spring info design CFF data
Generating Questions or Inquiries
In light of what the assignments are trying to accomplish, look at the data through the eyes of a beginner program administrator. Consider the following:
- what strikes you about the data as it relates to the learning outcomes?
- do you see any immediate patterns?
- how do the tags related to outcomes align with the actual outcomes?
- what impact might labor have on this data?
- what do some of the actual comments point to about teaching and learning?
These are just some initial thought questions to help you start to ask questions about what the data may show. Consider different questions that you may ask about the data.
Also what other information may you need about the program to narrow and refine the questions or queries you are considering?