Teaching and Learning Department's Statement on reliance of quantitative data and learning analytics
The Teaching and Learning (TAL) Department at the University Library is a team of library workers that partners with faculty and students to create inclusive learning environments using practices that center students’ agency and cultural wealth. TAL has always focused efforts on aiding students’ development as critical creators and users of information. It is clear to the Department that fulfilling these commitments requires an introspection of how we organize and assess our work. As part of this interrogation, we recognize that a fundamentally inaccurate focus on quantitative assessment is harmful; in particular, such an approach is part of a cycle of epistemic violence enacted upon BIPOC students and TAL workers.
It is important to be clear about the terms used in this statement. Data refers to any type of information that can be gathered about a learning experience and/or environment. It can refer to quantitative, numerical data (such as the minutes of a class session) or qualitative data (such as speaking with a student/faculty about how a class went or reviewing students’ reflective writing). Data is often used to refer to primarily quantitative, numerical data to the exclusion of qualitative data and approaches. Learning analytics is the practice of using data about students’ behaviors, particularly those activities that require interaction with online tools that track users, in order to identify possible interventions that encourage desired behaviors in the learning environment. Quantitative assessment refers to measurement of student learning outcomes and/or professional performance assessment that relies on numerical and statistical analyses. By contrast, qualitative assessment makes use of reflective practices, individual interviews and anecdotes, and other means that do not rely on statistical analyses for insights on student learning. Learning analytics and quantitative assessment approaches are not the only ways student learning and librarian work can and should be evaluated, but they are commonly used approaches because they are widely believed to be effective and are less time-intensive.
Harmful and Inaccurate
Quantitative assessment is harmful and inaccurate when it is used as a single method of assessing value. It privileges the ways of knowing and the values of the creators/users of the data rather than the ways of knowing and values of the “assessed” (students, librarians, staff, etc.). It disconnects the rich backgrounds and contextual conditions of our students/librarians and reduces them to numbers.
Learning analytics has been put forward as a way to address inequities by identifying socio-economic disparities/inequities. The data collected by TAL is anonymized to protect the privacy of our students, therefore this type of disaggregation is not possible. The way TAL works with students is fundamentally relational; we have to do the work of understanding who our students are rather than relying on disaggregated, quantitative data.
Quantitative data is fundamentally useless beyond TAL
Some aspects of TAL work do lend themselves to being counted through quantitative measures (e.g. questions answered, number of guides produced). Analyzing these quantitative metrics to come to some conclusion about TAL work could be of use to those within the Department. For example, in recent years, the amount and type of questions answered at the Research Help Desk indicated changing student needs.
When this data is taken outside of the Department, quantitative measures not only have almost no use, but are fundamentally inaccurate because of the disconnect that occurs between the data and the context of data collection. This disconnect leads to the exclusion of the many qualitative data sources and relational information that are generated when librarians work with others. The quantitative data that TAL may generate is not intended for use by those outside of the Department because of this. D’Ignazio and Klein describe this as “...strangers in the data.” This phrase is used to describe the phenomenon of data being used outside of the community in which it was generated (p. 133). The ‘strangers’ lack the context and knowledge of the community to make use of the data appropriately and accurately. D’Ignazio and Klein further illustrate the problems caused by strangers in the data by referring to Spivak’s (2010) concept of epistemic violence; “the harm that dominant groups like colonial powers wreak in privileging their ways of knowing over local and Indigenous ways.” Once TAL’s data goes beyond the Department, it can be used in ways not intended. Conclusions could be made that miss huge pieces of information. What could seem to be innocuous information sharing carries with it the explicit expectation that the data will be interpreted and used somehow - most likely in ways it was never intended to be used.
TAL centers our data gathering on how it may be useful to us in our individual and collective endeavor to aid students to continue to progress in learning and education. Too often, data collection and analysis is driven by entities beyond the community in which it is generated. These external demands provide another example of strangers in the data, but this time, explicitly enforcing outsider views of what “counts” based on the goals or priorities of the external entities rather than the goals, values, and priorities of those being assessed.
What TAL does instead
For these reasons, TAL’s use of quantitative assessment and learning analytics, if at all, will be in service of the Department’s, individual librarian’s, and students’ learning goals. Indeed, placing students’ learning goals (rather than the assessor’s goals) as central is foundational to our Department’s assessment methodology.
The TAL department will not make the quantitative data we do collect available to those outside the department without accompanying contextual, qualitative data or a disclosure that the data is incomplete.
Following D’Ignazio & Klein’s example (2020), the TAL department aims to use reflexivity and transparency in all assessment efforts. In reflexivity, we will acknowledge our own positionality and recognize the limited nature of our perspectives. The Department will seek to be transparent with students about our own goals, methods, and approaches to assessment, while continuously seeking student input and feedback on the entire process.
We also recognize that quantitative assessment data is seen as an easy (if incorrect) way of demonstrating imagined “value.” Removing data from its original context to be analyzed by individuals without the background or knowledge of the collection process inevitably leads to inaccurate and/or harmful results (D’Ignazio and Klein, 2020, p. 171). As D’Ignazio and Klein explain, connecting data back to the context in which they were produced (classroom, lesson, instructor impressions, etc.) allows us to “better understand any functional limitations of the data and any associated ethical obligations, as well as how the power and privilege that contributed to their making may be obscuring the truth” (2020, p. 153).
Guiding questions that aid us in the practice of reflexive and transparent assessment include:
Who is benefiting from our data practices?
Whose goals are prioritized?
What are the factors shaping student responses to assessment questions that we should be considering in our analysis?
What assumptions underlie our interpretation of the “use” of data that the library is collecting?
Are we using learning analytics to actually measure ourselves and our performance (as a library) rather than student learning?
How committed are we to actually making positive changes in response to meaningful data?
How is the Association of College and Research Libraries (ACRL) using our statistics? (as an example of outside organizations that may request TAL data)
What TAL does instead with our students
TAL’s approach to data collection and analysis centers on how it may impact students. Student success and engagement must be defined by students. While faculty and administrators may have valid and important interpretations of student success, student agency is often neglected or only peripherally considered. The use of quantitative data in learning analytics illustrates this issue well with particular negative, insidious impact to BIPOC students. Learning analytics reduces students to quantitative data points, makes assumptions about what these data points mean, and can harm BIPOC (and other students from historically under-represented groups in higher education) through these assumptions and biases.
In our instructional efforts, TAL faculty use practices that aim to either center student agency or ensure its full consideration. Examples include but certainly are not limited to: working with students during librarian-led instruction to generate their own learning outcomes, teaching students critical information literacy strategies, and engaging students in reflection about their research efforts. These practices typically yield little in the way of quantitative data points that provide insight on student success yet provide rich insight on student engagement. In addition, TAL librarians may choose to make use of student evaluation of instruction (SEI), which is a common tool used by instructors of record. SEI can be a useful, indirect means of assessment; it can provide insight. However, there is a significant body of research that SEI is often biased against women and BIPOC. When librarians choose to use SEI, the Evaluating Teaching and Learning materials (available only to CSUSM librarians) provide guidance and how to approach including such information in one’s working personnel action file (WPAF) and the evaluation process.
In addition to considering the impacts to students of our data collection and analysis, TAL also supports individual librarian’s choices in this realm of our work. It is hard to overstate how different each librarian’s portfolio, teaching approach, and faculty partnerships are and ergo, how evaluation and assessment approaches must differ.
TAL does this among ourselves, as library faculty and library workers
A focus on quantitative data about librarian work can also have negative impacts on how TAL faculty and all library employees work and relate to each other. The act of counting the number of instructional sessions, time spent teaching, or number of guides produced erroneously implies that the more TAL faculty do these things, the better their job performance (and ergo, the better for students). This is not the case. For example, a strategy to flip the curriculum of a particular library instruction session may result in one robust online guide and a reduction in total teaching hours for the librarian. This strategy may be the most impactful for student learning, depending on the situational context. Indeed, each librarian’s assignment of responsibility and strategy for supporting students in their disciplines varies, which is reflected in their instructional approaches. This applies the same to tenure-track and lecturer librarians. A focus on quantitative data in employee evaluations is especially harmful for librarians with temporary appointments, who may experience the precarity of their employment situation as a pressure to demonstrate value through numerical data (e.g. the number of courses taught, reference interactions, etc.). This pressure may reduce the librarian’s ability to experiment with curriculum and guide their own teaching practices due to the time spent taking on greater numbers of tasks in the hopes of securing a subsequent contract.
For BIPOC librarians, reliance on quantitative data has similar and arguably more insidious impacts. Parallel to BIPOC students, BIPOC librarians work within institutional structures that communicate a need to conform to racist standards and practices. One way to combat this is for the TAL Department to explicitly support individual librarian’s choices in this area of work as part of department-wide efforts to become actively antiracist. Indeed, this statement aims to be one action in this regard.
Students are not customers. Librarians are not providing a service to them but rather guiding them through an educational process. Meulemans & Carr (2013) elaborate on this. Quantitative measures (e.g. the ‘Yelp’ review approach to assessing chat interactions) erroneously communicate that students’ initial feelings about their interaction are an accurate representation of their educational engagement and success; especially when they are in the midst of doing something they find especially difficult. Research clearly indicates that students often experience feelings such as frustration, apathy, and anger during challenging periods in their education that may negatively impact their engagement. Educators engage students with research assignments that challenge them. Academic librarians, in their role as educators, interact with students as they experience these valid feelings (that may result in potentially negative actions by the students such as disengagement or a ‘bad comment’ about the librarian.) A customer service model that centers a customer’s satisfaction is not applicable. This reliance on assessing students ‘positive’ feelings about interactions have particularly damaging impacts on library workers in precarious employment situations. It can be a quick step to conclude a librarian is performing poorly if students are rating a librarian only two out of four stars after a chat.
In place of a focus on quantitative measures of performance, the Department values and practices reflection upon our individual and program-level work. In the Department, individuals may bring challenges (e.g. figuring out how to create materials to engage students) to the group. Similarly, the Department relies on everyone’s insight to plan and assess our programmatic-level work. We have made the explicit choice to make use of our relationships as colleagues to reflect, assess, and reconsider our professional practice. This choice makes our work more complex as it takes time to create a culture that successfully supports it. That said, we believe this is far better for students and more sustainable for library workers.
Further reading/Bibliography
Gayatri Chakravorty Spivak, Can the Subaltern Speak? Reflections on the History of an Idea (New York: Columbia University Press, 2010)
Catherine D’Ignazio & Lauren Klein, Data Feminism (Cambridge: The MIT Press, 2020)
Beauchamp, Adam, and Mallary Rawls. “In Search of a Just and Responsible Culture of Assessment.” YouTube, uploaded by Beauchamp and Rawls, 31 Aug. 2020, Two critical alternatives to learning analytics in libraries (CLAPS 2020).
Yvonne Nalani Meulemans & Allison Carr (2013), Not at your service: building genuine faculty-librarian partnerships.
U.S. Department of Education (2012), Enhancing Teaching and Learning through Educational Data Mining and Learning Analytics: An Issue Brief.
Manuela Ekowo &. Iris Palmer (2016), The Promise and Peril of Predictive Analytics in Higher Education: A landscape analysis.
Eamon Tewell (2016), Putting Critical Information Literacy Into Context: How and why librarians adopt critical practices in their teaching.