22

As discussed in the Data Collection chapter, before you begin collecting data, you should have a plan for what you need to collect, why you need to collect it, and how you will collect it for the purposes of decision-making and reporting. That’s not all, though; the second step in your data strategy is to decide how larger measures, like student savings, are calculated, presented, and analyzed.

Calculation Principles

When you are calculating any savings data to present to the public or to your stakeholders (see Chapter 8, Building Familiarity on Campus, for more on getting to know your stakeholders, and see Chapter 21, Data Collection and Strategies for OER Programs, for more on gathering data to answer stakeholder questions), consider a few key principles in processing this data:

Be able to say “it’s at least this much.” Take every caveat into account when calculating and presenting savings data and report conservatively. An example of this is using only the direct reported savings from OER program grantees in calculations instead of a reported potential for more savings if the entire department adopts the resources. If your actual savings are higher than your reported potential savings, that’s great news for all if it’s revealed later on!

Use transparent methods whenever possible. Some of the numbers you have will be directly taken from surveys, reports, and faculty-reported savings estimates, but others (such as applying an average textbook cost savings estimate per student in one course) may have to be taken from external sources. If possible, use external sources who report their own data and estimates openly, and link to those sources within your reports and shared spreadsheets.

Adapt your methods to the needs of your stakeholders. If you notice an emerging critical question that you did not anticipate in your collection strategy, try to adapt your calculation methods. For example, in its first year, Affordable Learning Georgia started tracking annual savings estimates from its faculty teams; when it was clear that administrators needed semester-by-semester data to answer their questions about OER use on campus, we changed our new grantees’ reporting methods to include semester-by-semester estimates and checked in with our past grantees on this as well.

Reporting Principles

Be as transparent as possible, whenever helpful. While many of your stakeholders will just need a summary report to keep up-to-date on your program, providing all of the important data in its primary form (spreadsheets, a database) and sharing your methodologies will help in “sharing the work” on how you reached the numbers in your report and assist others in answering their more complex questions. Try to avoid sharing extraneous and/or sensitive data: in your tracking sheets, you may have things like grants office contacts, invoice and purchase order numbers, or personal identifiable information (PII) that does not need to be shared.

Balance this transparency with up-front simplicity. Media outlets and administrators do not have exorbitant amounts of time to parse your data, so while having unprocessed or highly-technical data available is a plus, providing data in only a non-usable or jargon-heavy way is like having a layer of opacity on top of your intended transparency. If your own communications office or upper administration wants to know about your numbers, consider giving a “short version” in reports and communications first, and then put the details below that or in an attachment for reference.

Provide a slow-moving metric for the public. Similar to keeping things simple up-front, media outlets and administrators will be confused by a constantly-moving set of impact numbers. For example, if you are adding student savings being reported by faculty on a rolling basis, and then removing student savings immediately when faculty say they have discontinued using OER, newer reports and articles will start looking “wrong” and creating confusion. In this case, you may want to provide consistent numbers annually in reports and keep per-semester numbers in your shared spreadsheets for those who want to dive deeper into the data.

Reach out with good news. You may have wonderful reports that share great news about your program through your website or a newsletter, but sometimes you need that to go further. While much of this chapter focuses on the creation of reports to suit various stakeholders and media outlets, be sure to use all possible channels to market your program’s successes. Provide appropriate visuals: if you’re sharing savings across multiple institutions or departments, use tables; if you’re tracking student savings across a period of time, use line graphs or bar charts; if you’re showing the percentage of your grant teams that saw positive, neutral, or negative changes in outcomes, use pie charts. For more on marketing, see Chapter 7, Marketing Your OER Program.

Using Direct Data from Grants

After planning your data collection strategy for course redesign / OER implementation grants and confirming a fully-completed round of projects, using the collected savings data or savings estimates from grant projects seems far simpler than having to find an overall per-course or per-textbook average for an entire institution. However, the complexities within using direct project data from grants are in the details:

What if a grant project is discontinued? Not every grant team will use OER or no-cost or low-cost materials forever, due to either individual decisions, changes in leadership, or changes in the commercial market. How will you find which projects have discontinued the use of these materials?

What if grantees expand their use of OER beyond your estimates? Similar to discontinuations, how will you find out which projects have proliferated the use of OER within the department? ALG addresses this through annual sustainability checks, which are explained in detail further in this chapter.

What if grantees start using different OER materials? Can you ascribe the savings being generated from a team using different OER or no-cost or low-cost materials to a grant if the team moves to a different set of OER? The grant project most likely introduced them to OER, and they’re now undergoing improvements to their course on their own with that work in mind. ALG does report these same-course cost savings if grantees move to different open materials.

What if grantees expand their use of OER to new courses? Can you ascribe the savings being generated from a team using OER in a different course that wasn’t part of the grant project? As with the different materials issue, there’s no easy answer. ALG does not report on this expanded use of OER in new courses directly, but this expansion could be measured through accurate no-cost and low-cost course markings.

What happens when grantees, especially project leads, leave their institution? ALG’s discontinued projects have largely been caused by turnover, whether it’s a project lead leaving for another institution (the most likely) or a new Department Chair mandating the use of a particular set of resources for a course (far more rare). Not only are projects discontinued due to turnover, but data is far harder to collect when the Project Lead for a grant team suddenly disappears. How will you manage turnover in grant projects? ALG checks with Project Leads yearly in sustainability checks, but due to Project Lead turnover complicating who is able to respond for a team, we are looking at department-focused methods via campus advocates moving forward.

Adoption Data and Campus Store Partnerships

The University of Missouri’s Affordable and Open Educational Resources Program has an integrated partnership between the program and the Mizzou Stores, the bookstores for the University of Missouri. Beyond the potential for bookstores to provide print-on-demand services for open textbooks, the Mizzou Stores actively reach out to faculty to report OER and no-cost material adoption within their collection of textbook adoption data. This is an example of a highly-effective campus partnership between what at first seems like an unlikely pairing but is actually an alliance with a common goal: student success (Bell, 2018).

Partnerships like the one at the University of Missouri have a common goal: collecting textbook adoption data per the nationwide mandate for higher education institutions within Section 133 of the United States Higher Education Opportunity Act (HEOA). Section 133 is not just a mandate for collecting adoption data: the section encourages institutions to work together to find ways to reduce the cost of textbooks to students. While campus store partnerships are not always prevalent or possible, it stands to reason that the common goal of reducing textbook costs and the mechanics of reporting adoption data would foster some partnerships between adoption-data collectors (often campus stores) and OER program managers (Stocker, 2018).

Adoption Data Caveats

Like nearly all data collected for reporting, adoption data will come with caveats:

Incomplete Adoption Data: By far the most common issue with using textbook adoption data is getting adoptions reported for 100% of courses. In practice, this is quite difficult to do, depending on the data collection methods, how mandatory it is to report textbook adoptions within policy, who reports the adoptions (instructors, chairs, deans, administrative assistants), and how departments account for last-minute instructor assignments to sections and low-enrollment cancellations.

OER Knowledge: If faculty are asked if they adopted an open textbook and haven’t received any training on what “open” means, it stands to reason that this OER adoption data will only be as accurate as each individual faculty member’s own knowledge of what it means for something to be open. Even if textbook adoption data at an institution is at 100%, a faculty knowledge gap on OER will render this data inaccurate.

What’s Actually Required or Optional: If syllabi are being analyzed for adoption data, this can be a more accurate, if more time-consuming, way of collecting textbook adoptions; however, what materials are labeled as “required” and “optional” may differ in practice, and there isn’t a way to determine this from just looking at syllabi.

A Hidden Cost of “Free:” Student Privacy: Just because a resource does not come with a dollar amount does not mean it’s open or even zero-cost. For example, some seemingly zero-cost proprietary platforms might require the student to create an account, and the platform’s company may give this student account data or even behavioral data to a third-party company for monetary compensation. The cost to the student, in this case, would be privacy; truly open educational resources, or even those marked as zero-cost, should not require this.

Print Versions of Open Textbooks: The same print-on-demand advantages of a campus store partnership can complicate adoption reporting as well:

• If an OpenStax textbook is available as a print copy for $50 but it’s optional due to the digital version being open, does every instructor know and report this as a $0 course material? Do they have guidelines on what to do in this situation?
• Do some instructors require the printed version of an open textbook?
• Does the Campus Store make it clear to students at registration and when shopping for texts that a free digital version is available?

Your campus store may be just one example of a partner on-campus that can share helpful data and analysis advice. Affordable Learning Georgia partners with the Research and Policy Analysis office for enrollment data and course marking reports. Departments and offices with a goal to provide good data stewardship can help strengthen your data collection, analysis, and reporting.

Sustainability Checks: An Example from Affordable Learning Georgia

Conclusion