1. An assessment of the potential benefits of Blackboard Analytics for Learn:
meeting the needs of administrators, instructors and researchers
Matthew Bernacki
Assistant Professor, Educational Psychology
Michael Wilder
Instructional Design Coordinator, Office of Online Education
Overview
As the University becomes increasingly reliant upon WebCampus’s Blackboard Learn
software (Blackboard) for face-to-face, hybrid- and online courses, instructors,
instructional designers, and administrators face new challenges with respect to 1)
monitoring student use of technology for learning and 2) instructors’ use of technology to
teach. Blackboard Analytics for Learn may have the potential to meet the needs of these
stakeholders, and may also facilitate research about the quality of learning and instruction
in technology-enhanced courses. The results of these studies could then be leveraged to
improve student-learning outcomes by adapting instructors’ practices. There may be
more precise and economical solutions. This document examines the individual needs of
University administrators, instructors, and researchers, identifies the functionality that
Blackboard Analytics for Learn can and cannot provide to address these needs, and then
renders a recommendation regarding the utility of the analytics package.
Statement of needs
Student Accountability
In hybrid and online courses, students are required to access course materials that are
hosted on Blackboard. At the course level, Blackboard Learn itself provides some data
for faculty. User activity by content area (folders, assessments, assignments, etc.), user
activity in forums, course activity overview is available. At times, the detail of this
information is insufficient. For example, an instructor may encounter a student who
claims to have conducted an action on Blackboard (e.g. completed a test), but no record
of the test can be found in the instructor’s course interface. Ideally, the instructor could
request a report from Blackboard and examine a log of student use of Blackboard for the
period in question. At present, the software provides very basic reports that show:
Report 1: whether a student logged in that day, but not where they went
OR
Report 2: How many students accessed the course content item in question, but
not which students accessed it (or when).
Stakeholder #1 need: Instructors of hybrid and online courses need a greater degree
of detail regarding student actions in the system in order to be able to confirm the

2. learning efforts students claim to have made and to be able to make informed
decisions regarding the efficacy of instruction.
Instructor Accountability
In hybrid and fully online courses, instructional designers and instructors collaborate to
produce and deliver content for students to use when learning in a course. In some
instances, instructional designers build courses that are used by multiple instructors, and
are tasked not only with designing the material, but also with monitoring the fidelity of
instruction (such as course logins, time engaged in a course, etc.). The Blackboard
software that is currently in operation at UNLV does not provide a complete record of
instructor actions in a course, limiting the ability of instructional design staff and
administrators in academic units to monitor instructors’ activity in courses that meet via
Blackboard.
Stakeholder #2 need: Instructional designers and Academic Unit administrators
need additional functionality from Blackboard software to ensure instructors are
providing a course experience of sufficient quality.
Opportunities to Understand and Improve Learning
In addition to being able to confirm students’ actions for the purposes of accountability,
instructors have a desire to assess the utility of the instructional materials the use in the
technology-enhanced courses. To evaluate whether materials are beneficial for learning,
instructors want to examine how individual students use Blackboard-hosted content
items, and which approaches result in superior learning outcomes. At present, instructors
cannot look at this data in sequence, or at the student level.
Stakeholder #3 need: A fine level of detail about student learning to improve
educational experiences.
Similar to the needs of faculty and instructional staff (Need #1), educational researchers
are in need of a more refined data management system that captures student learning at a
fine-grained level of detail. To better understand how students learn best when using
technology, researchers need to be able to see the sequence and duration of learning
events, and the relations between students' actions. As described above, the current report
structure available in Blackboard Learn does not provide this level of detail.
Affordances of Blackboard Analytics for Learn to address University needs
As stated in the first section, the needs of administrators and chairs of academic unit as
well as faculty and researchers can be addressed with two specific types of functionality.
In order to support all these stakeholders, we need a learning management system that
can:
1) Track student use of Blackboard features in a fine-grained fashion. By
obtaining a log of student actions, we can help faculty monitor students’ use of

3. learning materials for both student accountability and the evaluation of their own
instructional approaches. We can also facilitate research on the benefits of
learning activities in online and LMS-supported hybrid environments.
2) Track instructor use. A fine grained log of instructor use is necessary for the
Office of Online Education to monitor PTIs instruction in courses, and can be
useful for Department Chairs to evaluate faculty use of the LMS system, where
warranted.
Following is an example of possible reports obtained from Blackboard Analytics for
Learn for Learn:
• Student at-a-glance
• Faculty at-a-glance
• Most active instructors
• Aggregate activity by organizational unit
• Logins by day of week
• Days since last login
• Length of sessions
• Course tool use
• Average discussion posts per user
• Courses with no content
• Building block usage
Based on an evaluation of the literature we have received that describes the functionality
of Blackboard Analytics, it is unclear that the Analytics package can provide the fine
degree of detail that educational researchers may require. Most of the functionality comes
in the form of new reports, which summarize data, rather than provide fine-grained
details. Blackboard Analytics for Learn may provide useful information for instructional
design and for University administration, however, at an aggregate level. This could be
useful for administrators who wish to examine enrollment in courses (e.g. Registrar,
Provost, etc.) or for Academic Unit heads to plan course offerings, but it does not offer
much to individual faculty members.
Faculty members may find some other tools useful, such as their tools that ply logistics
regression models to predict student achievement based on completion of certain
activities in Blackboard (e.g. completing the first paper on time predicts a B or better on
final grade). However, many of these findings are likely going to be intuitive, and can be
supported with some simple support in other ways (e.g. partnering with an Educational
Psychology or assessment and evaluation faculty member) using either basic Blackboard
functionality or an available Blackboard building block.

4. Potential alternative solutions
Do-it-yourself data analysis
Following the model established by Cal State Chico, a wealth of information can be
obtained by querying the database directly. Given access to the data, a qualified database
administrator should be able to generate basic reports (such as student/faculty logins,
empty courses, basic tool usage, etc.) using standard data querying methods (SQL
statements).
Building block to log student behavior
At an OIT-sponsored meeting with a Blackboard engineer on 11/14, we were informed of
a building block designed for Financial Aid officers that tracks student behaviors online.
The building block will be available in the coming months, and is reported to log all
student actions for the purposes of verifying their involvement in courses in order to
approve their financial aid.
We have yet to verify the reports made available by this tool but are in touch with a
Blackboard engineer (Liam Ferris) who has offered to help us explore its functionality. If
the building block works as described, it should address faculty and researcher needs
regarding student use of the LMS.
Pass-through software to log student behavior
Because UNLV-sponsored coursework on Blackboard is hosted via the University’s
servers, we could also capture student use of the LMS via pass-through software. In this
approach, software would be set up so that, when students access Blackboard, their
requests are routed through software that logs the students’ Blackboard account
information and their actions, without influencing the way they use the LMS interface.
This transparent method would allow students to continue using the LMS as normal, but
would also produce a locally housed log of student behaviors. This log can be sorted by
course and provided to instructors as requested for the purposes of monitoring student
behavior for accountability or course improvement. It can also be anonymized and
processed for researcher use. Andy Stefik, a faculty member in Computer Science, has
expressed a willingness to partner with the co-authors and OIT leadership to set up this
system to facilitate a research proposal that is to be submitted to the National Science
Foundation in January (Research on Education and Learning program; REAL).
Outstanding Need
While our alternative solutions could meet the needs of faculty and researchers, solutions
to meet the needs of administrators who wish to examine LMS use by instructors and
examine LMS use across courses have not yet been identified. It is not clear that the
functionalities provided by the Analytics package will satisfy the needs of these
stakeholders.

5. Additional effort should be undertaken by the Office of Online Education to identify
potential data solutions for LMS user logging across courses, and some polling of
Academic Unit leads is warranted to determine their needs re: evaluation of course
offerings and instructor use.
Recommendation
Because the cost to acquire Blackboard Analytics for Learn is high, time to implement to
long, and the likelihood that the package meets all of our needs is low, we currently
recommend not moving forward with a purchase at present. Instead, it would be good for
the Office of Instructional Technology, the Office of Online Education and a core of
faculty and researchers (potentially from the CMSCC committee and others with
expertise) to pilot DIY database analytics, building blocks, and pass-through software in
a subset of courses or time periods to examine whether they offer satisfactory solutions. It
is possible to attempt these approaches in the Spring, spend the Summer considering their
viability and, if warranted, acquiring a third-party analytics package for Fall 2014.
Note: As University officials consider how UNLV will approach learning technologies,
we also provide a set of common questions governing the use of technologies at
postsecondary institutions.

6. Questions to consider
Following is a set of important questions that need to be considered prior to
investment in any learning analytics solution:
• What are the specific goals of the data analytics?
• Who will gather the data?
• Who will act upon the results?
• Who will maintain any processes/software used to consolidate the data?
• Who will provide funding?
If proprietary software is used to consolidate the data:
• Will findings be treated as proprietary and will not be made available to other
researchers?
• How are various factors weighted in an algorithm?
• Are the concepts being analyzed the rights ones?
• Can researchers adjust the algorithms of vendor tools to conduct experiments of
other factors that might impact learning?
Questions concerning ethics and privacy
• Should students be told that their activity is being tracked?
• How much information should be provided to students, faculty, parents, issuers of
scholarships and others?
• How should faculty members react?
• Do students have an obligation to seek assistance?
• What data is appropriate to collect about student? What data is inappropriate?
• Who should be able to access the data and view results? Which data should be
reported anonymously?
• What is the impact of showing faculty modeling results? Do any of the data bias
faculty instruction and evaluation of students?
Questions regarding the interpretation of data
• Who gets to interpret the data?
• Who owns the data?
• Who benefits from the data? The institution, the instructor, or the student?
• Can data be use against instructors/departments/institutions to determine
employment, tenure or funding?
• Is time in an LMS representative of engagement? Simply because educational
content pages are open in a browser, does it mean that a student is processing the
information?