Digital grading

Replacing pen and paper with a digital solution

About the Project

Top Hat is an education platform used by thousands of professors and millions of students across North America.

In August 2019, we acquired a print textbook publisher and began converting their print content into digital materials. Of this content, 90%  was for Labs, Humanities and Social Science courses; disciplines relying primarily on subjective assessments (in particular, long-form written responses).

Prior to this acquisition, we had catered to large introductory level courses with auto-graded questions that would save instructors time. Our challenge was to create a seamless manual grading experience in our product, one that would be preferred over pen and paper.

As a business our goal was to convert 30% of the acquired print textbook users to use our digital textbook solution by Fall 2020.

My Role

User Research, Interaction, Visual Design, Prototyping & Testing

Date

Sept 2019 - March 2020

Understanding The Problem

To understand our users and the problems they were facing, I focused my efforts on two key areas:

1. Digitally Grading Essay Questions In Top Hat

Top Hat had one essay type question (the Long Answer question) which was manually graded. I began by investigating the experience of creating, assigning and grading this question type.

2. Manually Grading Print Content with Pen and Paper

Next I wanted to understand the process of manually grading print materials (pen and paper). This was the experience most of the instructors were accustomed to and for us to ensure they would prefer digitally grading their content — I needed to understand the issues with their existing experience.

Gathering Insights

Given two weeks to gather data and better define the problems, I used the following methods to gather information.

Quantitative Data
Salesforce —  I looked at the feature requests and support tickets related to Long Answer questions.
Amplitude —  I looked at the percentage of Long Answer questions assigned to students that were subsequently graded.
Stakeholder Interviews
I spoke with internal stakeholders (User Researcher, Account Managers, Partnership Team) who had existing relationships with the publisher and their customers.
User Interviews
We spoke with end users to get more insight into the pain points they faced while grading our Long Answer question and to understand the pain points of manually grading print content.
“ When I’m grading a Long Answer question, I have no way to give my students feedback. I’m having to email my  students to give them feedback on their assignments and this is more time consuming for me.”
- Introduction to Psychology Professor
Competitor Analysis
I conducted a competitor analysis by looking at other digital manual grading products such as Learning Management Systems and Google Classroom.
Observational Analysis
Along with the User Researcher, I visited several labs to observe how students took notes and worked on their Lab Reports during the lab session.

Insights

From the research that I conducted, I identified the following key problems and insights:

1. Digitally Grading Essay Questions In Top Hat

Lack of Efficiency
Usability issues in Top Hat slowed down the efficiency of grading student submissions, compared to traditional pen and paper. Instructors were not able to see their overall grading progress, whether a given student had answered the question, and the primary button was not aligned with the flow of grading one student after another. These issues led to a low grading completion rate.
76%
Of all Long Answer questions created and assigned to students were not being graded by instructors.
Inability to Give Feedback
Top Hat did not allow instructors to provide unstructured feedback to student answers.
38%
Of feature requests for Long Answer questions were about providing feedback with grades. This was the most requested feature.

2. Manually Grading Print Content

Lack of Efficiency
Instructors expressed their main frustration with manually grading student assignments as being lack of efficiency.
Multiple Approaches to Grading
Instructors had multiple approaches to grading student assignments:
‍
1. One question at a time to focus on the question and eliminate context switching
2. One student at a time when questions in an assignment are related to each other

For the first approach of grading one question at a time, it was an inefficient process to flip through all assignments with a focus on one question at a time. Our platform supported the first approach, but we did not have a way for instructors to grade an entire assignment for one student at a time.

Solutions

These solutions were created over 4-5 months in iterative cycles to improve the product.

The work was divided into 3 main projects:
1. Usability and feature improvements to Long Answer question grading
2. Assignment grading to accommodate for multiple grading approaches
3. File Submission question

Key aspects of designs are shown below but the work involved creating wireframes for different approaches (e.g different options for how instructors could give granular feedback on assignments), full flows and prototypes.

Project 1 - Usability And Feature Improvements To Long Answer Question Grading

We began with a focus on improving the Long Answer question grading experience
Existing Experience
Problems
1. No indication of grading progress
‍
2.  Instructor has no way to give students feedback on their work

3.  Order of primary and secondary button makes grading inefficient

4. Instructor has to click on each student to see if they have answered
Solution
Solutions
1.  Added filter to select only students who answered the question
‍
2. Added counter to show grading progress

3. Added visual indicators to quickly identify grading status

4. Added ability for instructors to leave general feedback

5. Made primary CTA “Submit & Next” to align with grading workflow

Project 2 - Assignment Grading - Accommodate For Multiple Grading Approaches

We created an additional grading view allowing instructors to grade assignments one student at a time. Note: this is an example of a partially graded assignment
1. Counter to track assignment grading progress

2. Dropdown to view all students with their answer status and to search for a student

3. Buttons to navigate between students

4. Overall grade as both a fraction and percentage

5. Ability for instructors to leave general feedback

6. Counter to let instructors know how many questions they need to grade

7. Visual indicators for question status and ability to click and navigate to the question

8. Grade input field in same position as auto-graded questions for visual consistency

9. Ability for instructors to leave feedback on the question level

Project 3 - File Submission Question

To support courses with Labs we created a File Submission question. This question type allows students to submit their assignments as files in various formats.
File Grading View
1. Counter to track assignment grading progress
‍
2.  Ability to download a students file submission

3.  Grade input field

4. Ability for instructors to leave general feedback

5. Inline comment indicator (inline comment left by hovering over file and clicking)

6. Ability to edit and or delete a an inline comment

7. Selected state of an inline comment
Student Graded File View
1. Question title and attached documents (available for students to download)
‍
2. Student's submission (available for them to download)
‍
3. Overall grade

4. Overall feedback

5. Inline comments - students can click on the indicators to put the comment in focus
‍

Testing and Feedback

I tested my designs both with internal stakeholders and end users throughout the product development cycle. I used the following methods:

- Validated lo-fi prototypes with internal stakeholders on a regular basis
- Conducted moderated usability testing with instructors via Zoom
- Conducted unmoderated usability testing via Usertesting.com

Results

We expanded the utility of Top Hat as a platform, and helped our users save a lot of time spent on grading and reviewing student course materials. The immediate use case came from our newly acquired users, but the work helped any instructor speed up the process of reviewing and grading students’ work.

60%
Conversions from print content to digital
68%
Increase in Long Answer questions created being subsequently graded (only 8% not graded)
35%
Of Long Answer and File Submission questions graded included feedback
Next Project: Accessibility