About the Project

Middle Mile delivery work is complex. Carriers are responsible for efficient operations, reliable execution, and health of their fleet. It is common for disruptions to occur during execution of any delivery.

To ensure Carriers deliver loads safely and on time, Amazon Relay implemented Performance grades – ranging from A+ to F – for all carriers aggregated from four key pickup and delivery metrics:

‍

• On time rate
• Acceptance rate
• Disruption-free trips
• Driver app usage

‍

Performance grades update every Sunday, and are only impacted by work executed in the past 6 weeks.

Carriers with higher grades received early access to book highly-desirable work.

Carriers with low grades are only allowed to book work after high-performing carriers have their pick.

‍

‍

‍So, how is performance disputed?

Drivers who are assigned work through Amazon Relay are required to use the Amazon Relay mobile app to complete their trips.

When a trip disruption occurs (flat tire, late to destination, etc.), the issue is automatically identified and logged.

‍

‍

If carriers felt a performance issue had been logged incorrectly, they have the ability to dispute the performance issue to have it removed from impacting their performance grade.

‍

The core problem with disputes

Not only were carriers impacted by performance issues – accurate or not – but the process to have a disputed issue resolved was confusing, effortful, and rarely resolved within the time window (6 weeks or 45 days).

This meant a carrier's grade could be incorrectly impacted for 6 weeks without adjustment.

‍

Avg. Dispute Resolution Time

48 days

‍

Avg. Number of Disputes per 100 VRID (Trips)

5.2

Diving Deep

In July 2021, for my first project after joining Amazon, I was assigned to this problem with a tenured Product Manager.

‍

Disjointed workflows

In order to improve the carrier experience, I needed to audit the existing workflow for filing a dispute.

‍

‍

A few issues jump out:

‍

• Carriers need to switch between two separate pages (Performance, Support Center) to gather inputs needed for filing
  ‍
• Disputes were not specifically highlighted within the Support Center, creating problems with discoverability
  ‍
• To submit a performance dispute case, carriers needed to manually find specific trip data that was difficult to remember or locate
  ‍
• After submitting a performance dispute case, there was no guidance for what to expect next – leaving carriers confused and directionless
  ‍
• Often, duplicate disputes would be filed as carriers had no way of tracking previously filed disputes

‍

"That's what's wrong with the current process. I have to go back and forth and back and forth. Then you close a screen and lose everything."

– Carrier from user testing session

‍

‍

Working backwards from the customer

I began revising this workflow, and with a goal of creating a self-guided experience, came up with a new model to help carriers submit disputes. It employed a "understand", "inform", "assist" workflow.

‍

‍

My hypothesis: keeping carriers in context and guiding them through the dispute workflow with clarifying questions, we would:

‍

• See a decrease in incoming disputes (more informed)
  ‍
• Eliminate duplicate disputes
  ‍
• Streamline the incoming data passed on to our Relay support team

‍

From 54 use cases to 4 outcomes

I was able to create and define a guided structure for all 54 unique dispute use cases with the help from internal support reps.

My goal was to streamline and automate these 54 use cases to 1) assist carriers through a simplified, guided dispute process, and 2) standardize the process for how support receives and resolves issues.

‍

‍

Designing the experience

With our core spreadsheet locked in, I used Sketch (before we migrated to Figma) to create the new guided workflow for carriers to submit performance disputes.

The majority of the workflow lived within a progressive disclosure modal.

Part one of the experience included the affected metric, reason for being logged as an issue, load details, and a dropdown for the carrier to select the reason they would like to dispute.

‍

‍

Dispute reasons could have multiple categories and sub-categories. Each dispute reason was tied to one of four performance metrics, so the dropdown list varied by issue.

‍

‍

Once an initial dispute reason was selected, the carrier would be presented with a next step (follow-up questions, sub-categories, additional dropdowns, etc.).

‍

‍

After completing part one of the workflow, a carrier would be presented with one of the following four outcomes:

‍

‍

‍

‍

‍

Carriers can submit a formal dispute regardless of which outcome message they receive.

‍

Testing the workflow

I conducted four 60-minute carrier interviews with a UX Researcher to collect feedback about this proposed experience.

‍

Primary insights

• All carriers were able to discover the correct location for disputing performance scores
  ‍
• Carriers expect all applicable inputs to be auto-filled with information like dates, times, and load notes
  ‍
• After submission, carriers want to know the status of their dispute within the performance page
  ‍
• Consider allowing bulk disputes for similar performance issues

‍

‍

Launch experience

This new workflow launched in October 2021 to all US carriers. These are just a few of the screens.