Below is a fairly typical example of many usability test session write-ups. Test sessions ranged from simple over-the-shoulder observation to video documentation to the full one-way mirror lab tests with screen capture. The depth of the write-ups would necessarily reflect the quantity of information gathered. |
|
ProWatch Exchange 3.0 Usability Test: HCR
Test Date
August 5, 2004
Test Location
HCR
Test Protocol
Test Subjects
USER 1 Ben B. (BB): Configuration deployment. Never used current portal since it is not part of his job duties. Novice user.
USER 2 Jason S.(JS): Network analyst. Uses current portal frequently to run reports for his manager; some ticket tracking. Experienced user.
USER 3 Corey L. (CL): Senior network analyst. Handles escalation from callcenter, VIP support. Uses current portal mainly reports with some ticket tracking. Experienced user.
USER 4 Cheryl P. (CP): Product engineer. Supports all corporate servers. Uses current portal for ticket tracking. Experienced user. NOTE: Cheryl viewed the portal demo prior to her test session (she was a replacement subject).
Results Summary
The following is a summary of the main issues brought up during the usability testing. More detail is contained in the individual task notes (below). The summary items are ranked according to overall priority in terms of usability.
HIGHLIGHTS
LOWLIGHTS
AMBIENT
Time on Task Results
RecommendationsThe following recommendations are considered preliminary since this was only the first round of usability testing. Easy (non-Dev) fixes will implemented before the next round of testing. The recommendations are ranked according to usability priority.
Detailed Task Notes
Task 1: Describe the current overall health of the network.
USER 1 User flow: Right-clicked on tree and looked at Reports menu; selected Dashboard, clicked WAN tab; read ticket list, confused about non-date order; went back to tree and right-clicked to Reports; ran WAN Availability, cancelled; went to Network Details; end. Result: Success with issues. Time on task (TOT): 325 seconds. Analysis: Since he'd never used a portal as part of his job, he was unclear on the purpose of the site and tended to click around randomly. He still judged correctly that the network was doing well in general.
USER 2 User flow: Expanded network tree, clicked on red element; sent to Network Details, examined the list of open tickets, "Just Austin has problems, HQ is up with problems." Correct. Result: Success. TOT: 60 seconds.
USER 3 User flow: Expanded network tree manually, clicked on Austin (red); viewed ticket list but did not click for details; clicked on Home, read recent ticket list; correctly identified which sites were up, with and without problems as well as differentiating between fail and caution tickets. Result: Success. TOT: 95 seconds.
USER 4 User flow: Clicked Home/Dashboard, viewed Combined tab; saw the Health icon but interpreted the arrow pointing to the current health level as the desired health level (opposite of design); commented that she wasn’t sure of the context of health, what was the definition, and that “infrastructure” meant “everything” to her regardless of tab context; after clicking to several other Dashboard tabs she noticed the health level arrow changing and corrected her interpretation, commented that the tickets listed in each tab didn’t seem to match tab names; correctly judged network health. Result: Success with issues. TOT: 215 seconds. Analysis: The confusion with the status icons was the first time witnessed since portal testing began in 1/04 so it will not be weighted too heavily. Regardless, she correctly interpreted the icon once she saw several examples. The comment on the confusing “infrastructure” label is more valid and should be revised with Marketing input.
Task 2: Check the monthly average availability of the wide area network.
USER 1 User flow: Had already visited Reports so he went directly there and correctly selected the WAN Availability report. Result: Success. TOT: 40 seconds.
USER 2 User flow: Right-clicked on tree, selected Reports; sent to Reports; right-clicked on the WAN product icon in the Filters section; then used Reports menu to select WAN Availability report. Result: Success with issues. TOT: 125 seconds. Analysis: He assumed that he could select a particular network segment on to which to run the WAN report, but since the tree was hidden in Reports, he tried to click the WAN filter button. This was reinforced by being able to right-click on the tree--he thought he was running the report on just that element. When presented with the concept of business views (custom trees), he said that would satisfy his need for running reports on discrete network segments.
USER 3 User flow: Clicked on Reports; clicked on WAN Availability; entered date range and ran report. Result: Success. TOT: 35 seconds.
USER 4 User flow: Clicked to Reports; selected WAN Availability report, ran it; commented that if the report ran more than a minute, she would click the “Notify on Completion” button. Result: Success. TOT: 20 seconds.
Task 3: Look at any open ticket and read the most recent ticket activity.
USER 1 User flow: Clicked on Home; returned to Dashboard; clicked a ticket in open ticket list; viewed ticket in the Trouble Ticket sub-tab; scrolled down to last activity. Result: Success. TOT: 25 seconds.
USER 2 User flow: Clicked on filter button again; clicked on tree element; sent to Network Details, clicked on ticket list; read last activity. Result: Success with issues. TOT: 35 seconds. Analysis: Clicked on filter button since he felt is was a shortcut icon, did not see the button as a button in the down state. May be related to the fact that we shown the filter icons in the Network Details list as well.
USER 3 User flow: Clicked Home/Details; clicked Show All link; sent to Network Details; clicked first ticket in list; scrolled activity grid of details; clicked to reverse the grid sorting, momentarily confused since nothing happened, clicked again and sort reversed, most recent activity at top of list. Result: Success. TOT: 60 seconds. Analysis: Since the sort icon does not appear in grid initially, he had no way of knowing what column was being sorted and in what order. Sorting and the appearance of the icon happen at first click, then it takes a second click to do what he wanted (reverse the sort).
USER 4 User flow: Clicked to Reports; clicked on tree filter icon button; clicked to Home/Dashboard; clicked on a ticket in the Health ticket list, sent to Network Details; scrolled to bottom of details and read activity. Result: Success with minor issue. TOT: 40 seconds. Analysis: When asked why she clicked on the filter button, she said that she wanted to “see a WAN ticket” and that she thought that was a shortcut/link. Recommend changing look of buttons to make more button-like (color down state, stronger drop-shadow).
Task 4: Ask NetSolve to make changes to some part of the network.
USER 1 User flow: Returned to Home/Dashboard; right-clicked on ticket list to find "New Ticket" option; momentary deadend; noticed Support tab, clicked; clicked Request Service; filled out and submitted form. Result: Success with issues. TOT: 285 seconds. Analysis: Thought Support referred to portal support, not NS support.
USER 2 User flow: Clicked to Support/Problems and Requests; clicked on Service Request; filled out form; sent. Result: Success. TOT: 50 seconds. Note: Selected tree elements in the form text field look like they can be selected on form, caused some confusion.
USER 3 User flow: Clicked to Support/Problems and Requests; clicked Service Request; filled out form, directed to clicked Save instead of Send; when asked to edit the form just completed, he could not see the Edit link on the list; found it, opened form; clicked Send; "The confirmation screen looks ugly". Result: Success. TOT: 95 seconds. Analysis: He felt the Edit link was too subtle in the Saved Request list--the color and style was too similar to normal text.
USER 4 User flow: Clicked to Support/Problems and Requests; clicked Service Request; completed and sent form. Result: Success. TOT: 60 seconds.
Task 5: Check the reliability trend for all network infrastructure.
USER 1 User flow: Clicked on Reports; selected Engineering Analysis report, no information; selected Network Details, returned to Dashboard; clicked on Reliability. Result: Success. TOT: 170 seconds. Note: Wanted reliability to be a graph, not an icon.
USER 2 User flow: Clicked Reports; surveyed the reports menu; clicked Network Details; right-clicked on tree to return to Reports; clicked Home/Details; clicked Dashboard, never saw Reliability section. Result: Fail. TOT: 430 seconds. Analysis: He felt there had to be a reliability report and was frustrated that there wasn't one available. Although he ended up at the correct screen, he was still focused on finding a report. The fact that he uses the current portal almost exclusively for running reports may have influenced his behavior.
USER 3 User flow: Clicked on Home/Dashboard; clicked on ticket list for Health; sent to Reports; scanned all reports in menu; deadend. Result: Fail. TOT: 200 seconds. Analysis: He had (quite logically) decided that reliability was displayed in a report and since he'd left Dashboard without seeing anything, decided there was nowhere else to check.
USER 4 User flow: Clicked straight to Home/Dashboard; viewed Reliability section. Result: Success. TOT: 15 seconds.
Task 6: See if there is a serial number available for any network device.
USER 1 User flow: Viewed Dashboard (already open); clicked Home/Details, selected ticket; sent to Network Details; clicked on Inventory sub-tab; clicked on list icon, then link; hesitated, then drilled down on inventory details screen, found serial number. Result: Success. TOT: 80 seconds. Analysis: Was momentarily confused by hosted detail screen when he clicked on the upper list.
USER 2 User flow: Clicked on tree; sent to Network Details, clicked on Inventory sub-tab; clicked on inventory list; momentarily stumped by hosted inventory screen, he finally clicked open an item and found a serial number. Result: Success with minor issue. TOT: 45 seconds. Analysis: He expected to be presented with just a single screen for inventory (as in the trouble ticket details), not what appeared to be another list. Known problem with the hosted screen.
USER 3 User flow: Clicked Home/Details; clicked tree; sent to Network Details, clicked on Inventory sub-tab; clicked on device name in inventory list, viewed list and momentarily deadended; clicked open lower inventory list, found a serial number. Result: Success with an issue. TOT: 155 seconds. Analysis: As with Jason, he was puzzled by the presentation of a list/tree in the lower details pane when other detail screens showed just one pane of information. Commented that if the list was necessary, it should at least automatically scroll to and open the “tree” to the selected inventory device.
USER 4 User flow: Clicked on tree, sent to Network Details; clicked on Inventory sub-tab; clicked on device in upper list; clicked on inventory tree/list, found a serial number. Result: Success. TOT: 45 seconds.
Task 7: Tell NetSolve about a problem with the network element.
USER 1 User flow: Clicked Support; clicked Report Problem; filled out and submitted form with no errors. Result: Success. TOT: 50 seconds.
USER 2 User flow: Clicked Support; clicked Report Problem; filled out and submitted form with no errors. Result: Success. TOT: 65 seconds. Note: He used tabbing for all navigation in form.
USER 3 User flow: Clicked to Network Details/Trouble Tickets; right-clicked on upper list (no result); then right-clicked on tree (no menu for Support); momentary deadend on Trouble Ticket sub-tab; saw Support tab, clicked; clicked on Report a Problem button; filled out and submitted form with no errors. Result: Success with minor issue. TOT: 160 seconds. Analysis: Based on normal Windows application functionality and being an advanced user, he expected right-click functionality on the upper list, and then when the right-click worked on the tree, did not find the menu he needed. We should add a Support right-click menu to the tree before release.
USER 4 User flow: Clicked to Support/Problems and Requests; clicked Request Service button; filled out and submitted form with no errors. Result: Success. TOT: 45 seconds.
Task 8: Look at an alarm and read the time and date it occurred.
USER 1 User flow: Clicked Home/Details; clicked on USA in tree, sent to Network Details; selected Alarms sub-tab, viewed time/date. Result: Success. TOT: 25 seconds.
USER 2 User flow: Clicked on tip of network tree; sent to Network Details, he hesitated, then clicked Alarm sub-tab. Result: Success. TOT: 35 seconds. Note: He wanted to be able to search for the alarm in the Quicksearch.
USER 3 User flow: Clicked on tree, sent to Network Details; clicked on Alarms sub-tab, read timestamp. Result: Success. TOT: 20 seconds.
USER 4 User flow: Clicked to Home/Dashboard; clicked on product tab with green icon (all good) so there was no ticket in the Health ticket list, momentary deadend; clicked Home/Details; clicked ticket in Latest Trouble Ticket section, sent to Network Details; clicked on Alarms sub-tab, read timestamp. Result: Success with minor issue. TOT: 70 seconds. Analysis: Green tabs will never show tickets which is potentially confusing to users. For all other “empty” tab lists we provide a default string instructing the user what to do (i.e., “Click a tab with a checkmark.”)--we do not have a default string in the Dashboard lists. Recommendation: Add default text to all Dashboard lists.
Task 9: Look at a closed ticket and read the last ticket activity.
USER 1 User flow: Clicked Network Details, looked at Trouble Ticket sub-tab; clicked Support; clicked Home/Details, selected Show All; sent to Network Details; returned to Home/Details; end. Result: Fail. TOT: 450 seconds. Analysis: Thought Search only was for portal content, not for ticket/inventory information. Brainstormed for better terms for "search" but came up empty.
USER 2 User flow: Clicked to Network Details/Trouble Tickets; clicked on Home/Details; right-clicked on open ticket list; clicked on Search; performed search on closed tickets. Result: Success. TOT: 195 seconds. Analysis: He wanted a history option available on the home page, or a right-click option for listed tickets. Did not associate the Search tab with searching data, only searching portal. Could not offer a better term, agreed that "search" was the best word for the tab, but a history/search link on Home would help.
USER 3 User flow: Clicked on Network Details; right-clicked on upper list (no result); clicked on Search tab; clicked on Search Trouble Tickets menu item; completed and submitted query form. Result: Success. TOT: 55 seconds. Note: He wanted a “History” link/button on Trouble Tickets sub-tab that would send him to the appropriate search page.
USER 4 User flow: Clicked on tree, sent to Network Details; clicked on Reports, selected Ticket Metrics/Open vs. Closed Tickets, did not run report; clicked Search; selected Trouble Tickets, filled out and submitted query with no errors. Result: Success. TOT: 155 seconds.
Task 10: Close the ticket that was opened on the network element named in Task 7.
USER 1 User flow: Clicked on Home, selected Show All; sent to Network Details/Trouble Tickets, clicked on list, didn't see element name; clicked on tree element, sent to Network Details/Trouble Ticket; clicked Close button. Result: Success. TOT: 130 seconds.
USER 2 User flow: Clicked to Network Details, viewed Inventory sub-tab; clicked on Search; returned to Inventory, clicked on ticket link; sent to legacy ticket screen, no Close button available; deadend. Result: Fail. TOT: 315 seconds. Analysis: Never revisited Trouble Ticket sub-tab in Network Details, perhaps due to previous search task for closed tickets. Found a ticket on the inventory screen which would've worked if he had not been sent to the legacy ticket screen. We may see this often for users that rely on the Search screen or Inventory (Quicksearch properly sends the user to the new ticket screen).
USER 3 User flow: Clicked on Home tab; clicked on ticketed device in network tree; clicked on upper list item; in lower details pane, clicked on Close Ticket button. Result: Success. TOT: 40 seconds.
USER 4 User flow: Clicked Expand All tree control; clicked on device, sent to Network Details; clicked on ticket in upper list; clicked on Close Ticket button in ticket details. Result: Success. TOT: 35 seconds.
General Feedback
Best feature
USER 1 Liked the tree for its feedback on status and the ability to show "entire network".
USER 2 Liked layout and look&feel, tree status display. Would like tooltips/alt tags for main tabs to indicate functions.
USER 3 Liked the tree for status and navigation. Liked the Quicksearch functionality.
USER 4 Liked the Quicksearch functionality.
Most confusing or unclear feature
USER 1 None of the display lists showed the sort order when first viewed, had to clicked the column headings to get sort indicator.
USER 2 Checkmarks on Inventory Details sub-tab had no meaning. Inventory details (legacy screen) was confusing and somewhat overwhelming.
USER 3 In some places the hyperlink style looked too much like the standard text.
USER 4 Looking for old tickets was unnecessarily difficult. Did not understand the purpose of alarms. Questioned the need for the network tree since the Dashboard and Home/Details displayed status.
What's missing?
USER 1 Would like to see the list of all tickets (open, closed, etc.) and then filter and sort the list to desired view.
USER 2 Universal search--the ability to search for any piece of data on one screen. Would also like to add links/windows to search other databases as well.
USER 3 More right-click functionality.
USER 4 Nothing. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||