The document below is part of a package of materials that was produced while designing the BroadJump usability lab. This document was used partly to convince upper management of the value of the lab (an investment of approximately $160,00 in 2003) and to familarize them with the site plans. Working with a specialist builder, the lab was completed on time and on budget several months before BroadJump merged with a competitor.
BroadJump Usability Lab

 

Value of usability testing

 

Just as a quick overview, I've collected some interesting figures related to the costs and benefits of usability and usability testing.

 

Cost of bad usability

Sun Microsystems has compiled this useful list:

  • 63% of all software projects overrun their budgetary estimates, with the top 4 reasons all related to unforeseen usability problems (Lederer and Prassad 1992). 
  • The percentage of software code that is devoted to the interface has been rising over the years, with an average of 47-60% of the code devoted to the interface (MacIntyre et al. 1990). 
  • Ricoh found that 95% of the respondents to a survey never used three key features deliberately added to the product to make it more appealing. Customers either didn't know these features existed, didn't know how to use them, or didn't understand them (Nussbaum and Neff 1991). 
  • 80% of maintenance is due to unmet or unforeseen user requirements; only 20% is due to bugs or reliability problems (Martin and McClure 1993; Pressman 1992)

 

Benefits of good usability
  • Design changes due to usability work at IBM resulted in an average reduction of 9.6 minutes per task, with projected internal savings at IBM of $6.8 Million in 1991 alone (Karat 1990).
  • Usability engineering has demonstrated reductions in the product-development cycle by over 33-50% (Bosert 1991). 
  • Usability reduces engineering/development costs and facilitates speed to market. 
  • Usability reduces testing and quality assurance costs. 
  • Usability reduces sales costs and shortens sales cycles. 
  • Usability can decrease production costs while improving profit margins. 
  • Usability improves customer Return on Investment. 


 

Usability Lab roles and users

 

With the significant investment BroadJump is making in the Usability Lab, it is imperative that we publicize and engage all departments in using the lab so that it will be used at all times. I will take ownership of this task for the near term and evaluate the time demands as we get up to speed. Between recruiting test subjects, scheduling tests, and editing/publishing the collected data, this will likely become a full-time job. At a higher level, the Director of Best Practices will have ownership of the lab management and processes.

 

Customers for the lab will span all BroadJump departments:

 

  • Tech Pubs: Document verification--observe subjects following procedures and instructions to determine validity of documentation.
  • Training: Develop/enhance CBT content by observing subjects using training materials.
  • Product Management development research: Product prototype testing--using HTML or Flash prototypes, bring in subjects to evaluate the basic usability of product concepts as well as feature sets.
  • Marketing: Focus group testing on current or proposed BroadJump products; competitive analysis of other company's products.
  • MarCom: Verify corporate website navigation, content and graphics.
  • Professional Services: "Smoke test" new usability features mandated by customers on BroadJump reference products, use feedback to confirm features or steer customer to better solution.
  • Sales Engineering/Customer Satisfaction: Evaluate deployment issues (i.e., leaky pipe problems) with subjects to identify both customer fixes as well as product fixes.
  • Test/QA: Confirm that products heading to GA conform to PRD and identify problems before final build.
  • Sales: Demonstrate usability of BroadJump products via test subject viewing by customers.
  • Partner Program: Evaluate potential partner's products for viability.

 

In addition to internal customers, at some later date we may also rent the lab out to local groups (other software companies or usability labs) which will provide a small revenue stream.

 

Ongoing costs

 

I have reviewed the lab plan developed by Operations and am working with Gary Kloc on refinements, specifically the media capture equipment and general floorplan. I should point out that there will also be several items that will incur ongoing costs for the lab:

 

  • Media--Tapes, CDs, potential printing costs if a report is to be widely distributed.
  • Test subject compensation--Test subject payment can range from free to $200 a session, depending whether we are using friends&family or professional admins for subjects. We will also have to supply snacks for the subjects while on site.
  • Recruiting service--This may be optional, but there are subject recruiting services that maintain and manage lists of potential test subjects. We must keep in mind that test subjects tend to be "consumable" in that once they evaluate a product, we cannot use them again on the same product. There is also the issue of finding subjects of the appropriate type for testing--novice, expert, modem user, broadband user, etc.
  • IT support and maintenance--IT resources will be required to keep the lab connections maintained as well as introducing faults in order to test the software. Eventually, this may be the responsibility of the Usability Lab manager.