A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://doi.org/10.3163/1536-5050.97.2.008 below:

Web usability testing with a Hispanic medically underserved population

Abstract Objectives:

Skilled website developers value usability testing to assure user needs are met. When the target audience differs substantially from the developers, it becomes essential to tailor both design and evaluation methods. In this study, researchers carried out a multifaceted usability evaluation of a website (Healthy Texas) designed for Hispanic audiences with lower computer literacy and lower health literacy.

Methods:

Methods included: (1) heuristic evaluation by a usability engineer, (2) remote end-user testing using WebEx software; and (3) face-to-face testing in a community center where use of the website was likely.

Results:

Researchers found standard usability testing methods needed to be modified to provide interpreters, increased flexibility for time on task, presence of a trusted intermediary such as a librarian, and accommodation for family members who accompanied participants. Participants offered recommendations for website redesign, including simplified language, engaging and relevant graphics, culturally relevant examples, and clear navigation.

Conclusions:

User-centered design is especially important when website developers are not representative of the target audience. Failure to conduct appropriate usability testing with a representative audience can substantially reduce use and value of the website. This thorough course of usability testing identified improvements that benefit all users but become crucial when trying to reach an underserved audience.

Highlights Implications INTRODUCTION

Lack of health literacy (the ability of the individual to find and use information to make informed health decisions) has been named one of the biggest problems in the United States by the surgeon general and the Institute of Medicine [1,2]. Low health literacy affects 90 million people nationwide and costs the United States $58 billion each year [2]. A 2008 report confirmed that the odds of seeking Internet health information were lower for those with lower household income [3].

The question remains, if a website were specially designed for those with low health and computer literacy, would it be used? Would use of this specialized site be different from typical web usage? Most importantly, if this population is more likely to report poor health and less likely to seek treatment [4], could such a website help individuals take preventive action and seek earlier interventions?

This report describes the development and testing of such a website, starting with the initial conceptualization and first draft of the site and continuing with a heuristic evaluation by a web usability expert and two subsequent rounds of usability testing with those in the intended audience. Although the concept of usability testing is not new in the health sciences library literature [5,6], no peer-reviewed reports have described thorough and multifaceted usability testing with those who need health information so much but remain outside the realm of typical web users. This comprehensive course of usability testing helped the developers avoid errors that would have limited use of the site, and it identified features that can benefit all users but are especially crucial when creating a website for an underserved audience such as the one described here.

Beginning of the Healthy Texas website

Starting in 1999, a coalition of forty community health organizations and community members, called the Women and Family Health Information Network (WFHIN), began meeting in San Antonio to address the difficulties that medically underserved individuals faced in obtaining health information and locating health resources. By 2002, this group had begun to develop a web gateway, called Healthy Texas <http://www.healthytexas.org>, for medically underserved individuals who lived in the thirty-eight counties of South Texas [7]. WFHIN found comprehensive national health websites but believed South Texans needed basic, clear, and regionally relevant information that was not available from other sites. Considerable investment was made in the early development of the site: WFHIN spent $25,000 on basic web layout and design services. Although untested with the target audience, this design was not embraced by WFHIN members, who felt it looked like a typical commercial website.

Later in 2002, librarians at the Briscoe Library at the University of Texas Health Science Center at San Antonio joined the WFHIN effort, helping to develop a site that would not duplicate established sites, such as MedlinePlus, and building a database of regional health care resources. The next three years were spent obtaining grant support, meeting with multiple groups to clarify priorities, and developing the initial site in English.

Target audience

It is imperative to have a clear understanding of demographics when designing and evaluating such a website. South Texas includes 38 counties south and west of San Antonio, extending to the Texas–Mexico border. According to information compiled by the South Texas Area Health Education Center, 34 of these counties have been designated as medically underserved areas by the US Department of Health and Human Services [8]. The population is 65% Hispanic, compared to 12.5% in the United States as a whole. A significant number (32%) have no high school diploma. This group often has unmet health needs and is less likely than the average Texan to visit a physician. Most South Texans speak a language other than English at home. Health practitioners say children may serve as interpreters for older family members, presenting another set of challenges [9] to health care and information seeking.

The following description of the intended audience for Healthy Texas was developed by WFHIN to guide website development:

Our target audience is family caregivers in the 38 counties of South Texas. They are mostly women, often moms, and often bilingual. In some families, the children help their parents find information on the web and translate it into Spanish. Our audience prefers clear, plain language or easy-to-read information. To avoid reinventing the wheel, Healthy Texas provides regionally relevant, culturally appropriate information. We are a gateway to health information, not a comprehensive website like MedlinePlus … Everything is audience tested. If it doesn't work for the audience, we do it again. [10]

The developers envisioned that the site would be used at community centers and public libraries and could become a basis for joint community programs with health organizations, providers, schools, churches, libraries, and more. It was thought that users might access the site with the help of others (called “connectors” here), including family members or lay health tutors, called promotoras, who had proved helpful in other library outreach initiatives [11,12].

Website description

The Healthy Texas website was envisioned as one that would deliver basic health information about the health priorities for the region, with local examples and links to other consumer health sites for more comprehensive information. It highlighted research studies that were conducted regionally. A unique feature of the site was community contributions, including art, poetry, and stories by individuals documenting their health experiences. The site was written at the sixth-grade level following principles of plain language and health literacy [13], reviewed with plain language software, and scored using the Flesch-Kincaid Readability Index [14]. It included:

  1. the Health Topics section with forty health topics drawn from areas of concern listed in Healthy Border 2010 [15]; eight individuals wrote the content, which was then edited by health professionals; The section provided links to MedlinePlus and EBSCO Health Library, if more in-depth information was desired

  2. a directory database of health resources for the region with 11,000 entries for health providers, facilities, and support groups in South Texas

In 2005, WFHIN and the library obtained grant funding for usability testing, as it was deemed especially important to gather usability data and perfect the site before further expenditures were made for translation into Spanish. As the characteristics of the target audience differed from those of the developers, it was agreed that usability testing could help reduce mistakes, saving time, effort, and potential embarrassment [16]. Realizing the difficulty of designing a website for individuals who are relatively unlikely to even use the web, the library contacted the School of Information at the University of Texas at Austin to locate a web usability expert. The recommended individual, who had twenty-five years of experience researching interface design, and his team of evaluators worked with the developers to conduct a course of user-centered design [17] of the site.

This report chronicles a two-year effort of three rounds of usability testing:

  1. heuristic evaluation, conducted by a usability expert

  2. usability study I, conducted remotely with a fairly representative sample of users

  3. usability study II, conducted at a representative location with users representative of the target audience

OVERALL OBJECTIVES

The objectives of the studies were to answer the questions:

THE HEURISTIC EVALUATION

Heuristic evaluation was employed to find obvious flaws before the Healthy Texas website was announced to end users. In heuristic evaluation, an expert conducts a professional review of the website using commonly accepted rules of good judgment [18]. The primary disadvantage of employing a heuristics-only evaluation is it does not employ representative users of the site. However, like usability testing, it is conducted by a usability expert, who brings experience from testing the usability of many other websites, and, compared to usability testing, it has several advantages:

  1. It can be completed quickly as opposed to usability testing, which requires planning and preparation, including locating and scheduling participants and evaluators as well as equipment and locations, designing the course of tasks to be performed; conducting the testing; and analyzing and reporting the results.

  2. The usability expert systematically explores every corner of a website, impractical in end-user testing, which typically must limit tasks to those that can be completed in sessions of no more than a few hours.

  3. It may be less expensive than complete usability testing.

Heuristic evaluation methods

In April 2005, the usability expert conducted the heuristic evaluation. With the target audience in mind, he reviewed all sections of the Healthy Texas website. He identified issues and made recommendations, prioritized as to severity (Table 1, online).

Heuristic evaluation results

The usability expert found twenty-three problems in navigation, terminology, inconsistency, and violation of web design standards. He reported the usability problems in four categories of severity: zero critical, five major, seven moderate, and ten minor. In addition, he found one “bug” (error in hypertext markup language [HTML] coding) and several “good things.” Usability experts include mention of good aspects of the design, both to help build credibility of the usability expert and to encourage website designers and developers not to change these parts of the design.

As the primary purpose of this paper is to address the generalizable findings associated with testing such a population, only a few examples of findings follow. Of course, the usability expert reported all problems to WFHIN leaders and the library website developers, and approaches to corrections were discussed. Among problems that were identified and corrected were:

  1. Navigation: The first version of the page included navigation bars at the left of the page and across the top. Some options were duplicated in both navigation bars. To simplify the appearance of the page and reduce confusion, the left navigation bar was removed.

  2. Images: The original images on the home page were of women. The evaluator noted that some might think the page was only for women and held nothing of interest for others. After looking for an appropriate image of a family without success, the images were replaced with a more engaging photo of a boy and other images were rotated.

  3. Inconsistent terminology: “Fitness” was used as a heading in one location, whereas “Health and Fitness” was used in another. Terminology was changed for consistency.

Heuristic evaluation discussion

The heuristic evaluation identified many problems at even the highest and most obvious levels of the site. Without the heuristic evaluation, users would have been confused and might have abandoned the site before exploration. Most problems were easily remedied. However, as in all heuristic analyses, the study did not entail actual user data, and, in this case, the usability expert was not a member of the target audience. The developers corrected most of the identified problems, but all agreed that representative end-user testing would be required to move forward, before the WFHIN leaders and the library website developers could be confident of the effectiveness of the design.

USABILITY STUDY I Study I objectives

Possible research foci and questions were formulated in a preliminary meeting between researchers and developers and in subsequent telephone conversations. These questions guided an end-user test in the summer of 2006.

Study I methods Participants

According to research conducted by Nielsen and Landauer, 31% of web usability problems can be identified by testing a single user and 85% of web usability problems can be discovered with a sample of 5 users. However, successful usability testing does require a representative sample, and subsequent testing (iterative design) is important [19]. The developers therefore requested that participants represent a cross-section of the target user base. From their previous experience with use of consumer health websites by this type of specialized audience, the developers defined 2 targeted user types:

  1. end users who were proficient at using a computer and the web independently

  2. connectors, who were experienced computer users and community opinion leaders who might introduce end users to the website or navigate the website at the direction of the end user.

Three participants from the end-user group and two from the connector group were recruited from possible users identified by WFHIN. All participants used English on a day-to-day basis. The end users were twelve, thirteen, and thirty-six years old and had least five years of computer experience; two had experience searching the web for health information. The connectors were forty-four and fifty years old, one of each gender, one a native English speaker, and one with at least five years' experience of searching the web for health information. Participants were not compensated directly but were offered grocery gift cards after usability testing in appreciation of their assistance.

Location

The location of the test participants made remote computer-based testing the most cost-effective approach. The remote sessions were hosted at the University of Texas Health Science Center at San Antonio Library and remotely administered from offices at the University of Texas at Austin. Each session lasted one hour and was attended by one participant and a session technician. The participant environment was a private room with a desk, computer, webcam, and speaker phone. The computers were installed with Windows Internet Explorer, webcam software, and the WebEx browser client for web-based conferencing.

Procedures

Prior to scheduling the WebEx conference, the session administrator sent each participant an email thanking the participant and providing instructions for the session. At the appointed time, the participant followed the uniform resource locator (URL) to the conferencing application and then telephoned the conference call number. The application allowed participants to share computer screens, so the session administrator observed and interacted with the participant remotely.

After logging into the WebEx conference, the participants were welcomed by the researcher and asked to consent to the session. The participants received an orientation to using WebEx and were asked to complete a questionnaire that asked for demographic and computer experience information. The researcher first asked participants for initial impressions of the Healthy Texas site. Then the researcher asked each participant to complete a series of tasks based on three scenarios [20]: a neck injury sustained in a motor vehicle accident, healthy eating, and diabetes.

During the tasks, participants were encouraged to work without assistance, but the researcher provided guidance if a participant needed help. Participants were encouraged to “think out loud” (verbalize their thoughts and rationales for performing actions) [21]. When the assigned tasks were completed, the researcher interviewed the participant. In the final step, participants completed a post-evaluation questionnaire to rate the website and describe their experiences.

Study I results

None of the participants had seen the Healthy Texas website before, and all reacted favorably to the aesthetics of the home page. They were comfortable with the density of information on the page. Two participants questioned the credibility of the site until they scrolled down and noticed its affiliation with the university. When asked what link interested them the most, four of five indicated they would click either the Learn about Health or a Health Problem link or the Health Topics top navigation link. Three of the participants thought the website was intended for a general audience, but two participants thought the imagery implied that the site was targeted at women or children. One participant commented that imagery could be more inclusive, showing a family or a diverse group of people.

Then participants were asked to use Healthy Texas to complete the series of tasks. Two participants expected a search feature, in which they could type in keywords based on the topic they were seeking. All participants commented that language throughout the site included medical jargon. Links from the page to other sites, without any indication that these sites were external, confused participants.

In each scenario, evaluators asked participants how they would go about finding specific information on the topic and then prioritize whom or what they would consult. A clear, ordered pattern emerged, all listing people first:

  1. family

  2. friends

  3. family doctor

  4. other health care professional

  5. books

  6. online resources

In the final questionnaire, participants' satisfaction ratings were largely positive, but there were negative ratings on important points, such as content comprehension. Participants commented that the information could still be clearer. One participant asked for more diagrams and illustrations to complement the text.

The participants liked several parts of the site. Three commented positively on the quality and reliability of the information on the site. The participants were unanimous that they would return to Healthy Texas and recommend it to a friend, family, or colleague and that they would use the website as a personal resource.

Developers had witnessed the growth of interactive features on health websites and speculated that Healthy Texas might need more features, such as the ability to calculate life expectancy or caloric intake. When asked whether participants would prefer greater interactivity or more clearly written information that was well organized and easy to find, participants chose the latter, four to one.

Based on the participants' actions and comments, the evaluators' recommendations were to:

Study I discussion

Study I corroborated some findings from the heuristic evaluation and supplemented them with additional findings that had not arisen in the earlier analysis.

The researchers and developers speculated that awareness of a website tailored for users with special needs, such as Healthy Texas, might actually change information-seeking behavior among the target audience. Participants had originally said that they were least likely to consult the web when looking for health information, and yet after using the site they reported they liked the Healthy Texas website, would use it again, and would recommend it to others.

The researchers thought that remote testing with this audience had worked well enough but felt that because personal relationships were so important to this audience, a local trusted expert, such as a librarian, would be needed at the remote site. The researchers were eager to corroborate and extend findings in face-to-face testing in the next phase.

In addition, researchers and developers agreed they were still concerned that participants were not sufficiently representative of the target audience, and so a second face-to-face end-user study was undertaken, in November 2006. The second study focused specifically on the usability of the website's health topics and directories and the overall reaction of users to the home page.

USABILITY STUDY II Study II methods Participants

Participants representing a wide range of computer and Internet experience, level of education, ethnicity, native language, and age were recruited through a neighborhood community center in southern San Antonio. Participants, some accompanied by family members, thus represented the target audience of the Healthy Texas website: diverse with a large percentage of Spanish speakers.

Location

The test was conducted in a neighborhood community computer lab in southern San Antonio. The test site was selected because it represented an actual location where the target audience might access the website.

Procedures

Participants signed a permission slip and completed a background questionnaire. They were given a brief orientation, in which they were informed of their right to end the test at any time. Following the orientation, participants gave their initial impressions of the Healthy Texas home page and then were asked to complete the following tasks, while thinking aloud:

Researchers observed the participants and noted any errors and difficulties. If participants were successful at completing tasks, they were asked to rate its difficulty. Otherwise, the task was recorded as incomplete. At the end of the test, participants received a follow-up questionnaire to evaluate their overall experience and were interviewed. Each test session lasted forty to sixty minutes.

Study II results

The researchers compiled a prioritized list of observed problems and recommended remedies. The problems were assigned a rating using the same system as in study I (Table 1, online). Of the 6 tasks presented to the 8 participants, or 48 tasks in all, a total of 38 were attempted and 27 (71%) were completed successfully. Due to time limitations, the remaining 10 tasks were not even attempted, yielding a lower completion rate than the researchers expected. Only 1 of the 6 tasks received a below-neutral rating on a 5-point scale rating tasks from very easy (5) to very difficult (1). Three others yielded median ratings just barely above “neutral” (3.5), 1 received a median rating of 5 (very easy), and 1 received a 4 (easy). Overall, the researchers characterized these scores as modestly successful, with room for improvement.

Participants had greater difficulty with the information-seeking tasks than the directory tasks and had particular difficulty understanding where to search for nutrition-related information, possibly because of the terminology used.

The satisfaction data were far more positive than the performance data, suggesting that participants had positive impressions of the website, despite any difficulties (Table 2). All questions yielded above-average ratings. Only one participant gave below-average ratings on as many as two questions.

Table 2.

Satisfaction rating results (medians) for study II (n = 8)

The lowest satisfaction ratings were given to questions related to the ease of use and ability to locate information in the Health Topics section. However, it was later determined that these participants were not searching for this information on the Healthy Texas website but rather had followed a link to an external site.

Participants gave the most positive rating to the question that asked about the likelihood of using the Health Topics section (specific to Healthy Texas and not an external link) to search for information in the future. This finding underscored the appeal of the Health Topics section of the website, despite some problems in usability, while also further emphasizing the need to make this section of the website as usable as possible.

As before, the findings were assigned one of four ratings. Fewer problems were identified in this round (eighteen compared to twenty-three in study I), and still no “critical” ones were identified. Six “major” problems and twelve lesser problems were identified.

Medical jargon continued to be a problem in this round of usability testing. As an example of a problem of moderate severity, participants found that some terms in the directory, for example, “cardiologist,” had not included a plain language equivalent. They were able to complete the task in this case, but completion required a moderate amount of effort. The phrase “heart doctors” was subsequently added to the category.

Study II discussion

Researchers were gratified to see improvements in performance, justifying the redesigns based on the heuristic evaluation and study I. Many lessons were still learned about usability testing with less experienced web users:

OVERALL DISCUSSION

This was a thorough, multifaceted course of user-centered evaluation of an emerging website, tailored to the challenges of a distinct audience that included individuals with language barriers and low computer literacy and health literacy. Following initial heuristic evaluation, the researchers tested five participants in study I and eight more in study II. This seems a small sample size, but usability testing typically does not require a larger sample. It does require iterative testing, as performed here [23], and further testing after translation to Spanish is in order.

Given that study II was associated with an outreach partnership, the researchers accepted all volunteers. However, the sample included a cross-section of possible users, so that at the end of that study, there was a good understanding of the strengths and weaknesses of the health information and resources website.

Although not in a peer-reviewed publication, Nielsen reported a study that showed lower-literacy online behavior was very different from that of higher-literacy users, and this may be relevant to the intended Healthy Texas audience. According to his research, these individuals do not scan text but read word for word, in a narrow field of view. They skip over text when it becomes too dense or complicated [24]. This finding supports allocating more time for usability testing of audiences with lower health and computer literacy.

Nielsen further stated that sites that target broad audiences, such as health sites, must make lower-literacy users a priority. His studies of a website for a pharmaceutical product for lower-literacy audiences that was redesigned according to his guidelines showed improvements in success rate in finding information, total time on task, and satisfaction for both lower-literacy and higher-literacy audiences. Even those who were capable of understanding complex information preferred the streamlined site.

Websites such as Healthy Texas, which include both locally developed content and links to externally created information, have high potential to confuse inexperienced users with inconsistent navigation, terminology, and style. Usability test administrators must be especially observant during usability tests and must ask clarifying questions during summary interviews. Otherwise, participants' reactions to sites not being tested (e.g., the external links in this study) may be mistaken for reactions to the test site itself. For this reason, it may be advisable to avoid remote usability testing with this audience.

Family was important to participants in these tests. They suggested that images should include more pictures of families. In addition, usability testing for this audience should accommodate family members who may serve as both translators and computer experts.

CONCLUSIONS

These studies demonstrate the value of conducting usability studies and consulting usability experts, especially when there are no preexisting design guidelines for the particular subject matter and audience and when the website developers are not representative of the target audience. To make best use of usability testing, standard practices had to be modified and confusing test results resolved. The resulting improvements will no doubt benefit all users of the site, but they may make a substantial difference in actual usage by individuals with computer and health literacy challenges.

The goal of Healthy Texas was to encourage medically underserved South Texans to find health information resources and early health care interventions. Costs of usability evaluation have been justified if they have helped Healthy Texas reach its potential.

Electronic Content

Table 1. Description of severity of problems.

Acknowledgments

The authors gratefully acknowledge the assistance of the Alamo Women and Family Health Information Network (WFHIN)—Deborah Kaercher, Carolyn Marshall, Jerry York, Joaquin Arriaga, and Yanira García-Bárcena—and the staff of the Briscoe Library, University of Texas Health Sciences Center at San Antonio. The WFHIN is to be commended for bringing such extensive and accessible health information to this deserving population and for allowing their emerging designs to be scrutinized by usability testing. Further, we thank the University of Texas at Austin School of Information for access to the Information eXperience (IX) Lab. This work was supported by donations from the SBC/AT&T Foundation and Frost Bank.

Footnotes Contributor Information

Mary Moore, Executive Director, Louis Calder Memorial Library, University of Miami, Miller School of Medicine, P.O. Box 016950, (R-950), Miami, FL 33101 mmoore@med.miami.edu.

Randolph G. Bias, Associate Professor, School of Information, The University of Texas at Austin, Mail Code D7000, Austin, TX 78712 rbias@ischool.utexas.edu.

Katherine Prentice, Education and Information Services Coordinator, Dolph Briscoe Library, MSC 7940, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX 78229-3900 prenticek@uthscsa.edu.

Robin Fletcher, Graduate Student, Department of Linguistics, The University of Texas at Austin, Austin, TX 78712 robin8tor@austin.rr.com.

Terry Vaughn, Senior Information Architect, Vertive, 3721 Executive Center Drive, Suite 100, Austin, TX 78731 tbvaughn@gmail.com.

REFERENCES Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Table 1. Description of severity of problems.


RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.3