RootED NWA Website Usability and Product Test
RootED NWA Website Usability and Product Test
Embedded Files
Executive Summary
Executive Summary
This report evaluates the usability and effectiveness of the RootED NWA non-profit website in conveying its mission and available resources. RootED NWA aims to empower parents to take an active role in their children's education, especially in the diverse Northwest Arkansas community.
The study recruited 5 participants who speak English as a second language to represent the target audience. Participants performed tasks on the website related to finding information, scheduling appointments, and making donations while thinking aloud. They then answered interview questions and completed the System Usability Scale (SUS) survey.
Key findings include confusing page titles and navigation, the contact form and appointment scheduler being inappropriately combined, lack of content on some pages, and overall poor information architecture. The average SUS score of 60.8 indicates below average usability.
Recommendations include reorganizing the navigation, separating the appointment process onto its own page, removing empty pages, utilizing images better, simplifying text, and implementing other design improvements.
While the site conveys RootED's mission reasonably well through text and images, the significant usability issues are likely impacting conversions and engagement. A more thorough redesign should be considered to improve the user experience.
Introduction
Introduction
The primary aim of this study is to evaluate the usability and effectiveness of the RootED NWA non-profit organization's website in achieving its intended goals. Specifically, the study seeks to assess the clarity and impact of the written content in conveying the organization's mission and available resources, the effectiveness of the digital images in supporting and reinforcing the organization's messaging, and the overall usability and user experience, particularly for the target audience of local parents, including those who may not be native English speakers.
The study employs a user-centered evaluation approach, combining task-based user testing with representative participants performing typical user flows on the website, post-task interviews to gather qualitative feedback on the user experience, including challenges, pain points, and areas for improvement, as well as the System Usability Scale (SUS) questionnaire to quantify participants' perceptions of the website's usability.
The significance of this study lies in its potential to identify specific areas where the RootED NWA website may fall short in effectively communicating its purpose, offerings, and meeting the needs of its target audience. The findings can inform data-driven recommendations for enhancing the website's content, design, and overall user experience, ultimately enabling the organization to better serve the local community, particularly parents seeking educational resources for their children. By combining multiple evaluation methods and gathering insights from representative users, the study aims to provide a comprehensive understanding of the website's strengths, weaknesses, and opportunities for improvement, ensuring that the organization's online presence aligns with its mission and resonates with its intended audience.
Background:
Background:
RootED NWA is a non-profit organization based in Springdale, AR, located within the larger North West Arkansas metropolitan area, a historically underserved area. [See apendix 1] The mission of RootED is to empower community initiatives to extend education beyond the classroom and to provide parents with the tools they need to take an active role in their children’s education. According to the existing text on the website:
“We work closely with under-resourced families of all socio-economic and cultural backgrounds to help them become familiar with the resources and support accessible to their children. We serve as a bridge between parents and public school districts, charter schools, independent schools, and private schools to help parents become confident and empowered champions for their child’s education.”
In essence, RootED NWA is seeking to bring the gap between formal educational institutions and the local community and help to close the achievement gap through community initiatives. The purpose of this website is to make the services the organization offers more accessible and to act as a platform for community dialogue.
Goals
Goals
The goals of this test are as follows:
● Assess the findability of information about homeschooling options and resources on the website.
● Evaluate the ease of connecting with the organization through consultation sessions.
● Assess the ease of making donations on the website.
● Determine if the written copy and images on the website successfully convey the mission of the organization.
● Evaluate the branding elements of the website and their impact on the users' feelings towards the organization.
● Obtain qualitative data on the users' experiences from tasks performed.
● Assess the users' understanding of the mission of the RootED NWA.
● Evaluate the perception of services offered by the organization and the reaction of the target audience to these services.
● Measure the impact of the website's images on the users' emotions and associations.
Methods
Methods
The target population for this study is Springdale, Arkansas, and the surrounding communities. This study will focus on those whose first language is not English. 25% of the population of Springdale is foreign-born, according to the most recent census (2020). Therefore, this will represent more than an insignificant portion of website visitors. Further, it is explicitly the goal of the stakeholder to target this demographic for their services.
Therefore, the following criteria were used to recruit test participants: Participants must speak English as a second language and be over the age of 25. It is essential they speak English as a second language to represent the significant foreign-born population of Springdale and the target audience of RootED NWA. [see appendix 4]
Recruitment was complicated by a variety of factors. These included but were not limited to, the following: the short time frame available to complete the study, the lack of financial funding from RootED for the study, and the necessity of participants not already being familiar with RootED NWA. Therefore, a convenience sample of 5 people within the defined demographic was used. The remaining three participants were recruited online, using appropriate ethical and informed consent procedures.
Test Design
Test Design
Participants were asked to follow the “speak aloud protocol” as they performed a series of tasks on the website. Following the test, users were asked several follow-up interview questions to assess the written copy and images on the website. Successfully conveying this mission is vital for increasing conversions on the website. In this context, conversions refer to donations, volunteer applications, consultation appointments, ect. Further, these interview questions evaluated the branding elements of the website. Finally, a System Usability Scale (SUS) was given to each participant to address overall system usability. [see appendix 5] Tests were conducted using the test hosting platform Looback.io. This platform allows for recording of both the user’s screen as well as webcam.
Figure 10: Table of SUS results
The Severity of the Findings
The Severity of the Findings
Severity was judged based on a five-point rating scale taken from Nielsen’s “Heuristic Evaluation” chapter.
• 0 - I don’t agree that this is a usability problem at all.
• 1 - Cosmetic problem only (need to be fixed if extra time is available on the project).
• 2 - Minor usability problem (fixing this should be given low priority).
• 3 - Major usability problem (important to fix, should be given high priority).
• 4 - Usability catastrophe (imperative to fix this before the app can be released).
Key Findings
Key Findings
Key Finding 1: Page Titles on the Navigation Menu
Key Finding 1: Page Titles on the Navigation Menu
Severity: 3
During the usability testing, participants had difficulty understanding the names on the navigation menu, making it difficult for them to find what they were looking for. Categorization of items is not intuitive for users, resulting in sever difficulty completing tasks.
Implications: The naming of pages on the navigation menu creates a barrier for users, who are unable to easily access the information they need. This can result in a negative user experience and a higher likelihood of users leaving the site.
Evidence: Participant P 3 took 4:48 seconds to locate information about homeschooling on the website. One of RootED’s primary missions is to offer information on educational options to users. To be 6 in line with the highest performing function tested, making a donation, users should be able to locate the page with information on homeschooling within one to one and a half minutes.
Further, 4 out of 5 participants selected at least one wrong page heading before finding the correct page for task 1. 3 out of 5 selected more than 2 incorrect pages prior to successfully completing the task.
Task 2 took participants an average of 3:39 seconds to complete, which is more than twice the amount of time for task 3. This is in spite of both tasks having a similar complexity. Participants expressed confusion about making appointments with staff under a page titled “Contact US.” For example, participant P1 stated the following:
“I would say right now I feel relieved that I found it [the appointment selector], but I wouldn't see ‘contact us’ in the same [category] as performing an action. Contact us sounds more like I asked a specific question?”
Figure 1: Current Navigation Bar with a gray border added for contrast
Recommendations:
● Review the names on the navigation menu to ensure they are clear and descriptive.
● Consider using alternative labels, which are more clear and more concise. These should accurately reflect the content that can be found within each section.
● Consider adding tooltips or hover text to provide additional information about each section on the navigation menu.
● Conduct a card sorting session to understand what categories of information users associate with alternative Navigation Menu titles.
Key Finding 2: Separate the Contact Form and Appointment Selector
Key Finding 2: Separate the Contact Form and Appointment Selector
Severity: 3
The usability of the Appointment Selector is severely impacted by its being located immediately next to the Contact Form
Figure 2: The Contact Form (left) is located on the same page as the Appointment Selector (right), with little distinguishing them as separate functions.
Implications: Users may fill out the incorrect form when they are trying to perform one of the functions. This has a serious impact on the usability of both, which are essential functions of the website. A parent who is attempting to make a consulting appointment with RootED NWA staff may, in error, submit a contact form rather than using the Appointment Selector. This could cause serious delays in their getting the assistance they need, or not receiving the aid they need entirely.
Evidence: 3 out of 5 participants either filled out the Contact Form or enquired if that was the correct action, before interacting with the Appointment Selector. Further, all 5 expressed confusion at the two forms being located directly next to one another. Participant 1 stated:
“I get this like two things at the same time so I don't know if I should uh fill this one to fill this one like right now after looking at it a bit I understand there are two different things but they should be like right next to each other.”
The two forms are unrelated in function, meaning it is entirely not necessary for users to complete the Contact Form to schedule an appointment
As shown in Figure 3 (right) the required information to make an appointment is to be input into a separate form after the appointment time has been selected.
Recommendations:
● The appointment Selector should be on a separate page from the Contact Form.
● As a separate page, the “name,” “email,” “phone,” and “note” fields should not appear as a drop own menu. Instead, they should be present on the page with the calendar elements.
● The words “checkout” and “cancel” need to be centered within their respective buttons.
Figure 3: A screen capture of the right-hand side of the screen showing the drop-down menu that appears when an appointment time has been selected.
Key Finding 3: Information Architecture
Key Finding 3: Information Architecture
The overall architecture of the site causes confusion amongst users and does not suite users’ existing mental models and associations.
Severity: 3
Implications: The navigation architecture on the website is ineffective, leading to a suboptimal user experience. The structure of website is organized in an unclear way, making it difficult for users to locate the information they need. As a result, users are often frustrated by the website and are sometimes unable to perform the tasks they set out to complete.
Evidence: Participant P5 failed to complete task 2 because he became too frustrated and stated that he could not locate the Appointment Selector. Prior to giving up on the task, he filled out the newsletter subscription form and the volunteer form, believing that one of these fields would complete the task.
Further, 4 of 5 participants were either visibly frustrated or stated their frustration while attempting to complete task 2. They expressed distress at not being able to locate the Appointment Selector where they thought it would logically have been located. Participant P1 even expressed his relief after locating the form, indicating the level of tension the failure to locate the form was causing.
On tasks one and two, nearly all participants selected multiple incorrect tabs while performing the basic function of the website. One of RootED’s primary missions is to connect families with educational resources. However, only 1 out of 5 participants correctly selected the page to obtain information on homeschooling on the first try.
Recommendations
● The navigation structure of the website should be reorganized to make it more intuitive and easy for users to find the information they need. This could include adding breadcrumb trails, using clear headings and labels, and ensuring that common tasks and information are easily accessible from the homepage.
● Information such as the Appointment Selector should be placed in a more logical and predictable location, to minimize user frustration. Consider using a search function or adding links to frequently accessed information in prominent places on the site.
Key Finding 4: “Education Environment”
Key Finding 4: “Education Environment”
Figure 4: “Education Environments” page
The “Education Environment” page consists of a single element, which lacks content. When users click on the link “read more,” the existing page immediately reloads.
Severity: 4
Implications: The lack of content and broken links on this page cause confusion and frustration for users. Further, it affects conversions of the site by causing this confusion. Users may select “Education Environments” when looking for information or resources on the site, and failing to find what they are looking for, may leave the site.
Evidence: 4 out of 5 participants clicked on the “Education Environments” during either task 1 or task 2, and expressed confusion at the consent of the page. Participant P3 summed up this frustration and confusion while performing task 1. While attempting to locate information on homeschooling they stated:
“I click on 'education environment' and then I would expect more text to be here to actually know what kind of education environments they are talking about.”
This evidence suggests that the existence of this pages devoid of content interferes with users attempting to access important information on the site by misleading them to an incorrect location.
Recommendations
● Consider eliminating the page entirely, as it currently serves no purpose and functionally is an obstruction to users of the site.
Key Finding 5: “Issues” page lacks content
Key Finding 5: “Issues” page lacks content
Similar to key finding 4, the “Issues” page lacks content and functions as an obstruction to users of the website
Severity: 4
Implications:
The lack of content on this page causes confusion and frustration for users. Further, it affects conversions of the site by causing this confusion.
Evidence:
The page has only a title and three lines of text, followed by three rows of space-holding elements.
Recommendations
Consider eliminating the page entirely, or at least until such a time as it is content to display.
Incorporate the “Issues” page as an element under another heading.
Figure 5: “Issues” page showing 1st of 3 rows of empty content.
Additional Design Recommendations
Additional Design Recommendations
General
Correct page overflow
Figure 6: Landing Page with Page Footer
Landing Page
Top image - Participants noted that the tone of the image of the home page creates a feeling of community action, which is what is intended. However, the image should be smaller within the viewport.
Image text - add padding to the right side of the text. The text should not be directly bordering the edge of the image.
Consider adding content - It may increase the usability of the site to locate some content from and links to some of the most frequently accessed pages. EX: Upcoming events, school options, about us.
Place the name and logo of the organization more prominently on the page.
Footer
Add a logo, name of the organization, and site map.
Add copy writen information.
Add text to the contact information, to increase accessibility for those using screen readers.
Consider general footer redesign.
Buttons:
Make sure all text on buttons is centered - some of the text in buttons, such as “cancel” and “checkout” are not centered.
Reverse location of “checkout” and “cancel” on Appointment Selector - “Canel” on the left-hand side, “Checkout” on the right.
All asterisks need to be located near the note saying “required,” or similar.
Add logo and name of organization prominently to Navigation Bar,
Forms
All labels for required fields should be in bold to increase the visibility and prominence of the text.
Add space and padding around elements (labels, text field, etc)
On the volunteer form, change “required” to an asterisk for consistency Areas of Interest button label. Further, organized as a vertical list, rather than a horizontal one.
Figure 7: Volunteer Form
Figure 8: Lower Portion of “Contact Us”
Images:
Participant feedback indicates that the tone of the images that appear on the website illicit feelings of community action, education, and generally positive feelings. This reaction meets the requirements for the tone of the site, and effectively communicates the mission of the organization
However, text-heavy pages, such as “About Us” and “School Options,” would benefit from the addition of more similar images.
Consider moving images from places where they serve no explicit purpose, such as the button “Contact Us” and place utilize them to break up text-heavy pages. See figure 10.
Written Copy:
The general tone and content of the website copy effectively convey the mission of the organization, as well as the resources available on the site.
However, the content of pages, especially text-heavy pages, should be simplified and made more concise. Long paragraphs of text can cause users not to read the content, thereby missing vital information. Further, participants expressed that as non-native speakers of English, the larger amounts of text could be difficult to understand.
Consider linking to outside resources, additional pages, or to “contact us” on text-heavy pages, especially those with resources such as “School Options.”
“Our Story” page needs to be more explicit about the mission and resources of RootED NWA and what they are trying to achieve, it is unclear here do not prevent their overall ability to utilize the system, however, they also indicate that the issues are severe enough that participants would not be likely to use the system frequently.
Figure 9: Example of using images to break up text
System Usability Scale (SUS) results
System Usability Scale (SUS) results
For a SUS, anything below 68 is considered below average. The average score on this website is 60.8 by the results of this study. Analyzing the results as broken down by question, it is apparent that participants felt the system was usable overall, but contained significant flaws. Participants ranked the systems as being usable for someone without a high level of technical skill, but it ranked poorly in terms of integration. This is likely to do the difficulty of navigating the site architecture, as well as the issues of non-functional pages. Participant feedback indicates that the usability issues within the system
Conclusion
Conclusion
Therefore, it can be concluded from this data that the usability issues present on the site are likely to impact conversions and the website's stickiness. That is, these issues will prevent users from utilizing functions on the website, and from returning more than once. Only more dedicated users will be repeated visitors to the site.
By implementing the design suggestion contained within this document, both the conversion and stickiness of the site could be improved. The outlined design suggestions should alleviate the majority of these issues. However, a more thorough system re-design should be considered, due to the depth of the issues present.
Appendix
The community of Springdale has a population of 87,609. 1 The NWA area has a metropolitan population of 534,904,. The city of Springdale is 40% Hispanic, 7.4% Pacific Islander, 3% Black, and 54% White. The average home price was $169,300, as compared to the national average of $357,319. The realities of local education, however, do not necessarily reflect the community at large. 94% of teachers in the NW Arkansas Metropolitan area are White. This also is in spite of school-aged children being even more diverse than the region overall. While the school district has pledged to increase the number of “people of color in teaching positions” by 50% in the next year, that would only bring the total to 9% of teaching staff.
Tabel detailing time on each task by participant
3.Table of successful and successful completion of tasks by the participant. Check marks signify successful completions.
4. Approximately half of the immigrants in the United States are considered not to be proficient in English. 4 Therefore, to adequately address the usability of the site, it is important to address its written copy with this demographic. Secondly, they had to be over the age of 25. Given that only 7% of women in the United States under the age of 20 are foreign-born, it is more likely that our focus population is going to be over the age of 20. The likelihood increases over the age of 25, as that percent increases into the 30 - 40 age bracket. 5 25 was chosen, rather than 30, due to the importance of including young families who may be in greater need of community support organizations, such as RootED. Further, according to a report published by the Pew Research Center “…6% of all new U.S.-born mothers were teens in 2014, this share was just 2% among the foreign-born.” From this data, it can reasonably be assumed that the percentage of immigrant parents with school-age children (at least 5 years of age) under the age of 25 in Springdale is approximately 2%, in reflection of the national average.
5. System Usability Scale questions, as in the order seen on the questionnaire.
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
System Usability Scale(SUS) Results
Upon completion of the 3 tasks and 4 interview questions, participants filled in a System Usability Scale. A System Usability Scale(SUS) is a widely-used tool for measuring the overall user satisfaction and usability of a system. It consists of a set of Likert scale questions that assess different aspects of the user experience, such as ease of use, usefulness, and overall satisfaction.each question is answered on a five-point Likert scale, ranging from "Strongly disagree" (1) to "Strongly agree" (5). The five-point scale is to allows respondents to rate their level of agreement with each statement, providing a way to quantify the user experience An SUS score of 68 is considered to be average.
6. Script and tasks for RootED NWA User Test 2023
Introduction and instructions
Introduction and instructions
“I am going to ask you to perform a series of 3 tasks, and then ask you 4 follow-up questions. For the questions, you may also need to look around on the website. Then, I will ask you to fill out a 10-question questionnaire. Don't worry its all on a one to five scales, so there is no short answer!
As you go through the tasks, I want you to speak out loud and tell me what you're thinking and doing as you go through the tasks. I want to know what you like about it, what you don't like, and any problems or issues you encounter. This is called the think-aloud protocol.
For example, as you use the website, if you're trying to find a specific item, tell me what you're looking for and how you're trying to find it. If you're confused by a certain button or feature, let me know. If you're having trouble completing a task, tell us what's difficult.
It's important to be as specific as possible and to speak your thoughts as they come to you, don't try to remember them. We want to know what's on your mind in real time.
It's also important to know that your feedback is valuable and it will help us improve the product or service. Don't worry if you encounter problems or if you don't know how to do something, that's the whole point of this testing session.
When you believe that you have completed a task, please verbally tell me that you have done so.
Please let us know if you have any questions or concerns before we start. And remember, don't hesitate to speak up, we're here to listen."
Tasks
Tasks
Find out more about homeschooling on the website.
Sign up for a counseling session on March 7 at 3 PM. Again, please fill out the entire form, but do not hit the submit button. You can just verbally tell me when you would do so.
Make a $50 dollar donation. Select the “test donation” option before hitting the submit button.
Questions
Questions
Can you tell me what you think the mission of the organization is as specific as you can?
What services does this organization offer, and to whom?
How do you feel about this organization after going through this website?
How did the images you saw on the sight make you feel? Did they remind you of anything?
Questionnaire
Questionnaire
“Great, thank you so much for that. The last thing here is a quick questionnaire that will help me understand your experience using the website. It's just 10 questions, all on a scale of 1 to 5, so it should only take a few minutes.”
Sources Cited
U.S. Census Bureau quickfacts: Springdale City, Arkansas. https://www.census.gov/quickfacts/fact/table/springdalecityarkansas/PST045221 (accessed March 4, 2023).
Rosa, J.D. (2023) Northwest Arkansas continues to lead state growth, 13th fastest-growing U.S. Metro over past decade, Talk Business & Politics. Available at: https://talkbusiness.net/2020/03/northwest-arkansas-continues-to-lead-state-growth-13th-fastest-growing-u-s-metro-over-past-decade/ (Accessed: March 4, 2023).
Rosa, J.D. (2023) Northwest Arkansas continues to lead state growth, 13th fastest-growing U.S. Metro over past decade, Talk Business & Politics. Available at: https://talkbusiness.net/2020/03/northwest-arkansas-continues-to-lead-state-growth-13th-fastest-growing-u-s-metro-over-past-decade/ (Accessed: March 4, 2023).
English proficiency among U.S. immigrants, 1980-2017 (2019) Pew Research Center's Hispanic Trends Project. Pew Research Center. Available at: https://www.pewresearch.org/hispanic/chart/immigrant-statistical-portrait-english-proficiency-among-u-s-immigrants/ (Accessed: March 4, 2023).
Golden, A. (2022) NW Arkansas is getting more diverse, Axios. Available at: https://www.axios.com/local/nw-arkansas/2022/07/29/nwa-diversity-racial-ethnicity (Accessed: March 4, 2023).
Affairs, A.S.for P. (2013) System usability scale (SUS), Usability.gov. Department of Health and Human Services. Available at: https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html (Accessed: March 4, 2023).
Page updated
Google Sites
Report abuse