Conducting Accessibility Research In An Inaccessible Ecosystem
Email Newsletter
Weekly tips on front-end & UX.
Trusted by 200,000+ folks.
Flexible CMS. Headless & API 1stEnsuring technology is accessible and inclusive relies heavily on receiving feedback directly from disabled users. You cannot rely solely on checklists, guidelines, and good-faith guesses to get things right. This is often hindered, however, by a lack of accessible prototypes available to use during testing.
Rather than wait for the digital landscape to change, researchers should leverage all the available tools they can use to create and replicate the testing environments they need to get this important research completed. Without it, we will continue to have a primarily inaccessible and not inclusive technology landscape that will never be disrupted.
Note: I use “identity first” disability language (as in “disabled people”) rather than “people first” language (as in “people with disabilities”). Identity first language aligns with disability advocates who see disability as a human trait description or even community and not a subject to be avoided or shamed. For more, review “Writing Respectfully: Person-First and Identity-First Language”.
Accessibility-focused Research In All Phases
When people advocate that UX Research should include disabled participants, it’s often with the mindset that this will happen on the final product once development is complete. One primary reason is because that’s when researchers have access to the most accessible artifact with which to run the study. However,
“
Prototype Research Should Include Disabled Participants
In general, the iterative prototype phase of a project is when teams explore various design options and make decisions that will influence the final project outcome. Gathering feedback from representative users during this phase can help teams make informed decisions, including key pivots before significant development and testing resources are used.
During the prototype phase of user testing, the representative users should include disabled participants. By collecting feedback and perspectives of people with a variety of disabilities in early design testing phases, teams can more thoughtfully incorporate key considerations and supplement accessibility guidelines with real-world feedback. This early-and-often approach is the best way to include accessibility and inclusivity into a process and ensure a more accessible final product.
If you instead wait to include disabled participants in research until a product is near final, this inevitably leads to patchwork fixes of any critical feedback. Then, for feedback not deemed critical, it will likely get “backlogged” where the item priorities compete with new feature updates. With this approach, you’ll constantly be playing catch-up rather than getting it right up front and in an elegant and integrated way.
Accessibility Research Can’t Wait Until The End
Not only does research with disabled participants often occur too late in a project, but it is also far too often viewed as separate from other research studies (sometimes referred to as the “main research”). It cannot be understated that this reinforces the notion of separate-and-not-equal as compared to non-disabled participants and other stakeholder feedback. This has a severe negative impact on how a team will view the priority of inclusive design and, more broadly, the value of disabled people. That is, this reinforces “ableism”, a devaluing of disabled people in society.
“
The problem is that even when a team is on board with the idea, it’s not always easy to do inclusive research, particularly when involving prototypes. While discovery research can be conducted with minimal tooling and summative research can leverage fully built and accessible systems, prototype research quickly reveals severe accessibility barriers that feel like they can’t be overcome.
Inaccessible Technology Impedes Accessibility Research
Most technology we use has accessibility barriers for users with disabilities. As an example, the WebAIM Million report consistently finds that 96% of web homepages have accessibility errors that are fixable and preventable.
Just like websites, web, and mobile applications are similarly inaccessible, including those that produce early-stage prototypes. Thus, the artifacts researchers might want to use for prototype testing to help create accessible products are themselves inaccessible, creating a barrier for disabled research participants. It quickly becomes a vicious cycle that seems hard to break.
The Limitations Of Figma
Currently, the most popular industry tool for initial prototyping is Figma. These files become the artifacts researchers use to conduct a research study. However, these files often fall short of being accessible enough for many participants with disabilities.
To be clear, I absolutely applaud the Figma employees who have worked very hard on including screen reader support and keyboard functionality in Figma prototypes. This represents significant progress towards removing accessibility barriers in our core products and should not be overlooked. Nevertheless, there are still limitations and even blockers to research.
For one, the Figma files must be created in a way that will mimic the website layout and code. For example, for screen reader navigation to be successful, the elements need to be in their correct reading order in the Layers panel (not solely look correct visually), include labeled elements such as buttons (not solely items styled to look like buttons), and include alternative text for images. Often, however, designers do not build iterative prototypes with these considerations in mind, which prevents the keyboard from navigating correctly and the screen reader from providing the necessary details to comprehend the page.
In addition, Figma’s prototypes do not have selectable, configurable text. This prevents key visual adjustments such as browser zoom to increase text size, dark mode, which is easier for some to view, and selecting text to have it read aloud. If a participant needs these kinds of adjustments (or others I list in the table below), a Figma prototype will not be accessible to them.
Table: Figma prototype limitations per assistive technology
| Assistive Technology | Disability Category | Limitation |
|---|---|---|
| Keyboard-only navigation | Mobility | Must use proper element type (such as button or input) in expected page order to ensure operability |
| Screen reader | Vision | Must include structure to ensure readability:
|
| Dark mode/High contrast mode | Low Vision Neurodiversity | Not available |
| Browser zoom | Low Vision Neurodiversity Mobility | Not available |
| Screen reader used with mouse hover Read aloud software with text selection | Vision Neurodiversity | Cannot be used |
| Voice control Switch control device | Mobility | Cannot be used |
Inclusive Research Is Needed Regardless
Having accessibility challenges with a prototype doesn’t mean we give up on the research. Instead, it means we need to get creative in our approach. This research is too important to keep waiting for the ideal set-up, particularly when our findings are often precisely what’s needed to create accessible technology.
Part of crafting a research study is determining what artifact to use during the study. Thus, when considering prototype research, it is a matter of creating the artifact best suited for your study. If this isn’t going to be, say, a Figma file you receive from designers, then consider what else can be used to get the job done.
Working Around the Current State
Being able to include diverse perspectives from disabled research participants throughout a project’s creation is possible and necessary. Keeping in mind your research questions and the capabilities of your participants, there are research methods and strategies that can be made accessible to gather authentic feedback during the critical prototype design phase.
With that in mind, I propose five ways you can accomplish prototype research while working around inaccessible prototypes:
- Use a survey.
- Conduct a co-design session.
- Test with a similar system.
- Build your own rapid prototype.
- Use the Wizard of Oz method.
Use a Survey Instead
Not all research questions at this phase need a full working prototype to be answered, particularly if they are about the general product features or product wording and not the visual design. Oftentimes, a survey tool or similar type of evaluation can be just as effective.
For example, you can confirm a site’s navigation options are intuitive by describing a scenario with a list of navigation choices while also testing if key content is understandable by confirming the user’s next steps based on a passage of text.
- Image description
Just be sure you build a WCAG-compliant survey that includes accessible form layouts and question types. This will ensure participants can navigate using their assistive technologies. For example, Qualtrics has a specific form layout that is built to be accessible, or check out these accessibility tips for Google Forms. If sharing a document, note features that will enhance accessibility, such as using the ribbon for styling in Microsoft Word.
Tip: To find accessibility documentation for the software you’re using, search in your favorite search engine for the product name plus the word “accessibility” to find a product’s accessibility documentation.
Conduct Co-design Sessions
The prototyping phase might be a good time to utilize co-design and participatory design methods. With these methods, you can co-create designs with participants using any variety of artifacts that match the capabilities of your participants along with your research goals. The feedback can range from high-level workflows to specific visual designs, and you can guide the conversation with mock-ups, equivalent systems, or more creative artifacts such as storyboards that illustrate a scenario for user reaction.
For the prototype artifacts, these can range from low- to high-fidelity. For instance, participants without mobility or vision impairments can use paper-and-pencil sketching or whiteboarding. People with somewhat limited mobility may prefer a tablet-based drawing tool, such as using an Apple pencil with an iPad. Participants with visual impairments may prefer more 3-dimensional tools such as craft supplies, modeling clay, and/or cardboard. Or you may find that simply working on a collaborative online document offers the best accessibility as users can engage with their personalized assistive technology to jot down ideas.
Notably, the types of artifacts you use will be beneficial across differing user groups. In fact, rather than limiting the artifacts, try to offer a variety of ways to provide feedback by default. By doing this, participants can feel more empowered and engaged by the activity while also reassuring them you have created an inclusive environment. If you’re not sure what options to include, feel free to confirm what methods will work best as you recruit participants. That is, as you describe the primary activity when they are signing up, you can ask if the materials you have will be operable for the participant or allow them to tell you what they prefer to use.
The discussion you have and any supplemental artifacts you use then depend on communication styles. For example, deaf participants may need sign language interpreters to communicate their views but will be able to see sample systems, while blind participants will need descriptions of key visual information to give feedback. The actual study facilitation comes down to who you are recruiting and what level of feedback you are seeking; from there, you can work through the accommodations that will work best.
I conducted two co-design sessions at two different project phases while exploring how to create a wearable blind pedestrian navigation device. Early in the project, when we were generally talking about the feature set, we brought in several low-fidelity supplies, including a Braille label maker, cardboard, clay, Velcro, clipboards, tape, paper, and pipe cleaners. Based on user feedback, I fashioned a clipboard hanging from pipe cleaners as one prototype.
Later in the project when we were discussing the size and weight, we taped together Arduino hardware pieces representing the features identified by the participants. Both outcomes are pictured below and featured in a paper entitled, “What Not to Wearable: Using Participatory Workshops to Explore Wearable Device Form Factors for Blind Users.”
Ultimately, the benefit of this type of study is the participant-led feedback. In this way, participants are giving unfiltered feedback that is less influenced by designers, which may lead to more thoughtful design in the end.
Test With an Equivalent System
Very few projects are completely new creations, and often, teams use an existing site or application for project inspiration. Consider using similar existing systems and equivalent scenarios for your testing instead of creating a prototype.
By using an existing live system, participants can then use their assistive technology and adaptive techniques, which can make the study more accessible and authentic. Also, the study findings can range from the desirability of the available product features to the accessibility and usability of individual page elements. These lessons can then inform what design and code decisions to make in your system.
One caveat is to be aware of any accessibility barriers in that existing system. Particularly for website and web applications, you can look for accessibility documentation to determine if the company has reported any WCAG-conformance accessibility efforts, use tools like WAVE to test the system yourself, and/or mimic how your participants will use the system with their assistive technology. If there are workarounds for what you find, you may be able to avoid certain parts of the application or help users navigate past the inaccessible parts. However, if the site is going to be completely unusable for your participants, this won’t be a viable option for you.
If the system is usable enough for your testing, however, you can take the testing a step further by making updates on the fly if you or someone you collaborate with has engineering experience. For example, you can manipulate a website’s code with developer tools to add, subtract, or change the elements and styling on a page in real-time. (See “About browser developer tools”.) This can further enhance the feedback you give to your teams as it may more closely match your team’s intended design.
Build a Rapid Website Prototype
Notably, when conducting research focused on physical devices and hardware, you will not face the same obstacles to inaccessibility as with websites and web applications. You can use a variety of materials to create your prototypes, from cardboard to fabric to 3D printed material. I’ve sewn haptic vibration modules to a makeshift leather bracelet when working with wearables, for instance.
However, for web testing, it may be necessary to build a rapid prototype, especially to work around inaccessible artifacts such as a Figma file. This will include using a site builder that allows you to quickly create a replica of your team’s website. To create an accessible website, you’ll need a site builder with accessibility features and capabilities; I recommend WordPress, SquareSpace, Webflow, and Google Sites.
I recently used Google Sites to create a replica of a client’s draft pages in a matter of hours. I was adamant we should include disabled participants in feedback loops early and often, and this included after a round of significant visual design and content decisions. The web agency building the client’s site used Figma but not with the required formatting to use the built-in screen reader functionality. Rather than leave out blind user feedback at such a crucial time in the project, I started with a similar Google Sites template, took a best guess at how to structure the elements such as headings, recreated the anticipated column and card layouts as best I could, and used placeholder images with projected alt text instead of their custom graphics.
The screen reader testing turned into an impromptu co-design session because I could make changes in-the-moment to the live site for the participant to immediately test out. For example, we determined that some places where I used headings were not necessary, and we talked about image alt text in detail. I was able to add specific design and code feedback to my report, as well as share the live site (and corresponding code) with the team for comparison.
The downside to my prototype was that I couldn’t create the exact 1-to-1 visual design to use when testing with the other disabled participants who were sighted. I wanted to gather feedback on colors, fonts, and wording, so I also recruited low vision and neurodiverse participants for the study. However, my data was skewed because those participants couldn’t make the visual adjustments they needed to fully take in the content, such as recoloring, resizing, and having text read aloud. This was unfortunate, but we at least used the prototype to spark discussions of what does make a page accessible for them.
You may find you are limited in how closely you can replicate the design based on the tools you use or lack of access to developer assistance. When facing these limitations, consider what is most important to evaluate and determine if a paired-down version of the site will still give you valuable feedback over no site at all.
Use Wizard of Oz
The Wizard of Oz (WoZ) research method involves the facilitators mimicking system interactions in place of a fully working system. With WoZ, you can create your system’s approximate functionality using equivalent accessible tools and processes.
As an example, I’ll refer you to the talk by an Ally Financial research team that used this method for participants who used screen readers. They pre-programmed screen reader prompts into a clickable spreadsheet and had participants describe aloud what keyboard actions they would take to then trigger the corresponding prompt. While not the ideal set-up for the participants or researchers, it at least brought screen reader user feedback (and recognition of the users themselves) to the early design phases of their work. For more, review their detailed talk “Removing bias with wizard of oz screen reader usability testing”.
This isn’t just limited to screen reader testing, however. In fact, I’ve also often used Wizard of Oz for Voice User Interface (VUI) design. For instance, when I helped create an Alexa “skill” (their name for an app on Amazon speech-enabled devices), our prototype wouldn’t be ready in time for user testing. So, I drafted an idea to use a Bluetooth speaker to announce prompts from a clickable spreadsheet instead. When participants spoke a command to the speaker (thinking it was an Alexa device), the facilitator would select the appropriate pre-recorded prompt or a generic “I don’t understand” message.
Any system can be mimicked when you break down its parts and pieces and think about the ultimate interaction for the user. Creating WoZ set-ups can take creativity and even significant time to put together, but the outcomes can be worth it, particularly for longer-term projects. Once the main pieces are created, the prototype set-up can be edited and reused indefinitely, including during the study or between participants. Also, the investment in an easily edited prototype pays off exponentially if it uncovers something prior to finishing the entire product. In fact, that’s the main goal of this phase of testing: to help teams know what to look out for before they go through the hard work of finishing the product.
Inclusive Research Can No Longer Wait
Much has been documented about inclusive design to help teams craft technology for the widest possible audience. From the Web Content Accessibility Guidelines that help define what it means to be accessible to the Microsoft Inclusive Design Toolkits that tell the human stories behind the guidelines, there is much to learn even before a product begins.
However, the best approach is with direct user feedback. With this, we must recognize the conundrum many researchers are facing: We want to include disabled participants in UX research prior to a product being complete, but often, prototypes we have available for testing are inaccessible. This means testing with something that is essentially broken and will negatively impact our findings.
While it may feel like researchers will always be at a disadvantage if we don’t have the tools we need for testing, I think, instead, it’s time for us to push back. I propose we do this on two fronts:
- We make the research work as best we can in the current state.
- We advocate for the tools we need to make this more streamlined.
The key is to get disabled perspectives on the record and in the dataset of team members making the decisions. By doing this, hopefully, we shift the culture to wanting and valuing this feedback and bringing awareness to what it takes to make it happen.
Ideally, the awareness raised from our bootstrap efforts will lead to more people helping reduce the current prototype barriers. For some of us, this means urging companies to prioritize accessibility features in their roadmaps. For those working within influential prototype companies, it can mean getting much-needed backing to innovate better in this area.
The current state of our inaccessible digital ecosystem can sometimes feel like an entanglement too big to unravel. However, we must remain steadfast and insist that this does not remain the status quo; disabled users are users, and their diverse and invaluable perspectives must be a part of our research outcomes at all phases.
(yk)
