Augmenta11y

An augmented reality (AR) based app which helps people with learning disabilities like dyslexia easily read signage, paper, and other material in the real world. This research is a part of my bachelor degree's capstone project. Along with three batchmates, I'm working on making the real world more accessible to people with learning disabilities like dyslexia.

Project Details

Duration: July 2018 to Present
Platform: iOS and Android
What I Did: Project Lead, Requirement Gathering, Requirement Analysis, Conducting competitive research, Creating wireframes, Cross-platform app development, Usability Testing

Problem

According to Dyslexia International, at least 1 in 10 people are affected by dyslexia, i.e. more than 700 million children and adults worldwide.

On the web, there are some handy products that help such individuals navigate and understand the internet better. For our final year project, my team and I wanted to translate the same concept to the real world where text could be more accessible to people on the fly with their smartphones.

Requirement Gathering

Since the idea had been validated by the existence of the web-based accessibility tools already, we needed was to figure out if this need translated to real-world textual information as well. For this, we audited how accessible such information was and prepared various use-cases where our app could help. Some examples included reading station names on a subway station, comprehending a restaurant menu, and reading the news or magazines.

We made mockups of how the text would look inside the app, and showed these to people of various age groups suffering from learning disabilities and noted down their feedback.

Requirement Analysis

The response we receieved was vital in determining our target audience. It was inferred that the primary audience which would benefit from such a product were children in the age group of 8-14. This was also backed by research, which stated that children coping with learning disabilities benefit greatly from the right kind of interventions and support provided by technology. Ages 15-60 were found to have found ways to deal with reading comprehension already, and hence didn’t believe that the app helped them greatly.

Target Audience

  • Primary Audience: Children aged 8-14.

  • Secondary Audience: Ages 60+ who don’t necessarily suffer from dyslexia, but liked the idea due to ease of readability.

Factors affecting readability

We also read research papers and academic journals to identify factors affecting readability for dyslexic impairments:

  • Usage of sans-serif fonts, especially OpenDyslexic – a research-backed font known to help people with dyslexic impairments

  • Better contrast between text and background (Yellow Text and Black Background)

  • Line Height & Spacing

Solution

Based on our requirement analysis, we moved on to making low fidelity mockups of the mobile application.

Since we were targeting a niche audience, the interface needed to do a few things really well:

  • Simple mental model

  • Child-friendly functionality

  • Accessibility first

Owing to existing solutions, we identified a few key features that were a must for this solution:

  • Real-time text recognition: This was one of the features which we found to be absent in most existing solutions, and felt like a real bottleneck in providing a seamless experience.

  • Reader Mode: Tapping a block of text opens a new screen with the text coupled with a text-to-speech mode.

  • History Mode: All tapped text blocks are saved for easy retrieval.

  • Personalized Settings & Onboarding Experience: Since learning disabilities are subjective, it’s important to have an on-boarding experience which understands user preferences and provides easy customization.

UML Sequence Diagram

Prototype

Having experience with Web Development and JavaScript individually, we opted to use React Native, a cross platform app development solution maintained by Facebook.

Using open source libraries and some trained machine learning models, we developed the prototype application’s basic features. This version was showcased at Dutch Design Week 2018, where we got some valuable feedback about the overlapping bounding text boxes, user personalization preferences, and some experience pain points.

Dutch Design Week 2018

Testing and Evaluation

We tied up with health organizations from various cities in India to validate our research prototype. The test subjects with a group of 39 students aged 8–14 years. The students were provided with two passages appropriate for each age group. The task given was to read one paragraph normally, and the other using the app. To ensure one paragraph was not inherently easier to read than the other, we alternated between the paragraph read using the app. The time taken by the students to read each passage, and their text styling preference recorded.

Students during testing & evaluation sessions

Outcomes

We saw a 21.2% reduction in the amount of time taken to read text using Augmenta11y. It was also observed that 85.7% of the students found the OpenDyslexic font helpful and 76.9% of them preferred to have a yellow background to the text.

Visualizing & analyzing the test data

We also presented a poster at India HCI 2018 – an ACM SIGCHI conference and will be publishing this research as a paper at the 9th International Conference on Computer Communication and Informatics (ICCCI 2019) in the HCI track.

Here's me at India HCI 2018 👋

Public Beta

Augmenta11y is currently in public beta where we’re ironing out bugs and improving the user experience of the app.

Download it here from the Google Play Store!

What’s next for Augmenta11y?

This is an on-going project and I couldn’t be more excited to keep working on it. Here are some of the things under development:

  • Translations into various languages

  • Customizable Reader Mode with in-depth settings

  • Improved on-boarding experience that would allow users to figure out and set their personal text styling preferences instead of using commonly accepted default settings

  • Zooming feature in Reader Mode