Tap to Sign: Towards using American Sign Language for Text Entry on Smartphones

Image credit: Unsplash

Abstract

Soon, smartphones may be capable of allowing American Sign Language (ASL) signing and/or fingerspelling for text entry. To explore the usefulness of this approach, we compared emulated fingerspelling recognition with a virtual keyboard for 12 Deaf participants. With practice, fingerspelling is faster (42.5 wpm), potentially has fewer errors (4.02% corrected error rate) and higher throughput (14.2 bits/second), and is as desired as virtual keyboard texting (31.9 wpm; 6.46% corrected error rate; 10.9 bits/second throughput). Our second study recruits another 12 Deaf users at the 2022 National Association for the Deaf conference to compare the walk-up usability of fingerspelling alone, signing, and virtual keyboard text entry for interacting with an emulated mobile assistant. Both signing and virtual keyboard text entry were preferred over fingerspelling.

Publication
Proceedings of the ACM on Human-Computer Interaction, Volume (7) Issue (MHCI)
Saad Hassan
Saad Hassan
Assistant Professor

My research interests include human-computer interaction (HCI), accessibility, and computational social science.

Abraham Glasser
Abraham Glasser
Assistant Professor at Gallaudet University
Max Shengelia
Max Shengelia
Former Research Assistant at RIT
Thad E. Starner
Thad E. Starner
Professor at Georgia Institute of Technology
Sean Forbes
Sean Forbes
CEO DPAN
Nathan Qualls
Nathan Qualls
DPAN Staff
Sam S. Sepah
Sam S. Sepah
Program Manager at Google