Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL Learners
Abstract
As they develop comprehension skills, American Sign Language (ASL) learners often view challenging ASL videos, which may contain unfamiliar signs. Current dictionary tools require students to isolate a single sign they do not understand and input a search query, by selecting linguistic properties or by performing the sign into a webcam. Students may struggle with extracting and re-creating an unfamiliar sign, and they must leave the video-watching task to use an external dictionary tool. We investigate a technology that enables users, in the moment, i.e., while they are viewing a video, to select a span of one or more signs that they do not understand, to view dictionary results. We interviewed 14 American Sign Language (ASL) learners about their challenges in understanding ASL video and workarounds for unfamiliar vocabulary. We then conducted a comparative study and an in-depth analysis with 15 ASL learners to investigate the benefits of using video sub-spans for searching, and their interactions with a Wizard-of-Oz prototype during a video-comprehension task. Our findings revealed benefits of our tool in terms of quality of video translation produced and perceived workload to produce translations. Our in-depth analysis also revealed benefits of an integrated search tool and use of span-selection to constrain video play. These findings inform future designers of such systems, computer vision researchers working on the underlying sign matching technologies, and sign language educators.
| Video | URL | Start Time | End Time | Duration | Description |
|---|---|---|---|---|---|
| Conversation 1 | https://www.youtube.com/clip/UgkxrbvAix-NlFzHxgpqahNmL-RIsPF4PgSE | 0:26 | 0:44 | 17 | A clip from a work advertisement posted on NTID’s YouTube page. |
| Conversation 2 | https://youtu.be/ZMs93GJS8Pk | 0:01 | 0:33 | 32 | A clip from sign language learning material. |
| Conversation 3 | https://youtu.be/hOgjwKdQ44A | 0:05 | 0:34 | 19 | A clip from a conversation between two Deaf individuals (not actors). |
| Education 1 | https://youtu.be/l42omYV4ReQ | 0:59 | 1:20 | 21 | A clip from a deaf queer YouTuber who talks about the deaf experience and queer topics. |
| Education 2 | https://www.youtube.com/clip/UgkxJ44B6umUjYcGCQSEpEsSLwterYAX0e3g | 0:44 | 1:03 | 19 | A clip from an ASL Story video shared as a part of project N.A.D.I.N.E. to promote childhood literacy. |
| Education 3 | https://youtu.be/E-6EOV1TmVU | 0:44 | 0:58 | 13 | A clip shared at CATIE center consisting of an expert signer talking about their college experience. |
| Theatre 1 | https://youtu.be/BSf4i4h5eIQ?t=20 | 0:20 | 1:00 | 39 | A clip of an expert signer performing the poem “The moon in my bedroom” by Jolanta Lapiak. |
| Theatre 2 | https://youtu.be/uhqKgwZMjf8 | 0:00 | 0:31 | 32 | An expert signer performing the poem “Mushroom” by Clayton Valli. |
| Theatre 3 | https://www.youtube.com/watch?v=OkjdTB3MHZc | 0:51 | 1:12 | 21 | An expert signer performing the poem “Tears of Life” by Clayton Valli. |
Saad Hassan
Assistant Professor
My research interests include human-computer interaction (HCI), accessibility, and computational social science.
Caluã de Lacerda Pataca
PhD Student at RIT
Akhter Al Amin
Software Engineer at Amazon
Laleh Nourian
PhD Student at Rochester Institute of Technology
Diego Navarro
Former Research Assistant at RIT
Sooyeon Lee
Assistant Professor at NJIT
Alexis Gordon
Former Research Assistant at RIT
Garreth Tigwell
Assistant Professor at Rochester Institute of Technology
