FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams

Arshad Nasser; Taizhou Chen; Can Liu; Kening Zhu; Pvm Rao

doi:10.1007/978-3-030-49062-1_32

FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams

Arshad Nasser
, Taizhou Chen^*
, Can Liu
, Kening Zhu
, Pvm Rao

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

Traditional tactile diagrams for the visually-impaired (VI) use short Braille keys and annotations to provide additional information in separate Braille legend pages. Frequent navigation between the tactile diagram and the annex pages during the diagram exploration results in low efficiency in diagram comprehension. We present the design of FingerTalkie, a finger-worn device that uses discrete colors on a color-tagged tactile diagram for interactive audio labeling of the graphical elements. Through an iterative design process involving 8 VI users, we designed a unique offset point-and-click technique that enables the bimanual exploration of the diagrams without hindering the tactile perception of the fingertips. Unlike existing camera-based and finger-worn audio-tactile devices, FingerTalkie supports one-finger interaction and can work in any lighting conditions without calibration. We conducted a controlled experiment with 12 blind-folded sighted users to evaluate the usability of the device. Further, a focus-group interview with 8 VI users shows their appreciation for the FingerTalkie’s ease of use, support for two-hand exploration, and its potential in improving the efficiency of comprehending tactile diagrams by replacing Braille labels.

Original language	English
Title of host publication	Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings
Editors	Masaaki Kurosu
Publisher	Springer
Pages	475-496
Number of pages	22
ISBN (Print)	9783030490614
DOIs	https://doi.org/10.1007/978-3-030-49062-1_32
State	Published - 2020
Externally published	Yes
Event	Thematic Area on Human Computer Interaction, HCI 2020, held as part of the 22nd International Conference on Human-Computer Interaction, HCII 2020 - Copenhagen, Denmark Duration: 19 Jul 2020 → 24 Jul 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12182 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Thematic Area on Human Computer Interaction, HCI 2020, held as part of the 22nd International Conference on Human-Computer Interaction, HCII 2020
Country/Territory	Denmark
City	Copenhagen
Period	19/07/20 → 24/07/20

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Keywords

Audio-tactile diagram
Blind
Finger-worn device
Offset point and click
Visually impaired

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-030-49062-1_32

Cite this

Nasser, A., Chen, T., Liu, C., Zhu, K., & Rao, P. (2020). FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams. In M. Kurosu (Ed.), Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings (pp. 475-496). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12182 LNCS). Springer. https://doi.org/10.1007/978-3-030-49062-1_32

Nasser, Arshad ; Chen, Taizhou ; Liu, Can et al. / FingerTalkie : Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams. Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings. editor / Masaaki Kurosu. Springer, 2020. pp. 475-496 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{883c35bc57184d4b805408597d4b902f,

title = "FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams",

abstract = "Traditional tactile diagrams for the visually-impaired (VI) use short Braille keys and annotations to provide additional information in separate Braille legend pages. Frequent navigation between the tactile diagram and the annex pages during the diagram exploration results in low efficiency in diagram comprehension. We present the design of FingerTalkie, a finger-worn device that uses discrete colors on a color-tagged tactile diagram for interactive audio labeling of the graphical elements. Through an iterative design process involving 8 VI users, we designed a unique offset point-and-click technique that enables the bimanual exploration of the diagrams without hindering the tactile perception of the fingertips. Unlike existing camera-based and finger-worn audio-tactile devices, FingerTalkie supports one-finger interaction and can work in any lighting conditions without calibration. We conducted a controlled experiment with 12 blind-folded sighted users to evaluate the usability of the device. Further, a focus-group interview with 8 VI users shows their appreciation for the FingerTalkie{\textquoteright}s ease of use, support for two-hand exploration, and its potential in improving the efficiency of comprehending tactile diagrams by replacing Braille labels.",

keywords = "Audio-tactile diagram, Blind, Finger-worn device, Offset point and click, Visually impaired",

author = "Arshad Nasser and Taizhou Chen and Can Liu and Kening Zhu and Pvm Rao",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; Thematic Area on Human Computer Interaction, HCI 2020, held as part of the 22nd International Conference on Human-Computer Interaction, HCII 2020 ; Conference date: 19-07-2020 Through 24-07-2020",

year = "2020",

doi = "10.1007/978-3-030-49062-1\_32",

language = "English",

isbn = "9783030490614",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "475--496",

editor = "Masaaki Kurosu",

booktitle = "Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings",

address = "Germany",

}

Nasser, A, Chen, T, Liu, C, Zhu, K & Rao, P 2020, FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams. in M Kurosu (ed.), Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12182 LNCS, Springer, pp. 475-496, Thematic Area on Human Computer Interaction, HCI 2020, held as part of the 22nd International Conference on Human-Computer Interaction, HCII 2020, Copenhagen, Denmark, 19/07/20. https://doi.org/10.1007/978-3-030-49062-1_32

FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams. / Nasser, Arshad; Chen, Taizhou; Liu, Can et al.
Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings. ed. / Masaaki Kurosu. Springer, 2020. p. 475-496 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12182 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - FingerTalkie

T2 - Thematic Area on Human Computer Interaction, HCI 2020, held as part of the 22nd International Conference on Human-Computer Interaction, HCII 2020

AU - Nasser, Arshad

AU - Chen, Taizhou

AU - Liu, Can

AU - Zhu, Kening

AU - Rao, Pvm

PY - 2020

Y1 - 2020

N2 - Traditional tactile diagrams for the visually-impaired (VI) use short Braille keys and annotations to provide additional information in separate Braille legend pages. Frequent navigation between the tactile diagram and the annex pages during the diagram exploration results in low efficiency in diagram comprehension. We present the design of FingerTalkie, a finger-worn device that uses discrete colors on a color-tagged tactile diagram for interactive audio labeling of the graphical elements. Through an iterative design process involving 8 VI users, we designed a unique offset point-and-click technique that enables the bimanual exploration of the diagrams without hindering the tactile perception of the fingertips. Unlike existing camera-based and finger-worn audio-tactile devices, FingerTalkie supports one-finger interaction and can work in any lighting conditions without calibration. We conducted a controlled experiment with 12 blind-folded sighted users to evaluate the usability of the device. Further, a focus-group interview with 8 VI users shows their appreciation for the FingerTalkie’s ease of use, support for two-hand exploration, and its potential in improving the efficiency of comprehending tactile diagrams by replacing Braille labels.

AB - Traditional tactile diagrams for the visually-impaired (VI) use short Braille keys and annotations to provide additional information in separate Braille legend pages. Frequent navigation between the tactile diagram and the annex pages during the diagram exploration results in low efficiency in diagram comprehension. We present the design of FingerTalkie, a finger-worn device that uses discrete colors on a color-tagged tactile diagram for interactive audio labeling of the graphical elements. Through an iterative design process involving 8 VI users, we designed a unique offset point-and-click technique that enables the bimanual exploration of the diagrams without hindering the tactile perception of the fingertips. Unlike existing camera-based and finger-worn audio-tactile devices, FingerTalkie supports one-finger interaction and can work in any lighting conditions without calibration. We conducted a controlled experiment with 12 blind-folded sighted users to evaluate the usability of the device. Further, a focus-group interview with 8 VI users shows their appreciation for the FingerTalkie’s ease of use, support for two-hand exploration, and its potential in improving the efficiency of comprehending tactile diagrams by replacing Braille labels.

KW - Audio-tactile diagram

KW - Blind

KW - Finger-worn device

KW - Offset point and click

KW - Visually impaired

UR - https://www.scopus.com/pages/publications/85088752109

U2 - 10.1007/978-3-030-49062-1_32

DO - 10.1007/978-3-030-49062-1_32

M3 - Conference contribution

AN - SCOPUS:85088752109

SN - 9783030490614

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 475

EP - 496

BT - Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings

A2 - Kurosu, Masaaki

PB - Springer

Y2 - 19 July 2020 through 24 July 2020

ER -

Nasser A, Chen T, Liu C, Zhu K, Rao P. FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams. In Kurosu M, editor, Human-Computer Interaction. Multimodal and Natural Interaction - Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Proceedings. Springer. 2020. p. 475-496. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-49062-1_32

FingerTalkie: Designing a Low-Cost Finger-Worn Device for Interactive Audio Labeling of Tactile Diagrams

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this