Downloads

Du, T. (2026). Toward an LLM-Driven Multimodal Accessibility Framework for Diverse Curatorial Contexts. Journal of Integrated Social Sciences and Humanities, 3(4), 0043. https://doi.org/10.62836/jissh.v3i4.0043

Toward an LLM-Driven Multimodal Accessibility Framework for Diverse Curatorial Contexts

Current research on museum accessibility has widely adopted sensory substitution technologies such as haptic feedback systems, virtual reality (VR), augmented reality (AR) and spatial audio technologies; however, most existing studies focus on a single technology and lack a unified framework capable of coordinating different assistive technologies, addressing diverse disability needs, and interpreting curatorial semantics. To address this research gap, this paper proposes a conceptual framework for a Curatorial Accessibility-oriented Large Language Model (CA-LLM) that integrates sensory substitution technologies. Grounded in sensory substitution theory and drawing on Research through Design (RtD) and participatory co-design methodologies, this framework constructs a three-layer system architecture comprising an input perception layer, a cognitive hub layer and a sensory output layer. It enables the intelligent coordination of curatorial knowledge, accessibility expertise and multimodal assistive technologies, thereby providing contextualised and personalised accessible curatorial experiences for people with diverse disabilities. This paper further discusses the framework’s potential contributions to the theory of accessible curation and museum practice, whilst critically examining the framework’s philosophical foundations, the boundaries of current AI capabilities, the practical constraints on its scalability across diverse curatorial contexts, and the ethical and cultural tensions its deployment would generate.

accessible curation large language models sensory substitution co-design research through design multimodal interaction

References

  1. World Health Organization & the World Bank. World Report on Disability; World Health Organization: Geneva, Switzerland, 2011.
  2. National Endowment for the Arts. Museums and the Americans with Disabilities Act at 25: Progress and Looking Ahead. Available online: https://www.arts.gov/stories/blog/2015/museums-and-americans-disabilities-act-25-progress-and-looking-ahead (accessed on 17 September 2026).
  3. Vaz R, Freitas D, Coelho A. Perspectives of Visually Impaired Visitors on Museums: Towards an Integrative and Multisensory Framework to Enhance the Museum Experience. In Proceedings of the DSAI 2020: 9th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-Exclusion, Online, 2–4 December 2020; pp. 17–21. https://doi.org/10.1145/3439231.3439272.
  4. Adams GK. Museums Need to Do More to Welcome Disabled Visitors. Museums Association. Available online: https://www.museumsassociation.org/museums-journal/analysis/2017/01/01022017-museums-need-to-do-more-to-welcome-disabled-visitors/ (accessed on 17 September 2026).
  5. Museums Association. Museums Fail to Improve Online Access Information for Some Disabled Visitors. Museums Association. Available online: https://www.museumsassociation.org/museums-journal/news/2022/10/museums-fail-to-improve-online-access-information-for-some-disabled-visitors/ (accessed on 17 September 2026).
  6. Poria Y, Reichel A, Brandt Y. People with Disabilities Visit Art Museums: An Exploratory Study of Obstacles and Difficulties. Journal of Heritage Tourism 2009; 4(2): 117–129. https://doi.org/10.1080/17438730802366508.
  7. Union of the Physically Impaired Against Segregation (UPIAS). Fundamental Principles of Disability; UPIAS: London, UK, 1976.
  8. Oliver M. The Politics of Disablement; Red Globe Press: London, UK, 1990; p. 185. https://doi.org/10.1007/978-1-349-20895-1.
  9. Eardley AF, Thompson H, Fineman A, et al. Devisualizing the Museum: From Access to Inclusion. Journal of Museum Education 2022; 47(2): 150–165. https://doi.org/10.1080/10598650.2022.2077067.
  10. Kleege G. More than Meets the Eye, 1st ed.; Oxford University Press: New York, NY, USA, 2017. https://doi.org/10.1093/oso/9780190604356.001.0001.
  11. Smith H, Ginley B, Goodwin H. Beyond Compliance? Museums, Disability and the Law. In Museums, Equality and Social Justice; Routledge: London, UK, 2012; pp. 59–71. https://doi.org/10.4324/9780203120057.
  12. Šebová Ľ, Lazurová I, Marčeková R, et al. Socio-Economic Aspects of Accessibility to Museums and Galleries in Europe by Removing Barriers. Theoretical and Practical Research in Economic Fields 2024; 15(2): 375. https://doi.org/10.14505/tpref.v15.2(30).17.
  13. Western Museums Association. A Call for Action—Museum Accessibility Survey Results. Available online: https://westmuse.org/call-action-museum-accessibility-survey-results (accessed on 17 September 2026).
  14. Wang Z, Yuan LP, Wang L, et al. VirtuWander: Enhancing Multi-Modal Interaction for Virtual Tour Guidance through Large Language Models. In Proceedings of the CHI '24: CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 11–16 May 2024; p. 612. https://doi.org/10.1145/3613904.3642235.
  15. Ye W, Zheng T, Qi Y, et al. ICH-Qwen: A Large Language Model towards Chinese Intangible Cultural Heritage. arXiv 2025. https://doi.org/10.48550/arXiv.2505.23831.
  16. Fallows E, White D, Brownsword N. Design and Development Approach for an Interactive Virtual Museum with Haptic Glove Technology. In Proceedings of the Academic Mindtrek 2022: 25th International Academic Mindtrek Conference, Tampere, Finland, 2022; pp. 242–255. https://doi.org/10.1145/3569219.3569382.
  17. Tan S, Peshkin MA, Klatzky RL, et al. Toward Human-Resolution Haptics: A High-Bandwidth, High-Density, Wearable Tactile Display. Science Advances 2025; 11(47): eadz5937. https://doi.org/10.1126/sciadv.adz5937.
  18. Zhang K. Towards Low-Cost Spatial Haptics: Brake-Based Shape Displays and Automatic Content Authoring. Ph.D. Thesis, Stanford University, Stanford, CA, USA, 2020.
  19. Vidal-Verdu F, Hafez M. Graphical Tactile Displays for Visually-Impaired People. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2007; 15(1): 119–130. https://doi.org/10.1109/tnsre.2007.891375.
  20. Souradi A, Lecomte C, Romeo K, et al. Towards the Tactile Discovery of Cultural Heritage with Multi-Approach Segmentation. In Proceedings of International Conference on Image and Signal Processing (ICISP 2020), Marrakesh, Morocco, 4–6 June 2020; Volume 12119, pp. 14–23. https://doi.org/10.1007/978-3-030-51935-3_2.
  21. Slater M, Wilbur S. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 1997; 6(6): 603–616. https://doi.org/10.1162/pres.1997.6.6.603.
  22. Gerling K, Meiners AL, Schumm L, et al. An Equitable Experience? How HCI Research Conceptualizes Accessibility of Virtual Reality in the Context of Disability. ACM Transactions on Accessible Computing 2025; 18(4): 1–35. https://doi.org/10.1145/3770755.
  23. Wolf M, Gerling K, Alexandrovsky D, et al. Understanding Accessibility for Physically Disabled Users in VR: Interplay of Physical, Digital, and Experiential Layers. In Proceedings of the ASSETS'25: The 27th International ACM SIGACCESS Conference on Computers and Accessibility, Denver, CO, USA, 26–29 October 2025; pp. 1–15. https://doi.org/10.1145/3663547.3746357.
  24. Zhao Y, Cutrell E, Holz C, et al. SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Glasgow, UK, 4–9 May 2019; pp. 1–13. https://doi.org/10.1145/3290605.3300341.
  25. Zhang J, Wei Q, Wei X, et al. Exploring the Design of Virtual Reality Museums to Support Remote Visitation with Older Adults. arXiv 2024. https://doi.org/10.48550/arxiv.2410.15092.
  26. Azuma RT. A Survey of Augmented Reality. Presence Teleoperators Virtual Environ. 1997; 6(4): 355–385. https://doi.org/10.1162/pres.1997.6.4.355.
  27. Creed C, Al-Kalbani M, Theil A, et al. Inclusive Augmented and Virtual Reality: A Research Agenda. International Journal of Human–Computer Interaction 2024; 40(20): 6200–6219. https://doi.org/10.1080/10447318.2023.2247614.
  28. Huo J, Keung KL, Lee CKM, et al. Hand Gesture Recognition with Augmented Reality and Leap Motion Controller. In Proceedings of the 2021 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), Singapore, 13–16 December 2021; pp. 1015–1019. https://doi.org/10.1109/ieem50564.2021.9672611.
  29. Camgoz NC, Hadfield S, Koller O, et al. Neural Sign Language Translation. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 7784–7793. https://doi.org/10.1109/cvpr.2018.00812.
  30. Baumeister J, Ssin SY, ElSayed NAM, et al. Cognitive Cost of Using Augmented Reality Displays. IEEE Transactions on Visualization and Computer Graphics 2017; 23(11): 2378–2388. https://doi.org/10.1109/tvcg.2017.2735098.
  31. Wang X. Co-Design of a Voice-Driven Interactive Smart Guide for Museum Accessibility and Management. Journal of Audiovisual Translation 2024; 7(1): 1–24. https://doi.org/10.47476/jat.v7i1.2024.267.
  32. Connors EC, Chrastil ER, Sánchez J, et al. Virtual Environments for the Transfer of Navigation Skills in the Blind: A Comparison of Directed Instruction vs. Video Game Based Learning Approaches. Frontiers in Human Neuroscience 2014; 8: 223. https://doi.org/10.3389/fnhum.2014.00223.
  33. Bach-y-Rita P, Kercel SW. Sensory Substitution and the Human–Machine Interface. Trends in Cognitive Sciences 2003; 7(12): 541–546. https://doi.org/10.1016/j.tics.2003.10.013.
  34. Zimmerman J, Forlizzi J, Evenson S. Research through Design as a Method for Interaction Design Research in HCI. In Proceedings of the CHI’07: CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, 28 April–3 May 2007; pp. 493–502. https://doi.org/10.1145/1240624.1240704.
  35. Charlton J. Nothing about Us without Us: Disability Oppression and Empowerment; University of California Press: Oakland, CA, USA, 1998. https://doi.org/10.1525/california/9780520207950.001.0001.
  36. Sanders EBN, Stappers PJ. Co-Creation and the New Landscapes of Design. CoDesign 2008; 4(1): 5–18. https://doi.org/10.1080/15710880701875068.
  37. Spinuzzi C. The Methodology of Participatory Design. Technical Communication 2005; 52(2): 163–174.
  38. Luetzenburg G, Kroon A, Bjørk AA. Evaluation of the Apple iPhone 12 Pro LiDAR for an Application in Geosciences. Scientific Reports 2021; 11(1): 22221. https://doi.org/10.1038/s41598-021-01763-9.
  39. Stamnes G. 3D Gaussian Splatting in Reality Capture Workflows. Photogrammetric Engineering & Remote Sensing 2026; 92(1): 11–15. https://doi.org/10.14358/pers.92.1.11.
  40. Marty PF, Jones KB. (Eds.) Museum Informatics: People, Information, and Technology in Museums; Taylor & Francis: New York, NY, USA, 2008. https://doi.org/10.4324/9780203939147.
  41. Wobbrock JO, Kane SK, Gajos KZ, et al. Ability-Based Design: Concept, Principles and Examples. ACM Transactions on Accessible Computing 2011; 3(3): 1–27. https://doi.org/10.1145/1952383.1952384.
  42. Ji Z, Lee N, Frieske R, et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 2023; 55(12): 248. https://doi.org/10.1145/3571730.
  43. Huang L, Yu W, Ma W, et al. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems 2025; 43(2): 42. https://doi.org/10.1145/3703155.
  44. Gaver W. What Should We Expect from Research through Design? In Proceedings of the CHI '12: SIGCHI Conference on Human Factors in Computing Systems, Austin, TX, USA, 5–10 May 2012; pp. 937–946. https://doi.org/10.1145/2207676.2208538.
  45. Druin A. Cooperative Inquiry: Developing New Technologies for Children with Children. In Proceedings of the CHI '99: SIGCHI Conference on Human Factors in Computing Systems, Pittsburgh, PA, USA, 15–20 May 1999; pp. 592–599. https://doi.org/10.1145/302979.303166.
  46. Sanders L, Stappers PJ. From Designing to Co-Designing to Collective Dreaming: Three Slices in Time. Interactions 2014; 21(6): 24–33. https://doi.org/10.1145/2670616.
  47. Ellcessor E. Restricted Access: Media, Disability, and the Politics of Participation; New York University Press: New York, NY, USA, 2016. https://doi.org/10.18574/nyu/9781479813803.001.0001.
  48. Theil A, Creed C, Shaqura M, et al. Multidisciplinary Perspectives on Designing Accessible Systems for Users with Multiple Impairments: Grand Challenges and Opportunities for Future Research. In Proceedings of the ASSETS '22: The 24th International ACM SIGACCESS Conference on Computers and Accessibility, Athens, Greece, 23–26 October 2022; pp. 1–6. https://doi.org/10.1145/3517428.3550405.
  49. Fricker M. Epistemic Injustice: Power and the Ethics of Knowing; Oxford University Press: Oxford, UK, 2007. https://doi.org/10.1093/acprof:oso/9780198237907.001.0001.
  50. Mladenov T, Dimitrova I. Epistemic Injustice as a Bridge between Medical Sociology and Disability Studies. Sociology of Health & Illness 2022; 45(6): 1146–1163. https://doi.org/10.1111/1467-9566.13479.
  51. Bjögvinsson E, Ehn P, Hillgren PA. Design Things and Design Thinking: Contemporary Participatory Design Challenges. Design Issues 2012; 28(3): 101–116. https://doi.org/10.1162/desi_a_00165.
  52. Gray ML, Suri S. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass; Harper Business: New York, NY, USA, 2019.
  53. Auvray M, Myin E. Perception with Compensatory Devices: From Sensory Substitution to Sensorimotor Extension. Cognitive Science 2009; 33(6): 1036–1058. https://doi.org/10.1111/j.1551-6709.2009.01040.x.
  54. Proulx MJ, Stoerig P. Seeing Sounds and Tingling Tongues: Qualia in Synaesthesia and Sensory Substitution. Anthropology & Philosophy 2006; 7(1–2): 135–150.
  55. Hurley S, Noë A. Neural Plasticity and Consciousness. Biology and Philosophy 2003; 18(1): 131–168. https://doi.org/10.1023/a:1023308401356.
  56. Ward J, Meijer P. Visual Experiences in the Blind Induced by an Auditory Sensory Substitution Device. Consciousness and Cognition 2010; 19(1): 492–500. https://doi.org/10.1016/j.concog.2009.10.006.
  57. Merabet LB, Pascual-Leone A. Neural Reorganization Following Sensory Loss: The Opportunity of Change. Nature Reviews Neuroscience 2010; 11(1): 44–52. https://doi.org/10.1038/nrn2758.
  58. Weinstein S. Intensive and Extensive Aspects of Tactile Sensitivity as a Function of Body Part, Sex, and Laterality. In The Skin Senses; Charles C. Thomas: Springfield, IL, USA, 1968; pp. 195–222.
  59. Hu J, Yan C, Zheng Y, et al. Position: Assistive Agents Need Accessibility Alignment. arXiv 2026. https://doi.org/10.48550/arxiv.2605.13579.
  60. Morris MR. AI and Accessibility. Communications of the ACM 2020; 63(6): 35–37. https://doi.org/10.1145/3356727.
  61. Morozov E. To Save Everything, Click Here: The Folly of Technological Solutionism; Public Affairs: New York, NY, USA, 2013.
  62. Shew A. Ableism, Technoableism, and Future AI. IEEE Technology and Society Magazine 2020; 39(1): 40–85. https://doi.org/10.1109/mts.2020.2967492.
  63. Kagkelidou M. Embracing Technology in Cultural Heritage: Overcoming Barriers to Engagement and Accessibility. Available online: https://heritagemanagement.org/embracing-technology-in-cultural-heritage-overcoming-barriers-to-engagement-and-accessibility/ (accessed on 17 September 2026).
  64. Falk JH, Dierking LD. The Museum Experience Revisited; Routledge: New York, NY, USA, 2016. https://doi.org/10.4324/9781315417851.
  65. Mishra A, Aboulafia A. Inclusive Innovation: How to Incorporate Privacy into Inclusive Design for Assistive Technologies. 2025. Available online: https://cdt.org/insights/inclusive-innovation-how-to-incorporate-privacy-into-inclusive-design-for-assistive-technologies/ (accessed on 17 September 2026).
  66. Grech S, Soldatic K. Disability in the Global South; Springer: Cham, Switzerland, 2016; pp. xiii–xxviii. https://doi.org/10.1007/978-3-319-42488-0.

Supporting Agencies

  1. Funding: This research received no external funding.