A Multimodal Affect Recognition Adaptive Learning System for Individuals with Intellectual Disabilities
Keywords:
Artificial Intelligence, multimodal sensor data, intellectual disabilities, adaptive learning system, machine learning
Abstract
Educational tools using Artificial Intelligence (AIEd) have been implemented to provide automated learning support for typical students. This innovative field focuses on using data and machine learning to detect a student's emotional state, with the goal of shifting them from unproductive emotions to more positive, learning-enhancing ones like engagement. However, AIEd systems that include emotion recognition often overlook students with intellectual disabilities. Our system employs multimodal sensor data and machine learning to identify three key emotional states related to learning (engagement, frustration, boredom). It then adjusts the educational content to keep the student in an ideal emotional state, optimizing learning effectiveness. To evaluate this adaptive learning system, we conducted studies with 67 participants aged 6 to 18, who served as their own controls, in sessions that used the system. These sessions alternated between using the system for both emotional state detection and learning progress to choose content (intervention) and relying solely on learning progress (control) for content selection. Remarkably, a lack of boredom was most strongly linked to better learning outcomes, while both frustration and engagement also showed positive correlations with achievement. Sessions using the intervention showed significantly more engagement and less boredom compared to control sessions, although there was no significant difference in achievement. These results indicate that customizing activities based on the learner's emotional state can boost engagement and foster emotions that are beneficial for learning. Nevertheless, longer-term studies are needed to assess the impact on actual learning achievements.
References
Aist, G., Kort, B., Reilly, R., Mostow, J., & Picard, R. (2002). Experimentally augmenting an intelligent tutoring system with human-supplied capabilities: Adding human-provided emotional scaffolding to an automated reading tutor that listens. In Presentation at the 4th IEEE International Conference on Multimodal Interfaces (pp. 483–490). Pittsburgh, PA.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.
Aslan, S., Alyuz, N., Tanriover, C., Mete, S. E., Okur, E., D’Mello, S. K., & Esme, A. A. (2019). Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. Presentation at Computer Human Interaction (CHI) 2019 (paper 304), Glasgow, Scotland.
Athanasiadis, C., Hortal, E., Koutsoukos, D., Lens, C. Z., & Asteriadis, S. (2017). Personalized, affect and performance-driven computer-based learning. Proceedings of the 9th International Conference on Computer Supported Education—Volume 1: CSEDU. Porto, Portugal, April 21-23, 2017.
Baker, R. S. J., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241.
Barros, M., Galea, M., Leiva, V., & Santos-Neto, M. (2018). Generalized Tobit models: Diagnostics and application in econometrics. Journal of Applied Statistics, 45(1), 145–167.
Basawapatna, A. R., Repenning, A., Han Koh, K., & Nickerson, H. (2013). The zones of proximal flow: Guiding students through a space of computational thinking skills and challenges. In Proceedings of the ninth annual international ACM conference on International computing education research (pp. 67–74). San Diego, CA.
Bertini, E. S., & Kimani, S. (2003). Mobile devices: Opportunities for users with special needs: Mobile HCI. In L. Chittaro (Ed.), Lecture notes of computer science (Vol. 2795, pp. 486–491). Udine, Italy: Springer Verlag.
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Henry, M., … White, J. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology & Evolution, 24(3), 127–135.
Boulton, H., Brown, D., Standen, P., Belmonte, M., Kwiatkowska, G., Hughes-Roberts, T., & Taheri, M. (2018, March 5–7). Multi-modalities in classroom learning environments. INTED2018 Proceedings: 12th Annual International Technology, Education and Development Conference. Valencia, Spain: IATED Academy.
Bozdogan, H. (1987). Model selection and Akaike’s Information Criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370.
Chickerur, S., & Joshi, K. (2015). 3D face model dataset: Automatic detection of facial expressions and emotions for educational environments. British Journal of Educational Technology, 46(5), 1028–1037.
Colley, A. (2013). Personalised learning for young people with profound and multiple learning difficulties. London, UK: Jessica Kingsley.
Craig, S., Graesser, A., Sullins, J., & Gholson, J. (2004). Affect and learning: An exploratory look into the role of affect in learning. Journal of Educational Media, 29, 241–250.
Cukurova, M., Kent, C., & Luckin, R. (2019). Artificial intelligence and multimodal data in the service of human decision-making: A case study in debate tutoring. British Journal of Educational Technology, 50(6), 3032–3046.
D’Mello, S., & Graesser, A. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modelling and User Adapted Interaction, 20(2), 147–187.
D’Mello, S., & Graesser, A. (2012b). Dynamics of affective states during complex learning. Learning and Instruction, 22, 145–157.
D’Mello, S., Dieterle, E., & Duckworth, A. (2017). Advanced, Analytic, Automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123.
du Boulay, B., Poulovasillis, A., Holmes, W., & Mavrikis, M. (2018). Artificial intelligence and big data technologies to close the achievement gap. In R. Luckin (Ed.), Enhancing learning and teaching with technology (pp. 256–285). London, UK: UCL Institute of Education Press.
Grawemeyer, B., Mavrikis, M., Holmes, W., Gutiérrez-Santos, S., Wiedmann, M., & Rummel, N. (2017). Affective learning: Improving engagement and enhancing learning with affect- aware feedback. User Modeling and User-Adapted Interaction, 27(1), 119–158.
Hamari, J., Sherno, D. J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179.
Johansson, T. (2011). Hail the impossible: P-values, evidence, and likelihood. Scandinavian Journal of Psychology, 52(2), 113–125.
Kort, B., Reilly, R., & Picard, R. W. (2001). An affective model of the interplay between emotions and learning. In Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT‘01) (pp. 43–46). Madison, WI: IEEE Computer Society.
Luckasson, R., Borthwick-Duffy, S., Buntinx, W. H. E., Coulter, D. L., Craig, E. M. (P.), Reeve, A., … The AAMR AD HOC Committee on Terminology and Classification. (2002). Mental retardation: Definition, classification, and systems of supports (10th ed.). Washington, DC: American Association on Mental Retardation.
Nakamura, J., & Csikszentmihalyi, M. (2002). The concept of flow optimal experience and its role in development. In C. R. Snyder & S. J. Lopez (Eds.), Handbook of positive psychology (pp. 89–105). New York, NY: Oxford University Press.
Nakamura, J., & Csikszentmihalyi, M. (2009). Flow theory and research. Handbook of positive psychology. In C. R. Snyder & S. J. Lopez (Eds.), Handbook of positive psychology (pp. 195–206). New York, NY: Oxford University Press.
Nesta. (2019). Educ-AI-tion rebooted? Exploring the future of artificial intelligence in schools and colleges. Retrieved from https://www.nesta.org.uk/repor t/education-reboo ted/
Nganji, J., & Brayshaw, M. (2017). Disability-aware adaptive and personalised learning for students with multiple disabilities. International Journal of Information and Learning Technology, 34(4), 307–321. https://doi.org/10.1108/IJILT -08-2016-0027
Ocumpaugh, J., Baker, R. S. J., & Rodrigo, M. M. T. (2012). Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual Version 1.0 (Technical report). New York, NY: EdLab.
Orekhova, E., & Stroganova, T. (2014). Arousal and attention re-orienting in autism spectrum disorders: Evidence from auditory event-related potentials. Frontiers in Human Neuroscience, 8, 34.
Porayska-Pomsta, K., Alcorn, A. M., Avramides, K., Beale, S., Bernardini, S., Foster, M. E., … Smith, T. J. (2018, December). Blending human and artificial intelligence to support autistic children’s social communication skills. ACM Transactions on Computer-Human Interaction, 25(6), Article 35, 35 pages. https://doi.org/10.1145/3271484
Rose, R. M., & Blomeyer, R. L. (2007). Access and equity in online classes and virtual schools. North American Council for Online Learning. Retrieved from https://www.inacol.org/resource/access-and-equity-in-online-classes-and-virtual-schools
Schalock, R. L., Borthwick-Duffy, S. A., Bradley, V. J., Buntinx, W. H. E., Coulter, D. L., Craig, E. M., … Yeager, M. H. (2010). Intellectual Disability: Definition, Classification, and Systems of Supports (11th ed.). Washington, DC: American Association on Intellectual and Developmental Disabilities. http://www.aaidd.org/ISBN-978-1-9353-0404-3
Scheiter, K., Schubert, C., Schuler, A., Schmidt, H., Zimmermann, G., Wassermann, B., … Edera, T. (2019). Adaptive multimedia: Using gaze-contingent instructional guidance to provide personalized processing support. Computers & Education, 139, 31–47.
Sumi, A. I., Zohora, M. F., Mahjabeen, M., Faria, T. J., Mahmud, M., & Kaiser, M. S. (2018). fASSERT: A fuzzy assistive system for children with autism using internet of things. In Brain informatics, lecture notes in computer science (pp. 403–412). Arlington, TX: Springer International Publishing.
Taub, D. A., McCord, J. A., & Ryndak, D. L. (2017). Opportunities to learn for students with extensive support needs: A context of research-supported practices for all in general education classes. The Journal of Special Education, 51(3), 127–137.
Thompson, N., & McGill, T. J. (2017). Genetics with Jean: The design, development and evaluation of an affective tutoring system. Education Technology Research and Development, 65, 279–299.
Tsatsou, D., Pomazanskyi, A., Hortal, E., Spyrou, E., Leligou, H., Asteriadis, S., … Daras, P. (2018). Adaptive learning based on affect sensing. Artificial Intelligence in Education, AIED 2018. Lecture Notes in Computer Science, 10948.
Vrieze, S. I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228–243.
Wang, L., Zhang, Z., McArdle, J. J., & Salthouse, T. A. (2009). Investigating ceiling effects in longitudinal data analysis. Multivariate Behavioral Research, 43(3), 476–496.
Yadegaridehkordi, E., Noor, N. F. B. M., Ayub, N. M. B. A., Affal, H. B., & Hussin, N. B. (2019, December). Affective computing in education: A systematic review and future research. Computers & Education, 142, 103649. https://doi.org/10.1016/j.compe du.2019.103649
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.
Aslan, S., Alyuz, N., Tanriover, C., Mete, S. E., Okur, E., D’Mello, S. K., & Esme, A. A. (2019). Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. Presentation at Computer Human Interaction (CHI) 2019 (paper 304), Glasgow, Scotland.
Athanasiadis, C., Hortal, E., Koutsoukos, D., Lens, C. Z., & Asteriadis, S. (2017). Personalized, affect and performance-driven computer-based learning. Proceedings of the 9th International Conference on Computer Supported Education—Volume 1: CSEDU. Porto, Portugal, April 21-23, 2017.
Baker, R. S. J., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241.
Barros, M., Galea, M., Leiva, V., & Santos-Neto, M. (2018). Generalized Tobit models: Diagnostics and application in econometrics. Journal of Applied Statistics, 45(1), 145–167.
Basawapatna, A. R., Repenning, A., Han Koh, K., & Nickerson, H. (2013). The zones of proximal flow: Guiding students through a space of computational thinking skills and challenges. In Proceedings of the ninth annual international ACM conference on International computing education research (pp. 67–74). San Diego, CA.
Bertini, E. S., & Kimani, S. (2003). Mobile devices: Opportunities for users with special needs: Mobile HCI. In L. Chittaro (Ed.), Lecture notes of computer science (Vol. 2795, pp. 486–491). Udine, Italy: Springer Verlag.
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Henry, M., … White, J. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology & Evolution, 24(3), 127–135.
Boulton, H., Brown, D., Standen, P., Belmonte, M., Kwiatkowska, G., Hughes-Roberts, T., & Taheri, M. (2018, March 5–7). Multi-modalities in classroom learning environments. INTED2018 Proceedings: 12th Annual International Technology, Education and Development Conference. Valencia, Spain: IATED Academy.
Bozdogan, H. (1987). Model selection and Akaike’s Information Criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370.
Chickerur, S., & Joshi, K. (2015). 3D face model dataset: Automatic detection of facial expressions and emotions for educational environments. British Journal of Educational Technology, 46(5), 1028–1037.
Colley, A. (2013). Personalised learning for young people with profound and multiple learning difficulties. London, UK: Jessica Kingsley.
Craig, S., Graesser, A., Sullins, J., & Gholson, J. (2004). Affect and learning: An exploratory look into the role of affect in learning. Journal of Educational Media, 29, 241–250.
Cukurova, M., Kent, C., & Luckin, R. (2019). Artificial intelligence and multimodal data in the service of human decision-making: A case study in debate tutoring. British Journal of Educational Technology, 50(6), 3032–3046.
D’Mello, S., & Graesser, A. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modelling and User Adapted Interaction, 20(2), 147–187.
D’Mello, S., & Graesser, A. (2012b). Dynamics of affective states during complex learning. Learning and Instruction, 22, 145–157.
D’Mello, S., Dieterle, E., & Duckworth, A. (2017). Advanced, Analytic, Automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123.
du Boulay, B., Poulovasillis, A., Holmes, W., & Mavrikis, M. (2018). Artificial intelligence and big data technologies to close the achievement gap. In R. Luckin (Ed.), Enhancing learning and teaching with technology (pp. 256–285). London, UK: UCL Institute of Education Press.
Grawemeyer, B., Mavrikis, M., Holmes, W., Gutiérrez-Santos, S., Wiedmann, M., & Rummel, N. (2017). Affective learning: Improving engagement and enhancing learning with affect- aware feedback. User Modeling and User-Adapted Interaction, 27(1), 119–158.
Hamari, J., Sherno, D. J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179.
Johansson, T. (2011). Hail the impossible: P-values, evidence, and likelihood. Scandinavian Journal of Psychology, 52(2), 113–125.
Kort, B., Reilly, R., & Picard, R. W. (2001). An affective model of the interplay between emotions and learning. In Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT‘01) (pp. 43–46). Madison, WI: IEEE Computer Society.
Luckasson, R., Borthwick-Duffy, S., Buntinx, W. H. E., Coulter, D. L., Craig, E. M. (P.), Reeve, A., … The AAMR AD HOC Committee on Terminology and Classification. (2002). Mental retardation: Definition, classification, and systems of supports (10th ed.). Washington, DC: American Association on Mental Retardation.
Nakamura, J., & Csikszentmihalyi, M. (2002). The concept of flow optimal experience and its role in development. In C. R. Snyder & S. J. Lopez (Eds.), Handbook of positive psychology (pp. 89–105). New York, NY: Oxford University Press.
Nakamura, J., & Csikszentmihalyi, M. (2009). Flow theory and research. Handbook of positive psychology. In C. R. Snyder & S. J. Lopez (Eds.), Handbook of positive psychology (pp. 195–206). New York, NY: Oxford University Press.
Nesta. (2019). Educ-AI-tion rebooted? Exploring the future of artificial intelligence in schools and colleges. Retrieved from https://www.nesta.org.uk/repor t/education-reboo ted/
Nganji, J., & Brayshaw, M. (2017). Disability-aware adaptive and personalised learning for students with multiple disabilities. International Journal of Information and Learning Technology, 34(4), 307–321. https://doi.org/10.1108/IJILT -08-2016-0027
Ocumpaugh, J., Baker, R. S. J., & Rodrigo, M. M. T. (2012). Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual Version 1.0 (Technical report). New York, NY: EdLab.
Orekhova, E., & Stroganova, T. (2014). Arousal and attention re-orienting in autism spectrum disorders: Evidence from auditory event-related potentials. Frontiers in Human Neuroscience, 8, 34.
Porayska-Pomsta, K., Alcorn, A. M., Avramides, K., Beale, S., Bernardini, S., Foster, M. E., … Smith, T. J. (2018, December). Blending human and artificial intelligence to support autistic children’s social communication skills. ACM Transactions on Computer-Human Interaction, 25(6), Article 35, 35 pages. https://doi.org/10.1145/3271484
Rose, R. M., & Blomeyer, R. L. (2007). Access and equity in online classes and virtual schools. North American Council for Online Learning. Retrieved from https://www.inacol.org/resource/access-and-equity-in-online-classes-and-virtual-schools
Schalock, R. L., Borthwick-Duffy, S. A., Bradley, V. J., Buntinx, W. H. E., Coulter, D. L., Craig, E. M., … Yeager, M. H. (2010). Intellectual Disability: Definition, Classification, and Systems of Supports (11th ed.). Washington, DC: American Association on Intellectual and Developmental Disabilities. http://www.aaidd.org/ISBN-978-1-9353-0404-3
Scheiter, K., Schubert, C., Schuler, A., Schmidt, H., Zimmermann, G., Wassermann, B., … Edera, T. (2019). Adaptive multimedia: Using gaze-contingent instructional guidance to provide personalized processing support. Computers & Education, 139, 31–47.
Sumi, A. I., Zohora, M. F., Mahjabeen, M., Faria, T. J., Mahmud, M., & Kaiser, M. S. (2018). fASSERT: A fuzzy assistive system for children with autism using internet of things. In Brain informatics, lecture notes in computer science (pp. 403–412). Arlington, TX: Springer International Publishing.
Taub, D. A., McCord, J. A., & Ryndak, D. L. (2017). Opportunities to learn for students with extensive support needs: A context of research-supported practices for all in general education classes. The Journal of Special Education, 51(3), 127–137.
Thompson, N., & McGill, T. J. (2017). Genetics with Jean: The design, development and evaluation of an affective tutoring system. Education Technology Research and Development, 65, 279–299.
Tsatsou, D., Pomazanskyi, A., Hortal, E., Spyrou, E., Leligou, H., Asteriadis, S., … Daras, P. (2018). Adaptive learning based on affect sensing. Artificial Intelligence in Education, AIED 2018. Lecture Notes in Computer Science, 10948.
Vrieze, S. I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228–243.
Wang, L., Zhang, Z., McArdle, J. J., & Salthouse, T. A. (2009). Investigating ceiling effects in longitudinal data analysis. Multivariate Behavioral Research, 43(3), 476–496.
Yadegaridehkordi, E., Noor, N. F. B. M., Ayub, N. M. B. A., Affal, H. B., & Hussin, N. B. (2019, December). Affective computing in education: A systematic review and future research. Computers & Education, 142, 103649. https://doi.org/10.1016/j.compe du.2019.103649
Published
2023-12-28
How to Cite
Bhatti, I., Tariq, M., Hayat, Y., Tariq, A., & Rasool, S. (2023). A Multimodal Affect Recognition Adaptive Learning System for Individuals with Intellectual Disabilities. European Journal of Science, Innovation and Technology, 3(6), 346-355. Retrieved from https://ejsit-journal.com/index.php/ejsit/article/view/343
Issue
Section
Articles
Copyright (c) 2023 Iftikhar Bhatti, Mehtab Tariq, Yawar Hayat, Aftab Tariq, Saad Rasool
This work is licensed under a Creative Commons Attribution 4.0 International License.
ISSN 
