Paras Sharma

General-purpose automatic speech recognition (ASR) systems do not always perform well in goal-oriented dialogue. Existing ASR correction methods rely on prior user data or named entities. We extend correction to tasks that have no prior user data and exhibit linguistic flexibility such as lexical and syntactic variations. We propose a novel context augmentation with a large language model and a ranking strategy that incorporates contextual information from the dialogue states of a goal-oriented conversational AI and its tasks. Our method ranks (1) n-best ASR hypotheses by their lexical and semantic similarity with context and (2) context by phonetic correspondence with ASR hypotheses. Evaluated in home improvement and cooking domains with real-world users, our method improves recall and F1 of correction by 34% and 16%, respectively, while maintaining precision and false positive rate. Users rated .8-1 point (out of 5) higher when our correction method worked properly, with no decrease due to false positives.

Culturally responsive computing (CRC) curricula engage learners in reflections on power and identity as they build technologies. Open-design tasks, with learner-chosen goals and multiple pathways to achieving them, are common in CRC and could be enhanced using adaptive technologies. Current adaptive technologies function best in well-defined learning trajectories. However, it is unclear how to design these technologies to respond to individual learners’ ideas in open-design settings. In this paper, we prototype a learning system that uses multimodal sensing, log data, and reflective dialogues to build explanatory learner models in open-design settings. We deploy our system in a 2-week summer camp with middle school girls and evaluate the system’s effectiveness to understand learner goals and activities. We show the importance of multiple modalities in making inferences about learner goals and activities.

Open-design environments, under open-ended learning environments, provide high agency to learners to define their goals and pathways toward those goals. However, such environments could be difficult to navigate through for some learners due to this openness in goals and activities. Our long-term goal is to build intelligent pedagogical agents to support learner activities within these environments using different dialogue strategies. In this work, we propose to build a Simulated Students system to emulate learner activities in an open-design environment. We hope to use this simulated data in future work to distill knowledge from Large Language Models (LLMs) to build adaptive, and context-based reinforcement learning dialogue models for learner support. We present the early results and proposed directions of our ongoing work and seek advice on how the different strategies that we propose to use in this work could be further used to build adaptive and context-based dialogue models for effective learning in open-design environments.

Open-ended learning environments (OELEs) involve high learner agency in defining learning goals and multiple pathways to achieve those goals. These tasks involve learners transitioning through self-regulated learning (SRL) phases by actively setting goals, applying different strategies for those goals, and monitoring performance to update their strategies. However, because of the flexibility, how learners react to impasses and errors has a critical influence on their learning. An intelligent pedagogical agent (IPA) continuously modeling learner activities could help support learners in these environments. However, this continuous comprehension of behaviors and strategies is difficult in OELEs with evolving goals, ill-defined problem structures, and learning sequences. In this paper, we draw from the literature on SRL phases and cognitive states to investigate the utility of two different methods, Sequence Mapping, and Hidden Markov Models, in building learner activity models from log data collected from a summer camp with 14 middle school girls in an open-design environment. We evaluate the effectiveness of these models separately, and combined, in identifying 7 states: Forethought, Engaged Concentration, Acting, Monitoring, Wheel Spinning, Mind Wandering, and Reflect and Repair. Lastly, we recommend dialogue intervention strategies for an IPA to support learning in OELEs.