Retrieval-Augmented Generation Enhanced Immersive System for Education
A Lightweight Web-Based Intelligent Learning System with RAG, MCP, and DH
📄 Paper: Under Review
🎥 Demo Video: Google Drive
🌐 Project Page: (coming soon)
💻 Code Release:
The RISE system is fully implemented and demonstrated in the submitted video. The full codebase is currently undergoing internal review. A partial or full release may be made in a later phase depending on institutional approval and licensing constraints.
RISE is a next-generation intelligent education system that integrates:
- Retrieval-Augmented Generation (RAG) for curriculum-grounded learning
- Model Context Protocol (MCP) for multi-agent orchestration
- Digital Human (DH) technology for immersive teaching
- 360° virtual classrooms, real-time student attention monitoring, and adaptive Q&A
- Learning analytics, virtual home-visit summaries, and online assessment
RISE is designed to run fully on lightweight web-based infrastructure, making high-quality intelligent education accessible even on low-end devices and in resource-limited regions.
- Interactive panoramic scenes (e.g., virtual field trips, cultural learning scenarios)
- Digital human instructor narrating auto-generated lesson content
- Scene switching synchronized with educational narratives
- Webcam-based head pose & gaze estimation
- Lightweight on-device inference
- Non-intrusive attention reminders to maintain engagement
- RAG-based retrieval grounded in textbooks, syllabi, or course documents
- Contextualized explanations aligned with curriculum and student level
- Supports adaptive questioning and misconception correction
- Unified context memory for digital human, RAG agent, and tool calls
- Transparent orchestration of teaching, retrieval, and assessment tasks
- Extensible architecture for new educational agents
- Multi-modal learner analytics
- Virtual home-visit summaries for parents
- LLM-powered exam generation, scoring, and personalized feedback
RISE consists of two major layers:
1. Immersive Delivery Layer
- Digital Human instructor
- 360° classroom
- Attention monitoring
2. Intelligent Processing Layer
- Educational RAG module
- LLM-based generation
- MCP agent coordinator
The two layers form a closed-loop learning workflow, enabling continuous adaptation and personalized instruction.
👉 Watch the full demonstration
The video includes:
- Immersive classroom tour
- Digital human live teaching
- Attention monitoring demo
- Learning analytics dashboard
- Online assessment generation
The full implementation (frontend + backend + data pipeline) will be released soon.
Planned release includes:
- Web-based immersive classroom engine
- MCP multi-agent orchestration server
- RAG retrieval pipeline (OpenAI / Qwen / Llama compatible)
- Digital human rendering modules
- Attention estimation module (Lightweight)
- Demo scripts and teacher evaluation tools
Current Status: ✔ System implemented ✔ Demo video released ✔ Open-source release scheduled ✖ Repository undergoing cleanup & IP review
TBA
RISE adheres to ACM’s policy on research involving human participants.
- Webcam-based signals never store biometric data
- All data is anonymized or processed locally
- Users may opt out at any time
- Demo video uses synthetic or consented samples
This project is developed by members of our research lab.
All work represents the effort of the authors and contributors only.
The system and its materials do not represent the views, policies, or official positions of any institution.
No institutional endorsement is implied.