Machine learning meets education and VR

Task description:

Implementing a Virtual Reality Classroom using Unity with Multimodal Data Collection and Generative Models

Objective:

The objective of this task is to create a virtual reality classroom using Unity and integrate it with multimodal data collection features such as eye-tracking. Additionally, generative models will be used to create virtual avatars for teachers and students, providing a more personalized and immersive experience for the users.

Key deliverables:

• Virtual reality classroom: Develop a virtual reality classroom environment that mimics the physical classroom setting. The virtual classroom should include features such as desks, chairs, whiteboards, projector screens, and other classroom equipment.
• Multimodal data collection integration: Implement eye-tracking technology to capture data such as gaze direction, pupil dilation, and fixation points of the user. This feature will enable educators to analyze and evaluate students' cognitive processes during the learning process.
• Generative models for avatars: Use generative models to create virtual avatars for students and teachers in the virtual classroom. The generative models should allow for a high degree of customization to provide a more personalized experience for the users.
• Interactive elements: Create interactive elements that enhance the immersive experience of the virtual classroom. For example, a teacher could use a virtual whiteboard to write and draw diagrams, and students could use virtual hand-raising or gesture-based features to interact with the teacher.
• User interface: Develop a user interface that enables users to navigate through the virtual classroom environment seamlessly. The user interface should also allow users to access additional features such as the virtual whiteboard, student assessment tools, and class schedules.
• Testing and debugging: Conduct extensive testing and debugging of the virtual classroom environment to ensure that it functions seamlessly with the multimodal data collection and generative model features.

What we offer:

• Flexible working time.
• Cosy working place at Munich city center (Marsstraße 20-22, near Hbf).
• Luxurious hardware (laptops with RTX 3080Ti, desktops with RTX 4080, Varjo XR-3 VR/AR headset, Tobii eye trackers, etc.).
• First-hand data.

Skills required:

• Basics & experience in Unity and C#.
• Basics & experience in Python.
• Familiarity with multimodal data collection techniques such as eye-tracking.

Skills that are good to have:

• Experience in developing virtual reality applications.
• Proficiency with machine learning frameworks like Pytorch, Tensorflow, and Keras.
• Understanding of generative models and experience with their implementation.
• Understanding of user interface design and user experience.
• Ability to test and debug complex applications.

Others:

Estimated timeframe: The coarsely estimated timeframe for this task is approximately 6-8 months with possible extension, depending on the complexity of the virtual reality classroom, the level of integration with the multimodal data collection and generative model features, and the degree of customization of the virtual avatars.