Student Feedback chatbot Hubert best practice – evidence from IT undergraduate courses

For teachers, getting feedback from students is crucial to improve various elements of the course: teaching practices, learning materials, or communication with students or preparing them better for the future workplace. Collecting feedback traditionally comes in the form of a traditional pencil-paper survey or online survey. Those ways of evaluations students usually find boring and fill them out quickly, just to do the last activity for the course. For teachers, answers, and comments written on the paper demands a lot of time to read and make useful conclusions. Online surveys often have low response rates since they are anonymous and voluntary. Typically, students fill them out, rate statements, but open-ended questions that require their opinion in few questions are left empty. 

This kind of information is not much use, so it is better to approach students innovatively: using a chatbot that will guide students in a natural, talking way to express their opinions about the course.

Chatbot Hubert was used in three undergraduate IT courses at the Faculty of Organization and Informatics, University of Zagreb at the end of the semester. Teaching and learning activities have switched to a completely online environment due to the global COVID-19 pandemic outbreak. Teachers had to react quickly to adjust lectures, laboratory materials, and ways of examining knowledge for online context. The process was challenging and constructive feedback was needed to see what works well and what should be improved on the course. 

Students were invited via LMS to chat anonymously with Hubert. Overall, 76 students participated in course evaluations. From their answers, it is obvious that they accepted this form of collecting feedback very positively: they were very talkative, gave objective comments – mostly positive ones about their teachers and courses. Even though they had an opportunity to give negative feedback, while expressing it they were constructive and zero rude/bad words appeared in their answers. On average, students spend about 8 minutes while talking with the chatbot which is simlar to the time required for other forms for collecting feedback.

Figure 1: Hubert chatbot interface in course evaluation


From the experience with Hubert for collecting students feedback, the following lessons were learned for creating course evaluation via chatbot:

In their answers students were rather objective and open, taking into consideration the current pandemic situation and fully aware of challenges during the semester both for teachers and students. If they initially provided just a short answer, chatbot Hubert encouraged them to express their opinion in more detail, for example: “Sure you don’t have anything to add...”, “What else could you say about that?”

Teachers involved in Pilot 1 of Edubots project find collected feedback as very valuable since online teaching continued in 2020. and 2021. Improvement is always needed and the best way to investigate the good and bad sides is to suggest students to start a conversation with a chatbot. 

Automated analysis is a powerful  Hubert's functionality that allows a more comprehensive and simple understanding of student responses and finding connections in them. From the teacher’s point of view, summary visualization of student’s comments is quite helpful for making steps for improvement of the course, but also individual answers are easy to read. Furthermore, teachers can add topic categories or emotions to students' statements.

 

Dashboard showing the overall student’s course experience


Summary visualization of student's answers about positive aspects of the course


Summary visualization of student's answers about positive aspects of the course


Example of students individual answers with Hubert's topic tag for sentiment analysis


About Edubots
Edubots is an ongoing Erasmus+ Knowledge Alliance, funded by the European Commission for 2020-2021. The aim is to document and develop best practice use of chatbots in Higher Education, and to create a learning community of Educators in Europe.

Consortium partners

The project was initiated by 6 partners from Norway, Sweden, UK, Croatia, Cyprus and Spain.

EDUBOTS - Best Practices of Pedagogical Chatbots in Higher Education
Reference Number: 612466-EPP-1-2019-1-NO-EPPKA2-KA

Consortium leader
EdTech Foundry AS
Hoffsveien 6a
0275 Oslo
Norway
Organizational number: 998 716 454
Project Manager: Gregor Jarisch  ( send email )