For teachers, getting feedback from students is crucial to improve various elements of the course: teaching practices, learning materials, or communication with students or preparing them better for the future workplace. Collecting feedback traditionally comes in the form of a traditional pencil-paper survey or online survey. Those ways of evaluations students usually find boring and fill them out quickly, just to do the last activity for the course. For teachers, answers, and comments written on the paper demands a lot of time to read and make useful conclusions. Online surveys often have low response rates since they are anonymous and voluntary. Typically, students fill them out, rate statements, but open-ended questions that require their opinion in few questions are left empty.
This kind of information is not much use, so it is better to approach students innovatively: using a chatbot that will guide students in a natural, talking way to express their opinions about the course.
Chatbot Hubert was used in three undergraduate IT courses at the Faculty of Organization and Informatics, University of Zagreb at the end of the semester. Teaching and learning activities have switched to a completely online environment due to the global COVID-19 pandemic outbreak. Teachers had to react quickly to adjust lectures, laboratory materials, and ways of examining knowledge for online context. The process was challenging and constructive feedback was needed to see what works well and what should be improved on the course.
Students were invited via LMS to chat anonymously with Hubert. Overall, 76 students participated in course evaluations. From their answers, it is obvious that they accepted this form of collecting feedback very positively: they were very talkative, gave objective comments – mostly positive ones about their teachers and courses. Even though they had an opportunity to give negative feedback, while expressing it they were constructive and zero rude/bad words appeared in their answers. On average, students spend about 8 minutes while talking with the chatbot which is simlar to the time required for other forms for collecting feedback.
From the experience with Hubert for collecting students feedback, the following lessons were learned for creating course evaluation via chatbot:
- create questions short and clear questions that cover both polarities, e.g. positive effects/benefits and negative effects/disadvantages
- Keep in mind that too long surveys can discourage participants.
- Stick to 5 to 6 core open-type questions for which students won’t need more than 10 minutes to answer.
- Ask twice. A chatbot should imitate real-life conversation. Give students a chance to express their opinions freely upon one topic in more than one question.
- Suggest using the assistance of some online translation service for expressing the opinions in a foreign language.
- Test the survey before administering it to students.
In their answers students were rather objective and open, taking into consideration the current pandemic situation and fully aware of challenges during the semester both for teachers and students. If they initially provided just a short answer, chatbot Hubert encouraged them to express their opinion in more detail, for example: “Sure you don’t have anything to add...”, “What else could you say about that?”
Teachers involved in Pilot 1 of Edubots project find collected feedback as very valuable since online teaching continued in 2020. and 2021. Improvement is always needed and the best way to investigate the good and bad sides is to suggest students to start a conversation with a chatbot.
Automated analysis is a powerful Hubert's functionality that allows a more comprehensive and simple understanding of student responses and finding connections in them. From the teacher’s point of view, summary visualization of student’s comments is quite helpful for making steps for improvement of the course, but also individual answers are easy to read. Furthermore, teachers can add topic categories or emotions to students' statements.
.png)
