Organization of the University Signal Processing Course
Pedagogy interested me for a very long time and, for many years, as a student, I was educated, but at the same time plagued and torn by the existing organization of education, I thought about how to improve it. Recently, I have more and more often been given the opportunity to test some of the ideas in practice. In particular, this spring I was given the opportunity to read the course “Signal Processing” at the Polytechnic University (SPBPU). His organization, especially the organization of reporting, is the first experiment, the results of which seem to me to be somewhat successful, and in this article I want to tell about the organization of this course.
I still do not have a clear understanding of what should be read in the course with such a name, but in general it is a course about what and how you can automatically do with images, sound, text, video and other examples of natural and artificially generated signals . By what was read earlier and would be most useful, this is a solution to problems with a semantic gap between the input signal and what one wants to understand. This article is not about course content - even in Russian there are quite a few videos of good courses on similar topics.
here at least in the near future the current link to the course presentations that are on my google disk . Most of what is there is torn from Anton Konushin's courses, csc and various Internet articles that are in the top most relevant ones. Nevertheless, in some places there are things that I could not find clear descriptions and tried to invent my own, in some places there are Russian descriptions of what I could find only in English - this applies especially to clustering, for example, to the mcl algorithm.
The outline of the article is approximately the following: first, the course organization I selected is briefly described, then there is a story about the problems that I consider useful to solve, then how I tried to do this while reading the “signal processing” course and how I evaluate the results, what problems I see what ideas to solve them are. All this is nothing more than my thoughts and ideas, and I will be very happy with comments, objections and more ideas! Moreover, this is all written in many respects precisely in the hope of receiving your ideas and comments. Still, perhaps, this text will help someone to find interest in high-quality teaching, in spite of all that is happening around.
General course organization chart
The course has two components: theoretical and practical. Both parts are very important: a theoretical one gives a great overview of existing algorithms and ideas for constructing them for solving problems with a semantic gap; practical should give at least some review of existing libraries, as well as practice skills in constructing your own algorithms. Accordingly, for both parts, there was a need for stimulating accountability for their study, which sets the main line of work for students.
As usual, the theoretical part consisted of lectures. After each lecture, home students were given a wide list of questions on the lecture, consisting of both routine questions about the details of the narration and creative questions about how and in what cases certain ideas could be improved and where they can be used before students are asked to come up with their own questions. according to the lecture (and you can also answer them). All questions were laid out in a post in the VKontakte group, the answers had to be written in the comments: one could either answer a question that had not yet been raised by anyone, or comment on / add an already existing answer, including one made by another student. The scope for closely related to the subject of creativity, in my opinion, was huge!
The addition to the answers to the questions should have been the ranking: after the deadline, the students had to drop me the names of the respondents ranked according to the deserved ratings. Also, comments on rankings were welcomed. After all this, I finally gave points for the lecture. According to the results of these points and a number of additional buns, including those growing from the practical part of the course, marks for the semester were given. Dissenters and idlers could try to raise the mark on the harsh exam (you can use absolutely everything, but I strongly ask for understanding).
The general message of the theoretical part was something like this: I try to give insanely much material, hoping that all students will find in it a lot of new and useful things. However, I do not require them to delve into everything, they can either choose interesting / useful moments for themselves and go deeper into them, or do a little bit of everything. I take the exam more as a fine for those who worked poorly in the semester than as a norm.
The practical part consisted of
- three mini-labs in which students had to run ready-made code actively using different libraries and pick up data on which it works well or does not work well,
- term paper, in which students were required to independently solve the problem with a semantic gap. They could take the initial task either from the proposed ones, or they could choose and agree with me. Then they had to come up with a solution, encode it, see that it works the first time, it does not work well and then try to improve it, following my own and my advice. The ideal would be to achieve a really good quality, convincing students that even in this area, patience and work in the right direction will grind everything, but, naturally, this can not be hoped for all the time.
All this was necessary to do on the test. The quality of work and the amount of effort expended could vary significantly. Due to the great effort it was possible to get more extra points in addition to the lecture notes.
It happened in the spring semester of the 4th course, when the semester ends a little more than a month earlier because of the bachelor's. That is, I had about 10-11 weeks.
I also had an insider in the form of a sister, who studied in one of two groups, in which I lectured. My sister sometimes could stop my crazy ideas with stories about her vision of the real situation in the group and loading in other subjects. In combination with the successful topic of the course, fate really favored the conduct of experiments more than ever!
Reflections on the problems you want to solve
In this section I try to tell about the problems, reflections on which led me to the described structure of the course. These problems are mainly related to two facts:
- There are creative and active students who are able to independently organize their studies in the direction they really need. Driving everyone under the average level, the existing system of education in higher education institutions often creates for such students difficult, nervous and meaningless conditions.
- Many teachers, unfortunately, are not interested in the quality of their work. Often such disinterest is a result of frustration in students. But the bad work of students cannot but be a consequence of the poor work of teachers. The situation can improve if high-quality work will benefit the teachers themselves, not just the students.
Of course, there are still many problems that are not very connected with both the first and the second. For example, what to do with those students who are not able to organize themselves on their own? Or those who seem to be trying, but still can not do anything?
The problems associated with the two facts described above have been most suffered by me, and I thought a lot about solving them. It seems to me that there is at the same time the decisive “silver bullet” of them: if smart students are in comfortable conditions, then they can be of great benefit to teachers.
Let's start with the motivation of the teacher. Naturally, it is necessary for a good course. So, from the course, the teacher can receive:
- Money. Ours is often symbolic. Especially for those who teach IT well, this money is absolutely ridiculous. As a rule, these people have or may have several times greater earnings in another job. And they just can not teach well just for the sake of wages.
- The stimulus is much better to immerse yourself in the material. I was very excited about the popularity of my lectures. And, at least for now, I was very afraid of the condemning views of the students and their negative opinion: “there’s nothing else to do but to force us to waste time on some nonsense with which he himself couldn’t or didn’t find it necessary to understand” .
- The results of student immersion in the material. You can create an atmosphere that will stimulate reasonable questions from students in lectures. Such questions can greatly help the teacher: to point out some mistakes and flaws, to encourage people to look at things from the other side, and maybe even make them understand something new.
- You can stimulate the activities of students, beyond the scope of the material readable in lectures. Then they can collect a lot of new information and produce results in an already somehow processed form. Yes, it is all the same then difficult to understand and verify. But it is precisely in the course of such checks that the horizons are broadened. And there is one more bonus: if something is not clear, sometimes you can ask a student instead of understanding it yourself. Such a question will also be a test of how well the student has figured out.
- Training to communicate with people. Training assessing people, understanding what can be expected from them, including depending on their own actions. You can try to assess in advance which student will cope with the task well and on time, which one is bad, which one will do what is needed, but for a very long time. Train different management approaches (reminders, etc.). Understand how easy it is and how exactly students (and probably not only them) can manipulate you. The space for experimentation is huge. The results of the experiments can be seen relatively quickly.
- Practice competent presentation of thoughts, lectures and other public speaking skills. Training understanding of poorly formulated answers and questions by students (sometimes all this has to be done on the fly - you can practice your own reaction).
- The results of testing simple ideas in practice by the hands of students. Both the results of testing your own ideas, and ideas that have occurred to the student can be useful. If you find a task that is really interesting for a student, it is likely that the student will generate good ideas and check them qualitatively.
- 'Free' use of students to solve their practical problems.
Распространено мнение, что с этого преподаватели получают наибольшую выгоду. Я в это довольно долго верил, но с каждым следующим экспериментом моя вера в это убывает. Пока у меня был только один студент, от сотрудничества с которым я в итоге получил именно то, что хотел, вовремя, и действительно сэкономил свое время. Этого же студента у меня, наверное, получилось научить лучше, чем других. Правда и тут, потом, в ходе проекта, выяснилось, что решение этой задачи мне нужно в немного другом виде, но это уже однозначно моя вина.
За всеми остальными студентами, с которыми я сталкивался, без конца приходилось бегать, напоминать им о научной работе, объяснять им одно и тоже по нескольку раз. В конце концов, я получал от них что-то очень странное, и зачастую в тот момент, когда я эту задачу уже решал самостоятельно. Плохо понимаю, насколько такой формат полезен для них (вроде они и тренируются чего-то делать, но как-то очень некачественно). У меня же этот процесс съедает много нервов и времени. Единственный плюс: иногда, в ходе обсуждений, мое внимание обращается на некоторые детали задачи, которых я раньше не замечал.
- Fame, prestige - with high-quality teaching
- Visibility of the results of their activities and grateful students. True, it is often difficult to understand the truth, students are often grateful not at all for what they need.
- Meet future specialists in their field. It is better to understand them, to understand what the new generation lives on. You can select those who liked, and then invite to work.
That's all that I managed to collect. For myself, I try to realize as clearly as possible what, besides pleasure and prestige, I hope to gain from reading the course. What should it be so that I would be ready to pay for it with my time for the entire semester? Without this understanding it is hard to believe that it is possible to conduct a course well. Own motivation must be considered when thinking through the course structure.
Comfortable conditions for advanced students
The second part of the requirements for the structure of the course is focused on creative and active students who have a good idea of what they need. Despite the fact that many teachers confidently deny even the possibility of the existence of such students, they definitely exist in advanced universities. By senior courses, their number increases significantly, especially with high-quality training. And it is smart students who are the hope of our fatherland and science.
In almost all universities, training is far from being as effective as it could be. At lectures, students are often told something that might be interesting, but strange: if necessary, then in some world, to which students have not yet grown. It often happens that advanced students have already heard or read about these things, understood, and then forgot - now they are forced to listen again. Often, students have to do strange practical tasks that the teacher came up with just because he believed that students need at least something to download. To write and correct reports that teachers often do not accept from the first time simply because it seems to them unworthy, and you should at least teach something.
If this all falls on people who otherwise would not do anything - this is probably not bad. As practice shows, by the end of training these people understand something, most of them are quite suitable for working in their specialty.
But it happens that such a system is applied to advanced students who already have their own plan of action, their own work, their own understanding of where to go. Moreover, this understanding is generally correct, and the work can be made very popular, if a little bit corrected. And here these students are lectured with abstract theoretical material, ill-considered practical tasks and reports that need to be written and corrected without end. Even if all this is necessary, it is much more effective to link this with the student’s scientific interests. So that he understands how this information will help him in practice.
Otherwise, if the student does not understand, only a small part will be learned. Yes, and it will soon be forgotten if it is not used closely in other courses. There will be only a general idea. As well as from non-core uninteresting school subjects or in nothing interested students. There may still be an understanding of where to run to figure it out.
But getting this information from students takes quite a lot of personal time. Many advanced students could spend it with great benefit. Such people are ready to absorb the knowledge that they need almost on the fly and with amazing efficiency, especially at senior courses.
Yes, maybe your course is exactly what an advanced student lacks. And he, poor fellow, does not understand. But abstract theoretical lectures are unlikely to help him. If you understand the essence of some interesting work and in the right place, advise to apply at least a small piece of knowledge that you give, - the student will surely understand and appreciate. Especially if your proposal for improvement will help to achieve a better result.
In fact, of course, everything is somewhat more complicated. Not all useful knowledge can be applied in an area that is interesting to the student. Then, especially if it happens at senior years, it would be good to try to understand what is more useful for a student: to do what you think is necessary, or what he considers necessary for himself. And act in accordance with this.
In this course, I almost did not have such a problem: the course on solving problems with a semantic gap seems to me applicable everywhere and useful to everyone. In fact, this is a course on designing algorithms and models in difficult situations. I believe that it is useful for everyone to understand that this exists, and how it works at least at the top level. The course also trains well modeling skills and a reasonable approach to solving many problems.
Much more I was afraid to tell only what many students already know. I didn’t want to force them to solve tasks that would not teach anything. I wanted advanced students not to be forced to do tasks for a tick, just to get credit.
For this you need to understand good students, to understand what they know and what they aspire to. To interrogate them, to get opinions, to look at the results of work, to understand something by them. Make sure that students do not fear me. Not afraid to answer the question correctly. Not afraid to criticize my line.
But you have to be not only not scary, but demanding. Even to advanced students, reasonable demands helps, they are built. The time allotted for the assignment helps to understand which way to choose, how deep to dig in, when to ask for help. Result requirements help to understand what to concentrate on. And all this organizes, helps to prioritize between a lot of fallen cases.
Being not scary and demanding for a teacher is far from easy. Especially if there are many students. Lazy people are more demanding. With them you will suffer to be fair in each particular case. For advanced students - the opposite. They are much more afraid of the tyranny of teachers than others. Because they have more at stake, more dependent on offset and departure. The very first unreasonable demand casts doubt: “Is the teacher intelligent? Will it adequately respond to my criticism? ”. Each following this doubt strengthens, the teacher in the eyes of the student turns into a madcap who needs to be pleased by spending as little time as possible.
It seems that only a reasonable rigid reporting system can solve the problem. Premeditated, which will not change in the semester. Compliance with this system should become more important than the teacher's opinion, no matter how strange it may sound. This dictates a high level of requirements for the reasonableness of the original system. It is clear that it is impossible to foresee everything, and I don’t want to waste time. Therefore, it is possible to clearly indicate the boundaries at which the teacher acts at his discretion. For example, the lab submitted after the deadline will be checked when it is not known, and after two lab failures in time, the consequences can be unpredictable. Further, depending on the reasons that led to this, you can either pardon or punish. But if the done satisfies the requirements, the teacher must do what he promised.
So, it was necessary to come up with a rigid rational reporting system. It needs to be more loyal to reasonable students. Positively taken into account all the useful things that they can come to mind and that will be associated with the course. But she did not give good marks for anything, but rather stimulated precisely to quality work.
It is also important that the reporting system be trusted and calmly felt with it. So that the student could set himself the task of doing everything at the beginning of the semester, get an assessment and feel at ease. Do not be afraid that the teacher in the middle of the semester will think: “he is doing something all too well. Probably, it is possible to give more complex tasks and make the assessment depend on them. ”
Also, as follows from the previous section, the reporting system should take into account the wishes of the teacher. And it turned out that many of the requirements have already been taken into account: they coincided with the requirements of loyalty to reasonable students and to high-quality work. If advanced students are free to ask questions, they will be asked, including what the teacher does not know. If you can go beyond the course - they will come out and find new information. If they understand what they are doing and why they will do it qualitatively. And information about the results of such experiments naturally broadens the teacher’s outlook. Let not immediately, but sooner or later there will be something new, useful for him.
Contented smart student - contented teacher!
The reporting system cannot motivate students without a reasonable assessment of their activities. How to evaluate the results of the semester, which student deserves a greater assessment, and which one less?
The most commonly used criterion with us is the grade for the exam. The teacher is trying to understand in the course of some communication or written how well the student understands the topic at the time of the exam. This in itself is difficult. Often, almost everyone who understands, but is timid and unable to speak, students receive lower marks than students who do not know the subject, but dodgy and arrogant. The written exam allows you to lower the degree of arrogance that a student can use. But interactivity is lost: it is impossible to understand whether a student understands what he did not write (and what he wrote). Another problem is cheating. I am familiar with some masters of pedagogy, whose assessments inversely correlated students' knowledge: the tasks covered an insane amount of material and even one who prepared well could not pass this on a normal grade. But the copiers received 5 and the teacher based on them confidently concluded that you can cope - if you prepare.
Ideas to solve these problems exist. But even if these problems can be solved, it will still not be possible to assess the residual knowledge of the student.
The probability of increasing the amount of residual knowledge increases if the knowledge in the student’s head is not only at the time of the exam, but also during most of the course. And if knowledge is still supported by practical activities, they will most likely remain. It turns out that a student’s knowledge would be well appreciated several times in a semester. And in the end, put the mark on the machine if the student worked well during the semester. But at the same time, a general overview of the course that the student was supposed to receive during the preparation for the exam is lost.
The problems do not end there: all students are different, it happens that one thing is obvious, and another one has to think long about it. Perhaps, it is fair to evaluate not only their final knowledge, but also the amount of effort expended? How to evaluate them? What is better: overestimate a student or underestimate? Is it advisable when evaluating students to compare their level with the level of a group / stream? On the one hand, it seems yes: if a problem is with the whole flow, it means that the teacher has worked poorly. On the other hand, lowering the bar will contribute to a drop in the level of students.
There are systems in which students are initially placed in conditions of dependence on other students: for example, as I understood, in CSC courses on similar subjects, all students are clustered and the student receives an assessment according to which cluster his score turned out to be. Such approaches increase competitive interest, but create uncertainty that can irritate students once again, and can also hinder teamwork.
All this I was so normal and could not think about. As a person who himself was a student most recently, it seems to me that the main thing is to ensure that a person with great difficulty in the semester can provide himself with a greater assessment - the one he wants. There should be a lot of ways to get this assessment: for practice and for theory in a variety of formats. But, if the course is important, it is necessary that the student can get a good grade only if he either really did a good job and was very advanced, or knows the course initially at the teacher level. I tried to think of such a system.
So I tried to make the course as comfortable and useful as possible for diligent students. From them, I waited for questions and messages that would push my knowledge further. But the problem, how not to forget about the rest, of course, was also relevant. Here the situation is very unfavorable: I knew that as a result of a number of reasons, by the 4th year many groups come in an already very collapsed state: most of the students still close the previous semester; There are those who can no longer bring themselves to do almost anything to study on time and who have gotten away with it for years. Timely feedback is incredibly important for the teacher: you can rebuild in time.
Detailed course organization chart
Possible patterns of reporting and teacher's behavior, solving the problems listed above, I began to actively think out when I was in my fifth year. I have already tried to test some of them, but there were a lot of reasons for which it was impossible to get relevant estimates. With all this in mind, I put together a course and told you exactly what happened.
First question: what do I want from this course? First of all, it was interesting for me to try my ideas in practice and really wanted something good to come out of them. The second most important argument was the improvement of one’s own knowledge, but in general, to some extent, all of the above teacher’s goals from pleasure to prestige took place.
In particular, in connection with the goal of improving knowledge, I would like the students not to be afraid of me, to be free to ask questions and openly express discontent with what is happening - all this would be good incentives for me. I also wanted to receive knowledge from them - I wanted to stimulate them to collectively expand the information received and not to limit the scope of their activities. Try to avoid thoughtless repetitions in their activities.
Thus, there appeared a consideration that students should answer various questions on the course (including creative ones and those to which I do not know the answers), see each other's answers and complement them. But not to duplicate - in this way, I do not need to figure out who has written off and who is not, and for students there is an extra reason to expand my knowledge, to go beyond what has already been told in lectures and written by group mates. It also becomes necessary to understand what they wrote ahead of them. It can also help stimulate early answers: the initial choice of possible questions is a little more.
A VKontakte group was created, numbered questions were laid out in it after each lecture (15 pieces, rather voluminous). to which students answered in the comments, complementing each other's answers.
- На повторение рассказанного на лекции. Иногда ответ на такой вопрос можно было найти напрямую в презентации лекции, передаваемой студентам после её прочтения.
- На придумывание практических примеров использования рассказанного.
- На выявление затронутых на лекции проблем в рассказанных алгоритмах. А также на продумывание алгоритмов, решающих обозначенные на лекции проблемы. Подразумевалось, что студенты могут как тащить алгоритмы из других источников, так и придумывать свои.
- На оценки эффективности рассказанных алгоритмов — в том числе, для лучшего понимания самих алгоритмов.
- На сравнение алгоритмов, решающих схожие задачи.
- На математические доказательства некоторых использованных или связанных фактов (например, теоремы о свертке, теоремы Котельникова).
Нужно сказать, что на лекциях я формальные доказательства почти не рассказывал, больше использовал доказательства “на пальцах” с большим количеством приближений и упрощений. Во-первых, потому что я сам формальные доказательства в практической жизни не особо использую и, как следствие, в них не очень хорошо разбираюсь; во-вторых, считаю, что на 4-ом курсе основной упор уже надо делать на практическое понимание, а не на теорию, без которой в целом можно жить.
- Еще причина: просмотренные мной курсы лекций по данной тематике, обильно снабженные теоретико-математическими определениями и доказательствами мне казались либо очень сложными для понимания всего сразу, либо затрагивающими слишком маленький объем информации — погружение в них мне сейчас кажется закапыванием в то, что почти не будет использоваться.
- На личные впечатления от курса и идеи по его улучшению — после последней лекции.
It was also possible to reasonably bring the students' answers and my remarks into a single readable document — this was also scored. And the document itself would later be useful to both students and me.
The main issue that confused me was: well, everyone will really like it and they will start really a lot and write well. But after all this all someone has to check - do I have enough time for it? In addition to reading these lectures, I had the main work, postgraduate study + scientific work, which I, however, almost abandoned this semester as a result. It seemed that this problem could be tried to be solved by a scheme that allows shifting at least part of the test from the teacher to the students. In addition to facilitating the work of the teacher, it is also undoubtedly useful for students: through the search for mistakes and the vision of another person, a significantly better understanding often comes. Some students are also interested in such “ala teaching” activities.
In the current case, I stopped at the ranking of the results of students:
(из исследований онлайн-образования, например Waters, A.E., Tinapple, D., and Baraniuk, R.G.: "BayesRank: A Bayesian Approach to Ranked Peer Grading", 2015)
And the ranking would help me a lot. Accordingly, after the deadline for answers, the students had to send me ranked lists of their colleagues, from the comments in these lists were welcomed. In principle, I did not insist on ranking, but only recommended it, whoever wanted it - he could send it. At the end of the course, it turned out that after the full ranking, the most common form of the answer was the top k who wrote the most useful answers.
Следующей важной частью являлось смысловое наполнение курса. План теоретической части курса получился такой:
- Нулевая лекция — введение, о чем курс, какие акценты я собираюсь делать + отчётность (её правила гигантские и на их рассказ я отвел чуть ли не пол лекции)
- 1-3 лекция о том, как в общем случае решали задачи обработки изображений до появления машинного обучения. Свёртки для поиска перепадов интенсивности и сглаживаний, canny, морфологическая обработка изображений, просмотр изображения в разных пространствах (преобразование Фурье / вейвлеты), ransac, преобразования Хафа / Родена, детекторы особых точек, блобов, дескрипторы, построение распознающего алгоритма.
- 2-3 лекции (сколько будет нужно) о идеях машинного обучения, основные принципы, как оно помогает решить проблемы придуманных алгоритмов. Автоматический перебор значений параметров, условий, их последовательностей, что можно делать с данными и чего нужно бояться, какие модели лучше брать за основу, понижение размерности, аппроксимирующие данные сети, кластеризация. Первую часть этого планировал рассказать довольно быстро (она встречается и в других курсах), про кластеризации подробнее (почему их опасно использовать, какой алгоритм выбрать и о чем нужно не забыть).
- Лекции, на которых разбираются примеры реальных задач (как минимум, распознавание лиц и обработка видеопотока, а так — смотря на сколько хватит времени, может у студентов будут идеи или желание рассказать что-то свое). Предполагался полусеменарный формат, в котором сначала мы пытаемся поставить задачу, потом довести студенческие идеи до решающих её, потом перейти к реально используемым и ещё неугаданным ими методам. Например, в задаче идентификации лица по изображению используются идеи PCA и LDA (метрики Фишера), до чего додуматься, по крайней мере на лекции, сложно.
Практическая часть должна иллюстрировать некоторые аспекты теоретической, знакомить студентов с библиотеками и заставить самостоятельно решить сложную задачу. Соответственно, было три мини-лабораторных, в них надо было взять набор готовых скриптов и запускать их достигая попутно различные цели:
- установить python, pycharm и различные библиотеки. Скрипты для запуска самые простые: загрузка картинок, какие-то простейшие фильтрации по цветам и местоположению пикселей.
- набор скриптов иллюстрировал часть рассказанного на лекциях 1-3, студентам надо было подобрать картинки, на которых скрипты бы хорошо или плохо работали, объяснить почему. Правда, на скрипты к этой лабораторной меня не хватило и они получились довольно куцыми.
- на машинное обучение: нужно было выбрать одну из двух библиотек: catboost или tensorflow и посмотреть, что они дают на простеньких задачах (задачи и датасеты были взяты из примеров библиотек почти без изменений, у меня также не хватило времени). Сначала я хотел дать обе библиотеки вместе, но потом показалось, что на это может уйти слишком много времени.
Все три лабы я старался подобрать так, чтобы их можно было сделать часа за 3 — за один вечер. Проверяемым результатом лаб были подобранные либо наборы картинок и результатов работы на них, либо значения параметров библиотечных функций в скрипте. Все лабы были обязательны, но это можно было делать либо качественно, либо некачественно, за качественное выполнение и спецзадания к лабам можно было получить доп баллы, повышающие оценку за семестр.
Сложную задачи студенты могли выбрать как сами: например, взять что-то связанное с бакалаврской или работой, так и из предложенных. Было важно, чтобы эта задача была задачей с семантическим разрывом. Было важно, чтобы решение задачи не требовало программирования в больших объёмах. Сложность была не очень важна — я считал, что плохой результат также будет результатом. Было выделено 5 этапов работы над задачей, итоги каждого этапа надо было согласовывать со мной.
- Выбор задачи
- Подбор данных: важный этап, в ходе которого, как правило, заодно формируется куда более реалистичное представление о задаче, рождаются гипотезы решающих её алгоритмов.
- Составление первого приближения: алгоритма, который бы хоть как-то решал поставленную задачу, от которого можно было бы оттолкнуться и дальше улучшать.
- Итерационное улучшение решения задачи.
- Неформальный отчёт, описывающий итоговый алгоритм и модификации алгоритма исходного алгоритма, предпринимавшиеся для его получения.
Сама задача, как и мини-лабы, была обязательной; за её качественное выполнение можно было получить много доп баллов.
Примерно за неделю до зачётной я добавил альтернативный вариант задачи, при решении которой можно было рассчитывать максимум на 4ку: я беру сигнал, описывающийся сложной математической функцией, и генерирую студентам по нему данные для обучения/тестирования. Их задача — аппроксимировать сигнал чем угодно. Таким образом, они избегают этапа сбора данных и решают искусственную задачу.
Above, I wrote a lot about points, it is time to explain what they gave.
There were several areas of activity for which points could be obtained. At the end, points for all directions were multiplied and raised to the degree of “1 / <number of lectures in the semester>”. Directions:
- Each lecture is a separate direction.
- Large (complex) laboratory
- Organizational issues
сюда идут баллы за советы и работу, помогающую организовать курс, например за объективное замечание о том, что чего-то не хватает, что-то делается плохо или за попытку переписать описание отчетности так, чтобы оно стало более читаемым. Количество баллов варьировалось на мое усмотрение в зависимости от полезности, актуальности, внятности формулировки и тп.
- Everything else related to the topic of the course
например, если студент хочет затронуть аспект обработки сигналов, про который я ничего не рассказывал — баллы пойдут сюда. Затронуть что-то можно, например, подготовив фрагмент лекции по этой теме; в зависимости от качества сделанного и ситуации со временем я могу разрешить или не разрешить проводить такое на лекции, но в любом случае поставлю хоть какое-то количество баллов и напишу часть возникших замечаний — у студента будет возможность следующей итерации, углубляющей его познания и приносящей новые баллы.
Initially, for each direction the student had 1 point (so that when multiplying it just did not turn out 0). Another 1 point could be obtained for coming to a lecture (in the direction corresponding to this lecture), it was not so easy - the lectures were at 8 am. I could not systematize the amount of points received for everything else, so I put it on my own discretion, often making an unequivocal mistake. There was only a general picture, according to which a student who perfectly understood the lecture could receive points 25, well understood - points 10, tolerantly understood - points 5, less was put to the one who at least did something. Naturally, when evaluating, I could only rely on what the student wrote, although more often he could be lazy or something else, as a result of which his real knowledge did not reach me.
It is important to write about deadlines. Lectures were held on Tuesdays at 8 am. First, the deadline for the answers to the lectures was set for the nearest Sunday, for the rankings - the next Thursday for Sunday. Then, from the side of the students, what I came up with for the first couple of lectures was clearly expressed myself: I need to write feedback on the answers, and after it it is desirable to give students the opportunity to correct themselves. At the same time, voices began to be heard that there were very few 5 days to answer. As a result, I, contrary to the concerns expressed by other students, added a week to the answers to the questions, and began to comment on the answers that came before the first Sunday. The decision was unequivocally wrong: they no longer responded, and for the increased period new lectures took place, and even I was confused about what it was. But he didn’t change anything: he decided that there were so many changes.
At the end of the semester for those who received practice credit, the scores corresponded to the final grade for the course. This assessment could be improved on the exam, which was supposed to pass like this:
Four complex questions on different topics are given for understanding (I will choose topics at my own discretion). Everything that was said at the lectures or got into the group in VK. Fully read out the answer to the question +1 point for those dialed in the semester (if a person understands only part of the question, then 0 points are put for the question, whichever part is). It will be possible to use everything, anything, but the questions will be really complicated - a deep understanding.
The prohibition of the use of materials on the exam very often leads to the fact that students instead of understanding bison or write off.
I saw the dynamics of recruiting points during the semester like this: advanced students will score enough for 5 automatic points for about the first 6-7 lectures. That is, somewhere by the end of March, just when I will tell the basic information and move on to examples of setting and solving real problems. With practice, I hoped that the diligent will understand, too, by April, maximum to the middle, if its priority will be reduced by the requirements of other courses. I evaluated it on my own: I think that when I was a student of the 4th course, I would have passed such a course in about a specified time frame, if nothing unexpected had happened. I expected from less advanced students that at least questions like the possibility of obtaining an automaton would interest many of them, they would read the answers of their colleagues and fragments of lecture presentations. Topics are generally interesting,
I would like to make a remark about the selected multiplicative combination of points between directions, and not additive (the root of the work, and not the sum divided by a number). This corresponds to the need to deal with a large number of directions at about the same level; even very, very deep knowledge in a couple of directions will not provide the student with a good grade for the course in the absence of his knowledge in other directions. For example, multiplicativeness protects from being able to get 5, having filled me with suggestions for improving the organization of the course: each next sentence, bringing as many points as the previous one, would have made less and less contribution to the final grade.
One of the immediately noticeable shortcomings of this system is its complexity. But since the course itself is quite complex and solving problems with a semantic gap requires the construction and understanding of complex algorithms, I believe that students should be able to understand this easily. Moreover, this reporting system is in itself similar to the solution of a problem with a semantic gap: some problems arose in the course model, the most important were selected from them, approximations were sought for their solution.
Another disadvantage of the system: it can really take a lot of time for students. Therefore, I tried to apply a long-standing idea: to offer students who know the material well and without a course or who consider themselves busy with more important things, contact me in the first month. I am ready to talk with them, and, depending on their level of knowledge and the reasons for displacing my course, offer them an automatic or simplified, adjusted for them, method of passing the course. After the first month, the offer is removed - otherwise it may be used at the end of the semester by weak students who could not force themselves to do something, but, potentially, would like to.
Approximately it was presented to students in the first lecture. Then I promised myself not to change it, even if I see that it works poorly and the students do significantly less or worse than expected. The course has begun.
The results were much worse than my expectations, although a number of hopes came true. I remember, after the first list of questions on the introductory lecture, I waited in fear: would there be at least some answers and whether they would be comprehended. And now, finally, the first answers began to appear, in the comments there was even some kind of discussion, but rather on a philosophical topic. Further, during the semester, the students continued to answer; however, as a rule, there were a couple of dominant students, who brought in about 70% of everything useful that was written.
By the end of the semester, activity decreased significantly, after the penultimate lecture, they sent me a ranked list consisting of one last name — the only person who answered at least some questions about that lecture. The reasons for this, I think, could have been general fatigue, perhaps some disappointment, inadequate assessment, an unsuccessful change in the deadlines, which led to the need to wait 3 weeks to get the final result of the lecture, the increased load on other subjects.
I was also more and more disappointed with the quality of the answers: it often seemed that much from somewhere was torn off without understanding, and the volume of new ideas was far from as great as I had hoped. Even from the students there was a remark that the current system stimulates at least some answers; on the degree of how deeply the student figured out, the scores do not depend so much. But for sure there were those who understood.
Since no one fit into my set plans for scoring and this threatened that everyone except for a couple or three people would have to pass an exam, I began to try to put higher points. It seemed that I overestimate the points too much to those who responded only with examples of tasks and the difference between these formal replies and those who really tried too little. By the end of the semester, I was increasingly embraced by the feeling that there are many students who do not understand almost anything of what was said, although they have relatively acceptable points. This feeling became even more at the last lecture, when I began to try to ask everyone in a row hoping to better understand the final level and add the points correctly to the respondents - it turned out that many people do not know basic things, for example, what neural networks are or particular points in the image.
The hopes for ranking were not justified either: there were very few comments in the ranked lists, by the end they had completely disappeared. Often, it seemed to be evaluated visually rather than carefully reading. However, I remember, at least a couple of times, when the ranking really helped and I looked at him and adjusted my ratings. But that it evaluated for me was not the question. The assessment took quite a lot of time, but I could do it on the way to the subway and in the end I was more likely to have timely answers than students.
Большую лабораторную никто не сдал даже в апреле. И я толком не понимал, сложная ли она или все никак не могут взяться и не знал, надо ли что-то менять и как, что в итоге требовать. Придумал задачу на 4 максимум, но и она не изменила ситуации. В лучшем случае к концу апреля студенты выбрали себе задачи и поприсылали данных. Некоторые выбранные задачи оказывались откровенно нерешаемыми на имеющемся у студентов уровне знаний. Например, студент хотел распознавать раковые опухоли, но при этом сам не понимал, чем именно они должны отличаться — я тут, естественно, никак не мог помочь.
С мини-лабораторными дело обстояло существенно лучше, первые две многие сдали в срок или не сильно вылезая за него; третью тоже почти все сдали, но в самом конце. Некоторые делали их качественно и лучше, чем я ожидал. Но основной практический упор я-то хотел сделать на большой лабораторной.
Вопрос, можно ли требовать от студентов на практике того, что еще не рассказано на лекциях, волновал умы многих знакомых мне преподавателей. Казалось, что формальный правильный ответ: конечно, нет — ведь это значит сначала отнимать у студентов лишнее время на самостоятельное изучение того, что потом будет рассказано, а потом рассказывать то, что им и так понятно. Но сейчас я думаю, вреда от этой формальной позиции получается существенно больше: своевременно попробовать самое сложное на практике уже не представляется возможным. Вместе с тем, понятно, что разбираться с материалом самостоятельно студенту необходимо, а повторение материала можно провести оригинально, например, предложить хорошо разобравшемуся студенту самому тщательно подготовиться и прочитать этот фрагмент лекции.
Did such a system ultimately give more than, for example, a classical system with an exam? The question is complicated, I hope that, nevertheless, yes, quite a lot of material has been given, while preparing for the exam, a part of it would surely drop out of consideration even from good students. Although the additions to the course in the answers were not so much as I had hoped.
Связана она с тем, что бывает, случается чудо и преподавателю удается научить студентов чему-то глобально новому. Например, на моих глазах студент начинает существенно разумнее подходить к решению задачи с семантическим разрывом. Шаги он предпринимает в целом правильные, получает приемлемый результат, но объяснять не умеет. И тут я, преподаватель, пытаюсь разобраться в том, что он сделал. Объясняет он непонятно — я задаю массу странных вопросов, делаю странные предположения, наконец, продираюсь через терминологию студента и понимаю. Предлагаю советы по улучшению, иногда плохие, как замечает уже разбирающийся в задаче студент. И тут получаю реакцию схожую с обычной: “Зачем это еще нужно делать?” и “не нужны мне ваши советы” до “Я и сам все прекрасно умел и без вас”.
Особенно сильно это может проявляться, когда начинается примерно так: студент исходно приходит со своим самоуверенным и непродуманным предложением по решению задачи вида “тут надо просто взять нейронную сеть и обучить”. Ты говоришь, что просто так сделать нельзя, надо еще, как минимум, много подумать, да и вообще эту задачу лучше не нейронными сетями решать. Студент, бывает, продумывает, мучается, но, молодец, действительно разбирается и приносит хорошо продуманное решение, основанное все-таки на нейронных сетях и всем своим видом говорит “я так бы и без ваших советов сначала сделал”. Прошу прощения у тех студентов, которые так не делают, вы существуете и я с некоторыми из вас знаком, спасибо вам. Тем не менее проявляющие такую неблагодарность студенты существуют, да я и сам, к сожалению, тоже так себя неоднократно вел.
Проблема выражения такой неблагодарности многими преподавателями легко решается с позиции силы: можно навязывать свое решение задачи, прерывать студента, если он говорит не то, что хочется услышать и тп. Это бывает действенно, особенно для плохих студентов, но хороших это лишает возможности подумать и осознать неправильность своих идей, гипотез — и получить опыт, который действительно запомнится. Сумасбродные ультимативные требования к решению задачи без понятных объяснений в таком предмете вызывают отторжение, основной задачей студента становится угодить преподавателю, а не получить знания или решить задачу. Лояльность приводит к тому, что ленивые студенты многого не делают, а некоторые еще и обижают преподавателя.
Замечал я эту особенность и ранее, но после этого семестра я как-то больше её прочувствовал, пережил. Возможно, потому, что действительно научил некоторых студентов. Такая неблагодарность, видимо, проистекает от внутренней гордости таких студентов, их комплексов, желанию показать себя опустившемуся почти до их уровня преподавателю. Помимо усложнения организации учебного процесса, такое поведение и показная неблагодарность студентов часто выводит из себя: страшно хочется как-нибудь вразумительно показать студенту, что он перешел черту. При этом умом понимаешь, что по сути студент разобрался, оценка должна быть положительной. Оказываешься почти в безвыходной ситуации, остается только посмотреть на это дело с юмором и списать все на глупость студента, но это сложно. У меня получалось плохо и я обижался.
Таким образом, неблагодарность студентов способна очень часто отравить настроение именно чему-то научившему их преподавателю. Схожих отравляющих настроение вещей может быть очень много. Они особенно больны, если все, что преподаватель надеялся получить от обучения этих студентов — удовольствие. Меня такая ситуация лишний раз укрепила в уверенности, что хорошо читать целый курс на одном удовольствии нельзя, нужно рассчитывать получить что-то еще, хотя бы мечту.
What I’m sure about is that the course was very successful in terms of advancing and systematizing my knowledge. Of course, most of what I said, I generally and so represented, but I felt many things more deeply. There were algorithms about the existence of which I knew and even used, but did not fully understand how they work, did not know many alternatives, or knew only the names. When preparing the course I had to figure it out. There was also a number of new things that I noticed under the clear influence of students, for example, on autoencoders. I got a lot, maybe not very often used, but definitely needed for a good orientation in the subject area. I think that the improvement in knowledge that has already happened has even influenced some of the decisions that I took in my work while thinking over the algorithms, I hope for the better.
It may be that I will fall to read this course again, for example, next year. Not for all the problems, I have ideas for a solution, but for some I have, and I will try to describe them.
- The main problem: the lack of timely progress on a complex task, I think to solve with the help of discussion of similar fragments of other tasks at seminars and clear homework with short deadlines. Each of the homework assignments will require a small fragment of a large laboratory, such as the formulation of a problem statement, the first selection of data, the elaboration of a quality criterion, ... For each fragment performed on time, points will be awarded. If a student is lagging behind, he will have to catch up to start receiving them.
- I also plan to more clearly and often pronounce the main idea of the course in various contexts. Although not sure that it will help: often, when you say the same thing, it starts the opposite to cause rejection. The main idea, if anything, was that the problem solving skill is not a thoughtless enumeration of various ML models in various configurations, but the manual construction of an individual model for a problem using suitable pieces of existing models with reasonable modifications to the task. For some reason, many people either do not understand this, or diligently make this look. Maybe some people can realize this idea only through practice, through stuffed bumps.
- I also plan to stop putting 1 point to everyone who came to the lecture; and set, by default, significantly less, for example 0.1. In order to get more points, it will be necessary to send or show me the notes of the main lecture theses or their photos on the day of the lecture. Almost anything can be written, the format and volume do not interest me. But for good notes I am ready to bet significantly more than 1 point.
Добавить это хочется, чтобы дополнительно стимулировать студентов слушать лекцию, а не спать и заниматься своими делами. Ещё многие существенно лучше запоминают то, что запишут. Интеллектуальная нагрузка для создания таких заметок не очень нужна. Также кажется, что это не будет слишком сильно нагружать студентов, которые не ведут конспект, те, которые его ведут — смогут просто его предоставлять.
Правда, критиками этой идеи выступают все опрошенные студенты. Они, в частности, указывают на то, что не так трудно списать эти заметки у соседа в конце лекции или просто записать что-то со слайдов, не особо обращая при этом внимание на лекцию. Вдобавок, необходимость писать некоторых может отвлекать от понимания.
Так что, возможно, хорошо бы как-нибудь поменять форму. Но в целом такая форма отчётности мне нравится, она применялась, например, на курсе мат статистики в csc: надо в день лаб прислать небольшую доделанную лабу — и, как мне кажется, многих студентов это стимулировало сесть за неё сразу и доделать. Хотя находились конечно и те, которые говорили, что в этот вечер они никак не могут и оказываются в невыгодном положении. Тут, мне кажется, может помочь другая идея: давать каждому студенту возможность сдвинуть дедлайны на несколько дней за семестр.
- There was a thought to replace the flat structure of answers to woody questions. So that the answers to all the questions would not be a complete list, but at least two-level: then the answers to one question will be next to each other, and not mixed up with the answers to other questions. Two-level structure of comments to posts is supported, for example, facebook. But it is much less likely to enter it and you do not want to make it the main means of communication. Simultaneously to lead two groups: VKontakte and facebook is strange. I would be glad if someone advises any other solution.
There are many problems that I don’t know how to solve and I don’t know if it is possible at all. The main thing that worries:
- too simple answers of students to my questions
- poor evaluation of answers: my assessment does not always correlate with reality
- a ranking that almost does not help: until the verification of student responses by the students themselves is very far away
In general, I definitely do not consider in vain the time spent on preparing and conducting the course; at least it was very useful for me.
That's all, it seems, and so it turned out too overloaded.
I want to thank:
- for reviewing: my mother, Margarita Melikyan (classmate, now MSU graduate student), Andrei Serebro (classmate, now a Yandex employee)
- all students who participated in this and passed the survey / wrote reviews
- Yes, and everyone who ever taught me something good