February 13, 2009 at The perfect storm. Post-mortem unannounced project.
Today we went with Vovcheg, our game designer guru, to take him a loan to buy an apartment and discussed our current project.
They agreed that it is surprising that both of us have not yet been fired. We managed to make a huge number of very significant mistakes: I - on management, Vovcheg - on design. The project is not finished yet, but certain conclusions can already be drawn.
The project lasts for the 8th month, and will last another 2-3 months with all the numerous platforms. Moreover, we now clearly see what kind of mistakes the company cost 2-3 months of development and tens of thousands of dollars (I hope that the bill will not work for hundreds of thousands).
TEAM The
main error in my management is the error in choosing the command:
- Betting on a young team that couldn’t, in fact, “turn around” on this very complex project in all aspects.
- A bunch of young PM + experienced PMA, which failed immediately. It seemed to me that I would be able to realize what, then, seemed to me to work in Nival.
- We were fatally mistaken in assessing the complexity of the game design task, which led to the fact that the designer-programmer team was removed, which is almost a disaster for a project of our complexity.
It took me 2 whole months to realize how bad the situation is, it justifies me a little bit that out of these 2 months 2 weeks I was on vacation in a warm place. What I saw after the vacation was a disaster. I realized how categorically and extensively I screwed up. It seems like never before I was mistaken SO cool.
The correction consisted in a complete change of the entire core team and its concentration in one room of the Minsk office. Only after that the project actually began at the speed with which it was supposed to begin 2 months ago.
Surprisingly, I was 1.5 months with the project almost every day, I controlled the key points, at the same time I could turn on the brain and see the obvious only after vacation.
Once again I was convinced that in a difficult situation, I am not able to correct the situation with PM directing. The only way I can solve such problems on projects is to solve them by myself. Close holes with yourself. Unfortunately, I have not turned out to be better than all other people in this aspect.
Vovcheg’s mistake in design was that it was impossible to throw an insufficiently developed idea into the remote team. We saw very well what happens to the project when the team does what it DOESN’T understand and DO NOT accept. None of the team even then understood what we really want to do. Really, I didn’t even get close! At the same time, over the weeks seeing, on the one hand, what unplayable garbage is in the build, on the other, a break with the Vision that we want to implement with the Vovcheg.
3 very important conclusions on this part, which, of course, look obvious.
- The most difficult things I will do myself, only myself. When we get another cool pm, that will change.
- The core team of complex projects will _always_ be in the same room of one office. And damn me if I ever try to break it again.
- Never again will I “throw people into the water with the hope that they will emerge”, I will “teach to swim accurately and consistently,” this concerns building teams and developing PMs.
- You can’t do something if you don’t have at least 1 person who supports and understands the idea, we didn’t have it then.
HOW THE PERFECT STORM BORN
We messed up Vovcheg very much in the entities where he and I were considered experienced guys - Pre-production and Vertical Slice (they also call him reference playable).
We perfectly went through the concept phase. We had a good and clear vision. We have been thinking about the concept for a very long time and by many people. I will not say that we came up with something incredible, but the concept phase was passed really well. A good start was made.
A clear and correct vision was the only ray of hope. The realization that there is a goal, that it is achievable, but we still don’t know a clear way how to get there, is what saved the project. If it weren’t, it would be a typical Death March Project, or, as I call it, the Disaster Project. For 8 years the company had many such projects. I have not yet learned to avoid them. A clear concept phase allowed us to stay late for 2-2.5 months.
Pre-production was completely failed.Despite the fact that formally some documents were being made, some (unplayable) Vertical Slice was developing, some plans were being written, it was reported that some risks had been removed - in fact, all this was a fake. The main thing has not been done - the certainty of what we are doing. Without this, plans and GDD could not have real content.
All this led to disaster.
We started production.
A lot of people came to the project. Reserves have already been eaten. Instead of concentrating on solving the main problems of the project - getting a real Vertical Slice, introducing certainty into key features - we had to spray a lot to provide a large team with work. We put pressure on the programmer of the engine to make changes even faster. When a large production team was already producing results, integrating this whole thing into a build only exacerbated the problem, we saw that nothing worked at all. I had to redo a lot, people howled at it, the spiral of problems twisted.
The changes that we made to the engine began to be very expensive. This has already become apparent to outside people; very simple changes cannot cost days. So something is wrong with the system inside. The new lead programmer of the Dimusik project, unfortunately, did not realize how deep we were then.
I found myself in a situation where I had fundamental problems everywhere. In design, we don’t know what we are doing. We have nothing but vision and a bad Vertical Slice build. We already have some kind of production trait and a lot of people who are embittered by the fact that we have to make a lot of changes that need to be given tasks, but there are no normally thought out tasks. We have drawn a lot of art, but the quality of many elements was far from the “Wow” we were counting on.
In order to somehow solve this, we had to change the Vertical Slice. I understood that there would be no more chances to be on time. To be on time is to come out simultaneously with the big release on American TV. It was a real deadline that could not be moved.
At this moment, the lead programmer comes to me and says that I need to stop any work and throw out 80% of the entire code base. What is the only solution. We both understand that this is true. In the best case, if we rewrite the current complex elements in the simplest form, for example, full random, instead of the current complex AI, we will do it all in 2 weeks. We will return to the working build in 2 weeks. We understood that we would return to the previous point soon in a month. And this will only allow us to make at least some changes to the project.
Remember, I recently talked about the fact that the project is similar to a play where the manager’s task is to “cram the invisible” and make everyone a winner.
My project has become a TRAGEDY. My personal tragedy. The tragedy of the team and our entire company.
I clearly realized that we were at least 2 months late and that we were actually at the beginning of pre-production.
Transferring the deadline for 2 months, according to the big brand, where we have big guarantees for revenue, it was CATASTROPHE.
LIFE AFTER THE DISASTER
Only when the realistic dates returned to the project, when we clearly realized that we were at the beginning of pre-production, when a very experienced and strong team, sitting in the same room, took up the project, it began to move forward very rapidly. Ideas were generated not by just one designer, but by the whole team, that is, PM, designer, artists, programmers, later - qa. Changes were made very quickly. The project was changing very quickly.
A competent lead programmer Dimusik then read out McConnell's Perfect Code and was able to effectively implement the ideas presented there, a very clear and very simple architecture of 3 states was formulated (instead of a complex and confusing of 10-20 states). These 3 states became the core of the engine. Everyone, starting from me, ending with QA, knew this state.
I first saw then myself that correctly asking questions to a customer (designer) can seriously help a designer better understand what he wants, you can help find a solution much easier and better.
We have implemented a scripting system at many levels of projects, a real data driven approach. What took 2 weeks and 10 iterations of the work of the designer programmer was now done in 3 days by the programmer, immediately right and almost without bugs, and the designer himself then very easily and quickly set up the feature in a couple of days. It was a real shock to me.
Later we learned that Nival does the same for scripting magic in Allods Online. Hardcode 200 spells in heroes 5 - it was an old ineffective approach, which we also left.
It was 10x. We all know that the best people are 10x averages. Now I was able to measure it very accurately, and indeed it was 10x. This was a shock to me again.
THE DARK SIDE OF AGILE
I knew well before how evil Agile could be. On the project, I saw with concrete examples that HOW MUCH EVIL can be Agile in the wrong hands. We did some features in agile style, that is, to be honest, without really thinking about what we want and what will change (then iterations will be made for changes).
When at a certain moment we had to make a big next pack of changes, the lead programmer Dimusik behaved in a completely different way. We both realized that there must be a better way.
Dimusik for 2 days did not write a single line of code. He talked a lot with the designer for a long time, they discussed and thought over different options, Dimusik proposed another way to implement the feature, which was radically simpler, caused noticeably fewer problems, but which, in fact, did the same thing. Dima drew diagrams, drew tables of various interaction options, worked out all the nuances in great detail. Fortunately, I had the intelligence to understand that this is not “2 days of doing nothing,” but our future success.
After these 2 days, Dima in 1 day implemented this already simple feature. And when she earned from the very first iteration, completely without bugs (with only 2 very small blots, which were more likely modifications), the whole team was in shock. This was fixed for 3 days, of which only 1 day the code was written, it was 1 iteration, instead of 2-3-4 weeks and 3-5-10 iterations.
It became obvious to everyone how much better the good old forgotten (?) All (?) Waterfall approach could be. Only after that I truly, on my blood, realized how important it is to always maintain a subtle balance between the Agile and Waterfall approaches. I spoke about this before, at the last CWI , for example, but only now I saw such a egregious example.
“Keep it simple” slogan Dimusik, tough and very simple architecture, rejection of Agile in many elements of the project, scripts and data driven approach in almost all elements of the project - this was the foundation of the revival of the project.
WHERE ARE WE?
I have already spent 8 months and 50+ person-months on a portable project for many platforms (multiply by the cost of a person-month and get the cost). The best people of the company work on the project, the maximum of people whom I was able to “effectively occupy” are 4 programmers, 4 artists, 2 dedicated teams.
It's 11:29 p.m., Thursday February 12th. We received 6 crushing alpha play tests yesterday. Never in all 5 years and 20+ projects have play-tests been so bad. All players are united in the fact that “nothing is clear what is happening on the screen, the goal is not clear, it is boring to perform the same actions.”
It’s ridiculous to say, but we actually didn’t pass alpha, we have already done a lot on beta, I have 3.5 weeks or 21 working days left. We are in a hurry to finish submission materials for all key Tier1 American channels, otherwise we will simply skip “our window” to launch.
I know we'll be late. It is impossible to complete the project in 3.5 weeks when the alpha stage has not yet been completed. We will be late for 1 to 3 weeks. We will fight for every day, do everything to bring release closer at least for a day, for each day.
DREAMS ARE TREATED
TODAY 4 hours ago, Vovcheg performed something very important for himself - he bought a good apartment in Minsk, and now, when I am writing these lines, Vovcheg is writing a new pack of changes at home, which we conditionally call “Death Changes” (before that there was “ Mad changes pack ”), which will be cheap and that will greatly improve the game. Despite the devastating alpha review, we believe that there are changes that will be good and cheap.
Our faith is our ray of hope, this is our Vision, we have come a long way in search of a goal, and we feel that we are already very close. We are almost here. Our intuition tells us that we are already very close. It seems we are almost happy.
Is there life after deadline? I dont know.
But I know for sure that I will be proud of our Game, proud of the team that has come this way, I know that when Vovcheg and I come back from San Francisco, listening to Kazima and his “Solid Game Design: Making the 'Impossible' Possible” , we will make our 'Impossible' Possible - we will make our new project much better than the current one.
Denis Voikhansky,
Executive Producer, where, at Reaxion, we create games we want to play ourselves.
PS Perhaps, after we announce our game, you will have a chance to evaluate what we have been creating all these months.
They agreed that it is surprising that both of us have not yet been fired. We managed to make a huge number of very significant mistakes: I - on management, Vovcheg - on design. The project is not finished yet, but certain conclusions can already be drawn.
The project lasts for the 8th month, and will last another 2-3 months with all the numerous platforms. Moreover, we now clearly see what kind of mistakes the company cost 2-3 months of development and tens of thousands of dollars (I hope that the bill will not work for hundreds of thousands).
TEAM The
main error in my management is the error in choosing the command:
- Betting on a young team that couldn’t, in fact, “turn around” on this very complex project in all aspects.
- A bunch of young PM + experienced PMA, which failed immediately. It seemed to me that I would be able to realize what, then, seemed to me to work in Nival.
- We were fatally mistaken in assessing the complexity of the game design task, which led to the fact that the designer-programmer team was removed, which is almost a disaster for a project of our complexity.
It took me 2 whole months to realize how bad the situation is, it justifies me a little bit that out of these 2 months 2 weeks I was on vacation in a warm place. What I saw after the vacation was a disaster. I realized how categorically and extensively I screwed up. It seems like never before I was mistaken SO cool.
The correction consisted in a complete change of the entire core team and its concentration in one room of the Minsk office. Only after that the project actually began at the speed with which it was supposed to begin 2 months ago.
Surprisingly, I was 1.5 months with the project almost every day, I controlled the key points, at the same time I could turn on the brain and see the obvious only after vacation.
Once again I was convinced that in a difficult situation, I am not able to correct the situation with PM directing. The only way I can solve such problems on projects is to solve them by myself. Close holes with yourself. Unfortunately, I have not turned out to be better than all other people in this aspect.
Vovcheg’s mistake in design was that it was impossible to throw an insufficiently developed idea into the remote team. We saw very well what happens to the project when the team does what it DOESN’T understand and DO NOT accept. None of the team even then understood what we really want to do. Really, I didn’t even get close! At the same time, over the weeks seeing, on the one hand, what unplayable garbage is in the build, on the other, a break with the Vision that we want to implement with the Vovcheg.
3 very important conclusions on this part, which, of course, look obvious.
- The most difficult things I will do myself, only myself. When we get another cool pm, that will change.
- The core team of complex projects will _always_ be in the same room of one office. And damn me if I ever try to break it again.
- Never again will I “throw people into the water with the hope that they will emerge”, I will “teach to swim accurately and consistently,” this concerns building teams and developing PMs.
- You can’t do something if you don’t have at least 1 person who supports and understands the idea, we didn’t have it then.
HOW THE PERFECT STORM BORN
We messed up Vovcheg very much in the entities where he and I were considered experienced guys - Pre-production and Vertical Slice (they also call him reference playable).
We perfectly went through the concept phase. We had a good and clear vision. We have been thinking about the concept for a very long time and by many people. I will not say that we came up with something incredible, but the concept phase was passed really well. A good start was made.
A clear and correct vision was the only ray of hope. The realization that there is a goal, that it is achievable, but we still don’t know a clear way how to get there, is what saved the project. If it weren’t, it would be a typical Death March Project, or, as I call it, the Disaster Project. For 8 years the company had many such projects. I have not yet learned to avoid them. A clear concept phase allowed us to stay late for 2-2.5 months.
Pre-production was completely failed.Despite the fact that formally some documents were being made, some (unplayable) Vertical Slice was developing, some plans were being written, it was reported that some risks had been removed - in fact, all this was a fake. The main thing has not been done - the certainty of what we are doing. Without this, plans and GDD could not have real content.
All this led to disaster.
We started production.
A lot of people came to the project. Reserves have already been eaten. Instead of concentrating on solving the main problems of the project - getting a real Vertical Slice, introducing certainty into key features - we had to spray a lot to provide a large team with work. We put pressure on the programmer of the engine to make changes even faster. When a large production team was already producing results, integrating this whole thing into a build only exacerbated the problem, we saw that nothing worked at all. I had to redo a lot, people howled at it, the spiral of problems twisted.
The changes that we made to the engine began to be very expensive. This has already become apparent to outside people; very simple changes cannot cost days. So something is wrong with the system inside. The new lead programmer of the Dimusik project, unfortunately, did not realize how deep we were then.
I found myself in a situation where I had fundamental problems everywhere. In design, we don’t know what we are doing. We have nothing but vision and a bad Vertical Slice build. We already have some kind of production trait and a lot of people who are embittered by the fact that we have to make a lot of changes that need to be given tasks, but there are no normally thought out tasks. We have drawn a lot of art, but the quality of many elements was far from the “Wow” we were counting on.
In order to somehow solve this, we had to change the Vertical Slice. I understood that there would be no more chances to be on time. To be on time is to come out simultaneously with the big release on American TV. It was a real deadline that could not be moved.
At this moment, the lead programmer comes to me and says that I need to stop any work and throw out 80% of the entire code base. What is the only solution. We both understand that this is true. In the best case, if we rewrite the current complex elements in the simplest form, for example, full random, instead of the current complex AI, we will do it all in 2 weeks. We will return to the working build in 2 weeks. We understood that we would return to the previous point soon in a month. And this will only allow us to make at least some changes to the project.
Remember, I recently talked about the fact that the project is similar to a play where the manager’s task is to “cram the invisible” and make everyone a winner.
My project has become a TRAGEDY. My personal tragedy. The tragedy of the team and our entire company.
I clearly realized that we were at least 2 months late and that we were actually at the beginning of pre-production.
Transferring the deadline for 2 months, according to the big brand, where we have big guarantees for revenue, it was CATASTROPHE.
LIFE AFTER THE DISASTER
Only when the realistic dates returned to the project, when we clearly realized that we were at the beginning of pre-production, when a very experienced and strong team, sitting in the same room, took up the project, it began to move forward very rapidly. Ideas were generated not by just one designer, but by the whole team, that is, PM, designer, artists, programmers, later - qa. Changes were made very quickly. The project was changing very quickly.
A competent lead programmer Dimusik then read out McConnell's Perfect Code and was able to effectively implement the ideas presented there, a very clear and very simple architecture of 3 states was formulated (instead of a complex and confusing of 10-20 states). These 3 states became the core of the engine. Everyone, starting from me, ending with QA, knew this state.
I first saw then myself that correctly asking questions to a customer (designer) can seriously help a designer better understand what he wants, you can help find a solution much easier and better.
We have implemented a scripting system at many levels of projects, a real data driven approach. What took 2 weeks and 10 iterations of the work of the designer programmer was now done in 3 days by the programmer, immediately right and almost without bugs, and the designer himself then very easily and quickly set up the feature in a couple of days. It was a real shock to me.
Later we learned that Nival does the same for scripting magic in Allods Online. Hardcode 200 spells in heroes 5 - it was an old ineffective approach, which we also left.
It was 10x. We all know that the best people are 10x averages. Now I was able to measure it very accurately, and indeed it was 10x. This was a shock to me again.
THE DARK SIDE OF AGILE
I knew well before how evil Agile could be. On the project, I saw with concrete examples that HOW MUCH EVIL can be Agile in the wrong hands. We did some features in agile style, that is, to be honest, without really thinking about what we want and what will change (then iterations will be made for changes).
When at a certain moment we had to make a big next pack of changes, the lead programmer Dimusik behaved in a completely different way. We both realized that there must be a better way.
Dimusik for 2 days did not write a single line of code. He talked a lot with the designer for a long time, they discussed and thought over different options, Dimusik proposed another way to implement the feature, which was radically simpler, caused noticeably fewer problems, but which, in fact, did the same thing. Dima drew diagrams, drew tables of various interaction options, worked out all the nuances in great detail. Fortunately, I had the intelligence to understand that this is not “2 days of doing nothing,” but our future success.
After these 2 days, Dima in 1 day implemented this already simple feature. And when she earned from the very first iteration, completely without bugs (with only 2 very small blots, which were more likely modifications), the whole team was in shock. This was fixed for 3 days, of which only 1 day the code was written, it was 1 iteration, instead of 2-3-4 weeks and 3-5-10 iterations.
It became obvious to everyone how much better the good old forgotten (?) All (?) Waterfall approach could be. Only after that I truly, on my blood, realized how important it is to always maintain a subtle balance between the Agile and Waterfall approaches. I spoke about this before, at the last CWI , for example, but only now I saw such a egregious example.
“Keep it simple” slogan Dimusik, tough and very simple architecture, rejection of Agile in many elements of the project, scripts and data driven approach in almost all elements of the project - this was the foundation of the revival of the project.
WHERE ARE WE?
I have already spent 8 months and 50+ person-months on a portable project for many platforms (multiply by the cost of a person-month and get the cost). The best people of the company work on the project, the maximum of people whom I was able to “effectively occupy” are 4 programmers, 4 artists, 2 dedicated teams.
It's 11:29 p.m., Thursday February 12th. We received 6 crushing alpha play tests yesterday. Never in all 5 years and 20+ projects have play-tests been so bad. All players are united in the fact that “nothing is clear what is happening on the screen, the goal is not clear, it is boring to perform the same actions.”
It’s ridiculous to say, but we actually didn’t pass alpha, we have already done a lot on beta, I have 3.5 weeks or 21 working days left. We are in a hurry to finish submission materials for all key Tier1 American channels, otherwise we will simply skip “our window” to launch.
I know we'll be late. It is impossible to complete the project in 3.5 weeks when the alpha stage has not yet been completed. We will be late for 1 to 3 weeks. We will fight for every day, do everything to bring release closer at least for a day, for each day.
DREAMS ARE TREATED
TODAY 4 hours ago, Vovcheg performed something very important for himself - he bought a good apartment in Minsk, and now, when I am writing these lines, Vovcheg is writing a new pack of changes at home, which we conditionally call “Death Changes” (before that there was “ Mad changes pack ”), which will be cheap and that will greatly improve the game. Despite the devastating alpha review, we believe that there are changes that will be good and cheap.
Our faith is our ray of hope, this is our Vision, we have come a long way in search of a goal, and we feel that we are already very close. We are almost here. Our intuition tells us that we are already very close. It seems we are almost happy.
Is there life after deadline? I dont know.
But I know for sure that I will be proud of our Game, proud of the team that has come this way, I know that when Vovcheg and I come back from San Francisco, listening to Kazima and his “Solid Game Design: Making the 'Impossible' Possible” , we will make our 'Impossible' Possible - we will make our new project much better than the current one.
Denis Voikhansky,
Executive Producer, where, at Reaxion, we create games we want to play ourselves.
PS Perhaps, after we announce our game, you will have a chance to evaluate what we have been creating all these months.