My lizard is the Lizard of Testing
The beta test is going well. I've already been able to make a few revisions based on the feedback I am getting.
If you were playing an older version, I would recommend downloading the newest and trying again. Here's a few significant changes since the first Beta release:
- The lizard's movement physics have been adjusted: you will now come to a stop very quickly after landing. The tendency to slide off the edge seemed to be the biggest source of difficulty for players.
- Several rooms have been slightly adjusted to reduce difficulty, e.g. the octopus boss has fewer eggs and she won't randomly do "extra" attacks anymore.
- Windows/Mac/Linux builds now automatically save your game when you quit. You can resume next time you open the program.
Probably most of you are familiar with the idea of beta testing, but there's a few stages of development that are fairly common practice in game production. Some quick definitions:
- Alpha: a playable version of a small part of the game, with the goal of demonstrating what the final product will be like, and working out the production techniques that will be needed to finish the full game.
- Beta: a complete or nearly complete version of the game, to be tested publicly rather than internally to find bugs and problems, and gather feedback before final release.
- Final: the game is finished; further revision may be possible (patches/DLC), but this is the first complete and definitive version of the game.
All of these terms are a little bit nebulous, because every company has their own process. Often a "beta" may be significantly incomplete, and there are terms like "vertical slice" or "prototype" that are similar to "alpha" but there's a bit of variation in what all these things mean.
Sometimes people ask what takes the most time in game development, and in my estimation testing uses an order of magnitude more labour than anything else. Many people at a game company do testing constantly; designers and programmers especially must test the things they are working on quite a bit just to see what they are doing! If I place an obstacle somewhere in Lizard, I usually have to then "play" that obstacle dozens of times, trying to think of all the ways people will succeed or fail with it. If I'm not satisfied, I make a change and the process repeats.
In addition to that more immediate "does the thing I just made work?" kind of testing, there's a much larger set of testing tasks mostly to answer the question "does everything still work?" This becomes very hard to answer as a game grows in size. New changes often affect old stuff in unexpected ways, especially with programming tasks. These are very time consuming; a lot of bugs like this don't surface without doing things on opposite sides of the world, distant tasks that you wouldn't have tried together in your more immediate tests.
That kind of testing work usually is given to teams known as Quality Assurance. It is tedious work, done by a small army of workers who are probably the lowest paid at the company. Despite the low status of the QA tester, though, it is valuable and essential work. It's critical work.
For Lizard, QA is mostly just me. I don't have any budget for employees to do it. I have frequently sent builds to friends and people I trust, and I am extremely grateful to everyone who has played Lizard over the course of its development, but I can't ask or expect such a person to do the kind of mind-numbing, tedious, diligent and ongoing work of a QA tester. Though, a few people in particular have really gone above and beyond in this respect, and I am exceptionally thankful to them.
I do think that getting to play a brand new game, even unfinished, has its own significant value, but if I tried to quantify how much value that is, what can I compare it to? For example, what would a week of full time testing labour cost? I don't think it's even close to that... not unless I spread it out among a lot of people. That's precisely why we have a beta test. It's "free" testing, but only as far as someone remains interested and motivated by the game itself. Get it out to enough people and it really adds up.
If you're a software developer, you're probably aware of the concept of "unit testing" or other types of automated tests. These can be helpful, but unfortunately they do not really address the main labour problems that game testing has. For most of it you really do need a human at the wheel, both to figure out and do the kinds of things that need to be tried, and to evaluate the results. Automated tests work best with clear cut cases where there's a very specific action and response. Their value is immeasurable in some kinds of software engineering, but unfortunately in game development they have a more minor role. Here's a recent article by Ron Gilbert that I think outlines and illustrates the issue well: Unit Testing Games
My bugs are the Bugs of Lizard
Since the beta test began, I've been very surprised by the lack of discovered bugs, actually. I didn't have any known bugs going into the beta, but I'm used to them popping up when testing goes broader. Maybe I've been extraordinarily lucky? I'm very skeptical of that. All software has bugs.
I've had a number of reports of things that happen in my game that seem a little weird, but so far all of them were things I did intentionally. I do not mind "false" reports like this at all. Even though you haven't found something I have to fix, it's also good indirect confirmation that you've played at least up until that point. Information about how you play is valuable to me too!
The first real bug reported to me was actually a bug in an older version of the SDL framework library I was using for the Windows build, and quickly resolved by just updating to a newer version, something I just hadn't bothered to do yet since the start of the project.
If you're wondering about the numbering of the builds: while I was working on the game I had an internal counter that automatically changed the version number. This was mostly to identify if I was accidentally playing an old version, or which version I may have sent out to someone to try. I didn't put it in until halfway through development, and it wasn't keeping track of every single build, but it started at 0.0.0.0 and made it to 188.8.131.52 before I was ready to create the beta. I've had to play a lot of different versions of Lizard over the past few years.
It's not just bugs, right?
Aside from finding bugs, i.e. things that happened by accident, the beta test is a time for me to collect and respond to feedback. I had many people try Lizard before I began the beta, but suddenly it is being played by more people than ever before, and people I don't know personally. I'm getting a lot of information about my game back that just I didn't have access to until now.
People often tell me about things they like or don't like after trying my game. If someone tells me "--- is hard", that's not normally a surprise, it's usually something I intended to be hard. If I have the opportunity, I might like to ask "did you eventually get through it, or give up?", "what compelled you to keep going?", "how long did it take?" or maybe ideally I can watch them try the hard thing and see exactly what they're trying to do and why it isn't working for them. All of these things are really difficult to put into words. I expect people to tell me that the hard thing is hard... but getting an understanding of how hard and why is much trickier. There's a huge amount of interpretation involved in making use of any feedback for things like this, but even the vague comments are still useful.
It's even valuable to get the same comments over and over again. Every redundant comment gives me more perspective about how much it's noticed. Sometimes just hearing the same complaint for the 50th time is the one you needed to finally decide it's worse than you intended, and that you need to do something about it.
People also tend to offer suggestions. I don't mind suggestions, but usually their value is that they are indirectly telling me about a problem. The problem is what's important; it's what I really need information about. Occasionally I do hear very good suggestions, but if I'm honest about this: most of them are ideas that I'm already well aware of, or are otherwise unusable to me. There's usually a reason that idea wasn't already there, and it may even have already been implemented and tried. Sometimes the reason is technical, sometimes it's a conflict with other design goals, but it's usually very complicated to explain why.
Every change to the game has to be evaluated against the million tiny pieces that make up the whole; everything affects everything else. If I wanted to tell you why I didn't do a thing, the real explanation probably requires having to educate you about every other thing in the game, or at least every thing that's relevant to the decision you're asking about. That can be a lot. That's usually too much. (All this before subjectivity even comes into the picture.)
Anyhow, suggestions are perfectly fine as feedback, please don't hesitate with those, just know I probably won't use it directly. They're still a good indication of what's bothering you and what your expectations were. There is of course a point where a suggestion can turn into a demand, or someone wants an account or explanation why, and I usually just can't oblige, but even this comes with a useful point of data about how they feel about my game.
I've been trying to put this gathered information to use to make improvements to the game.
Several people at this point have gone through the game thoroughly from start to finish, and this is very good confirmation that the game works and doesn't seem to have anything mechanically wrong with it. There are probably undiscovered bugs, you just can't catch 'em all, but so far nobody's caught anything major.
The reports from people struggling with my game are very interesting too. I've been aware for a long time that I've been making a game that is a bit difficult. Over the course of development, I've watched a lot of people attempt and overcome various challenges in it, but with the beta I've suddenly got more and better information about this, and I'm trying to use it to identify the roughest edges of the game and smooth them down a little.
The biggest thing I've done so far is just to change the movement physics. Specifically I adjusted how quickly you can stop after landing or turning around. I won't say too much about it right now, since this update is already pretty long, and it's a topic that deserves a whole article of its own.
The short story is that I really enjoyed the way it felt, and having to cope with that slide, but the more comments I got about it the more I began to think it was a mistake. Even though I had seen many people learn it, and I had carefully made sure you could still make any jump in the game without figuring out how to stop your slide, it seems like it was just too much for too many people.
This change in particular is just one number in a line of the lizard character control code, but it's a systemic change that applies to every single time you need to stop or turn around in the game. When more in the middle of development, it was hard to even consider a change like this, but now I have a perspective on the complete game that makes it easier to see what I can gain or lose by doing it. Ultimately, after trying it, I felt the negative consequences were low.
When will it end?
The amount of useful feedback I can squeeze out of this beta is finite. As I said above, it only works as far as people are interested in playing something new, and I know the majority of you probably want to wait until the cartridge is in your hands, too.
I think I've gotten the bulk of it by now. I've got a list of things to do for one more beta update, and after that I'm probably not going to change anything else unless it seems critical. I'm pretty happy with the shape the game is currently in, and don't foresee the beta continuing for any protracted length from there.
As soon as the final ROM is prepared, Infinite NES Lives will begin building and shipping out cartridges to all the backers. Preparing for the public release will take a bit longer, but since that's mostly out of my hands I will probably spend the time preparing the game for release on Steam, and creating a new demo version to replace the alpha prototype.
To everyone who has been willing to beta test the game, THANK YOU SO VERY MUCH. Several of you have been very thorough, and you should know that this has been critically important to the project. Thank you.
A few people have contacted me about translating Lizard, and I must apologize for the delay in response. There have been some small changes to the text of the game as part of the beta revisions, so I haven't gotten around to preparing materials for it yet. I will reply soon.