“Five hours to the deadline!” the team leader shouts. You must decide what feature you still have time to implement. But is it really aligned with the overall vision and the other stressed devs? Or should you test your previous code instead? When working under extreme pressure, there is no time to waste. How do you develop in this context?

The work reported in this paper is the result from typical skunk work – often the most entertaining research to do! My original idea was to supervise a MSc thesis project on software development projects under extreme time pressure, i.e., hackathons. I wanted to explore what best practices remain when there is no time at all for wasteful activities. Unfortunately, I couldn’t find any interested students, thus I had to get my own hands dirty – and decided to focus on game jams (hackathons for game development). I collected the survey data and called for volunteers to help me with the analysis in a tweet – Vahid and Nash responded and together with a RISE colleague and a local indie game developer we established a great team.

Game development, game jams, and crunching

Video game development has turned into a huge industry. The total revenue of the video game industry was about $30.4 billion in 2016. Development of a major game title is a challenging task that involves the many skilled professionals from various disciplines including software engineering, art, and music. Modern video games are comparable to large-scale conventional software systems, in terms of size and complexity. For example, the server-side code of World of Warcraft was reported to be about 5.5 MLOC.

Game studios are under fierce market pressure to release their titles. To manage the task, game development uses several conventional software engineering activities such as requirements management, configuration management, and verification and validation. On top of that, there are of course also game specific practices and activities.

There is a wide variety of video games though, and how they are developed obviously varies greatly. Not all popular games are developed by huge studios, many contemporary games are developed by a few indie developers. A “game jam” is a hackathon for game development where participants gather to develop games within a short time span. Taking part in a game jam means that enthusiastic participants enter a voluntary period of “crunch time”, i.e., long hours of overtime – a problematic phenomenon in proprietary game development. In this study, we explored what development practices remain when crunching.

A survey of the Global Game Jam 2017 participants

We conducted a questionnaire-based opinion survey in relation to GGJ2017. Based on lessons learned from a pilot run, we decided to refer to “software requirements” as “game expectations” and “quality assurance/testing” as “satisfaction assessment”. The questionnaire had a mix of closed and open questions.

To support data collection, we used two main strategies to publicize the survey. First, I attended a local game jam in Malmö, Sweden (the one I participated in 2016). I approached all teams present at the event in the afternoon of the second day, explained the purpose of the study, and shared a link to the online questionnaire. Second, the global GGJ organization team was kind enough to post a tweet about our survey from the official GGJ Twitter account (@GlobalGameJam). In total, we received 198 responses out of the 36,401 registered GGJ2017 participants – I honestly expected a higher response rate.

Looking at the demographics of the respondents, we find that many were game development students or software development students. However, a quarter of the respondents were working as game or software developers. Furthermore, most participants had taken part of multiple game jams, worked in teams of 3-8 people, and 75% used the game engine Unity3D during the game jam. The roles in the team were mostly programmers, but also designers and artists – fewer worked on audio. While we didn’t get that many responses, the mix of respondents looked really good!

Requirements engineering under extreme time pressure

When developers are super stressed, what does requirements engineering turn into? We explored this using two perspectives: RQ1) How is initial gathering of game expectations conducted? and RQ2) How is evolution and change of game expectations handled? For both questions, we did qualitative data analysis to categorize the freeform data. The full paper contains plenty of quotes from the respondents.

The left part of the figure below shows the results from RQ1. We found that iterative sessions of brainstorming were the main method (76%) used for capturing the initial set of requirements. Brainstorming dwarfs everything else, but we found some use of whiteboards, majority voting, and prototyping too.

The right part of the figure illustrates RQ2. The most common way (47%) for the teams to handle changes was to maintain a discussion in the team throughout the game jam and reach a consensus. Many respondents also reported that they had no approach at all to manage changes to the expectations. On the other hand, some had more structure like checkpoints or appointing a leader.

Quality assurance under extreme time pressure

Analogously, we studied what remains of quality assurance and testing when developing games under extreme time pressure. The two perspectives explored were: RQ3) What type of quality assurance is done during game development? and RQ4) What type of quality assurance is done on the final product?

The left part of the figure below shows RQ3. Compared to the dominance of brainstorming and discussions for requirements engineering, the approaches for quality assurance are more varied. Regular communication is reported as the most frequent approach to QA, used to ensure that the development progressed in line with the shared expectations on the game. Communication was followed by playtesting within the game jam team. Two quite different approaches to QA that many respondents brought up are related to project management: good planning and clear definitions of tasks – many jammers believe in some order in the jam chaos.

To the right, the figure shows QA approaches to the final version of the game. Again, internal playtesting is a popular approach to do concluding QA before the game jam deadline, reported by 28% of the respondents. Just before the game deadline, many teams organized some external playtesting as well – inviting spectators, other jammers, and passers-by. We also found that many teams also did no concluding quality assurance, or only a bare minimum of testing.

Best practices under extreme time pressure

Finally, we explored what software engineering and game development best practices remain under the extreme time pressure of a game jam. The figure below shows the results. The top five software development practices were: continuous integration, minimum viable product, scope management, version control, and stand-up meetings.

Continuous integration makes sense since there is no time to resolve problems from “big bang” integration close to the deadline. The two project management practices of minimum viable product and scope management are also logical as they help a lean mindset and mitigate the risk of not having a final game to deliver at the end of the jam. Version control and stand-up meetings are interestingly widespread, a technically-oriented practice and a human-oriented practice, respectively. Both practices could introduce some overhead, i.e., setting up a version control server and stopping all development activities for a short time to discuss the current status, but apparently, participants believe that the practices provide a good return on investment.

The primary finding from the survey is that the time pressure characterizing game jam participation obviously stimulates ad hoc approaches to video game development. This is an expected finding, as a game jam is a one-shot project with an extremely challenging deadline. In the paper, we generalize our findings to similar contexts, especially software startups – as startups also face intense time pressure and operate with a small number of developers in a chaotic, rapidly evolving context with substantial uncertainty. Our findings are similar to startup research by Paternoster et a (2014).

Implications for Research

• We show a number of development practices that remain under extreme time pressure – stressed developers cling to those.
• The development at GGJ is similar to development at software startups – and to some extent app and web development.
• Crunching is an issue in the game industry but game jam organizers are in a good position to explain that crunching should be a rare event.

Recommendations for Jammers

• Brainstorming, regular communication, and internal playtesting dominate requirements engineering and quality assurance at GGJ.
• The most frequent best practices used at GGJ are continuous integration, minimum viable product, scope management, version control, and stand-up meetings.
• Some lightweight practices could be considered, including static code analysis, code reviews, and simple coding conventions – they might help!

Citation details

Markus Borg, Vahid Garousi, Anas Mahmoud, Thomas Olsson, and Oskar Stålberg. Video Game Development in a Rush: A Survey of the Global Game Jam Participants, IEEE Transactions on Games, Early access, April 2019. (preprint)

Abstract

Video game development is a complex endeavor, often involving complex software, large organizations, and aggressive release deadlines. Several studies have reported that periods of “crunch time” are prevalent in the video game industry, but there are few studies on the effects of time pressure. We conducted a survey with participants of the Global Game Jam (GGJ), a 48-hour hackathon. Based on 198 responses, the results suggest that: (1) iterative brainstorming is the most popular method for conceptualizing initial requirements; (2) continuous integration, minimum viable product, scope management, version control, and stand-up meetings are frequently applied development practices; (3) regular communication, internal playtesting, and dynamic and proactive planning are the most common quality assurance activities; and (4) familiarity with agile development has a weak correlation with perception of success in GGJ. We conclude that GGJ teams rely on ad hoc approaches to development and face-to-face communication, and recommend some complementary practices with limited overhead. Furthermore, as our findings are similar to recommendations for software startups, we posit that game jams and the startup scene share contextual similarities. Finally, we discuss the drawbacks of systemic “crunch time” and argue that game jam organizers are in a good position to problematize the phenomenon.