Developing a Realtime Android Game - Lessons Learned - Part I

Lessons Learned - Part I

We just completed our 11 month journey of creating a realtime space shooter for Android using Scala programming language. We did the project in our freetime and you can check out the end result at www.scawargame.com.

We learned a lot during these 11 months. This is the beginning of a series of blog posts about things we think went well and things we would definitely do differently if we were to start again from the beginning. It will be a mixture of topics about coding, project management, agile and marketing.

Plan Well

We did pretty much everything in our project in an agile way: very low bureaucracy, development in sprints, work lists in backlogs, used pair programming and weekend hackathons. We also did stuff in the usual way, wrote production code while still very much in learning phase. All this is well and good, but what we forgot and paid dearly for is that design decisions have a so-called last responsible moment of making.

Planning too much ahead is risky but if you delay your design decisions past the last responsible moment, the decisions are made by default. So instead of a concious group decision, you just implement what think is needed. Soon after you realize you just implemented something that was just your reflex and if you would have thought this through with your team earlier, you would have come up with a better way of doing it. Now, you will come up with the better idea but it’s too late to avoid rewriting major part of your work. Repeat this often enough and it will get frustrating.

So plan well here means: plan enough, just early enough and plan it with your whole team, don’t just make quick decisions to get on with your implementation.

Hard Work Makes You Hate Your Game

One thing about projects that take a long time: you are going to hate them! They eat all of your free time and make your significant other mad. Working with the same thing for months get really boring very easily. And since we were using our free time, it was sometimes really hard to get motivation to start working again after a long day at work.

Best solution what we discovered was don’t work alone! When you have pressure from other developers, the pressure is the best motivation to make you work harder. Nothing makes you feel better than getting feedback from your fellow developer saying “Shit, that is awesome!”

Also remember to take some time off from your project. Fresh mind and body will give you new ideas and perspective on what should be done. Spend time with your family, they seem to nag less if you do so.

Motivation from Goals (SPEC)

Well planned goals are a key in enabling you to finish your game someday. In the very begining of our project, we didn’t really make goals, we just did stuff. Doing stuff without goals is a good way for getting to know your tools and platform, but for advancing your project, not so much. Better way to do proof-of-concept is to do Spikes: time-boxed research / development tasks that should be thrown away. You’ll avoid plenty of mistakes in your project code and design this way.

After a while in to our project we started planning what we should do and tried to prioritize what was important and what wasn’t. Working with the prioritized list made it easier to get the important features completed. Completing an important feature will boost your motivation through the roof. Prioritizing is a good way to help you create goals. Also when working with a team it’s easier to divide tasks with your fellow developers when you know where you need to focus.

Our first release version, which actually wasn’t released, changed our way of working more dramatically. We got a huge motivation boost from squeezing the first release candidate ready, or should I say the first actual beta. We had good features but... not enough content, not very polished version and loads of optimizing to be done.

This was the point where we started to do actual “sprints” with features that needed to be completed. We already had done the Spikes and the prioritizing, now all we needed was some planning poker and set of features to agree on. When planning and doing the estimation, it is really important to get your team into the same place. Talk about the features and do the estimations as a group. Every member of the team has to participate in the planning poker! It’s not important if every member knows how long it takes to program a feature, but they can and will provide some other useful information to the planning.

Finish your goals. Make plenty of release candidates. When you complete a full list of planned and estimated feature, you get a motivation boost. Completing a “release” ready product boosts your motivation even more.

So here’s our mantra on how you should do your stuff.

Spike - Prioritize - Estimate - Complete = SPEC

Make Builds Fast

Make your builds as fast as possible. This chapter might as well be known as “do what we say, not what we did”.

Your build times affect your productivity. If your build times go up, you spend longer periods coding blind and this way you’ll also spend more time debugging. More debugging means more test builds. You get the picture, it’s a nasty circle.

But we were building an Android game with Scala. And as long-time enterprise programmers, we were building with Maven. So we had few problems, namely:

• Scala is bad for your build times
• Android is bad for your build times
• Maven is bad for your build times

We should have never chosen Maven. But then again, building for Android involves lot of extra steps, dexing, signing your code and zipaligning your packages etc. So you definitely need automated builds. By the time we started our project, SBT, Scala’s not-so-aptly-named Simple Build Tool was constantly and completely rewritten, it seems, every few days or so. Well not quite, but enough to encourage us to stay away.

Scala’s library is pretty big. And maven scala plugin stopped supporting incremental builds. So compile times are pretty bad as you always load the whole language library and always compile your whole project. Dalvik, Android’s JVM, can not survive the amount of classes included in the Scala library. So you need to run Proguard, to shake off all classes that are not really needed by your application. On our machines this takes as long as compiling.

We tried several things to shorten the build time. Mainly how to skip the proguard phase. After proguarding your application, you can grab scala-part of the resulting classes, put them in jar and use that package as a dependency instead of the full scala library. We wrote a short blog entry about this a while ago. This allows you to skip the proguarding entirely, but every time you use a new API from Scala, or a new implicit conversion or something subtle like that, you need to create a new jar with the new required classes included. So this might have some uses in a stable and mature project project but it will get tedious really easy.

Other option was to hack the Android bootloader to load Scala libraries on startup along with other libs provided by the OS. We did this on the Android Emulator but were reluctant to do it on real devices. Partly because it’s a bit sketchy and partly because we then would not have had regular devices to test on.

To top things off, once your package is done, you need to uninstall the previous version from your phone and install the new package. You can’t use Android Emulator as it’s pretty much one of the worst pieces of software known to man.

End of Part I

Well that's it for part I. We got plenty of topics lined of next parts. As always, comments, diggs, +1:s etc are very welcome.

Proguard performance and Scala versions

We have been holding back from moving to Scala 2.9 for a long time now. The reason is that Scala builds for Android need to be treated with Proguard to reduce their size. And on Scala 2.9, Proguard is incredibly slow. I think this has to do with the trait bloat that was creeping up on Scala library while it gained more and more features. Now that it's "fixed" by introducing more abstract classes to Scala 2.10 I was interested in seeing if my guess was correct and if the proguard performance would back to 2.8 level.

After fixing some issues in our build introduced by Scala 2.10, I run our build for Scala versions 2.8.2, 2.9.1 and 2.10.0-M2 (the currently latest milestone release). Timing information was gained from Lasse Koskela's maven-build-utils which I've done some work with lately.
Our build includes 7346 lines of Scala and some Java. Here's the performance information:

Scala 2.8.2

[INFO]   compile                                  32.6s  45%

[INFO]     maven-compiler-plugin:compile           3.0s   9%

[INFO]     maven-scala-plugin:compile             29.6s  90%

[INFO]   process-classes                          18.7s  26%

[INFO]     android-maven-plugin:unpack             0.7s   3%

[INFO]     proguard-maven-plugin:proguard         18.0s  96%

Scala 2.9.1

[INFO]   compile                                  32.0s  19%

[INFO]     maven-compiler-plugin:compile           3.0s   9%

[INFO]     maven-scala-plugin:compile             29.0s  90%

[INFO]   process-classes                         110.3s  67%

[INFO]     android-maven-plugin:unpack             0.8s   0%

[INFO]     proguard-maven-plugin:proguard        109.5s  99%

2.10.0-M2

[INFO]   compile                                  32.8s  41%

[INFO]     maven-compiler-plugin:compile           3.0s   9%

[INFO]     maven-scala-plugin:compile             29.9s  90%

[INFO]   process-classes                          24.0s  30%

[INFO]     android-maven-plugin:unpack             0.8s   3%

[INFO]     proguard-maven-plugin:proguard         23.2s  96%

As you can see, Proguard takes over 6 times as long on 2.9.1 that it does on 2.8.2. And on 2.10.0-M2 it's almost back to what it was on 2.8.2. So that's really good news for Android developers using Scala. Compile time differences between different Scala library versions are negligible.

Integrating Scoreloop to your Maven build

I’m a big fan of automated build process. Especially on Android where creating a release-ready package is a really multi step process: you need to compile, strip with ProGuard, dex, package, sign, verify and what not.

When it came time to integrate Scoreloop to our game, we had some problems. Scoreloops offers a nice UI ready for use. But this means you can just put it in a jar and use it like normal dependency, because UI’s need graphics, layout xml’s and other resources. Finding info on how to get this to work with Maven proved difficult. Thanks to Julien Donguy at Scoreloop for pointing me to Android Library Projects.

Here’s how you would go about adding Scoreloop to your Maven build:

Download the Scoreloop SDK and unpack it somewhere. Inside you’ll find two directories: ScoreloopCore and ScoreloopUI. ScoreloopCore contains scoreloop-core.jar which you can install to your local maven repository like any other jar:

mvn install:install-file -Dfile=scoreloop-core.jar -DgroupId=com.scoreloop -DartifactId=scoreloop-core -Dversion=2.4 -Dpackaging=jar

Next you want to create a zip file from the contents of the ScoreloopUI directory (so that AndroidManifest.xml is in the root of the zip) and rename that zip to scoreloop-ui.apklib.

We can then add that package also to your Maven local repository with command:

mvn install:install-file -Dfile=scoreloop-ui.apklib -DgroupId=com.scoreloop -DartifactId=scoreloop-ui -Dversion=2.4 -Dpackaging=apklib

Note the different packaging parameter.

Next you’ll add both of these packages as dependeciens to your pom.xml.

        <dependency>
            <groupId>com.scoreloop</groupId>
            <artifactId>scoreloop-core</artifactId>
            <version>2.4</version>
        </dependency>
        <dependency>
            <groupId>com.scoreloop</groupId>
            <artifactId>scoreloop-ui</artifactId>
            <version>2.4</version>
            <type>apklib</type>
        </dependency>

Now running mvn clean package should create an apk that includes all necessary parts from Scoreloop. You can see the apklib getting included with something like this:

[INFO] --- android-maven-plugin:3.0.0-alpha-11:generate-sources (default-generate-sources) @ YourGame ---
[DEBUG] Expanding: C:\Users\someuser\m2\repository\com\scoreloop\scoreloop-ui\2.4\scoreloop-ui-2.4.apklib into C:\ideaworkspace\examples\YourGame\target\unpack\apklibs\com.scoreloop_scoreloop-ui_apklib_2.4

We did experience problems with earlier versions of maven-android-plugin so we suggest using Maven 3 and android-maven-plugin 3.0.0 alpha’s.

Atleast for Intellij IDEA, there’s one more step to go. Apklib’s, unlike regular jar dependencies, are not visible inside IDEA, so if you start coding, you’ll find none of the classes from the scoreloop-ui.apklib available. To fix this you can add the ScoreloopUI as a module dependency inside IDEA:

• Select File –> New module.. –> Import module from external model


• Point the file explorer to the location where you unpacked ScoreloopUI.


• When it’s imported, select your game’s module and “Open Module Settings”


• Select “Dependencies” tab and click Add.. –> Module Dependency and select ScoreloopUI.

Now the sources for the UI classes are available to your IDE, and this doesn’t affect the maven build process.

From here on forwards you can follow the User Guide that comes with Scoreloop.

Comments and feedback are welcome as usual!

Improve your maven android build speed

Here’s a nice way of shaving around 10 seconds off your average maven builds total time when building Scala Android applications.

You know the part when you need to unpack everything to have ProGuard go through all your classes to remove that extra fat. Well, even if you make a lot of changes to your code, most of the time is spent unpacking the libraries (Scala and Android mainly). And these libraries don’t change that often. So after you’ve done it once, and they have already been unpacked to your target directory, why do it again? Well, someone figured that out and now it’s added to maven-android-plugin in version 2.9.0-beta-5, so switch to that from 2.8.4 which is what you are probably using.

Then all you need to do is add the line:

<lazyLibraryUnpack>true</lazyLibraryUnpack>

under the <configuration> of your maven-android-plugin.

If this causes force closes on your device/emulator, check that your ProGuard settings are not too aggressive.

Your first scala-android apk

Ok, so far we've done some basic stuff. Here's some more before we really get to the good stuff. This post only contains the configuration for getting Scala working with Android. The next tutorial is going to include all the configurations to get a proper Android application packaged with Maven.

To get your environment running, please check this post by Janne.

Ok, let's get to the interesting part. Combining Scala, Maven and Android SDK with a working development environment.

Properties

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>5</maven.compiler.source>
    <maven.compiler.target>5</maven.compiler.target>
</properties>

New dependencies

• Android
• Andengine (2d engine)

<dependencies>
    <dependency>
        <groupId>com.google.android</groupId>
        <artifactId>android</artifactId>
        <version>2.3.3</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.andengine</groupId>
        <artifactId>andengine</artifactId>
        <version>1.0</version>
    </dependency>
</dependencies>

Since the Andengine package is not included in any Maven repository, you'll need to add the file by hand:

mvn install:install-file -Dfile=path-to-andengine.jar/andengine.jar -DgroupId=org.andengine -DartifactId=andengine -Dversion=1.0 -Dpackaging=jar

You can download the Andengine file here.

Remember to set packaging to apk!
<modelVersion>4.0.0</modelVersion>

<groupId>org.example</groupId>
<artifactId>first-application</artifactId>
<version>1.0</version>
<packaging>apk</packaging>
<name>first-application</name>

Plugins
The Android plugin that handles the apk generation.

<plugin>

<groupId>com.jayway.maven.plugins.android.generation2</groupId>
    <artifactId>maven-android-plugin</artifactId>
    <version>2.8.4</version>
    <configuration>
        <sdk>
            <!-- Don't forget to set your ANDROID_HOME environment variable to your SDK directory! -->
            <path>${env.ANDROID_HOME}</path>
            <!-- Platform 10 is Android 2.3.3 -->
            <platform>10</platform>
        </sdk>
        <jvmArguments>
            <jvmArgument>-Xmx1024m</jvmArgument>
        </jvmArguments>
        <!-- All of these go into the /src/main/android/ directory, we don't
        want to polute the project root directory. -->
        <androidManifestFile>${project.basedir}/src/main/android/AndroidManifest.xml</androidManifestFile>
        <resourceDirectory>${project.basedir}/src/main/android/res</resourceDirectory>
        <assetsDirectory>${project.basedir}/src/main/android/assets</assetsDirectory>
        <nativeLibrariesDirectory>${project.basedir}/src/main/android/native</nativeLibrariesDirectory>
        <resourceOverlayDirectory>${project.basedir}/src/main/android/overlay</resourceOverlayDirectory>

        <deleteConflictingFiles>true</deleteConflictingFiles>
        <undeployBeforeDeploy>true</undeployBeforeDeploy>
    </configuration>
    <extensions>true</extensions>
</plugin>

Last  we need the most important plugin of them all, ProGuard. Without this  plugin the apk generation will fail since the Scala dependency will generate too  many files.

<plugin>

<groupId>com.pyx4me</groupId>
    <artifactId>proguard-maven-plugin</artifactId>
    <version>2.0.4</version>
    <executions>
        <execution>
            <phase>process-classes</phase>
            <goals>
                <goal>proguard</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <injar>android-classes</injar>
        <libs>
            <lib>${java.home}/lib/rt.jar</lib>
        </libs>
        <obfuscate>false</obfuscate>
        <options>
            <option>-keep public class * extends android.app.Activity</option>
            <option>-keep public class * extends android.app.Application</option>
            <option>-keep public class * extends android.app.Service</option>
            <option>-keep public class * extends android.content.BroadcastReceiver</option>
            <option>-keep public class * extends android.content.ContentProvider</option>
            <option>-dontskipnonpubliclibraryclasses</option>
            <option>-dontoptimize</option>
            <option>-printmapping map.txt</option>
            <option>-printseeds seed.txt</option>
            <option>-ignorewarnings</option>
        </options>
    </configuration>
</plugin>

With these settings you should be able to create a working development environment for Android development with Scala. Make a test run by running the following command: mvn package to see if the environment compiles.
Congrats! You have an Android application that does nothing.
Also, if you can't get things working with the tutorial, here's the pom.xml you can use to get your application to compile.
From my point of view getting Android, Scala and Maven working together is a real pain in the ass and I couldn't have done the configuration without the following tutorials:
http://www.scala-lang.org/node/345 (Scala with Maven) Thanks to the Scala community.
Building Android applications with Maven and IntelliJ 11/26/2010 Thanks to Christian Bauer for an exellent tutorial.
Also some more information from Proguard:
http://code.google.com/p/maven-android-plugin/wiki/ProGuard?spec=svn331

Sharing Maven local repository with DropBox

Maven stores copies of libraries your projects depend on in to a local directory called local repository. Making this repository available to all the developers in your team can be beneficial, especially if you have multiple manually added libraries (for example Andengine doesn’t have a Maven repository so the libraries need to be manually added to your local repository.
Easy and free way of sharing this local repo is:

1. All developers register and install Dropbox.
   
2. You create a localRepository directory under your Dropbox folder.
   
3. You share the localRepository folder to rest of your developers.
   
4. All developers configure their Maven to use the Dropbox folder as their localRepository.

Location of your local Maven repository can be configured in settings.xml, either in the <user_home>/.m2 directory, or in the <maven installation>/conf directory. You need to add a new <localrepository> tag. For example:

<settings>  
...
<localrepository>C:/Users/someone/Documents/My Dropbox/localRepository</localrepository>
...
</settings>

That’s it! Thanks Eric Lefevre-Ardant for the tip on changing the repo location.

Your first Scala module with Maven

Start by:

• Installing IntelliJ IDEA
• Downloading and unpacking Maven
• Installing Scala
• Installing Scala plugin to IntelliJ
• Setting Maven home directory (either with M2_HOME or overriding the settings in IntelliJ)

Next up is creating a project and a module based on scala-archetype-simple:


File -> New project..
Select "Create project from scratch".

Give the project a name and the module a name. Remember that the module name is used as a package name for scala, so no dashes allowed!

Select module type: Maven Module.

Click next.

Make sure the create from archetype is selected and click "Add Archetype..".

Fill in the following details:

Select the archetype you added and click Finish.

In this archetype, the pom.xml contains <repositories> and <pluginRepositories> but they are commented out. Uncomment those blocks.

On the very right side of IntelliJ there's a vertical block that contains a tab called Maven Projects. Open it, open the project you just created and run the "package" maven task. This will compile, test and package your project and it will also download all dependencies required by a scala maven project.

Now you should be all set to create your first maven-based scala project.

Note that the archetype used, also created a simple object for you and some tests for that object to get you started.