Hiding your Mobile Secrets

When you open-source a project, you want to avoid leaking API keys and other secret sauce to the public. Best case scenario, someone will use your API keys to create dozens of apps and you will get bombarded with emails about them. Worst case scenario, they can rack up an enormous bill or steal all your data.

I've been working on a Kotlin/Native framework (which allows a common shared codebase between iOS and Android) and its two companion apps as an experiment. As with most experiments I do, I wanted to keep it open-source, but I also wanted to be able to put crash reporting with Crashlytics in there so I could see quickly when things broke.

There is not a Kotlin/Native Crashlytics SDK, so I was going to need to install the iOS and Android SDKs at app level. I decided to use this as an excuse to research how to keep secrets on apps for each platform.

In the end, with the use of a .secrets folder and some well placed environment variables on our Continuous Integration server, I was able to get the app building and running with Crashlytics on both platforms, in ways which would be reproducible even if we weren't using Kotlin/Native in the future.

If you only care about one platform, you can jump to reading about either Android or iOS, but the overall idea is to automate much of the following process:

1. Pull anything secret for a particular platform out of your public-facing items and into a file.
2. Stick that file into a git-ignored .secrets folder.
3. When running locally, load the ignored file.
4. When running on CI, load values from environment variables.
5. Take the secure values you retrieved from wherever, and put them into the file which needs those values.
6. Finish your build.
7. git reset the file which should not have secret values committed.

Android

We'll start with the easier of the two platforms - Android.

Android's easier because it already uses the Gradle build system, which makes creating new things to plug into it relatively easy, once you figure out how to hook into the system.

When you go create an Android application through Firebase, you'll be asked to download a google-services.json file that has all the information Firebase needs to spin up its SDK. This needs to be placed in the root of your repository.

Unfortunately, this file has a number of sensitive API keys in it, which should not be committed to a public repository. So how do you take them out?

I made an android_secrets.properties file and stuck it in the git-ignored .secrets folder. A .properties file is a plaintext-ish file which a format which looks like this:

YOUR_KEY_NAME=YourValue

YourValue is stored as a string, so quotation marks are not necessary (and often are not actually wanted).

I then took all the values out of the google-services.json file which I did not want to be public, and replaced them with the string REPLACE_AT_BUILD_TIME, so that the JSON looked something like:

{
    "project_info" : {
        "project_number": "REPLACE_AT_BUILD_TIME"
    },
    "client": [
        {
            "api_key" : [
                "current_key": "REPLACE_AT_BUILD_TIME"
            ]
        }
    ]
}

This makes it super obvious if what you're doing isn't actually working at build time - You'll get a crash if the values provided to the Firebase SDK are not in the proper format. This can also be committed to version control without worry.

I took the sensitive keys I'd removed and placed them into the .properties file I referred to earlier, which then looked something like this:

FIR_PROJECT_NUMBER=FirstSecureValue
FIR_CURRENT_KEY=SecondSecureValue

I decided to make a gradle Task to replace google services - that way I could set it up as a dependency for another task I knew would always run.

In the build.gradle at my Android project level, I added a new task below my android and dependencies closures:

task replaceGoogleServices {
}

To test that it was working, I gave it a very simple doLast block, which is what's executed as the last piece of the task you're setting up:

task replaceGoogleServices {

    doLast {
        println "REPLACE GOOGLE!"
    }
}

and then below the task's declaration, I set it up as a dependency of preBuild (which is run with pretty much any Android gradle command:

preBuild.dependsOn replaceGoogleServices

When that ran, I could see the "REPLACE GOOGLE!" print out in the console, and I could confirm it worked. Next, I knew I wanted to add some additional methods, so I started like by trying to define some initial methods:

task replaceGoogleServices {
    def loadSecrets() {
        def secrets = loadSecretsFromProperties()
        if (secrets == null) {
            secrets = loadSecretsFromEnv()
        }
        assert(secrets != null)
        return secrets
    }

    def loadSecretsFromProperties() {
        // TODO
    }

    def loadSecretsFromEnv() {
        // TODO
    }

    doLast {
        println "REPLACE GOOGLE!"
    }
}

The problem is that within a Gradle Task, you can't use the normal def syntax used to define a method. You must declare methods like so:

task replaceGoogleServices {
    ext.loadSecrets = {
        def secrets = loadSecretsFromProperties()
        if (secrets == null) {
            secrets = loadSecretsFromEnv()
        }
        assert(secrets != null)
        return secrets
    }

    ext.loadSecretsFromProperties = {
        // TODO
    }

    ext.loadSecretsFromEnv = {
        // TODO
    }

    doLast {
        println "REPLACE GOOGLE!"
    }
}

You can still call your methods in exactly the same way as you would with def defined methods, you just have to set them up as closures.

Then, I fleshed out the method to load the secrets from a file:

ext.loadSecretsFromProperties = {
    def secretsPropertiesFile = rootProject.file(".secrets/android_secrets.properties")
    def secretsProperties = new Properties()

    try {
        secretsProperties.load(new FileInputStream(secretsPropertiesFile))
        println "Loaded secrets from local file..."
    } catch (FileNotFoundException e) {
        // File was not found, try to load from env.
        return null
    }

    return secretsProperties
}

If that failed, I'd try to load the same keys from the environment:

ext.getEnvironmentValue = { key ->
    return System.getenv().get(key, null)
}

ext.loadSecretsFromEnv = {
    def url = getEnvironmentValue("FIR_PROJECT_NUMBER")

    if (url == null) {
        return null
    }

    println("Loading secrets from env...")

    def secretsProperties = new Properties()
    secretsProperties.put("FIR_PROJECT_NUMBER", getEnvironmentValue("FIR_PROJECT_NUMBER"))
    secretsProperties.put("FIR_CURRENT_KEY", getEnvironmentValue("FIR_CURRENT_KEY"))

    return secretsProperties
}

Once I had that down, I was able to go into the doLast method and use it (and Groovy's slightly insane JSON parsing library) to get the values from the secrets.properties file into the JSON:

doLast {
    def secrets = loadSecrets()
    assert (secrets != null)

    def servicesJSONFile = rootProject.file('android/google-services.json')
    def parsedJSON = new JsonSlurper().parseText(servicesJSONFile.text)
    def builder = new JsonBuilder(parsedJSON)

    def projectNumber = secrets.getProperty("FIR_PROJECT_NUMBER")
    assert(projectNumber != null)
    builder.content.project_info.project_number = projectNumber

    def currentKey = secrets.getProperty("FIR_CURRENT_KEY")
    assert(currentKey != null)
    builder.content.client[0].api_key[0].current_key = currentKey

    // [You get the idea, more code omitted for brevity]

    servicesJSONFile.write(builder.toPrettyString())
}

After this ran, the JSON was updated to look something like:

{
    "project_info" : {
        "project_number": "FirstSecureValue"
    },
    "client": [
        {
            "api_key" : [
                "current_key": "SecondSecureValue"
            ]
        }
    ]
}

Huzzah! The values were being replaced automatically. Now, all that was left was automatically resetting google-services.json so I didn't accidentally commit those values I'd gone through all this trouble to secure.

Figuring out how to get something to execute after a task is completed in gradle is a little bit harder than it seems on its face. However, I did find a workaround.

After some research I realized that the gradle task graph has an afterTask closure, as well as a list of all the tasks it's performing.

Below the end of my resetGoogleServices task, I added something to look for the last task so I could execute something after the last task had completed:

gradle.taskGraph.afterTask { Task task ->
    if (task == gradle.taskGraph.allTasks.last()) {
        println "DO THIS LAST!"
    }
}

Once I verified that was executing where it needed to, after some time trying to use another task, I realized things would be much easier if I just used a def method, which had caused me so much problem in working with a Task before.

Once I realized that, it was simple to access the file and then reset it with a method, and then call that method when the task graph completed its last task:

def resetGoogleServicesFile() {
    println "Resetting services JSON"
    def servicesJSONFile = rootProject.file('android/google-services.json')
    "git checkout HEAD -- ${servicesJSONFile.path}".execute()
}

gradle.taskGraph.afterTask { Task task ->
    if (task == gradle.taskGraph.allTasks.last()) {
        resetGoogleServicesFile()
    }
}

The full version of the build.gradle file I landed on can be found in our [DoorbellBot-Native] repo here.

iOS

As with anything Apple makes, iOS's build system is a bit opaque. It's also somewhat difficult to hook into without shell scripting. I was interested in a solution which did not involve shell scripting, for the very technical reason that I am terrible at it, and screw it up constantly.

Swift provides an interesting twist to this, particularly with the use of the Swift Package Manager, often referred to as SPM.

Swift files can be turned into executable scripts, but you also can create small command line applications using SPM which allow you to take advantage of imported frameworks. SPM is built into the version of Xcode that will be out this fall, but that wasn't even announced when I started working on this, so I needed to do things in a bit more of a complex fashion.

I'd built some Swift scripting tooling using John Sundell's Marathon layer on top of SPM. I really loved Marathon's ease of setup, but it's designed more for things that don't need to run on every build of your app. It takes a second to spin everything it needs up, so if you're running it on every build, it can cause the build to stutter slightly.

In my previous use of Marathon, I'd wound up hacking together some shell scripts to determine if I needed to run Marathon to prevent the hiccup, but that seemed to defeat the point of trying to run scripts in Swift in the first place.

I decided to try to build my own command line application with the Swift Package Manager directly to see if I could remove the overhead for something I wanted to run every time I built my app.

Ironically, John Sundell's own work on Marathon is part of what got me up and running with this quickly: John's got a great guide to setting up a command line application which walked through some of the process of Marathon's setup.

I'll leave that link as the definitive guide to setting up a command line tool, as it's very thorough. I wound up also using a couple of small libraries John put together for scripting, Files, to make dealing with the File System simpler, and ShellOut, to make running Bash commands simpler.

There was a small twist in the data formats this time as well: Instead of Google handing me JSON, then my script loading from a platform-specific data storage type, it was the reverse.

Google hands you a GoogleService-Info.plist file where all of your keys are stored. I went through a similar process of taking out the sensitive keys and replacing them with garbage information such as REPLACE_AT_BUILD_TIME as I did with Android, but this time I put the sensitive keys into a JSON file which was stored in .secrets, since JSON is super-easy to parse with Swift.

First, in the main bit of the script, I'd use Files to grab access to the folder represented by $SRCROOT in the environment - the source root of the iOS app (checking to make sure it was finding the right folder), and pass it into a function I'd made to update the Plist:

let rootPath = try shellOut(to: ShellOutCommand(string: "echo $SRCROOT"))
guard rootPath.hasSuffix("iOS") else {
    throw FileError.cantAccessSRCROOT
}

let rootFolder = try Folder(path: root)
try FirebasePlistUpdater.run(sourceRoot: rootFolder)

Similarly to the Android script, I'd check to see if there was a local file, and if there wasn't I'd try to load secrets from the environment. Here's what that function looks like:

enum Error: Swift.Error {
    case couldNotAccessGitRoot
    case noSecretsFileOrSecrets(String)
}

static func run(sourceRoot: Folder) throws {
    guard let gitRoot = sourceRoot.parent else {
        throw Error.couldNotAccessGitRoot
    }

    let secrets: [String: String]
    if self.exportJSONExists(gitRoot: gitRoot) {
        print("Secrets file exists locally, loading from that...")
        secrets = try self.loadSecretsFromJSON(gitRoot: gitRoot)
    } else {
        print("No local secrets file, attempting to load from environment variables...")
        secrets = self.loadSecretsFromEnvironment()
        guard !secrets.isEmpty else {
            throw Error.noSecretsFileOrSecrets("You don't have a secrets file or secrets set up in the environment - this ain't gonna work. Please make sure to set up secrets on CI." )
        }
    }

    try self.updateFirebasePlistValuesCommands(sourceRoot: sourceRoot, secrets: secrets)
}

One thing I particularly like about doing this in Swift is being able to add customizable errors. I wound up using a lot of print statements as well so I could see what was happening at certain levels on the CI server as well.

But when something went wrong, it's was really helpful to have a specifically typed error which eventually got printed out rather than a more generic exit code. It made debugging the whole thing much faster.

The Files library made checking if a file exists very simple:

private static func exportJSONExists(gitRoot: Folder) -> Bool {
    do {
        let secretsFolder = try gitRoot.subfolder(named: secretsFolderName)
        return secretsFolder.containsFile(named: secretsFileName)
    } catch {
        debugPrint("Error checking if export script exists: \(error)")
        return false
    }
}

I didn't want this function to throw, but to simply return false if there was an issue, since that issue could be handled the same way as a nonexistent file would be handled: By trying to load the secrets from the Environment.

I was able to use Files to cut down on code loading from JSON as well:

private static func loadSecretsFromJSON(gitRoot: Folder) throws -> [String: String] {
    let secretsFolder = try gitRoot.subfolder(named: secretsFolderName)
    let iosFile = try secretsFolder.file(named: secretsFileName)

    return try JSONLoader.loadStringJSON(from: iosFile)
}

If for some reason either the JSON file didn't exist or couldn't be loaded, I'd try loading the secrets from the environment. Fortunately Swift has an extremely simple way to do this:

private static func loadSecretsFromEnvironment() -> [String: String] {
    let envDict = ProcessInfo.processInfo.environment

    return envDict.filter { key, _ in
        return key.hasPrefix("FIR_")
    }
}

Then, once I had the secrets either from JSON or from the Environment, I made a wrapper for an Apple-provided tool called Plistbuddy (Note that link is to another site because Apple doesn't actually have any documentation for Plistbuddy beyond its man page) to replace replace data within the plist I got from Google.

With that wrapper, I was able to quickly go through and update values for different secrets in the plist using the data I'd loaded from either the JSON file or Environment variables:

private static func updateFirebasePlistValuesCommands(sourceRoot: Folder,
                                                      secrets: [String: String]) throws {

    let iOSAppFolder = try sourceRoot.subfolder(named: "DoorbellBotNative")
    let plistFile = try iOSAppFolder.file(named: "GoogleService-Info.plist")

    let keysToUpdate = [
        "AD_UNIT_ID_FOR_BANNER_TEST",
        "API_KEY",
        "CLIENT_ID",
        "DATABASE_URL",
        "GCM_SENDER_ID",
        "GOOGLE_APP_ID",
        "PROJECT_ID",
        "REVERSED_CLIENT_ID",
        "STORAGE_BUCKET"
    ]

    keysToUpdate.forEach { key in
        let secretKey = "FIR_\(key)"
        guard let secret = secrets[secretKey] else {
            print("No secret found for \(secretKey)")
            return
        }

        Plister.setValue(secret, for: key, in: plistFile)
    }

    Plister.save(file: plistFile)
}

Unlike the Gradle code being able to handle everything for the Android app, I did have to add a bash script with one line at the very end of my build process for the iOS app:

git checkout HEAD -- "$SRCROOT/DoorbellBotNative/GoogleService-Info.plist"

Unfortunately, there's no real way to do this as part of the Swift Script without writing a second script - it must wait to be run until after the app has been built with the plist that has all the secrets in it, and you can't tell the rest of the build steps to proceed until after your Swift script exits.

However, I'd now managed to get the amount of code written in untyped Bash down to a minimum, which was one of the primary goals of this exercise, so I figured that'd do for now.

If you're interested in seeing the whole mess, you can see the Scripty project in the DoorbellBot-Native repo, and the calls to it in the build phases of the main xcodeproj file.

Conclusion

It takes a little bit of banging your head on the desk to get tools doing exactly what you want them to do, but once a task like this is automated, you can share that automation throughout the projects which you work on.

Particularly when you use tools like the Swift Package Manager and Gradle that encourage the sharing of code and techniques, building something reusable across many projects can be both fun and super-useful.

And even if you're trying to build something cross-platform, you can still spend plenty of time noodling on something applicable to a single platform, and which can help you when building natively.