Recipe: Transform the JSON that your API Client decodes

I’m currently having a not-so-fun fight with the Firebase REST API while trying to bypass using the Firebase iOS SDK in a demo project whose purpose it is to not use Firebase, per se, but to implement a REST API Client on iOS. People say “Use Firebase” as it’s easy to get up and running. VERY debatable here. It does however highlight one aspect of my profession that almost seems to be a general rule:

“Mobile devs almost inevitably have to spend a lot of time figuring out why the backend never works as advertised.”

It’s been extremely rare in my career to be given an Backend API spec that can be implemented without any headaches, but I’ll save that for another post as to how best to tackle that. (In fact I did write the beginnings of such a post, many years ago).

Anyway, it seems that the firebase backend doesn’t return arrays of dictionaries. It just returns dictionaries. So if you request a collection, you’ll get a dictionary with a key count equal to the collection size.

This is a problem if you just want to parse an array of objects, because there is no array. And if you are not a backend developer like me, you just have to work with what you are given.

Here’s a use case coming from Firebase. I requested /users.json and got:

{
    "SomeRandomFakeUserId2": {
        "name": "Stephen the Admin"
    },
    "SomeRandomFakeUserId3": {
        "name": "Stephen the Tester"
    }
}

But ideally this would be an array of dictionaries with the key above embedded in the dictionary, like so:

[
    {
        "objectId": "SomeRandomFakeUserId2",
        "name": "Stephen the Admin"
    },
    {
        "objectId": "SomeRandomFakeUserId3",
        "name": "Stephen the Tester"
    }
]

But ideally I would like to keep my paradigm that generally works on my APIClient, which generally works like:

    enum APIResult<T> {
        case success(T)
        case error(Error)
    }
    
    @discardableResult
    public func sendThenDecodeJSON<T: Codable>(_ request: URLRequest,
                                               completion: @escaping (APIResult<T>) -> Void) -> URLSessionTask? {
        // session: URLSession
        let task = session.dataTask(with: request) { (data, response, error) in
            
            do {
                // Omitted:  Handle errors, etc.
                
                // You see here, all I have to do is create a struct for my responses, based on the JSON spec on the server, and I'm good.  Except... How do I use the Codable approach to json if I don't know what the key names will be?
                let result = try JSONDecoder().decode(T.self, from: data)
                
                DispatchQueue.main.async {
                    completion(.success(result))
                }                
            } catch let error {
                // Omitted for brevity: DO BETTER ERROR HANDLING HERE
            
                DispatchQueue.main.async {
                    completion(.failure(error))
                }
            }
        }
        task.resume()
        return task
    }

So what can be done here? I want to make my system flexible, but with a minimum of additional configuration. I know that on my endpoints, my existing solution gets the job done, but on a few, it doesn’t.

So I came up with the PayloadTransformable protocol:

protocol PayloadTransformable: Codable {
    static var requiresTransformation: Bool { get }
    static func transform(_ input: [String: Any]) -> [String: Any]
}

// Provide some default implementation for conformance that ultimately results in it working as before
extension PayloadTransformable {
    static var requiresTransformation: Bool { return false }
    static func transform(_ input: [String: Any]) -> [String: Any] {
        return input
    }
}

And then you have some data models, and just have to implement that:

struct Object: Codable {
    let objectId: String
    let name: String
}

struct ObjectsResponse: PayloadTransformable {
    let objects: [Object]
    
    static var requiresTransformation: Bool { return true }
    
    static func transform(_ input: [String: Any]) -> [String: Any] {
        
        var objectArray = [[String: Any]]()
        
        for key in input.keys {
            var newObject = [String: Any]()
            newObject["objectId"] = key
            guard let values = input[key] as? [String: Any] else {
                print("Failed.")
                return input
            }
            for (key, value) in values {
                newObject[key] = value
            }
            objectArray.append(newObject)
        }
        
        return ["objects": objectArray]
    }
}

And then you modify how the APIClient handles the incoming data:

@discardableResult
public func sendThenDecodeJSON<T: PayloadTransformable>(_ request: URLRequest,
                                               completion: @escaping (APIResult<T>) -> Void) -> URLSessionTask? {

        let task = session.dataTask(with: request) { (data, response, error) in

            do {
                // Omitted:  Handle errors, etc.

                var jsonData = data
                // Due to the default implementation listed in the Protocol extension, in general you have to do nothing in order to adopt this, other than make your relevant Codable types conform to PayloadTransformable (a quick find and replace)
                if T.requiresTransformation {
                    guard let json = try JSONSerialization.jsonObject(with: data, options: []) as? [String: Any] else {
                        fatalError()  // you'd throw an error here
                    }
                    let transformed = T.transform(json)
                    jsonData = try JSONSerialization.data(withJSONObject: transformed, options: [.prettyPrinted])
                }
                
                let result = try JSONDecoder().decode(T.self, from: jsonData)

                // we are using a delegateQueue on URLSession, so we want to complete on the main thread
                DispatchQueue.main.async {
                    completion(.success(result))
                }

            } catch let error {

                // Omitted for brevity: DO BETTER ERROR HANDLING HERE>

                // we are using a delegateQueue on URLSession, so we want to complete on the main thread
                DispatchQueue.main.async {
                    completion(.failure(error))
                }
            }
        }
        task.resume()
        return task
    }


And that’s it.

You could copy-paste this into a Playground and check it out:

protocol PayloadTransformable: Codable {
    static var requiresTransformation: Bool { get }
    static func transform(_ input: [String: Any]) -> [String: Any]
}

// Provide some default implementation for conformance that ultimately results in it working as before
extension PayloadTransformable {
    static var requiresTransformation: Bool { return false }
    static func transform(_ input: [String: Any]) -> [String: Any] {
        return input
    }
}
struct Object: Codable {
    let objectId: String
    let name: String
}

struct ObjectsResponse: PayloadTransformable {
    let objects: [Object]
    
    static var requiresTransformation: Bool { return true }
    static func transform(_ input: [String: Any]) -> [String: Any] {  
        var objectArray = [[String: Any]]()
        for key in input.keys {
            var newObject = [String: Any]()
            newObject["objectId"] = key
            guard let values = input[key] as? [String: Any] else {
                print("Failed.")
                return input
            }
            for (key, value) in values {
                newObject[key] = value
            }
            objectArray.append(newObject)
        }
        
        return ["objects": objectArray]
    }
}
func simulatedResponseHandler<T: PayloadTransformable>(_ data: Data) -> T {
        
        do {   
            var jsonData = data
            if T.requiresTransformation {
                guard let json = try JSONSerialization.jsonObject(with: data, options: []) as? [String: Any] else {
                    fatalError()
                }
                let transformed = T.transform(json)
                jsonData = try JSONSerialization.data(withJSONObject: transformed, options: [.prettyPrinted])
            } 
            let object = try JSONDecoder().decode(T.self, from: jsonData)
            return object
            
        } catch let e {
            fatalError("Failed!")
        }
}

// All that was definitions.  Now the code you execute:

do {
    
    let untransformed =  [
        "SomeID" : ["name": "SomeName"],
        "SomeOtherID" : ["name": "SomeOtherName"]
    ]
    
    let transformed = ["objects": [
        ["objectId": "SomeID", "name": "SomeName"],
        ["objectId": "SomeOtherID", "name": "SomeOtherName"],
    ]]

    let jsonInput = try JSONSerialization.data(withJSONObject: untransformed, options: [.prettyPrinted])

    let response: ObjectsResponse = simulatedResponseHandler(jsonInput)
    print(String(describing: response))
    
} catch let e {
    print(e.localizedDescription)
}



GDPR Compliance and Analytics

This post is more of a conceptual brainstorm about GDPR and still being able to acquire useful analytics data.

If I were to sum up the GDPR, I basically take it to mean that you cannot store/track data about your user without their consent, AND they have the right to be forgotten… i.e. they should be in control of their own data.  This basically means don’t track usage events using a user identifier (on iOS the venerable identifierForVendor property), as this would constitute gathering data about a person that can be inferred via this identifier.

If you can convince your user to opt-in, then great.  Business as usual just with some provisions to be able to delete any data they no longer want you to have/use.  But is there a way to still gather information about App usage without someone’s consent while still obfuscating who actually did the things you’re trying to track?  My understanding is that you can gather all the data you want so long as it’s not possible to trace that back to the user itself.

My thoughts are that if you as an analytics person are able to relax your requirements somewhat, I think it could be possible.  If you as an analytics person don’t need real-time updates on how your users have been using your app, but still want all the same insights, I think it’s still achievable…. if you have patience.

I think the solution is to keep all usage tracking on the device itself, then say once a month you upload it all in one go to some custom API endpoint that would parse all that data into usage stats, all without some user identifier.  The semantics of that are “There is a user – we don’t know who – who used the app in the following ways last month.”  Then you can see funnels.  Then you can see retention.  Then you can see all of those things over a time frame that is useful.

Currently, if we uploaded each and every event as they happened, you’d have no way to connect all those events to a user.  If you upload all of those in one go, with timestamps, you can still process all this data and get a picture of what a user does in a specific amount of time without caring about who it was, specifically.