GDPR Compliance and Analytics

This post is more of a conceptual brainstorm about GDPR and still being able to acquire useful analytics data.

If I were to sum up the GDPR, I basically take it to mean that you cannot store/track data about your user without their consent, AND they have the right to be forgotten… i.e. they should be in control of their own data.  This basically means don’t track usage events using a user identifier (on iOS the venerable identifierForVendor property), as this would constitute gathering data about a person that can be inferred via this identifier.

If you can convince your user to opt-in, then great.  Business as usual just with some provisions to be able to delete any data they no longer want you to have/use.  But is there a way to still gather information about App usage without someone’s consent while still obfuscating who actually did the things you’re trying to track?  My understanding is that you can gather all the data you want so long as it’s not possible to trace that back to the user itself.

My thoughts are that if you as an analytics person are able to relax your requirements somewhat, I think it could be possible.  If you as an analytics person don’t need real-time updates on how your users have been using your app, but still want all the same insights, I think it’s still achievable…. if you have patience.

I think the solution is to keep all usage tracking on the device itself, then say once a month you upload it all in one go to some custom API endpoint that would parse all that data into usage stats, all without some user identifier.  The semantics of that are “There is a user – we don’t know who – who used the app in the following ways last month.”  Then you can see funnels.  Then you can see retention.  Then you can see all of those things over a time frame that is useful.

Currently, if we uploaded each and every event as they happened, you’d have no way to connect all those events to a user.  If you upload all of those in one go, with timestamps, you can still process all this data and get a picture of what a user does in a specific amount of time without caring about who it was, specifically.

Level Up: Test-Driven Development

Until very recently, I’m used to being that hired gun who parachutes in, kills all the work tickets, asks the important questions about the product, makes an app work as desired on a tight deadline to a reasonable level of quality given the constraints, then is out again, all for a reasonably good sum of money. It’s exhausting because these companies often have no established procedures.  So in addition to being Lone Wolf: Kicker of Asses, I’m training junior developers so they can assist me without slowing me down too much, I’m creating an Adhoc QA Department (what is severity, what is reproducibility, how to write bug reports, what is a end-user test suite so you know how to regression test, and why you should use a ticketing system instead of just popping by my desk to effectively pull the plug on whatever 50 things I had in my head), I’m having to interpret incomplete designs, pull the assets out of the Design tools (Zeplin, Sketch, sometimes Photoshop) because many designers don’t know what exactly you need anyway, poke holes and/or fill in the gaps with the UX, and of course manage upwards. Oh yeah, and also do “Agile Waterfall” development, which is borne out of companies who only really do Waterfall but want to be “hip” to the new trends and demand we do Agile (with no scrum master or really anyone who knows how to lead that effectively). So then your time is further taken up with meetings and pushing work tickets around that actually don’t really encapsulate the work you actually need to do, but managers need you to do that so they can generate reports with some data that is meant to impress other people who really have no idea what’s going on anyway, and actually just increased levels of trust in your hires and “we’re good” would be equally effective/ineffective. (Ah, perhaps they don’t know how to make the right hires or can’t get them if they did.) In all of that, I have to “get ‘er done” because the deadline doesn’t change and surprise! All of your dependencies (Design, Backend API) also have the same deadline.

Yikes. A day in the life.  That above would be the worst day in the life.  It’s rarely all of those things.

So I’m grateful for my current freelance contract. It’s the first contract really in years where I felt I’m working in a proper software development company. The management overhead seems low, but the organization is not impacted. They have a process here that works. (I think it’s just because I hit the jackpot and they just placed a priority on the members of the team; personable yet very competent. Ultimately it’s a team who cares about what they do, and making sure there’s a nice team vibe. It also helps that they have corporate backing and therefore have a generous timeframe and budget it would seem.)

“Take the time you need to do a good job.” This is very much the culture here. For one of the first times in my career, I’ve been really exposed  to an office environment where you’re given time to think, and time to write a lot of tests while you develop. You can ask for specifications and those exist and are fixed. There are 2 other iOS devs here to bounce ideas off of, and of course to do code reviews with. It is so extremely satisfying when you get to refactor your original approach into ever more concise code that is more robust and less error prone. Time where you can write the API docs and the Unit Tests to basically freeze the specification. Normally there just isn’t enough time, given all the other tasks that Lone Wolf has, AND the product design always seems to be a moving target.

In short, it feels like I finally have the time and space to level up. Unit Tests are especially awesome when you work in teams and I’m glad for the opportunity to work in this environment for a while so I can establish some good habits and really reach a new plateau in my journey as a software developer.

No, I didn’t die

Looking at this blog, it’s been over a year since I wrote anything. I’ll just list my excuses:

  • I was overworked on some agency projects and there simply wasn’t enough time to write posts
  • It’s generally a grey area to write about problems solved for clients who make me sign an NDA
  • I could have made the solution to the problem look generic, thereby obfuscating my client’s identity and thus my attachment to the project, but didn’t.
  • And now THE main reason….

I spent 8 months of 2018 taking a sabbatical. I like to call it a sabbatical because this is how you don’t lose respect from people when compared to saying “I’m unemployed and am enjoying watching my beard grow”. But really it was more like a “I’m fed up with insane projects that are under-staffed, with poor management and deadlines that don’t change, so ultimately the bottom-feeding developer who is doing the job of a small team on his own is starting to wonder if it’s time to pack it in and open a bakery, or do some other kind of artisanal work that involves repetitive, meditative movements and a certain degree of craftsmanship. <blows sawdust off his handmade hope chest>”

Sabbatical has a nicer ring to it and doesn’t require a backstory.  “Ah, he must be a well-to-do kind of man.”

This sabbatical was not about going to some coding retreat so to level up my skills as a developer. It was more like a sanity check. The reason I like coding and software engineering is that I find it to be a highly creative medium. For me, being creative is what gives life some meaning and makes it fun. The process of having an idea, developing it in your mind, putting it down on paper, refining and iterating on that, solving the unknowns, then finally doing the implementation / building. Then it’s done, you pat yourself on the back, revel in your accomplishments, then say “next!!” I love it. So the sanity check was a) to get a little distance from a profession that was starting to become not fun for me anymore, and b) do something completely different but also creative.

So I did that. In 2018, I decided I wanted to build a camper out of some kind of vehicle, then drive around North America until I experienced some sort of epiphany. I gave myself a year. I used 8 months. I built a camper. I had an adventure. I drove 24,000km in 4.5 months and saw so many beautiful places and met tons of very lovely people. I had some profound realizations about my life that were restorative and empowering for me going forward.

In short it was the most useful year I’ve had in over a decade. If you have the opportunity to take a long period of time off and leave your normal surroundings behind, I highly recommend it.  You’ll return knowing what changes you need to make in life.

True, I have fallen a bit behind with the latest and greatest in iOS / Swift dev, but since I’ve been back working (since November 2018), I’ve been fortunate to work with great developers and I’ve actually leveled up in this short time. I can say I’ve become way more of a test-driven developer, and I’ve also got a few new tricks up my sleeve that you generally get to learn when you work with other developers that work on your platform; not something I’m all that used to as I tend to be either the only iOS dev on a project, or they quickly say “oh but you know the most”.

At any rate, I’m back and I’m still happy to be a developer, and now that my approaches have changed somewhat, I have new areas to grow. So I’m not stagnating like I felt I was, and I’ve been fortunate to get a few freelance contracts that have not turned about to be a complete and total clusterf*ck, which has often been the case in Berlin, Germany. I oftentimes have the feeling that when it comes to Software Development practices, Berlin is the wild west; there are no rules, OR Berlin is teenage love; just fumble around in the dark for a while and hope something good happens.

iCloud Drive – What a disaster!

So, I’ve been working on a new version of my app Songbook Simple and I want to improve some of the file sync code.  Up until now I’ve been supporting Dropbox, with the original idea that a user can edit and modify all his/her songs from the convenience of their desktop, and it would just sync on the device.  This *has* worked, and *still* works, though it always seemed like a quick and dirty solution to me.

Enter Apple TV.  I thought it would be fun to make Songbook Simple Apple Tv compatible, but in order to do that, one basically has to support iCloud for syncing data.  As the Apple TV won’t be an editor, but just a viewer, we need to get data from somewhere, and I’m not so sure Apple TV is entirely Dropbox friendly yet.

So, I’m trying to implement iCloud support in Songbook.  I followed the excellent 4 part tutorial series written by Ray Wenderlich himself.   It all seemed to work brilliantly.  Files were syncing across iOS devices.

But wait.  Now what about using my Mac to edit files?  Well, turns out this is where iCloud Drive is a complete and utter disaster.  On icloud.com, there is no folder associated with Songbook.  On iOS, the iCloud Drive app DOES show my app folder and its contents.  On OSX, in the iCloud Drive folder, you don’t see anything associated with Songbook.  So my user workflow, namely, editing songs on your Desktop, is broken.

I’m all for “Automagic” for the sake of better User Experience, but if it’s “Autobroken”, then get rid of your magic because you’re killing us.

In order to restore the ability to edit the contents of my SongbookSimple folder, I had to open a terminal window, and do the following:

cd ~/Desktop
ln -s ~/Library/Mobile\ Documents/iCloud~com~softwarebarn~SongbookSimple/Documents/

Then, from Finder, navigate to the Desktop, then either move that Alias that just got made (called Documents) to somewhere better, or simply drag it into my Finder’s sidebar.

Great feature, Apple!  All these years to mimic Dropbox and it’s still a failure.  Yes, I followed ALL the instructions, and spent 3 hours this morning making sure I didn’t miss any.

An Open Letter to (mobile) Server Devs

On basically every project I’ve ever worked on that involves a server, I seem to notice the same things that happen over and over and could be preventable with proper planning.  When mobile devs discover server bugs, it often blocks our work and halts development (not to mention the costs associated with that, which are NOT trivial).

So I thought I’d just list the things that I wish server developers would think about when they start building a system, and rant a bit about the mentality we’d hope for in any server dev.

 

Design for the failure case, not the ideal case.

Servers fail all the time.  You should consider what will first go wrong before you focus on what will go right.  If you set up your architecture correctly at the beginning, this becomes easier.  Yes, your project has more boilerplate at the beginning, and you can’t just dive into the things that make programming fun, but remember you are a server; you serve others.  The quality of your work is measured by how close to perfect your work is.  Nobody knows when things are working correctly, but they DO know when they’re not.  It’s a tough life as a server dev because you get very little love.  But you chose that, so what should I say?

Consider at the very start how you’re going to deliver errors to the clients.

 

Think of responses like a textual representation of an object.

I come from Object-Oriented Programming.  Think of responses as objects, and you need an application-wide “baseclass”.

{
    "data" : {
        /* YOUR ACTUAL RESPONSE PAYLOAD HERE */
    },
    "errors" : [
        {
            "code" : 401,
            "domain" : "HTTP",
            "description" : "Unauthorized",
            "recovery_suggestion" : "Your access token may have expired.  Try logging in again."
        },
        {
            "code" : 1001,
            "domain" : "Our Server Form Validation",
            "description" : "Email address invalid",
            "recovery_suggestion" : "Make sure your email is formatted correctly."
        }
    ],
    "message" : "Any arbitary message that is relevant to the response itself.  This response isn't supposed to make sense for any specific context.  They're example fields.",
    "response_time_in_ms" : 1234,
    "status_code" : 200
}

 

It’s not 1982

Mobile Clients prefer textual representations in the response.  Your responses can be verbose and at times redundant.  If you have to echo the http response status code in the payload, why not?  All the pertinent information is in the same place.  To an extent, of course.  It just makes your API that much more easy to work with.

 

The Importance of a Documented API

Your server may be down.  When your server is down, your mobile developer is often stuck when he doesn’t need to be.  Most of the time, mobile devs need any data, so long as it can be used to populate the contents of the UI.  The Server API is a contract.  It specifies what the server team will deliver, and what the mobile developer can expect.  It ensures that two mostly separate teams can work happily in parallel.

If you provide a documented API before it’s even implemented, the mobile dev can get his job done.

It just needs to be simple.  Send these parameters with this method to this endpoint, and you can expect a response of:  (most verbose response here that shows all possible fields that might be defined.  Mark in comments beside that value if NULL can be expected.)

Then you highlight the error scenarios.  Possible error codes and why.

 

Interactive API via Postman

Postman is a very valuable tool.  It’s generally how you resolve disputes with the server devs to the effect of “Server’s broken!  No it’s not!”  You can make your requests outside of the app and inspect how the server is working.  It also makes your bug reports more believable to a server dev, where he can’t really say “stupid user error”.  It’s of course up to the server dev to keep environment variables up-to-date and provide some documentation on how to use Postman in the way the server dev expects you to.

 

This is sort of a working draft, but on almost every project, I wish there would be devs that understand how their work makes its way into the end product.  And if you ensure that your role is one of service, and your customers are mobile developers, keeping a service-oriented mentality, like the Maître d’Hotel, where you derive your happiness when your customers are happy, this surely is the way for you to become a rockstar.

Excellent Definition of a Senior Software Engineer

Found this on Quora and it certainly justified my job title! 🙂

What does this mean for a software engineer: “You are not at a senior level”?

QUOTE:

The Perils of Hackers in your Team

I’ve currently joined a project as the only iOS Developer.  I’ve inherited a codebase that’s passed through the hands of 4-5 guys over the course of 3 years.  3 years is already at least a yellow flag in terms of iOS Development.  In my view, the only reason the same project should be allowed to live for 3 years is if you have older devices to support, or a massive userbase and a fat client architecture (making loss of data an issue).  For startups, you should always consider re-writing your Minimum Viable Product, as so many lessons are learned, so many pivots are made, so many corners are cut, to get you to where your codebase is today.  Cruft (look it up) is your biggest problem.

Poorly structured code will hurt your bottom line.  It becomes increasingly expensive to work with, and adding new features will not be easy, and the chances of introducing bugs will be high.  As much as one can talk about Test Driven Development (TDD), I’ve rarely heard of teams (in the startup world at least) that write tests, and if they do, they write them *after* the code has been written.  (This has proven to be the most expensive way to implement tests).  So, fine, the world is not perfect.  In these cases I would hope that if you are a developer reading this, you will at least put comments in your code and also NSAssert statements (the poor man’s sanity check / testing shortcut).

Let me now define what I mean by Hacker in the context of this post.  A Hacker is someone who may look good to any non-developer.  He gets the job done.  The code “works”.  He appears passionate as he types away in a frenzy, trying to make things ‘just right’.   The thing about a Hacker is that he may be self-taught and completely lack the principles of good software design.  He writes spaghetti code (that works).  He may say things like “I don’t need to put comments, the code should speak for itself.”  (If that’s true it is saying “Help me!” or “I’m taking hostages!” or “Your time now belongs to me!”).  But the worst thing about a Hacker on your team is that he will write code that is often a testament to every anti-pattern you have ever known, and to have to work with that person’s code is like being sabotaged before you can even begin.  Because to work with that code, you essentially have to rip out everything he’s done to make the codebase useable again.  So before you can begin to be awesome,  you have to weather the storm of management (who can’t possibly know your struggles) saying “I’m noticing progress is slow”, or “but it’s working now, what do you mean you need to rewrite it…?”

In short, a Hacker in charge of your codebase is basically signing a death warrant for your current codebase, or paying for a huge financial hangover after he’s gone.  A Hacker on your team can have some benefit (thinking outside the box, enthusiasm), but be wise and make sure he’s not in charge of architecture, and make sure you do code reviews.