Press "Enter" to skip to content

Drive Android API Integration – Part 2

0

This is the second part of an article about Drive Android API integration. In the first part, we talked about some Drive API features such as how to configure the environment and sign in to the Drive API,  how to use File Picker Component, as well as how to query, create and upload files to Drive programmatically. Now, we want to take a step further and talk about two more complex Drive API features: how to resolve conflicts that may occur when syncing data and also how to monitor a file/folder for changes.

For the demo app, we updated the same repository as in the first part, by adding all features covered on this article.

Resolving conflicts when syncing data

Initial Thoughts

In the first part of this article we added cloud feature to our demo app. With that, we were able to manipulate files in the cloud by using Drive API. Now we want to bring more power to the app by allowing our users to change their data on multiple devices, and later synchronize it.

When dealing with data synchronization, especially two-way sync, there are lots of challenges we have to deal with, such as different timezones, simultaneous access to the same file, file manipulation when a device is offline, a file that was deleted by an user was changed locally by another, etc, etc, etc. One thing we should keep in mind is that there is no the best approach to solve all these issues, since one strategy that works fine for a solution may not work for another. For example, in a certain solution, when syncing data, in case of a conflict, we might want to keep the newer change and discard all the old ones. But in another solution, we might prefer to keep server changes over client modification, or maybe let the user to decide which one to keep.

But the point is, before starting coding, you might be better to understand all your needs and then design a solution that best suits all possible conflicts, no matter how you will do that.

Usually we will use a library and that library should provide all the necessary information to solve such conflicts. And this is where Drive Android API comes in. When we request a sync, depends on the strategy we use, in case of any conflict that Drive cannot resolve, it sends us an event on which we can access some versions of our data:

  • localBaseData -> local data before any local change (i.e.: before adding, removing or changing any item)
  • localModifiedData ->local modified data (i.e.: items at the moment we requested a sync process like any new or modified item. Deleted items are not here
  • serverData -> remote data (i.e. items currently on the server)

Note: the default behavior is to overwrite the server content with the local content when a conflict arises. We will see more on this later.

With these three sets of data, we are able to perform a merge in order to resolve all the conflicts locally.

But what should we do with all three sets of data? Well, there is no rule for that. The only thing that the API expects, is that after we work on that data we commit our resolution to the server.

Before we move on, it is important to note there are two different parts when using Drive API to sync and resolve conflicts:

  1. The Drive API manipulation -> This is what Drive API provides to us (e.g. methods, configuration, etc) and we need to understand it in order to configure the flow accordingly.
  2. Conflicts Resolution -> This is specific for the application that can apply any arbitrary logic to resolve conflicts. There is no Drive API involved on this part.

About the demo app

Let’s take a break and quickly give you some additional information related to the demo app that may help you to better understand this article.

We based this example in the conflict project that is part of the Suite Android Demo. But, instead of deal with a list of strings (groceries items) as in the Google example, we decided to make things more interesting and work with a complex object called task. A task is our model, there is, an object of type TaskModel that holds a name and a description. You can create as many tasks as you want, change them and also remove any task at any time. Then, by running the demo app on multiple devices, you can add/change/remove tasks and, when performing a sync you will see the conflict resolution in action.

When we sync local data to Drive, a text file “driveApiDemoAppTaskList” is created. It holds a list of tasks. Each line is a task and it is represented as follow:

This is how “driveApiDemoAppTaskList” file with some tasks looks like:

By using a “complex” object (i.e. our TaskModel) instead of only a string, it makes our demo app closer to a real app. You can see how to serialize a list of tasks in a stream of bytes before send it to Drive as well as deserialize an inputStream into a list of tasks so that we can add it to the UI. You can use TaskModel class as a base to your own model structure by changing it the way you want, and all you have to do is to adapt TaskConflictUtil::getTaskListFromInputStream() and TaskConflictUtil::formatStringFromTaskList() methods to serialize/deserialize your data accordingly.

Also, notice we do not persist any data locally, so if you create, modify or delete a task locally and leave ConflictResolutionFragment without sync to the server, your changes will be lost. All the tasks are held in the adapter.

The Drive API Manipulation

As already mentioned, this part is related to the Drive API and it is the common part that should be implemented whenever we use Drive API. We need to understand how the API works and what it provides in order to properly use their methods, classes and interfaces as well as understand which conflicts strategies it offers, how to receive conflicts events, etc.

In order to be in sync, the first method we must call is DriveClient::requestSync(). Usually we call it when user wants to refresh its list of data. In the demo app we call it right after the Drive API connection is established (which should happen after ConflictsResolutionFragment is created). This method requests a synchronization with the server to download any metadata changes that have occurred since the last sync with the server.

As stated in the docs, “In order to avoid excessive load on the device and server, sync requests are rate-limited. If the request has been rate-limited, the operation will fail with the DRIVE_RATE_LIMIT_EXCEEDED status. This indicates that a sync has already occurred recently so there is no need for another. After a sufficient backoff duration, the operation will succeed when re-attempted.”  I could not find any specific information about maximum request rate nor backoff duration. Docs  just says “The limit varies depending on the kind of requests”, but based on my tests, I was able to call it up to 5 times within 30 seconds. The recommendation is to use an exponential backoff strategy in order to prevent our app to call it too often.

Once this method returns, performing a query will return fresh results. We need then to create a Query object with the filters we want and run it. In our case, we are looking for a text file named “driveApiDemoAppTaskList” (as already mentioned).

If our query returns an empty MetadataBuffer object, it means Drive could not find the file we are looking for (i.e. “driveApiDemoAppTaskList” text file). In this case, we need to create it. Even when MetadataBuffer is not empty, we still need to check whether the file is trashed or not. If so, also create a new one, since we do not want to work with a file that is trashed. The snippet below shows how to request Drive to create a file.

Note that, as we mentioned in the first part of this article, even when DriveResourceClient::createFile() fires addOnSuccessListener callback, it only means the file was created in the Drive client. We should create an ExecutionOptions object and call its setNotifyOnCompletion method with the value true and then pass the ExecutionOptions object to the createFile method. Later, when the file is created on the server, Drive API fires a CompletionEvent.STATUS_SUCCESS event. We are not relying on this, but you should consider it on your app.

Now, we can load the file content by calling DriveResourceClient::openFile() method and then deserialize the content into a list of TaskModel.

We should now have a fresh list of tasks (or an empty list in case of Drive just created a new “driveApiDemoAppTaskList” file). Note in the snippet above we stored the list content in the taskListContents variable, which is an object of type DriveContents. We will use it later when requesting a sync process.

Now, you can play around demo app by adding, removing or changing tasks the way you want.

When we want to perform a sync process, the first step is to call DriveResourceClient::reopenContentsForWrite() by passing the DriveContents we saved when first loaded our task list. It will close the current DriveContents and returns a new DriveContents opened in MODE_WRITE_ONLY. The returned contents are usable for conflict detection.

Then, since we want to override default sync strategy (which overrides server content by using the local changes) to resolve locally all the conflicts, we have to create an ExecutionOptions instance and define the CONFLICT_STRATEGY_KEEP_REMOTE conflict strategy to be used. As stated on the docs, this strategy “…keeps the remote version of the file instead of overwriting it with the locally committed changes when a conflict is detected.”. Also, in the docs we find that “…in case of conflict the remote state of the file will be preserved but through the CompletionEvent, the client will be able to handle the conflict, access the locally committed changes that conflicted (and couldn’t be persisted on the server), and perform the desired conflict resolution with all the data.”

Great, no? That is exactly what we want to do.

After that, just call DriveResourceClient::commitContents() method by informing the newer DriveContents and the strategy just configured. This will commit local data to the server. Then, if a conflict is detected on the server, a CompletionEvent.STATUS_CONFLICT event will be fired. In case of no conflict, a CompletionEvent.STATUS_SUCCESS event is received.

But before we go any further, let’s see what docs says about file conflicts: “A file conflict happens when the written contents are not applied on top of the file revision that Drive originally provided when the contents were read by the application. A conflict could happen when an application reads contents at revision X, then writes revision X+1 and, by the time X+1 is committed or updated on the server, the file version is not X anymore (because, e.g. another app or a remote change already modified the file to revision X’).”. A more simple definition of conflicts can be found here: “A conflict occurs when the local version of the base file contents do not match the current file contents on the server. In this case, a simple merge is not possible and a Conflict status is returned. ”

Now, in order to receive a CompletionEvent, we must create a service extending the DriveEventService and override its onCompletion method, where we can handle the CompletionEvent, as well as configure this service in the manifest file.

We can noticed in the snippet below that when we receive a CompletionEvent.STATUS_CONFLICT, we delegate the resolution to the ConflictResolver class. This is just to make code cleaner.

Inside ConflictResolver, the first thing we have to do is to silent signs in to the Google Drive.

Next, extract both local and modified data streams from the CompletionEvent. This is done by calling CompletionEvent::getBaseContentsInputStream() and CompletionEvent::getModifiedContentsInputStream() methods respectively. Then we have to deserialize those streams into lists of TaskModel. Note both methods may only be called once per CompletionEvent instance. Another point is that we can only use the returned InputStream from both methods until dismiss() or snooze() is called.

The next step is to call DriveResourceClient::openFile() method in order to open the current server task list version.

Now, we have three lists of tasks. As mentioned at the beginning of this article, they are:

  • localBaseTaskList -> local data before any local change (i.e.: before adding, removing or changing any task)
  • localModifieTaskList ->local modified data (i.e.: tasks at the moment we requested a sync process like any new or modified task. Deleted tasks are not here
  • serverTaskList -> remote data (i.e. tasks currently on the server)

We then delegate the conflict resolution to the class TaskConflictUtil.resolveConflict(…). Basically it handles all the conflicts and returns another list with the resolution. We will talk about it in the next section.

After TaskConflictUtil.resolveConflict() returns, we have a new list containing all the tasks after the conflict resolution. The last thing to do is to call DriveResourceClient::commitContents() in order to commit the resolution list to Drive. When calling it, there is a small possibility to have another conflict. As pointed out here, “It is not likely that resolving a conflict will result in another conflict, but it can happen if the file changed again while this conflict was resolved. Since we already implemented conflict resolution and we never want to miss user data, we commit here with execution options in conflict-aware mode (otherwise it would overwrite server content).”

This means that we will create an  ExecutionOptions object and call its  setNotifyOnCompletion method with the value true and then pass the ExecutionOptions object to the commitContents() method. In case of a second conflict, all this process will repeat.

See the entire resolve() method:

Conflicts Resolution

In the previous sections we explained the steps to request a sync process and all we have to do to handle a conflict event and commit the resolution back to the server. This was done by extensively using Drive Android API. But what Drive API does not define is how we can resolve conflicts. The only thing it expects is that after we receive a CompletionEvent.STATUS_CONFLICT event, we commit our resolution back, no matter how the resolution looks like.

We will demonstrate here one hypothetical strategy we used on the demo app. As we mentioned at the beginning, there is no the best strategy to solve conflicts, since one strategy may work for a certain solution but may not work for another. That being said, let’s see the steps we defined for our strategy:

  1. Tasks modified on the server will override local tasks. If the same task (i.e. a task with the same name) was modified locally, local changes will be lost.
  2. Any task removed on the server will also be removed locally. If the same task was modified locally, local changes will be lost.
  3. Task created on the server will be created locally. In case of a task with the same name was created on both server and local, the new local task will be lost.
  4. New local task will be synchronized to the server (unless a task with the same name was created on the server. See item #3.
  5. Tasks deleted locally will be also deleted on the server.
  6. Local tasks modification will override server content, unless the same task was removed or modified on the server. See item #1.

To make things clear we created a separate class (i.e. TaskConflictUtils.kt) to implement our strategy. If later we decide to change our conflict resolution logic, we can just replace TaskConflictUtils class for another one.

We are not getting into implementation details since the idea here is just to present you a simple strategy and show to code it. We tried to add as much comments as possible in the code in order to describe each step used in our strategy. This seems to be enough for you to understand the implementation.

Unit Tests

In order to test the conflict resolution algorithm we could make local changes (i.e. add, modify or remove local tasks) and also open “driveApiDemoAppTaskList” file on the Web and add, modify or remove any server task. Then, after saving the file, request a sync process on the device.

But why don’t we use unit tests instead? This would be a much more robust solution… and that’s what we did. These are some scenarios we implemented on our unit tests:

  • Tasks removed, changed and created locally
  • Tasks removed, changed and created on the server
  • The same task modified on both client and server
  • A task with the same name created on both client and server

. For details, check ConflictsResolutionUnitTest class into the demo app. If you change the strategy, you can simply add new tests to ensure all your strategy’s steps are working as expected.

Monitoring a file/folder

Now, let’s talk about file/folder monitoring feature provided by the Drive API.

When I started studying Drive Android API to implement file monitoring, I never thought this would cause me terrible headaches and endless nightmares. It’s serious! 

Drive API documentation states that “You can use change listeners to receive notifications whenever a specified file or folder has changes to its contents or metadata. A change listener implements the OnChangeListener interface for the ChangeEvent and receives a direct callback from the Drive service to a currently connected client application.” By reading this, I expected the obvious: start monitoring a file (or a folder) and when it changes outside my application (e.g. from an instance of the app running in another device, on the web, etc), I would receive a notification event. Either I missed something or it does not work as I expected. But so far, it seems the second option is the truth.

Although I never could make it work, I decided to add this example on this article (and in the demo app) in order to group together some information I found online (mostly from SO) from other developers that also ran into this issue. Also, I really expect someone one day corrects me and shows how to properly use Drive API to monitor a file/folder. That would be much appreciated.

Anyway, according to the docs, the process to start listening is pretty simple. Once we get the FileId from the file we want to monitor, all we need to do is to call DriveResourceClient::addChangeListener() by passing that ID and implement the callbacks. See the snippet below how to do it.

But when doing it, listener only receives events when changes are done from inside our app.

Let’s see a scenario that does not work:

  • We start monitoring a Drive file named “myFile”.
  • Then, we go to the web, open “myFile” file, change it and save it.
  • Nothing happen! Actually there is an event that always is fired once, but it seems to be related to something local.

Now, a scenario that works:

  • We start monitoring a Drive file named “myFile”
  • We keep using our app, and at a certain point, we change “myFile” file from inside our app.
  • When our change hits the server we receive a notification event.

But, what I do not understand is why would I want to receive callback events in my app when I change the file from inside it? For me it does not make sense, since when changing a file, if we use an ExecutionOptions properly, we receive a CompletionEvent when the change hits the server.

During my researches, I found many frustrated developers out there, but never anyone that could save the day. I found one or two comments saying events should come after some time, but it never worked for me, even after 24+ hours waiting for the events.

Let’s see some issues I found on SO about Drive API change events.

In this SO question, on the last response, OP suggests to “…force the event by making a requestSync.”. But this would make Change Events useless, since after call requestSync, performing a query will return fresh results. Also, by using requestSync() method, we may end up reaching the rate limit and the operation may fail with the DRIVE_RATE_LIMIT_EXCEEDED status.

In this SO question from Apr/15, OP clearly comments it does not work at all, and anybody never added any opposite comment nor any suggestion since then.

In this SO question (from Apr/14), OP also tried to use the Google sample with no success. There is a comment also suggesting to use requestSync().

This SO question from Jan/15, although it is not directly related to change events, in one of the answers, there is an OP complaining about the change events functionality. He also refers to some others questions that might worth to take a look.

Finally in this documentation, they do not mention whether it should work only for local changes or also for external changes, but based on all the SO links above (and you can also find a few more if you Google it), I assumed Change Events work only locally.

I would really appreciate if someone could show the right way to make it work. If you feel I am wrong, please, leave a comment below and we can start a discussion. It that happens, I will update this section accordingly, and of course, give all the credits to whoever show how to work with it.

Conclusion

Well, that is the end of the second article about Drive Android API. I hope you can have now a better idea on how to deal with conflicts when syncing your data with a Drive backend. As you could see, Drive Android API has its tricks and limitations, but it also works really well in many aspects. The best you can do before using it, is to dive into the docs, read as much code as you can and then, decide whether it works for you or not.

If you have any comments about this article, please, let a comment below.