We recently shipped a major new feature in the Bipsync Notes iOS app. We’re calling it Selective Sync, and it gives users full control over the data that is synced to their iOS devices.
Funds that use Bipsync to its full capability – drafting notes in its editors, forwarding in emails and clipping webpages of interest – soon amass sizeable databases of research. At many funds, hundreds of notes are created per day. This poses a challenge not only to us engineers as we seek to deliver that data to each device in a timely fashion, but also to our users as they try to keep abreast of the things that matter most to them.
Empowering users
We thought about this and realised that by allowing users to indicate their topics of interest, we could use that information to decide what should be sent to the device.
After considering several designs, we arrived at a solution akin to the way one would choose playlists to download in Spotify, or folders to sync in iOS’ Mail app: the user navigates to what they’re interested in, checks a box or taps a button, and everything within that context is downloaded and kept up-to-date going forward.
Selected contexts aside, we felt that at a minimum we should always sync the notes authored by the user. So someone using Selective Sync would aways have access to their own notes, plus any content tagged with the additional contexts they’d chosen for sync that had been written by others in the team. These tags could be tickers, investment ideas, a “read later” label, or any other custom tagging item created by the fund in Bipsync.
How it works
Before we get into the feature itself, let’s consider how Bipsync syncs data when Selective Sync is not enabled.
The data sync is two-way. First the device sends anything that the user has newly authored to the server, before the server sends the device any data the user should be able to access, but which is not yet available on the device.
For a long time we kept track of which items were on the device by a hash of their ID and revision number, and we stored this in the local database on the device. We included that hash each time we requested data from the server so it could determine what, if anything, needed to be sent. If you were reading this blog back in 2015 you may remember the following from an earlier post that described that process:
You might be thinking that as collection sizes grow we’ll begin to send a substantial amount of data with each [sync] request, which will increase the time they’ll take to execute. This is true. If and when this becomes an issue we’d investigate other solutions – perhaps by utilising timestamps… or by replaying an audit log constructed through event sourcing to arrive at eventual consistency. In the meantime… we elected to ship with the simplest approach that worked.
Well, three years later we did finally overhaul the way this worked, because potentially uploading megabytes of data each time we performed a sync clearly wasn’t ideal.
We introduced the concept of a session on each device. They’re similar to the way sessions work in web apps: when the user logs in to the iOS app we assign them a session ID when they first begin to sync. The server then keeps track of the data objects that were successfully transferred, storing a record of their ID and revision number alongside the corresponding session ID.
The sync process, “session-ified”
Consequently, when the iOS app asks the server for any documents that it doesn’t have, the server is able to filter out the documents it knows are redundant much more quickly, because instead of transferring this data over the network, we can achieve the same result with a simple database query.
These foundations gave us a fast, efficient sync mechanism on top of which Selective Sync could be built.
Advanced Syncing
At the heart of Selective Sync is a dictionary of criteria which dictates exactly what the user is interested in. This dictionary is sent along with each document request to the server, and the server incorporates it into the database queries it makes to find content the user should see.
The server-side logic that applies sync criteria.
Maintaining that dictionary is fairly simple from the application’s perspective. As the user navigates around the app, adding tags to their list of synced items, we update the stored criteria and trigger a new sync. Here’s a video of it in action:
One tough decision we had to make was what to do with notes created by other users whose tagging changes such that they’re no longer associated with a context that’s been selected for sync. At this point they no longer fall within the user’s sync criteria. Should we remove them from the device? Hide them from the user? Leave them alone?
Eventually we decided that once a note has been delivered to the device and potentially seen by a user, it should remain regardless of future Selective Sync changes. We felt that users would be confused or frustrated if they were unable to find notes they’d previously seen on the device, because it probably wouldn’t be obvious why those notes had disappeared.
It’s important that these notes continue to receive updates as they’re modified elsewhere – this is possible thanks to our session database on the server, because we can ensure that all previously delivered notes survive any filtering by our Selective Sync criteria.
Future Improvements
The feature works nicely as is, but there are a couple of improvements we have planned for the near future:
- We’d like to support a wider variety of tag types, to enable all the workflows we think our users will employ.
- An enhanced online search which is able to find notes that are not on the device, because they’ve been excluded by the sync criteria, but which are in theory accessible to the user.
- A way for users to opt in to removing “orphaned” notes that no longer meet the sync criteria that’s in place. This will aid users in keeping their local databases small, improving disk usage and app efficiency.
Turn it on!
Selective Sync is available now in Bipsync Notes iOS. For best results we recommend enabling it from a fresh install; this way the sync criteria is in effect from the start, keeping the downloaded content nice and trim. Any user who’d like to see faster sync times, or would like to find content more easily, should appreciate the feature.
We’d love to hear any feedback you may have so we can factor it in to future updates.
