Reducing The Risk Of Data Loss During Mobile Data Collection

The digitalization of data collection with mobile applications is becoming mainstream due to its many benefits, such as increased efficiency of the collection process and quality of the collected data.

However, a major concern by many users is the potential loss of valuable data during upload. This might be the result of poor network connectivity, unexpected interruption during data capture or the device battery running out.

Reducing the risk of data loss

Other concerns are that, like any software, the mobile app or overall data collection service may encounter errors. These errors are particularly disrupting when working remotely without a network connection. Therefore the reliability of the software and quick error resolution is of utmost importance.

We have addressed these challenges with several features in Poimapper. This blog is meant to alleviate concerns in transitioning to using mobile data capture. In addition, we give advise how you can yourself reduce the risk of losing data.

1. Minimizing data loss during upload caused by poor network connectivity

Since Poimapper Plus mobile application is built to support offline situations, network connectivity is not a challenge during data capture. Problems with connectivity occur mainly during data upload or download whereby data may be partially uploaded or downloaded when the network connection is interrupted.

Ideally, it should be possible for one to resume upload or download from the point of interruption, but in some cases, resume doesn’t always work perfectly especially when the network interruption takes longer periods of time.

Considering this, we changed how upload is implemented in Poimapper. Initially, we had a streaming approach whereby data was uploaded as a stream. If the connection was lost, the stream would continue from where it had stopped.

Sometimes, however, if the upload is disrupted for a long period of time, the data would be partially uploaded resulting in bad customer experience. Therefore we changed the approach to allow uploading one data point at a time.

This results in a clearer outcome from the user’s point of view. If the connection is lost certain data points are completely uploaded and some not at all; only one data point at most may result in partial upload i.e. just the data, but not the images.

Previously images would sometimes be uploaded, but the saving to the database would fail. For this reason, a further confirmation check was introduced, so that when the files are finally stored a confirmation message is sent to the app. In addition, if the app is closed while an upload is going on, the app will remember how far it got and upon restarting the app, it will automatically continue uploading.

As a final precaution, as network connectivity issues can be unpredictable, a continuous log of all upload activities noted by the server is saved for automatic issue reporting if upload failure for whatever reasons is noted.

2. Saving as a draft in case of unexpected closing of the app

The app may close unexpectedly for a number of reasons, for instance, the device battery runs out. Should there be an open form being edited at the time of closure, the open form is saved as a draft.

Drafts are also saved every time a page is changed or if the operating system closes the app. Particularly the Android operating system is prone to this, so typically when logging in, the app will ask whether to keep the saved draft version.

3. Continuous activity logging

To help handle unexpected issues, we have introduced continuous logging.

This ensures that all activities done in the app are logged. The log file can be uploaded for further investigation of the issue. If an issue is encountered, the user can go to the settings menu and simply tap “Stop Logging” button which automatically uploads the log file. Our technical team will then investigate and resolve the issue.

4. Regular updates and bug fixes

We are constantly updating Poimapper. If the update is critical, an update is enforced and a prompt to update the app is displayed upon login.

For optional updates, an update button appears in the main menu header and users can opt to update if they wish. This was implemented to serve as a notification to Poimapper app users that a new update is available.

5. Script error alert for advanced forms

Poimapper supports advanced form scripting. Advanced forms are essentially software programs, they have script statements for calculations, conditional logic, and data validation.

If there is an error in the script, to alert users of potential issues with the form itself, a triangle is shown in the header, showing where there are issues. This will help in fixing the issues quickly in the form builder. If script statements are not defined for some script fields, the app just ignores them.

6. Improved support and user guides

Previously on the support page of the app, we had an email and phone number available in case of required support.

This was limited as the support was available only during office hours in our time zone. To help answer a majority of support queries, we have recently introduced an offline user manual to the app, that is also available under the support button.

In 2019, we plan to increase our support to 12 hours during weekdays. Feedback is very important for any software provider, thus we strive to continuously improve our product and offer you the best possible support.

How to Reducing the risk of data loss

Most of the responsibility and work to avoid loss of data is with us, but there are some things the user can do to help.

1. Always use the latest version of the mobile application. (For critical updates, we enforce updates)

2. Split the form into pages, as the app saves the data point as draft every time you switch pages.

3. When making calculations and conditional statements, verify them in the form builder simulator before deploying them. Wrong statements should not cause loss of data, just that calculated results are wrong or missing, and that questions that should have been asked are not asked.

4. When deploying a form for data collection, change the status from test to deployed in the form builder, this prevents deleting fields from the form. This will avoid the loss of already collected data (Technically we do not delete the field, just hide it, but from an end-user point of view the data is not accessible anymore)

5. In the settings menu select Automatic upload. This will cause data to be uploaded as soon as each data point is saved. The downside of automatic upload is that the app may upload even when network connectivity is poor and possibly costly. Sometimes it may be better to wait until there is a good free Wifi connection.

6. Ensure your devices have sufficient capacity and quality. Running out of memory is maybe the most common way of crashing a mobile device. Even when the app is forced to close we try to mitigate issues by auto-saving. However, if the error is sudden without a controlled closing of the app, the app will not be able to store the data and data may be lost.

7. Use the latest version of Android OS in your device and avoid Android versions customized by the vendor. Older Android versions and Android versions modified by the device vendor may behave in ways that the app does not support and cause errors with loss of data.

8. Ensure that users are sufficiently trained to use the app so that data is not lost due to user mistakes. Training also avoids situations where a user may unnecessarily believe that data has been lost, causing avoidable disruptions to the whole data collection process.

As network connections improve, software gets better and memory capacities of mobile devices increase, we can be confident that data upload reliability will also continue to improve.

Reducing the risk of data loss

You can read more about like this, click here!

Follow us on Facebook, Twitter, LinkedIn and Youtube!