GSON JSON for Mobile App Storage

You will be surprised how GSON can propel your mobile app productivity especially release 1 candidates. In this approach you will represent the persistent state of your apps as JSON. You will use the "Beautiful" tool called GSON from google to convert your application objects to JSON and persist the resulting JSON string either in a shared preference or on internal storage as files.

This approach is really suitable for writing apps really quick. It is not unreasonable to release apps with in 1 to 2 weeks once you get seasoned at this. This would probably take a month or two otherwise. You may get a 2 to 3 times advantage on simpler apps.

Even when you consider complex apps you may get significant advantage while you are prototyping the idea and test it out on a limited release.

This semi article which will eventually be part of our upcoming Expert Android book documents the necessary elements and present the code snippets and answers the questions I ran into as we researched this topic. 


What does Android recommend for data storage options? The Official line!
The JSON solution
What is GSON?
  What is GSON's homepage?
  Is there a user guide for GSON?
  Can I save nested objects?
  Can I save nested collections of objects?
  How are characters in strings escaped?
How do you add external jar files to Android APK file?
What are shared preferences?
Difference between getSharedPreferences and getPreferences?
How to get a context independent of an activity?
Where are shared preference files stored?
How do you save JSON to a shared preference file?
What does the saved preference file look like?
How do you read JSON from shared preferences back as Java Objects?
How does escape characters work in android preferences?
What is internal storage?
How do you save to internal storage?
How do you restore objects from internal storage?
Should you use external storage like the SD card?
What are the useful references while working with this approach?
How do you code such a way that you can migrate in the future to SqlLite?
What Next?

Tell me about Data Storage Options

I have documented at the link above the choices of various data storage options. You can also use the references section at the top right to see the summary of 5 different options that Android suggests.

Summary of research is to use either shared preferences or internal files as the storage mechanism for your objects after converting them to JSON using GSON. These storage options have two advantages. First they are private. Second they are removed when the application is removed. The disadvantage is if the files become too large in terms of 10s of megabytes or have images or videos etc.

You use java objects to store them directly in persistent storage after converting them to JSON strings. The JSON strings can then be stored in internal files or in the shared preferences. We will use google's GSON as the tool to go between java objects and strings. It works really well!!

GSON is a java library (a single jar file) that you can use to convert java objects to JSON and back. You can see links to the home page, user guide, and the API on the right hand side of this document. You can also see a detailed research notes on GSON in those links.

Yes. You can have objects inside objects in a nested structure. If you mirror your java objects with a focus on their storage structure you can accomplish quite a lot. So you want these objects to represent primarily data and not full blown behavior.

Yes. as long as your member collections use java generics. When you use java generics you specify the type of the collections. This is what helps GSON to instantiate the right objects. See the user guide for more features and limitations.

Good question. GSON will escape the quote characters and any other character that is special for JSON. GSON also by default escapes the html and XML characters. In our research we found this behavior very satisfying. 


Go to the home page of GSON indicated at the right. 
download the gson jar file. 

Create a sub directory called "lib" in your eclipse project
  do this under the root of your project
  parallel to src.
Copy the gson jar to that lib directory

Go go project properties
Go to build properties
Add the gson jar as a dependency

Go to the order/export tab
Choose the gson jar to be exported as well

that's it. You will have the jar file in your APK when you build.

You may be able to link the external gson jar with out copying to your apk project. You will compile and link successfully. but when you run it you will have a run time class not found exception. So follow the steps indicated above as it did work for me that way. 


public class ChildObject {
   public String name;
   public int age;
   public boolean likesVeggies = false;
   
   public ChildObject(String inName, int inAge)
   {
      name = inName;
      age = inAge;
   }
}

public class MainObject 
{
   public int intValue = 5;
   public String strinValue = "st<ri>\"ng\"Value<node1>test</node2>";
   public String[] stringArray;
   public ArrayList<ChildObject> childList = new ArrayList<ChildObject>();
...
}


public class MainObject 
{
   public int intValue = 5;
   public String strinValue = "st<ri>\"ng\"Value<node1>test</node2>";
   public String[] stringArray;
   public ArrayList<ChildObject> childList = new ArrayList<ChildObject>();
   
   public void addChild(ChildObject co)
   {
      childList.add(co);
   }
   
   public void populateStringArray()
   {
      stringArray = new String[2];
      stringArray[0] = "first";
      stringArray[1] = "second";
   }
   public static MainObject createTestMainObject()
   {
      MainObject mo = new MainObject();
      mo.populateStringArray();
      mo.addChild(new ChildObject("Eve",30));
      mo.addChild(new ChildObject("Adam",28));
      return mo;
   }
   public static String checkTestMainObject(MainObject mo)
   {
      MainObject moCopy = createTestMainObject();
      if (!(mo.strinValue.equals(moCopy.strinValue)))
      {
         return "String values don't match:" + mo.strinValue;
      }
      if (mo.childList.size() != moCopy.childList.size())
      {
         return "array list size doesn't match";
      }
      //get first child
      ChildObject firstChild = mo.childList.get(0);
      ChildObject firstChildCopy = moCopy.childList.get(0);
      if (!firstChild.name.equals(firstChildCopy.name))
      {
         return "first child name doesnt match";
      }
      return "everything matches";
   }
   
}


public void testJSON()
{
   MainObject mo = MainObject.createTestMainObject();
   Gson gson = new Gson();

   //Convert to string
   String jsonString = gson.toJson(mo);
 
   //Convert it back to object
   MainObject mo1 = gson.fromJson(jsonString, MainObject.class);
}

Incredibly simple.

how to store and retrieve this string in a persistent place. we have two options. shared preferences and internal file storage. I will cover shared preferences first.

What are shared preferences?

use the link above to read more about shared preferences. But briefly a shared preference is an XML file that holds key value pairs in a persistent manner. The XML file is an implementation detail. Android may chose a different representation in the future should it chose so.

You can have as many shared preferences as you like. Each shared preference can have as many key/value pairs as you wish.

these xml files are private to your application. They are removed when your application is uninstalled.

you will need a context object to call the method getSharedPreferences(). However when you are in an activity you can just call getPreferences(). The later method merely calls the former with the activity name as the name of the xml file.

For our case we will use the getSharedPreferences() as the state of the entire application is not tied to a single activity. 


public class MyApplication extends Application
{
   public final static String tag="MyApplication";
   public static Context s_applicationContext = null;

   @Override
   public void onConfigurationChanged(Configuration newConfig) {
      super.onConfigurationChanged(newConfig);
      Log.d(tag,"configuration changed");
   }

   @Override
   public void onCreate() {
      super.onCreate();
      s_applicationContext = getApplicationContext();
      Log.d(tag,"oncreate");
   }

   @Override
   public void onLowMemory() {
      super.onLowMemory();
      Log.d(tag,"onLowMemory");
   }

   @Override
   public void onTerminate() {
      super.onTerminate();
      Log.d(tag,"onTerminate");
   }

}

you basically create your own application instance and store a static pointer to the context. 


public void testEscapeCharactersInPreferences()
   {
      String testString = "<node1>blabhhh</ndoe1>";
      SharedPreferences sp = getSharedPreferences();
      SharedPreferences.Editor spe = sp.edit();
      spe.putString("test", testString);
      spe.commit();
      String savedString = sp.getString("test", null);
      if (savedString == null)
      {
         mReportTo.reportBack(tag,"no saved string");
         return;
      }
      //savedstring exists
      mReportTo.reportBack(tag,savedString);
      if (testString.equals(savedString))
      {
         mReportTo.reportBack(tag,"Saved the string properly. Match");
         return;
      }
      //they dont match
      mReportTo.reportBack(tag,"They don't match");
      return;
   }
   private SharedPreferences getSharedPreferences()
   {
      SharedPreferences sp 
      = MyApplication.s_applicationContext.getSharedPreferences("myprefs", Context.MODE_PRIVATE);
      return sp;
   }

use of MyApplication to get the shared preferences.

and the way we call getSharedPreferences(). Notice the mode bit. 


int mode;
//One of 
MODE_PRIVATE (value of 0 and default)
MODE_WORLD_READABLE
MODE_WORLD_WRITEABLE
MODE_MULTI_PROCESS


SharedPreferences.Editor spe = esp.edit();
spe.put...();
spe.commit();

If this preferences file is shared among processes, a commit will take it to the persistent state and make it visible to the other clients. A single file will return a shared preferences object that is one or at least refer to the same file when in a different process. 


/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PREFS_NAME.xml
or the default is at
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml

In eclipse ADT use the file manager tool to see the files on your emulator or the device and pull the file to your local drive and view it. 


<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
<string name="test">&lt;node1&gt;blabhhh&lt;/ndoe1&gt;</string>
</map>


public void storeJSON()
{
    MainObject mo = MainObject.createTestMainObject();
    //
    Gson gson = new Gson();
    String jsonString = gson.toJson(mo);
    this.mReportTo.reportBack(tag, jsonString);
    MainObject mo1 = gson.fromJson(jsonString, MainObject.class);
    this.mReportTo.reportBack(tag, jsonString);
    
    SharedPreferences sp = getSharedPreferences();
    SharedPreferences.Editor spe = sp.edit();
    spe.putString("json", jsonString);
    spe.commit();
}

public void retrieveJSON()
{
    SharedPreferences sp = getSharedPreferences();
    String jsonString = sp.getString("json", null);
    if (jsonString == null)
    {
        mReportTo.reportBack(tag,"Not able to read the preference");
        return;
    }
    Gson gson = new Gson();
    MainObject mo = gson.fromJson(jsonString, MainObject.class);
    mReportTo.reportBack(tag,"Object successfully retrieved");
    String compareResult = MainObject.checkTestMainObject(mo);
    if (compareResult != null)
    {
        //there is an error
        mReportTo.reportBack(tag,compareResult);
        return;
    }
    //compareReesult is null
    mReportTo.reportBack(tag,"Retrieved object matches");
    return;
}


<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
<string name="test">&lt;node1&gt;blabhhh&lt;/ndoe1&gt;</string>
<string name="json">{&quot;childList&quot;:[{&quot;name&quot;:&quot;Adam&quot;,&quot;likesVeggies&quot;:false,
&quot;age&quot;:30},{&quot;name&quot;:&quot;Eve&quot;,&quot;likesVeggies&quot;:false,
&quot;age&quot;:28}],&quot;stringArray&quot;:[&quot;first&quot;,&quot;second&quot;],
&quot;strinValue&quot;:&quot;st\u003cri\u003e\&quot;ng\&quot;Value\u003cnode1\u003etest\u003c/node2\u003e
&quot;,&quot;intValue&quot;:5}</string>
</map>

Both have escape sequences. So we should not worry about our string values in objects containing special characters. They are automatically handled WELL both by GSON and Android preferences.

If you are further afraid that you want full control of the file where the JSON is stored you can use the internal storage option provided by Android. These internal files are stored on the device specifically for your app and are private by default. I will present some code snippets on how to do this. 


private String readFromInternalFile()
{
    FileInputStream fis = null;
    try {
        Context appContext = MyApplication.s_applicationContext; 
        fis = appContext.openFileInput("datastore-json.txt");
        String jsonString = readStreamAsString(fis);
        return jsonString;
    }
    catch(IOException x)
    {
        mReportTo.reportBack(tag,"Cannot create or write to file");
        return null;
    }
    finally
    {
        closeStreamSilently(fis);
    }
}
private void saveToInternalFile(String ins)
{
    
    FileOutputStream fos = null;
    try {
        Context appContext = MyApplication.s_applicationContext; 
        fos = appContext.openFileOutput("datastore-json.txt"
                                ,Context.MODE_PRIVATE);
        fos.write(ins.getBytes());
    }
    catch(IOException x)
    {
        mReportTo.reportBack(tag,"Cannot create or write to file");
    }
    finally
    {
        closeStreamSilently(fos);
    }
}

Nothing complicated at all. 


data/data/<your-pkg-name>/files/<your-internal-filename>


{"childList":[{"name":"Adam","likesVeggies":false,"age":30},
{"name":"Eve","likesVeggies":false,"age":28}],"stringArray":["first","second"],
"strinValue":"st\u003cri\u003e\"ng\"Value\u003cnode1\u003etest\u003c/node2\u003e",
"intValue":5}

Because typically the data you would represent as JSON is very specific to your application it doesn't make sense to make this available as external storage which is typically used for music files, video files or files that are common in format that is understandable by other applications.

Because external storage such as an SDCard can be in various states (available, not mounted, full, etc) it is harder to program this for simple apps when the data is small enough.

So I don't foresee for now the application state as being maintained on external storage.

May be a hybrid approach is meaningful if the application requires music and photos and those can go on the external storage while keeping the core state data in JSON and internal. 


This document
GSON user guide
Data Storage Options summary document from Android
various Android APIs pertaining to Shared preferences
various Android APIs pertaining to internal storage
Some basic java apis to read/write files

I have provided links for all of these on the right hand side.

This is a real good question. This GSON approach will be expensive if you are writing the file every time something changes. There is a break even point where you want to migrate your code to sqllite and use granular updates.

you can structure your code so that you can swap the persistence layer with out significantly altering your code.

To do this break your app into a persistence service layer and the rest. You will need to be disciplined enough to use the service layer as a stateless set of services that you can use a different implementation later. So remember interfaces and service interface firewalls.

Yes. See this link

This is an article that I wrote in 2006 about extreme prototyping. You can borrow the same principles to construct a service layer that can be swapped.

In conclusion GSON is a good approach for quickly deploying your apps to the market place. It is good to test out market strengths. It is good to release a whole lot of simple apps really fast with minimal work on persistence.