Menu Search
Jump to the content X X
Smashing Conf New York

We use ad-blockers as well, you know. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

Building A Simple Cross-Browser Offline To-Do List With IndexedDB And WebSQL

Making an application work offline can be a daunting task. In this article, Matthew Andrews, a lead developer behind FT Labs, shares a few insights he had learned along the way while building the FT application. Matthew will also be running a “Making It Work Offline” workshop1 at our upcoming Smashing Conference in Freiburg in mid-September 2014. – Ed.

We’re going to make a simple offline-first to-do application2 with HTML5 technology. Here is what the app will do:

  • store data offline and load without an Internet connection;
  • allow the user to add and delete items in the to-do list;
  • store all data locally, with no back end;
  • run on the first- and second-most recent versions of all major desktop and mobile browsers.

The complete project is ready for forking on GitHub3.

Which Technologies To Use Link

In an ideal world, we’d use just one client database technology. Unfortunately, we’ll have to use two:

Veterans of the offline-first world might now be thinking, “But we could just use localStorage6, which has the benefits of a much simpler API, and we wouldn’t need to worry about the complexity of using both IndexedDB and WebSQL.” While that is technically true, localStorage has number of problems7, the most important of which is that the amount of storage space available with it is significantly less than IndexedDB and WebSQL.

Luckily, while we’ll need to use both, we’ll only need to think about IndexedDB. To support WebSQL, we’ll use an IndexedDB polyfill8. This will keep our code clean and easy to maintain, and once all browsers that we care about support IndexedDB natively, we can simply delete the polyfill.

Note: If you’re starting a new project and are deciding whether to use IndexedDB or WebSQL, I strongly advocate using IndexedDB and the polyfill. In my opinion, there is no reason to write any new code that integrates with WebSQL directly.

I’ll go through all of the steps using Google Chrome (and its developer tools), but there’s no reason why you couldn’t develop this application using any other modern browser.

1. Scaffolding The Application And Opening A Database Link

We will create the following files in a single directory:

  • /index.html
  • /application.js
  • /indexeddb.shim.min.js
  • /styles.css
  • /offline.appcache

/index.html Link

<!DOCTYPE html>
<html>
  <head>
    <link rel='stylesheet' href='./styles.css' type='text/css' media='all' />
  </head>
  <body>
    <h1>Example: Todo</h1>
    <form>
      <input placeholder="Type something" />
    </form>
    <ul>
    </ul>
    <script src="./indexeddb.shim.min.js"></script>
    <script src="./application.js"></script>
  </body>
</html>

Nothing surprising here: just a standard HTML web page, with an input field to add to-do items, and an empty unordered list that will be filled with those items.

/indexeddb.shim.min.js Link

Download the contents of the minified IndexedDB polyfill9, and put it in this file.

/styles.css Link

body {
  margin: 0;
  padding: 0;
  font-family: helvetica, sans-serif;
}

* {
  box-sizing: border-box;
}

h1 {
  padding: 18px 20px;
  margin: 0;
  font-size: 44px;
  border-bottom: solid 1px #DDD;
  line-height: 1em;
}

form {
  padding: 20px;
  border-bottom: solid 1px #DDD;
}

input {
  width: 100%;
  padding: 6px;
  font-size: 1.4em;
}

ul {
  margin: 0;
  padding: 0;
  list-style: none;
}

li {
  padding: 20px;
  border-bottom: solid 1px #DDD;
  cursor: pointer;
}

Again, this should be quite familiar: just some simple styles to make the to-do list look tidy. You may choose not to have any styles at all or create your own.

/application.js Link

(function() {

  // 'global' variable to store reference to the database
  var db;

  databaseOpen(function() {
    alert("The database has been opened");
  });

  function databaseOpen(callback) {
    // Open a database, specify the name and version
    var version = 1;
    var request = indexedDB.open('todos', version);

    request.onsuccess = function(e) {
      db = e.target.result;
      callback();
    };
    request.onerror = databaseError;
  }

  function databaseError(e) {
    console.error('An IndexedDB error has occurred', e);
  }

}());

All this code does is create a database with indexedDB.open and then show the user an old-fashioned alert if it is successful. Every IndexedDB database needs a name (in this case, todos) and a version number (which I’ve set to 1).

To check that it’s working, open the application in the browser, open up “Developer Tools” and click on the “Resources” tab.

In the Resources panel, you can check whether it's working.

In the “Resources” panel, you can check whether it’s working.

By clicking on the triangle next to “IndexedDB,” you should see that a database named todos has been created.

2. Creating The Object Store Link

Like many database formats that you might be familiar with, you can create many tables in a single IndexedDB database. These tables are called “objectStores.” In this step, we’ll create an object store named todo. To do this, we simply add an event listener on the database’s upgradeneeded event.

The data format that we will store to-do items in will be JavaScript objects, with two properties:

  • timeStamp
    This timestamp will also act as our key.
  • text
    This is the text that the user has entered.

For example:

{ timeStamp: 1407594483201, text: 'Wash the dishes' }

Now, /application.js looks like this (the new code starts at request.onupgradeneeded):

(function() {

  // 'global' variable to store reference to the database
  var db;

  databaseOpen(function() {
    alert("The database has been opened");
  });

  function databaseOpen(callback) {
    // Open a database, specify the name and version
    var version = 1;
    var request = indexedDB.open('todos', version);

    // Run migrations if necessary
    request.onupgradeneeded = function(e) {
      db = e.target.result;
      e.target.transaction.onerror = databaseError;
      db.createObjectStore('todo', { keyPath: 'timeStamp' });
    };

    request.onsuccess = function(e) {
      db = e.target.result;
      callback();
    };
    request.onerror = databaseError;
  }

  function databaseError(e) {
    console.error('An IndexedDB error has occurred', e);
  }

}());

This will create an object store keyed by timeStamp and named todo.

Or will it?

Having updated application.js, if you open the web app again, not a lot happens. The code in onupgradeneeded never runs; try adding a console.log in the onupgradeneeded callback to be sure. The problem is that we haven’t incremented the version number, so the browser doesn’t know that it needs to run the upgrade callback.

How to Solve This? Link

Whenever you add or remove object stores, you will need to increment the version number. Otherwise, the structure of the data will be different from what your code expects, and you risk breaking the application.

Because this application doesn’t have any real users yet, we can fix this another way: by deleting the database. Copy this line of code into the “Console,” and then refresh the page:

indexedDB.deleteDatabase('todos');

After refreshing, the “Resources” pane of “Developer Tools” should have changed and should now show the object store that we added:

The Resources panel should now show the object store that was added.

The “Resources” panel should now show the object store that was added.

3. Adding Items Link

The next step is to enable the user to add items.

/application.js Link

Note that I’ve omitted the database’s opening code, indicated by ellipses (…) below:

(function() {

  // Some global variables (database, references to key UI elements)
  var db, input;

  databaseOpen(function() {
    input = document.querySelector('input');
    document.body.addEventListener('submit', onSubmit);
  });

  function onSubmit(e) {
    e.preventDefault();
    databaseTodosAdd(input.value, function() {
      input.value = '';
    });
  }

[…]

  function databaseTodosAdd(text, callback) {
    var transaction = db.transaction(['todo'], 'readwrite');
    var store = transaction.objectStore('todo');
    var request = store.put({
      text: text,
      timeStamp: Date.now()
    });

    transaction.oncomplete = function(e) {
      callback();
    };
    request.onerror = databaseError;
  }

}());

We’ve added two bits of code here:

  • The event listener responds to every submit event, prevents that event’s default action (which would otherwise refresh the page), calls databaseTodosAdd with the value of the input element, and (if the item is successfully added) sets the value of the input element to be empty.
  • A function named databaseTodosAdd stores the to-do item in the local database, along with a timestamp, and then runs a callback.

To test that this works, open up the web app again. Type some words into the input element and press “Enter.” Repeat this a few times, and then open up “Developer Tools” to the “Resources” tab again. You should see the items that you typed now appear in the todo object store.

03-step3-dev-tools-opt-50010

After adding a few items, they should appear in the todo object store. (View large version11)

4. Retrieving Items Link

Now that we’ve stored some data, the next step is to work out how to retrieve it.

/application.js Link

Again, the ellipses indicate code that we have already implemented in steps 1, 2 and 3.

(function() {

  // Some global variables (database, references to key UI elements)
  var db, input;

  databaseOpen(function() {
    input = document.querySelector('input');
    document.body.addEventListener('submit', onSubmit);
    databaseTodosGet(function(todos) {
      console.log(todos);
    });
  });

[…]

  function databaseTodosGet(callback) {
    var transaction = db.transaction(['todo'], 'readonly');
    var store = transaction.objectStore('todo');

    // Get everything in the store
    var keyRange = IDBKeyRange.lowerBound(0);
    var cursorRequest = store.openCursor(keyRange);

    // This fires once per row in the store. So, for simplicity,
    // collect the data in an array (data), and pass it in the
    // callback in one go.
    var data = [];
    cursorRequest.onsuccess = function(e) {
      var result = e.target.result;

      // If there's data, add it to array
      if (result) {
        data.push(result.value);
        result.continue();

      // Reach the end of the data
      } else {
        callback(data);
      }
    };
  }

}());

After the database has been initialized, this will retrieve all of the to-do items and output them to the “Developer Tools” console.

Notice how the onsuccess callback is called after each item is retrieved from the object store. To keep things simple, we put each result into an array named data, and when we run out of results (which happens when we’ve retrieved all of the items), we call the callback with that array. This approach is simple, but other approaches might be more efficient.

If you reopen the application again, the “Developer Tools” console should look a bit like this:

The console after reopening the application

The console after reopening the application

5. Displaying Items Link

The next step after retrieving the items is to display them.

/application.js Link

(function() {

  // Some global variables (database, references to key UI elements)
  var db, input, ul;

  databaseOpen(function() {
    input = document.querySelector('input');
    ul = document.querySelector('ul');
    document.body.addEventListener('submit', onSubmit);
    databaseTodosGet(renderAllTodos);
  });

  function renderAllTodos(todos) {
    var html = '';
    todos.forEach(function(todo) {
      html += todoToHtml(todo);
    });
    ul.innerHTML = html;
  }

  function todoToHtml(todo) {
    return '<li>'+todo.text+'</li>';
  }

[…]

All we’ve added are a couple of very simple functions that render the to-do items:

  • todoToHtml
    This takes a todos object (i.e. the simple JavaScript object that we defined earlier).
  • renderAllTodos
    This takes an array of todos objects, converts them to an HTML string and sets the unordered list’s innerHTML to it.

Finally, we’re at a point where we can actually see what our application is doing without having to look in “Developer Tools”! Open up the app again, and you should see something like this:

Your application in the front-end view12

Your application in the front-end view (View large version13)

But we’re not done yet. Because the application only displays items when it launches, if we add any new ones, they won’t appear unless we refresh the page.

6. Displaying New Items Link

We can fix this with a single line of code.

/application.js Link

The new code is just the line databaseTodosGet(renderAllTodos);.

[…]

function onSubmit(e) {
  e.preventDefault();
  databaseTodosAdd(input.value, function() {
    // After new items have been added, re-render all items
    databaseTodosGet(renderAllTodos);
    input.value = '';
  });
}

[…]

Although this is very simple, it’s not very efficient. Every time we add an item, the code will retrieve all items from the database again and render them on screen.

7. Deleting Items Link

To keep things as simple as possible, we will let users delete items by clicking on them. (For a real application, we would probably want a dedicated “Delete” button or show a dialog so that an item doesn’t get deleted accidentally, but this will be fine for our little prototype.)

To achieve this, we will be a little hacky and give each item an ID set to its timeStamp. This will enable the click event listener, which we will add to the document’s body, to detect when the user clicks on an item (as opposed to anywhere else on the page).

/application.js Link

(function() {

  // Some global variables (database, references to key UI elements)
  var db, input, ul;

  databaseOpen(function() {
    input = document.querySelector('input');
    ul = document.querySelector('ul');
    document.body.addEventListener('submit', onSubmit);
    document.body.addEventListener('click', onClick);
    databaseTodosGet(renderAllTodos);
  });

  function onClick(e) {

    // We'll assume that any element with an ID
    // attribute is a to-do item. Don't try this at home!
    if (e.target.hasAttribute('id')) {

      // Because the ID is stored in the DOM, it becomes
      // a string. So, we need to make it an integer again.
      databaseTodosDelete(parseInt(e.target.getAttribute('id'), 10), function() {

        // Refresh the to-do list
        databaseTodosGet(renderAllTodos);
      });
    }
  }

[…]

  function todoToHtml(todo) {
    return '<li id="'+todo.timeStamp+'">'+todo.text+'</li>';
  }

[…]

  function databaseTodosDelete(id, callback) {
    var transaction = db.transaction(['todo'], 'readwrite');
    var store = transaction.objectStore('todo');
    var request = store.delete(id);
    transaction.oncomplete = function(e) {
      callback();
    };
    request.onerror = databaseError;
  }

}());

We’ve made the following enhancements:

  • We’ve added a new event handler (onClick) that listens to click events and checks whether the target element has an ID attribute. If it has one, then it converts that back into an integer with parseInt, calls databaseTodosDelete with that value and, if the item is successfully deleted, re-renders the to-do list following the same approach that we took in step 6.
  • We’ve enhanced the todoToHtml function so that every to-do item is outputted with an ID attribute, set to its timeStamp.
  • We’ve added a new function, databaseTodosDelete, which takes that timeStamp and a callback, deletes the item and then runs the callback.

Our to-do app is basically feature-complete. We can add and delete items, and it works in any browser that supports WebSQL or IndexedDB (although it could be a lot more efficient).

Almost There Link

Have we actually built an offline-first to-do app? Almost, but not quite. While we can now store all data offline, if you switch off your device’s Internet connection and try loading the application, it won’t open. To fix this, we need to use the HTML5 Application Cache14.

Warning Link

  • While HTML5 Application Cache works reasonably well for a simple single-page application like this, it doesn’t always. Thoroughly research how it works15 before considering whether to apply it to your website.
  • Service Worker16 might soon replace HTML5 Application Cache, although it is not currently usable in any browser, and neither Apple nor Microsoft have publicly committed to supporting it.

8. Truly Offline Link

To enable the application cache, we’ll add a manifest attribute to the html element of the web page.

/index.html Link

<!DOCTYPE html>
<html manifest="./offline.appcache">
[…]

Then, we’ll create a manifest file, which is a simple text file in which we crudely specify the files to make available offline and how we want the cache to behave.

/offline.appcache Link

CACHE MANIFEST
./styles.css
./indexeddb.shim.min.js
./application.js

NETWORK:
*

The section that begins CACHE MANIFEST tells the browser the following:

  • When the application is first accessed, download each of those files and store them in the application cache.
  • Any time any of those files are needed from then on, load the cached versions of the files, rather than redownload them from the Internet.

The section that begins NETWORK tells the browser that all other files must be downloaded fresh from the Internet every time they are needed.

Success! Link

We’ve created a quick and simple to-do app17 that works offline and that runs in all major modern browsers, thanks to both IndexedDB and WebSQL (via a polyfill).

Resources Link

(al, ml, il)

Footnotes Link

  1. 1 http://smashingconf.com/workshops/
  2. 2 https://matthew-andrews.github.io/offline-todo/
  3. 3 https://github.com/matthew-andrews/offline-todo
  4. 4 http://caniuse.com/indexeddb
  5. 5 http://caniuse.com/sql-storage
  6. 6 http://caniuse.com/namevalue-storage
  7. 7 https://hacks.mozilla.org/2012/03/there-is-no-simple-solution-for-local-storage/
  8. 8 https://github.com/axemclion/IndexedDBShim
  9. 9 https://raw.githubusercontent.com/matthew-andrews/offline-todo/gh-pages/indexeddb.shim.min.js
  10. 10 https://www.smashingmagazine.com/wp-content/uploads/2014/08/03-step3-dev-tools-opt.jpg
  11. 11 https://www.smashingmagazine.com/wp-content/uploads/2014/08/03-step3-dev-tools-opt.jpg
  12. 12 https://www.smashingmagazine.com/wp-content/uploads/2014/08/05-step5-app-opt.jpg
  13. 13 https://www.smashingmagazine.com/wp-content/uploads/2014/08/05-step5-app-opt.jpg
  14. 14 https://developer.mozilla.org/en-US/docs/Web/HTML/Using_the_application_cache
  15. 15 http://alistapart.com/article/application-cache-is-a-douchebag
  16. 16 http://www.serviceworker.org/
  17. 17 https://matthew-andrews.github.io/offline-todo/
  18. 18 http://nparashuram.com/IndexedDBShim/
  19. 19 http://alistapart.com/article/application-cache-is-a-douchebag
  20. 20 https://jakearchibald.github.io/isserviceworkerready/
SmashingConf New York

Hold on, Tiger! Thank you for reading the article. Did you know that we also publish printed books and run friendly conferences – crafted for pros like you? Like SmashingConf Barcelona, on October 25–26, with smart design patterns and front-end techniques.

↑ Back to top Tweet itShare on Facebook

Matt works for the FT on the FT Web app, one of the most successful cross platform HTML5 offline web apps. He will also be running workshops on building offline-first applications in London and Freiburg in Autumn 2014.

  1. 1

    Well, it’s fine, but let’s imagine that we have the data in the database 100 thousand items, and we need to display 10 items per page, you will need to sort the elements of one or more of the fields? And if we need to filter the data according to any of the field?

    Please complete the following examples for the above tasks using IndexedDB. And run it, and then telling us how long it will take for the program.

    That is, the question is this – how fast will it work? Indeed, judging by your code sample data from indexedDB, filter have all 100,000 items using JS? And if they are 1 million?

    -22
  2. 4

    Eugene Ivanov – why would you have / need 100,000 items in your personal to-do list?

    This is a very good introductory tutorial on how to implement a basic offline web application using IndexedDB And WebSQL for a use case with a relatively small dataset, not a “how-to” on building a fully-fledged multi-user web-scale to-do list app!

    Saying that, it wouldn’t be too difficult to extend this to potentially hand off full / historic data to a back-end API when you are online, and only maintain the immediately relevant items for offline usage…

    8
  3. 5

    Maybe I’m over-simplifying this, but if I wanted an offline to-do list, I would just use TiddlyWiki.

    http://tiddlywiki.com/#TaskManagementExample

    They pretty much have all this functionality built into their system, and it will just be a single file I have to manage in the end, no database needed. You can check out how to implement the to-do list by viewing the source to the wiki entry, just click the edit/pencil icon.

    I get that this article was supposed to be an introduction to using these tools in tandem together, but I just feel like I still need to show this simpler solution.

    -11
    • 6

      That looks like a great product!

      So I think this is never really supposed to be ever used for someone’s real todo list – it’s just a demonstration of what’s possible with web technologies today. The key bit is that this prototype can work offline – so even if you have no internet connection at all you can still add items to and delete items from your todo list.

      12
    • 7

      I think you’re confusing the example used in the article for the purpose of the article. The Todo list app is completely irrelevant, it’s only used to demonstrate offline DB capabilities in a browser, which is the point of the article.

      It could be a Todo list, a daily time tracking, or a list of cat names you think of, it doesn’t matter.

      7
  4. 8

    simple but effective..great job!

    1
  5. 10

    Any experience with pouchdb, localforage or the new service workers? Thoughts?

    Thx!

    2
    • 11

      Service Workers are really exciting. I really can’t wait – they’re going to change everything (in a good way, I hope).

      There are a few very nice abstractions on top of IndexedDB out there – e.g. LocalForage and Dexie (https://github.com/dfahlander/Dexie.js). I think I were building a new app now I would definitely use one of them. (Though I still think it’s useful to *learn* how the underlying browser APIs work, even if you never use them)

      I like the ideas behind PouchDB and Hoodi.ie (http://hood.ie/) but in order to use them (as far as I know) you must use CouchDB on your web servers. So for that reason I don’t think I would ever use either.

      3
      • 12

        Wow, thank you for the extremely helpful response! I’m in at the position where I’m starting a smaller project with a blank slate so it’s very useful.

        Great writing! Thank you again.

        1
      • 13

        For hood.ie I have no idea.
        If you only want offline usage, no syncronisation with a server, you don’t need CouchDB for PouchDB. You would just use PouchDB as an abstraction Layer for indexedDB. But you will get a very powerful sync mechanism if you deside to either install CouchDB on your Server or use a Service like claudant as replication point.

        2
  6. 14

    There is a IndexedDB wrapper called “localForage” that implements a localStorage like syntax for IndexedDB.

    It works on multiple desktop/mobile browsers:
    https://github.com/mozilla/localForage#supported-browsersplatforms

    1
  7. 15

    Thanks for a great example of how to build offline JavaScript apps and not a todo app.

    2
  8. 16

    I have been working on an offline-capable webapp. http://wholesale.mizbeach.com

    I use Persist.js as an abstraction over the storage layer, though before I worked directly with localStorage and didn’t seem to have much trouble with it. After this project I’d be hesitant to ever build a website that can’t work offline. The speed and the simple back-end structure is far too good to pass up.

    0
  9. 17

    In 5. Displaying Items don’t you need ” ?

    function todoToHtml(todo) {
    return ”+todo.text+ ”;
    }

    0
  10. 20

    Laverne Hoiseth

    September 7, 2014 11:35 am

    Thanks for the excellent post against your blog, it genuinely provides me with a look about this topic.??;~.??

    0
  11. 21

    About the appcache, is it ok to leave `index.html` out of manifest, if you want the page to work offline?

    I realise that the guide were mostly about indexdDB and that part were great!

    0
    • 22

      Hi Emil, Great question!

      Basically you would probably never want to add ‘/index.html’ to the manifest (because it’s unlikely anyone would specifically type in your-domain.com/index.html into their web browser) but you might think you would need to add ‘/’ so get the home page of your website be cached.

      However, there is a slightly unexpected piece of behaviour in AppCache that I’ve written about before here: http://labs.ft.com/2012/11/using-an-iframe-to-stop-app-cache-storing-masters/.

      > “one of the ill thought out features of the HTML5 application cache is that the page which points to the manifest in its HTML tag – – will itself also be cached, whether you like it or not”

      So in this case you don’t need to add home page (because AppCache will do it magically for you). But if you do add it, it wouldn’t cause any harm.

      Hope this helps!

      Matt

      1
  12. 23

    I appreciate the clarity of the example. I see now how to use indexedDB in at least a simple manner.

    I read the article hoping to find an app that used JavaScript to write a file to the local filesystem. Instead I saw an app that writes to a local database which is not stored across browsers and is not stored in my web root directory.

    I am happy with what I have learned from the post though I have yet to find something that will act like a modularized, secure, persistent, browser-independent datastore that I can stick in my webroot directory with my other web resources.

    Mark, do you know of anything that could help me or point me in the right direction of what I am looking for?

    0
  13. 24

    This is an extremely well crafted sample for any one wanting to start using indexedDB and start on offline apps. Thanks !!

    0

↑ Back to top