Yearly Archives: 2014

Thing Tracker Network

Thing Tracker Site Template

One aspect of the Thing Tracker Network that is perhaps not terribly clear is that it is not dependent on any particular client, library or vendor. This means that anyone could start participating in the network straight away.

As the TTN client that I am working on is taking it’s time I thought it worthwhile to outline how anyone can start integrating with the network today.

At it’s heart the TTN is simply a web API, where meta-data about Things is stored in JSON formatted documents and made available on the web. To this end it would be simple for someone to create the document by hand and host it, either themselves or via a service such as Github, Google Drive, etc.

Whilst JSON is quite an easy format to read it isn’t meant for humans. The TTN client, and similar services, will happily pick up these JSON documents and work with them (searching, indexing, presenting, etc.) but it would be nice to be able to view the information in a human-friendly way.

Therefore I have thrown together a simple site template which when matched with a TTN Tracker JSON document will provide a web site suitable for publishing. It is hosted on Github (https://github.com/garyhodgson/thing-tracker-site-template) and there is a demo site on Github Pages and also via Google Drive (just to show an alternative option).

 

Screenshot of thing-tracker-site-template

 

Usage

The steps to set up a site are quite straightforward and are also listed in the Github project readme. I will list them here again as they are so few:

  • Clone or copy (zip) the thing-tracker-site-template project.
  • Modify the contents of tracker.json adding meta-data about your Things.
    • Most attributes are self-explanatory, and most are optional so if something is not needed it can usually be deleted. See the specification for details, and feel free to ask in the community for advice.
    • If JSON is new to you then have a look at the JSON website, or perhaps the lighter introduction here.
  • It is recommended to add thumbnail images in the thumbnails folder so they are loaded locally to the website. These can be referenced in the tracker directly, e.g.
    "thumbnailUrls":[
      "thumbnails/my_thumbnail.png"
    ]
  • Be careful to check that the tracker.json is valid – a wrong trailing comma can cause the site to fail to render.  The JSONLint validator may help.
  • Optionally feel free to modify the look and feel of the site. The site uses AngularJS and AngularUI Bootstrap, and the main css file (css\ttn-client.css) is relatively straight-forward to modify.
  • Once satisfied upload the site to your favourite hosting provider.

Hosting on Github Pages

  • Create a public repository on Github.
    • “thingtracker” would be a good name as the resulting website would be hosted at https://[user name].github.io/thingtracker
    • Remember also that custom domain names can be pointed at Github Pages.
  • Clone locally and create a branch named “gh-pages”.
  • Copy your thing-tracker-site-template into this folder.
  • Commit and push to github.
  • After a short while the site should be available via https://[user name].github.io/[project name].

Hosting on Google Drive

  • Create a folder in Google Drive.
  • Copy your thing-tracker-site-template into this folder.
  • Edit the share permissions so it is public.
  • Make a note of the folder id from the address bar, e.g. https://drive.google.com/#folders/0B1a6oY6Gw7BJRU9MSV9KNDg2SU0
  • The site should then be available via https://www.googledrive.com/host/[folder id]/

Caveats

Obviously this approach has several shortcomings.

Editing the tracker file in JSON is not terribly user friendly

At first I thought that having to edit the tracker.json file directly would be a show-stopper, but I’ve come around to the idea that there are several benefits: it can be edited in any text editor; it can be edited directly online in Github; the syntax is easy to pick up, etc. Plus, having to enter all the data in a web form is also quite a hassle, and also prone to error.

The disadvantages are that the format has to be correct in order for the site to render (trailing commas is a good example of where it’s easy to trip up). Double quotation marks have to be escaped. It is probably not very clear for non-techies how to approach editing the file.

Therefore the TTN client, and perhaps future iterations of the site-template will have a more accessible forms approach to entering and modifying the data.

Having all things listed in one tracker file is not terribly scalable

This is actually already solved but not yet documented or fully tested. Each Thing in the Tracker could instead consist of a “refUrl” attribute pointing to another JSON document which contains the Thing information. Take a look at the recently updated Specification page for more details.

The resulting website is very simplistic

There will be no design prizes awarded for the web site. However it was quick to set up, neutral in tone, and is similar in nature to the Githubiverse design. Anyone with basic skills in web design will be able to modify the CSS in order to spruce up the look and feel. Who knows, perhaps TTN Site “Skins” will start making the rounds?

The website is missing numerous useful features

Search being the foremost example, commenting perhaps next. Remember that this is simply a starting point in order to get data into the Network. Once services such as search engines become available it should be trivial to plug these in. Implementing a local keyword search would also be a reasonably easy task if it is desired.

If anyone has more suggestions I would be happy to hear about them. Feel free to raise issues in the Github project or simply comment below.

I should mention that I consider this to be a partial replacement of Githubiverse, which suffered from two flaws. Being a pure client side solution there is no (secure) way to provide keys for the Github API, and the default number of calls is simply not usable. Secondly, it assumes that projects are stored in Github, which of course may not be the case. The TTN Site Template requires more work to set up and maintain (at the moment) but it is more flexible, not vendor specific, and a step towards populating the Thing Tracker Network.

So I hope this article gives an idea of how accessible and open the Thing Tracker Network aims to be.  As ever, let me know your thoughts and opinions.

Thing Tracker Network

TTN Client – Brain Dump

A post by Jason Gullickson has prompted a long overdue status report/brain dump on TTN. Since the last update very little has happened and so I want to go over why that is, and also play around with some ideas of how to get going again.

Getting the TTN Client to beta quality (or even alpha-alpha quality) has proven to be a bit more of a challenge than I expected. Finding the time, and motivation, to tackle some of the challenges, on top of work and family commitments, has been hard.

A challenge I mentioned in my last post was how to reconcile having a distributed model where many of the participants may be off-line for a majority of the time? This led me develop a file-based model for the client which could be also used as a static website – allowing it to be transferred to an arbitrary hosting provider. This leads to the question: how to handle multiple revisions of a particular design? So as not to have to transfer and host copious amounts of data. This led me to investigate using a git-based model, which would have the benefit of integrating easier with Github, but sadly the state of Javascript git libraries is such that this is not a trivial thing to implement.

Another challenge is to handle gracefully the synchronisation and caching of information between TTN nodes. Another is how to secure the network, or at least provide safeguards against malicious nodes and and actors. Another is how to provide search capabilities… and so the list grows.

The client currently allows the following:

  • When starting the first time the client creates a set of keys and connects to the DHT via predefined bootstrap nodes.
  • Multiple Trackers can be created and Things added to them. Basic metadata can be defined which follows the TTN Specification. The metadata can be edited, but the screens have no CSS applied at the moment so they look very basic.
  • The Tracker and Things are made available over a basic REST-like Web Service. Separating the method of publishing the TTN data from the DHT network has the benefit of making it available to a wider range of potential clients.
  • Other nodes can be added to the client by their ID. The trackers of the remote node are then loaded and can be browsed – although I think there is a bug whereby the client loses track of the node when looking up it’s tracker and can’t find it’s things – but I have to double check.
  • The remote tracker and things are cached locally. The client does not yet periodically check to see if there are any updates. Currently the cache has to be purged by hand if the client should retrieve the data again.
  • A basic messaging system exists, whereby a node will forward a message it receives to each of it’s siblings – this would allow a way of nodes announcing updates to the network. An alternative may be to use RSS to publish updates, and it probably makes sense to do both so that a wider range of clients can interact.
  • The client can also read a Tracker which might not originate from a TTN Node, i.e. a website or something. I’m not terribly happy about how I have implemented it though (it uses the concept of a dummy TTN Node to wrap the site) and I don’t think it works very well yet. The goal would be that the client could read TTN trackers from any number of sources, not just participants of the DHT network.

Just writing these few lines reinforces to me just how how much the client is trying to achieve. One of my thoughts on tackling this is by modularisation. Currently the client has too many responsibilities, and is trying to cover too many use cases, I think. I’ll try and enumerate the basic requirements:

  • Store metadata about Things in the form of Trackers.
  • Provide a decentralised way of sharing this information.
  • Discover other Trackers and Things.
  • Present this information to the user.

The devil of course is in the detail. Expanding on these:

  • A Thing may have several revisions.
  • A remote Tracker may be off-line.
  • A remote Tracker may contain malicious content.
  • The client should run on various platforms, possibly running headless on a server, host, Raspberry Pi, etc.
  • The metadata should be accessible to non-TTN clients, i.e. via an API.
  • …etc…

So perhaps instead of the client attempting to do all of these things it should be split up into smaller modules which tackles each thing in turn, i.e. a module which does nothing else apart from manage the actual Trackers; another which provides an API for the Trackers; another which connects to the DHT; another which publishes an RSS feed; another which searches for other Trackers, etc.

I also suspect that I should try and not cover so many bases at first, and just try and produce a limited, but stable, solution that is perhaps only understandable by fellow programmers/hackers at first, and then expand in it to make it more suitable for a wider audience.

In any case I hope that I can get past this current blockage and start producing some tangible results soon.

Thing Tracker Network

TTN Relays

In part prompted by Marcus Wolschon’s blog post about leaving Thingiverse  I have been thinking a bit more about Githubiverse and it’s potential role in the Thing Tracker Client I am writing.

Githubiverse is a nice little hack to take advantage of Github Pages to serve a basic landing page for 3D printing projects. It has a nice advantage that it is all client based code and requires no web back-end (other than a Github repository of course), utilising the Github API to retrieve the data it needs. Unfortunately Github then decided (quite reasonably) to limit the number of anonymous requests that can be made by a client to 60 per hour, and considering each Githubiverse page refresh requires several API calls this is soon exhausted. Because all the code for Githubiverse is publicly viewable it cannot take advantage of any of the authorisation schemes available, such as OAuth.

I have used the page layout of Githubiverse as the basis for displaying things in the TTN client, and have for some time being considering how to add a relay (or mirror) function to the client which will allow a persons tracker to be available even if the client is off-line (which I presume may be the case for many people – even I shut off my PC for hours of the day!). The important requirement for me is that any relay function should be independent of any particular vendor or product, i.e. everyone should be free to choose where they host any TTN relay node.

My initial thoughts revolved around hosting solutions, perhaps something simple in PHP so it can be used in a wide variety of hosting providers, but even this adds a burden on the end user which would make the TTN client unattractive to some. I think my current plan should provide enough flexibility to satisfy the requirements I have.

The client will create a static snapshot of the tracker which is ready to upload or copy to a web server. This snapshot has two roles, it serves up the information required by other TTN clients, and also provides a basic website where people can browse the tracker and download the content. The relay site will be easy to modify for those wishing to change the look and feel of it, and a future feature may be to make it easily skinnable.

TTN Relay
In terms of hosting the relay there are several reasonable options (Note: all the examples below have the same content, simply hosted in different locations):

  • Google Drive / Dropbox / etc.: By copying the relay to a local folder shared by Google Drive, Dropbox and similar services, it can then be accessed as a regular static website. (I don’t know if this is still possible with recent Dropbox accounts, but with older ones such as mine which have a Public folder it is).
  • Github / Git: Integration with Github should be relatively straight-forward, pushing a local copy of the relay to a github-pages branch should provide a similar experience to the current Githubiverse solution. Interfacing with git should work over something like gift or maybe even git.js.  And of course it doesn’t have to be restricted to Github, other git repositories could also be used, opening up a host of other options.
  • FTP / Rsync: A third candidate is to upload the relay to a hosting site, and a plethora of synchronisation options are available.

The client should ensure that all internal references are valid for the machine readable aspect of the relay (via valid relative refUrl attributes I think). It could also provide integration to the wider Tracker Network potentially providing search capabilities.

This is all in addition to having the option to host the information directly from the TTN Node itself should one wish to.  The relay concept simply opens up more opportunities for sharing the data – which is the core aim of this project.

Thoughts and opinions are always welcome, either in the comments or on Google+