Yearly Archives: 2013

Thing Tracker Network

Thing Tracker Network Client : Status Report

A little while ago I started work on a Thing Tracker Network (TTN) Client, with the intention of having at least a working beta before the end of the year.  This proved to be rather optimistic, and whilst the work is still ongoing, I thought it useful to write a post outlining the goals I have in mind, and the challenges I believe have to be overcome.  At this stage I believe the core design is reasonable, and so feel comfortable asking for opinions, feedback and help.

The client concept differs from many alternatives by it being a distributed system.  Whereas the majority (all?) of the existing 3D printing repositories require registration to a central service, this client, in conjunction with the TTN, requires none, and the various Things are stored in the network itself.  This brings several interesting challenges as I shall list later.

A few of the factor driving the design decisions are:

  • The client should be cross-platform, and easy to install and use for a wide range of people. (Whilst recognising that the majority of makers are technically very competent.)
  • Should not rely on central services.
  • Should offer some degree of security, i.e. the ability to verify and trust nodes and their content within the network.

After several discussions with others about the design, the (current) outline of the client is as follows.

  • A standalone application, built on Node-Webkit, which uses a Distributed Hash Table (based on KadoH) to connect to other nodes in the Thing Tracker Network.
  • Information is passed between nodes via RESTful Web Services (using Restify), this includes sharing information about Things and lists of Things (i.e. Trackers) – following the TTN Specification.
  • Sharing of actual content, i.e. the actual design files, is planned to run over BitTorrent, however, lacking suitable BT client libraries, in the short term it will be implemented as direct transfer over HTTPS.
TTN Client Architecture

TTN Client Architecture

Even this simple sounding app brings with it challenges.

Redundancy

How to ensure that a node’s content is still available even if the originating node is offline?

It has to be assumed that only a small portion of nodes will be online 100% of the time. This is where using BitTorrent for the content distribution might have a real benefit, besides resource sharing etc, as the content would be available on another network.

The current design allows for content files to be stored anywhere, and so they could be made available in the cloud (Dropbox, personal web server, etc), but it would make for a better user experience when such remote storage was seamlessly integrated into the client in some way.

My current thinking is that each person can opt to have their node act as a relay for other peoples trackers. That is, I might decide to store and make available the tracker and content for, say RichRaps, designs, and should he be offline when someone wishes to download one then the client could ask the network for alternative locations. This fits with my vision of an ecosystem building up around the TTN – where a variety of clients interact in different ways: relays, aggregators, filters, and so on. Someone may decide to donate resources to relay particular designs on behalf of the community, for example the RepRap project could run a client which acts as a relay for all RepRap designs. This would differ from existing offerings in that the original data remains linked to it’s originating node for purposes of ownership and attribution, plus the data would be replicable by other nodes – i.e. the content is not exclusive to one particular server, rather it just happens to be regularly available at that node.

This is one of the major benefits of this design if we can pull it off: We shouldn’t be dependent on the services of others. If a repository falls offline sometime in the future there should be enough redundancy in the network to ensure that designs remain available.

Security

How to verify that a Thing originates from a given node?

The current design uses Public Keys and Signatures to build a web of trust within the network, to verify that a piece of information originates from the node it claims.  This is an area that will need a lot of work in order to make the system trustworthy.

How to store Things retrieved from possibly untrusted sources?

The app runs inside your network and has access to your local file systems, therefore it is very important that any interaction with external resources be controlled and secured as much as possible. For example, I expected my locally running Avast anti-virus to recognise when the client downloads an infected file, but preliminary tests seem to indicate this is not the case, at least with the original settings which do not scan every file by default. One might be able to use a portable virus scanner, such as ClamAV, but this may conflict with making the client easy to install, and is another component to have to manage.

Communication

How to announce new Things, or updates to existing Things, to the network?

This is an area I have yet to implement, but I suspect may turn out to be one of the more interesting parts of the client. Using a DHT to connect nodes may allow for interesting ways of passing information between the nodes. For announcing, some form of message passing approach may be best, or some form of publish-subscribe model. These updates could potentially be stored in the DHT itself.

How to discover new nodes?

The primary way of including new nodes would be by manually adding the node ID which is gained “out of band”, via forums, blogs, etc. The client would then look up the information for the node, and optionally traverse the trackers (and nodes) that that node itself is following. I can quite imagine that if I were to start following RichRaps node then I may be interested to follow those that he is following too. In this way the network effect should yield a large set of possible trackers to follow.

An alternative way might be to have a “listening” mode for the client, which listens on the DHT for new nodes and adds them to a list to be reviewed whenever you wish. Furthermore there could be a “promiscuous” mode which listens for new activity and automatically adds the trackers and things it picks up to the local tracker. This is how I imagine an indexing or search-engine client might operate – trawling the network for new information. Again, by promoting the TTN specification, rather than any particular solution, the possibilities as to the type and function of the participants of the network are boundless.

Storage

How best to store Thing and Tracker data locally?

If a thing is updated regularly then how to efficiently record these changes without having many redundant copies cluttering the local harddrive? The immediate idea might be to use something like git to store local Things, but that again may conflict with the ease-of-use/installation driving factor.

Versioning

How to handle the fact that Things may have multiple versions?
This adds another complication to the network, not only in terms of organisation and storage, as mentioned above, bit also in terms of hereditary and attribution. How can we effectively track the ancestors which make up a design?

Usability

How should the application look and feel? Could it be skinnable, open to extension, etc.?

By basing the design around the TTN, rather than a particular client, the hope is that others can take the client as a basis for their own designs, or build something from scratch. By making the information available over RESTful web services the barrier to creating a client is greatly reduced.

Interestingly the UI aspect of the client appears to take a large proportion of the time, not due to technical reasons, but because of trying to decide how to make the client accessible to non-technical people.

The application uses AngularJS for the front-end, and this certainly helps reduce the amount of boilerplate in getting the app up and running.

Work In Progress Screenshot

Work In Progress – Tracker Page

Use Case

The above gives a slight indication as to the aspects that I have been considering, and this is by no means an exhaustive list. I’d like now to run through a use case to show how I imagine the client to work in practice. It is quite broad, but also dives deep into the technical details now and again to exemplify aspects of it.

Setup

  • First of all someone downloads the latest version of the client and runs it on their local PC.
  • Because it is the first time it has run the client recognises that there is no local keys for the node and so generates both public and private keys, and uses the hash of the public key as the node id. These values will remain for the life of the node (although we need a way of handling the case when a node is compromised and the keys have to be regenerated – you no doubt see how these seemingly small features give way to ever more issues to deal with!)
  • The user is prompted to optionally provide information about the node, and set particular parameters, such as where to store files locally, whether to automatically cache trackers, etc.
  • The client comes with several well-known bootstrap nodes configured, and these are contacted to announce that the new node is joining the network.

Browsing

  • The client may optionally come with a set of nodes pre-installed, which are contacted and their trackers retrieved and displayed.
  • The user may choose to browse these trackers and their Things.
  • Should they choose to download the content locally, the target node is queried for the content and if available a zip file of the content is stored locally.
    • Optionally, if the node is verified (more later) then the zip file might automatically be unzipped in the client cache so that individual files can be accessed from within the client. Otherwise the user would have to confirm any action involving the remote content until it is deemed verified, e.g. after it has been scanned for viruses.
    • Here we see an opportunity to use the network to help verify the contents of a particular Thing. Some form of flagging scheme may be useful, whereby nodes can flag content as being malicious or objectionable, and if nodes I trust and have verified, my friends to speak, have marked a Thing as being suspicious I might take extra care when working with it, or ignore the Thing, Tracker, or even node, entirely.
  • The user may choose to manually add a node to his list to follow. The node id may have been retrieved via a blog post, email or other means, and is pasted into the client. The remote node has a status of “unverified” at this stage.
  • The remote node is contacted and a list of trackers is returned, these are displayed in the client.
    • If the node is not available then the client may ask the network if anyone else has a cached copy of the node’s trackers.
  • The remote node also returns information about itself, including its public key.
  • The user may then optionally choose to verify the node by providing the fingerprint of the public key belonging to the remote node. This would be at best obtained out-of-band, via a different medium to the source of the node id, and in a way in which the user can be sure originates from the node owner. Note that at this stage no claims are made as to the security of this method. This is one area which would require input and review from those with much more experience in application security. Personal contact, email, and multiple verified sources, may all help in instilling a level of trust that the fingerprint of the node key really originates from the node owner, and that to a certain degree the node can be verified. Furthermore, the option exists to build a web of trust, whereby a node verified by node I have verified has more weight than an unverified node. This is something to be explored further.

Publishing

The user then may choose to publish their own Thing in the network.

  • As this is a fresh install no Tracker yet exists, therefore the user would be prompted to create one.  The thinking being that people would be free to create several Trackers, each of which may track different types of projects.  For example one tracker for personal projects, one for RepRap related designs, another for the local hackerspace, etc.
  • To create a Thing, a form would prompt for information about the Thing, including the location of the files that make up the release.
    • The Thing could be saved in draft form so it can be worked on over a period of time.
  • When the release is ready to publish the client will zip the content into a file ready for others to download, add the Thing to the Tracker, and send an announcement to the network.
  • If the user then finds a typo, or other minor fault, in the release, they may choose to edit certain fields whilst the version remains unchanged.  I think that for certain changes it would be wise to ensure that the version is incremented.
Work In Progress - Thing Page

Work In Progress – Thing Page

 

Hopefully this gives an indication of the factors involved in designing and producing such a client.  There’s a whole lot more in addition to the above.  the challenge now is simply finding the time, and motivation, to produce a client that can at least give a reasonable experience for those willing to try it out.  I’m a firm believer that if the Thing Tracker Network Client can be made to work then it could make a huge impact in the maker world, bringing control back to the individual.

I should say that the code so far is available on Github, but I really can’t recommend trying to use it (although I’ll help anyone who wants to have a go) as the design is still in a state of flux, plus the code quality leaves a lot to be desired as several different approaches and techniques have been tried, and it really needs a good refactoring to make it a bit easier to delve into. However, with that warning, the project can be found here: https://github.com/garyhodgson/thing-tracker-network-client

There is also a test node running on an Amazon EC2 instance at ec2-54-234-89-153.compute-1.amazonaws.com:9880 with the node id “a0a371095c6193c4b15f366d31ac4911a072e634” if anyone is really adventurous! 🙂

 

 

 

Thing Tracker Network

Thing Tracker Network Update

Time flies! It’s been almost two months since I posted about the Thing Tracker Network (TTN). Recently progress has slowed down to a crawl, mainly due to other priorities (RepRap Magazine, work, family, imminent house move, etc, y’know the usual things), and also because I am taking a slight detour with regard to TTN which I want to briefly describe here.

After the initial excitement of announcing the proposal there have been a series of suggestions and improvements which I have attempted to merge into the specification. I think the current version is a reasonable basis for defining Things and the relationships between them. Expanding to include the possibility of adding detailed bill-of-materials and tracker metadata, for example.

A quick aside: there are now several resources that make up the infrastructure of TTN which I will quickly list:
  • The website is the public face of the proposal and will be updated on an ad-hoc basis to reflect the current state of the proposal.
  • The Github repository holds the live specification along with examples. In the near future it will hopefully hold reference libraries, tools, and example applications. There is also a wiki and issue tracker to handle documentation and tasks.
  • A Google+ community to hold discussions, and an associated G+ page which is the source of “official” statements (as opposed to my personal thoughts as a fellow hacker).

Several people have mentioned in passing that the technology of the fabled Semantic Web, now known as Linked Data apparently, may be of interest. A little internet research led me to a useful resource in the Linked Data Book, and the more I read of it the more I realised that the TTN sounds like it is basically trying to be a Linked Data vocabulary. Consequently I think it is worth exploring the idea further, to see whether the goal of the TTN should be to define such a vocabulary along with tools and libraries to support it.

There are several considerations should a Linked Data approach be taken.

  • Many of the concerns that have cropped up in discussing TTN have already been discussed in the Semantic Web world, such as identity, security, and reliability.
  • A set of tools and libraries have already been developed which could potentially be utilised.
  • Linked Data seems ideally suited for describing the types of things the TTN is all about.
  • Using the language and specifications of the Semantic Web might interface the TTN to the wider internet of things, as espoused by Tim Berners-Lee.
  • The Linked Data specifications (e.g. RDFa,RDFS,OWL) are, however, more complex than those originally envisaged by TTN, and the learning curve to fully understand how the network works may be steeper.
  • Using a recognised and widely adopted standard would seem preferable than devising a proprietary, albeit open, standard.

From my reading so far, I believe the following would be possible using Linked Data:

  • Define a vocabulary using RDFS (Resource Description Framework Schema) which would be similar (if not identical) to the current TTN json schema.
  • Mark up a Thing with metadata attributes using one of the following ways:
    • RDFa (Resource Description Framework in attributes) to embed the TTN attributes within a web page describing a Thing.
    • JSON-LD (JavaScript Object Notation for Linked Data) to describe the Thing in a JSON document.
    • Traditional RDF (Resource Description Framework) in an XML document.
  • Use existing libraries, or build new ones, to interact and work with this data.

In fact I hope that the RDFS & JSON-LD combination might already be very close to the work we have done so far.

To date my playing and experimenting has been limited. I have started formulating an RDF Schema (http://thingtracker.net/thingtracker.rdfs) and have added some RDFa to the example Githubiverse site (http://garyhodgson.github.com/githubiverse-tst), which appears to be readable according to the few tools I have tried out.

RDFa Source

Snippet from source html. New RDFa attributes highlighted.

Parsed RDFa

RDFa parsed using Green Turtle RDFa extension for Chrome browser.

There is a whole raft of things I still need to learn (help & guidance very welcome!), and this is still only an exploratory dive into Linked Data, but so far I feel it is worth pursuing further.

3D Printroom

3D Printroom

A small distraction I thought up a few days ago involves how we organise our STL and gcode files.  The Windows and Linux file explorers are rather limited when dealing with exotic files (no previews, no metadata). Additionally, I don’t know how other people manage their files, but mine are scattered across my main PC, my laptop and a NAS, within arbitrarily named folders filled with overly long file names (in an attempt to capture some metadata). It certainly has it’s limitations!

file chaos

Chaos! Chaos I tell you!

This seems ripe for an application of some sort (ignoring the patently absurd, if rational, argument to simply impose a little order on things).

I have always enjoyed using Adobe Lightroom to organise and work on my photos – it has several features that could be very useful when dealing with 3D printer files:

  • Directory scanning & sync
  • Arbitrary collections and groups
  • Adding additional metadata, including comments and ratings
  • Previews of non-standard files (e.g. RAW)
Adobe Lightroom

Adobe Lightroom – something to aspire to.

I wish I could announce that I have completely written a “Lightroom for 3D Printers” but at this stage I have only created a skeleton app with limited functionality:

  • Drop a directory in the top left panel to load it.
  • The top file view filters only STL files.
  • When an STL is selected the related gcode files are displayed below. This assumes the files start with the same name, e.g. cube.stl -> cube.gcode.
  • Any metadata in the gcode file is displayed in a table view (Slic3r only).
  • Double clicking a STL file displays a preview (thanks to Tony Buser’s Thingiview).
  • Double clicking a gcode file displays a preview (thanks to Joe Walnes’ gcode-viewer).
3D-Printroom

3D Printroom (Alpha!)

… and that’s it … so far.

The reason I am posting about this already is to share the idea, gauge interest, and to make the source code available in case anyone wants to pick it up and hack with it.

I chose to develop it with node-webkit, which allows you to develop with HTML + JS + CSS but still release a native binary. (This is necessary in order to use node.js so the local filesystem can be accessed). I’ve added a few notes on the github project on how to hack with it.  Or there is a Windows binary available to play with: 3D-Printroom-distribution.zip.

I’d be interested to hear if anyone thinks such an app would be useful.

 

 

Community Thing Tracker Network

Thing Tracker Network

A little time ago, RichRap started a discussion on the RepRap forum about a model repository for the project. My response to this crystallised a few thoughts I had on the subject since the Thingiverse debacle from last year. The basic concern being that, whilst creating a content repository is relatively straightforward from a technical perspective, there are other factors which will determine whether it will succeed. The most obvious being the catch-22 situation whereby users are attracted to a service that is being used by many other people, i.e.  where they are more likely to have their project/design discovered by other people. There is also a concern that any one provider evolves into a monopoly, depriving the community of diversity.

 

With all this in mind, I have been developing the idea a little more and would like to present the resulting proposal for consideration to the community. To do this I have created an information website which I hope can be used as a central hub for distribution and further development of the idea: thingtracker.net

Overview

The site should have enough information to get the rough idea across, and so I shan’t repeat it all in this blog post, however a TL;DR summary may be helpful. The proposal simply suggests a way in which an ad-hoc Network of Things could be created. This is enabled by a short, flexible, and easy to implement specification, which, when followed, would allow a wide range of individuals and groups to participate in the network. Furthermore, the open nature of the specification should promote development of an ecosystem within the maker community, allowing a diverse set of applications and websites to emerge.

 

Use Cases

A couple of use-cases may help explain the potential of the proposal.

  • As an individual, when I create a design that I wish to share, I have several services available to me (Thingiverse, GrabCad, etc.). These being walled gardens means that anyone searching within one will not find results from another. Furthermore, search engines such as Google may result in my Thing being found, but its generic nature means that this is unlikely unless people use the right search terms. Therefore my options are either to limit the number of people who can discover my Thing, or submit, and maintain, the Thing to multiple repositories. Neither option is optimal, and the latter is particularly inefficient should my Thing evolve and I have to update the information in several places. Ideally I would submit my Thing to my favourite (based on features that I value) repository, and then have it discoverable by others, independent of where it is hosted. Having the option to self-host the Thing would also be good. What is missing at the moment is a standardised way for web applications to exchange data about Things.
  • As a member of a hackerspace, or other group, we would like to host and share our designs ourselves but still have them discoverable by a wide audience. Furthermore, we would like to provide much more detailed information, such as: bill of materials; instructions; and other information. In this case there may be a sub-Network of Things which has it’s own ecosystem but is still a part of the wider network.

 

Where Next?

To start dissemination of the proposal I made a brief post on Google+ and was pleasantly surprised at the positive feedback the idea received. I shall be collating this information and reflecting it in further revisions of the proposal and specification. In order to facilitate further discussions, without over-thinking the infrastructure, I have created a dedicated page on G+ to act as a blog and to collect discussions, etc.  There is also an email address: contact@thingtracker.net

 

The proposal is at a very early stage and is completely open to everyone,  so feel free to share your ideas, thoughts and opinions. If you feel that an open, distributed, Network of Things would benefit the maker community then please share the links, spread the word and join the discussion.