As we covered last week, the internet needs to change, and there are several projects currently in the works with very ambitious but necessary steps to make that a reality. I will be spending considerable time in the coming weeks testing out software and investigating these projects in detail. My goal is to figure out the landscape. I want to know what these various projects really are, where they are in development, and get a good idea of their strengths and weaknesses.
My first project is to tackle IPFS, the Interplanetary File System. First off, the name is great. It is an homage to J.C.R. Licklider's mention of the concept of a universal network way back in the 1960's which he called the "Intergalactic Computer Network," an optimistic, but apropos name. The goal of IPFS is to complement or even replace HTTP as the primary method of transferring information. Juan Benet, the inventor of IPFS, describes it as a "Single bittorrent swarm, exchanging objects within one Git repository." It uses the combination of several existing technologies to create a generic framework for exchanging any kind information across the internet, whether it is websites, messages, distributed apps, or media files. If done correctly, it could form the backbone of a much more efficient internet of the future.
For detailed information and background on IPFS, check out www.ipfs.io.
I am spending this week installing, testing, and playing with IPFS and taking copious notes of my journey. I hope it serves to introduce you to the technology, its strengths and weakness, and will serve to guide you in the process should you want to use it yourself. I’m stumbling around looking for the easiest, most straightfoward way of getting it working properly. Because this software is in such an early stage, documentation is incomplete and a bit scattered. I’m putting everything I do here step-by-step in hopes of having everything in one place for easy reference. I really don’t know what I’m doing, so this will be a learning experience for all involved!
ASUS laptop running Ubuntu 16.04.
12G RAM
Intel i7-2670GM CPU
NVIDIA GT560M
I decided to use the Go implementation of IPFS because it is the main implementation and is the most mature. I installed Go using the instructions here: https://golang.org/doc/install. I first tried using apt-get to install Go. That doesn’t seem to be updated or set up properly, so I ended up using the instructions on the golang.org website.
My next step was to install IPFS. There are several install methods but installing IPFS from the pre-built package is recommended, and worked very well. I downloaded the go-ipfs.tar.gz from here: https://dist.ipfs.io/#go-ipfs, unpacked it into my executables $PATH, and ran the install.sh script. Now, I needed to update IPFS and, though manual updates are simple, I went with the ipfs-update tool.
Navigate to somewhere within $PATH and download the ipfs-update binary with 'go get':
$ go get -u github.com/ipfs/ipfs-update
Check what version of IPFS you have installed:
$ ipfs-update version
Check what versions are available:
$ ipfs-update versions
Install latest version:
$ ipfs-update install <version>
That’s it! IPFS is installed.
Now that IPFS is installed, it’s time to start using it. First, we initialize the repo and generate our keypairs:
$ ipfs init
initializing ipfs node at /Users/justin/.go-ipfs
generating 2048-bit RSA keypair...done
peer identity: QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u
to get started, enter:
ipfs cat /ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG/readme
The “peer identity” hash, “QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u” is the ID that identifies a particular node.
Now, we can start the daemon and get online!
$ ipfs daemon
Several lines appeared, ending with “Daemon is ready.” Probably a good sign.
IPFS is now running on the machine and we’re ready to start surfing the new internet. I ran the daemon in the background so I don’t accidentally kill it. I use Screen so I have easy access later.
Now let’s do a few simple things like see what peers we are talking to, download and upload simple files, and pull up the web console.
$ ipfs swarm peers
This command will give you a listing of the peers your computer is currently talking to. Wow - a huge list of peers showed up. Looks like people are using the software. Awesome!
I can also access peers, a file list, and some other stuff on the webui at http://localhost:5001/webui
Now let’s try giving the network some files, and retrieving some files from the network.
$ echo “hello world” > hello // Create a file called “hello” and add the text “hello world”
$ ipfs add hello // Add “hello” to ipfs
added QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o hello
That last command will return a hash, in this case: QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o. This is the hash of the contents of the file. So, if you create another file, say “hello1” and add the same text, it will give you the same hash. This allows IPFS to de-duplicate information. If something has the same information, it has the same hash, the same address, and can be stored only once on a given machine.
Now, if we want to retrieve that file from ipfs, we can use “cat” or “get”.
$ ipfs cat /ipfs/QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
hello world
Remember, IPFS works on a single global repository, so that same command will work to retrieve a file from anywhere in the world. For example let’s try:
$ ipfs get /ipfs/QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ/cat.jpg
How cool is that? It went out into the wide world and found the file using nothing but the hash of the contents. On the down side, I now have a random picture of a cat on my computer that definitely wasn’t there before. Oh well.
You can also wrap file and directory information into the file with the -w flag.
$ ipfs add -w hello
added QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o hello
added QmYRMUVULBfj7WrdPESnwnyZmtayN6Sdrwh1nKcQ9QgQeZ
This will preserve file and directory information so you can reference it that way. The first line returned the same hash as before – the hash of the contents. The second line returned a different hash which was the content plus file and directory information.
Speaking of file and directory information, we can also wrap subdirectories in as well, and reference the directory and file names under the parent hash. I put the file “hello” into a directory “test_dir.”
$ipfs add -r test_dir
added QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o test_dir/hello
added QmYRMUVULBfj7WrdPESnwnyZmtayN6Sdrwh1nKcQ9QgQeZ test_dir
You’ll notice the first hash is the same as it was before for the file “hello.” The second hash wraps the directory and contents into a single hash. We can list the contents of the directory:
$ipfs ls QmYRMUVULBfj7WrdPESnwnyZmtayN6Sdrwh1nKcQ9QgQeZ
QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o 27 hello
It lists the hash, the size, and the file name of each object within the directory. We can also display the contents of the file within the directory.
$ipfs cat /ipfs/QmYRMUVULBfj7WrdPESnwnyZmtayN6Sdrwh1nKcQ9QgQeZ /hello
hello again, world
Ok, so that’s the basics, but let’s now move on and try some really fun stuff. I bet we can make IPFS do all kinds of interesting things.
What we explored above was ok for basic file sharing. I’ve been reading through the documentation, and only marginally understand it. But, it looks like IPFS can do a variety of interesting things, so we’re going to explore some of them now. I really don’t know what I’m doing at this point, so you might want to stand back a little.
The first thing I’d like to do is figure out mutability. Obviously, for the internet to work properly, we need a way to give someone an address and have it persist across changes. Otherwise, I must somehow contact all the readers of my blog with a new address every time I update it with new content. That seems to be where IPNS comes in. It stands for Interplanetary Name System. IPNS is a PKI interface that allows mutable names anywhere in the world with no central authority. A node gets assigned a private/public keypair. The private key authorizes writes to the IPNS namespace, and the hash of the public key is the “peer id” which identifies the node and serves as the mutable name. Let’s figure out how this works in practice.
Using the “hello world” file from earlier, we run the following command to associate that file with our peer id:
$ ipfs name publish QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
Published to QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u: ipfs/ QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
That first hash is my peer id, the second is the hash of the file that was added. Now, we can get the contents of the file like this:
$ ipfs cat /ipns/QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u
hello world
You’ll notice that we replaced “/ipfs/” with “/ipns/” and replaced the file hash with my peer id. Now, lets try changing the contents of that file.
$ echo “hello again, world” > hello
$ ipfs add hello
Added QmZyLKm2tJiKuDR1qCxap1fSTHVqjTTVnBbCc8XzfXNQM1 hello
$ ipfs name publish QmZyLKm2tJiKuDR1qCxap1fSTHVqjTTVnBbCc8XzfXNQM1
Published to QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u: /ipfs/QmZyLKm2tJiKuDR1qCxap1fSTHVqjTTVnBbCc8XzfXNQM1
Now, let’s get that file again using just my peer id:
$ ipfs name resolve QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u
/ipfs/QmZyLKm2tJiKuDR1qCxap1fSTHVqjTTVnBbCc8XzfXNQM1
You’ll notice the peer id now resolves to the new hash.
$ ipfs cat /ipns/QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u
Hello again, world
So, with nothing but a peer id, you can get whatever newly updated information is in that file. Mutability achieved!
What about accessing these files from a regular web browser? IPFS comes with an HTTP gateway built in. It is accessible on the node via http://localhost:8080/ipfs/<hash> or http://localhost:8080/ipns/<peerid>. Ipfs.io also hosts a gateway on port 80 for those who don’t have their own (http://ipfs.io/ipns/<peerid> or http://ipfs.io/ipfs/<hash>).
Going to my web browser, I type in:
Localhost:8080/ipfs/QmZyLKm2tJiKuDR1qCxap1fSTHVqjTTVnBbCc8XzfXNQM1
And the webpage displays: hello again, world
Awesome.
Or, we’ll type in:
Ipfs.io/ipfs/QmZyLKm2tJiKuDR1qCxap1fSTHVqjTTVnBbCc8XzfXNQM1
And get the same thing! We are hitting the gateway at ipfs.io, so we can even do this on a computer without IPFS installed!
Finally, we can type in:
Localhost:8080/ipns/QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u
Here, we replace “ipfs” with “ipns” and replace the object’s hash with my peer id. Low and behold, the same text pops up in the browser window. This also works through the ipfs.io gateway or any other gateway you care to use. I’ve set up one computer running IPFS, and I use it as the gateway for my other computers. The gateway is blocked from outside access by default, but opening the config file and changing the “gateway” IP address from 127.0.0.1 to 0.0.0.0 will open it up.
It wasn’t clear to me when I started messing with this, but I’m starting to think that we could build a functional website with this. Let’s give it a shot.
I’ll make a directory called “blog” and put this article in it as an html file: “ipfs_first_foray.html.
Then I’ll make a second file called page2.html and link it as a relative link with the path “page2.html”. Click here to see a picture of my dog.
Finally, I add the website to ipfs and publish to ipns:
$ ipfs add -r website/
$ ipfs name publish QmZRiAJqzq2uwsX3bVYkCCfkJjxyJjFDBiA9pP4KBuQrh1
Now, you can visit this page and website at http://ipfs.io/ipns/QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u. Or, if you have IPFS running on your machine, you can get the files using "ipfs get" or view it in your browser at localhost:8080/ipns/QmasyafXpkUBor1v5QV15j5mwdwHqFf8AQVdHJyS4Q237u.
And there it is! It certainly isn’t fancy, but it’s a website with a mutable address and links to resources. Honestly, I’m quite impressed with IPFS so far. It’s clearly an alpha stage product, but for something so fundamentally different than traditional internet technologies, I’m finding it easier to use than I expected. We have only covered a small part of what can be done with IPFS, and exploring more will be the topic of future articles.
I’d like to figure out how to introduce human readable names both for ease of use and to allow interfacing with existing technologies. I’ll be investigating interfacing with traditional name services like DNS, as well as blockchain name services like Namecoin, along with anything else I discover along the way.
There will also need to be tools for building more complex websites. Traditional CMS solutions like Wordpress are out of the question as far as I know, because the way that data is stored and dynamically served will not work with IPFS. That said, data structures and messaging interfaces could certainly be developed that will step in to fill that void. I’ll be looking to see if there are projects in development to serve those needs.
Good browser support will be a must before this technology can be broadly adopted. It looks like there has already been some work in this area, and I’ll be looking into that in more detail. It will be necessary for a user to be able to run a node and browse IPFS resources without going through an HTTP gateway or using other obnoxious hacks.
Thanks for joining me! I hope you learned a little about IPFS and are as excited as I am about taking the next steps into the web of tomorrow.