Hyphanet

For other uses, see Freenet (disambiguation)

File:Freenet screenshot the freedom engine jan06.png

Screenshot showing a 'freesite', a freenet-hosted page. The page displayed is an indexing site called "The Freedom Engine".

Freenet is a decentralized censorship-resistant peer-to-peer distributed data store. Freenet works by pooling the contributed bandwidth and storage space of member computers to allow users to anonymously publish or retrieve various kinds of information. Freenet uses a kind of key based routing similar to a distributed hash table to locate peers' data.

Freenet is currently under development, and a version 1.0 has not yet been released. Freenet is considered by many to be fundamentally different from other peer-to-peer networks; it is still somewhat more difficult to use and slower, and it does not have integrated search functionality. Currently, Freenet cannot be used to create or distribute dynamic content, such as content that utilizes databases and scripting. According to the Freenet Project group, such tradeoffs are expected since Freenet's primary goals are neither ease-of-use nor performance. Unlike other peer-to-peer networks, Freenet is primarily intended to combat censorship and allow people to communicate freely and with near-total anonymity.

Purpose

Although many nations censor communications to different extents, they all share one commonality in that a body must decide what information to censor and what information to allow. What may be acceptable to one group of people may be considered offensive or even dangerous to another. Freenet is a network which, putatively, removes the possibility of any group imposing their beliefs or values on any other. In essence nobody is allowed to decide what is acceptable for anybody else. Tolerance for each others' values is encouraged and failing that, the user is asked to turn a blind eye to content which opposes his or her views.

Technical design

The type of routing protocol Freenet uses is key based routing. While the idea emerged independently, Freenet's routing algorithm is similar to that employed by distributed hash tables (DHTs). The main differences are that Freenet nodes do not have fixed specialisations, and the routing algorithm is heuristic in nature. Therefore, it does not guarantee that it will find a given piece of data. Freenet can also be viewed as a small world network.

The Freenet file sharing network is designed to be highly survivable, with all internal processes completely anonymized and decentralized across the network. The system has no central servers, is peer-to-peer, and is not subject to the control of any one individual or organization. Even the designers of Freenet do not have any control over the overall system. The system is designed so that information stored in the system is encrypted and replicated across a large number of continuously-changing anonymized computers around the world. It is extremely difficult for an attacker to find out which participants are hosting a given file, since the contents of each file are encrypted, and can also be broken into sections that are distributed over many different computers. Even the participants themselves don't know what they are storing.

The end goal of the Freenet network is to store documents and allow them to be retrieved later by an associated key, as is now possible with protocols such as HTTP. The network is implemented as a number of nodes that pass messages among themselves peer-to-peer. Typically, a host computer on the network will run the software that acts as a node, and it will connect to other hosts running that same software to form a large distributed network of peer nodes. Certain nodes will be end user nodes, from which documents will be requested and presented to the human user. But these nodes communicate with each other and with intermediate routing nodes identically—there are no dedicated "clients" or "servers" on the network.

The Freenet protocol is intended to be implemented on a network with a complex network topology, such as the Internet (Internet Protocol). Each node knows only about some number of other nodes that it can reach directly (its conceptual "neighbors"), but any node can be a neighbor to any other; there is no hierarchy or other structure. Each document (or other message such as a document request) in Freenet is routed through the network by passing from neighbor to neighbor until reaching its destination. As each node passes a document to its neighbor, it does not know or care whether its neighbor is just another routing node forwarding information on behalf of another, whether it is the source of the document being passed, or whether it is a user node that will present the document to an end user. This is intentional, so that anonymity of both users and publishers can be protected.

Each node maintains a data store containing documents associated with keys, and a routing table associating nodes with records of their performance in retrieving different keys.

To find a document in the network given a key, a user sends a message to a node (probably one running on the same machine as the client program) requesting the document, providing it with the key. If the document is not found in the local data store, the node then finds the node in its routing table that it thinks will be able to locate the key most quickly, and forwards the request to that node, remembering that it has done so. Note that this is a change from the behavior of earlier versions of Freenet nodes: it represents the "Next Generation Routing" protocol. The old behavior was to remember which keys were retrieved from what nodes, and to route based on which node gave us the key closest to the one we were looking for. The effect is largely the same, but NGR, as it is called, results in better overall performance.

The node to which the request was forwarded repeats the process until either the key is found or the request passes through a set maximum of nodes, known as the "Hops To Live" value. Along the route, if a node is visited more than once (and it will know this because it remembered forwarding the request the first time) then that node cuts off the loop by sending a message to the node that sent it the second request telling it to try the next-best choice, then the next-next-best, and so on.

Eventually, either the document is found or the hop limit is exceeded. The terminal node sends a reply that works its way back to the originator along the route specified by the intermediate nodes' records of pending requests. The intermediate nodes may choose to cache the document along the way. Besides saving bandwidth, this also makes documents harder to censor as there is no one "source node."

Essentially, the same path-finding process is used to insert a document into the network: a request for the nonexistent document is made, and once it fails, the document is sent along the same path as the request. This ensures that documents are inserted into the network in the same place as requests will look for it. If the initial request doesn't fail, then the data already existed, and the insert "collides."

Initially, each node has no information about the performance of the other nodes it knows about. This means that routing of requests will be essentially random. But since different nodes have different randomness, they will disagree about where to send a request, given a key. So the data in a newly-started Freenet will therefore be distributed somewhat randomly.

As more documents are inserted by the same node, they will begin to cluster with data items whose keys are similar, because the same routing rules are used for all of them. More importantly, as data items and requests from different nodes "cross paths", they will begin to share clustering information as well.

The result is that the network will self-organize into a distributed, clustered structure where nodes tend to hold data items that are close together in key space. There will probably be multiple such clusters throughout the network, any given document being replicated numerous times, depending on how much it is used. This is a kind of "spontaneous symmetry breaking", in which an initially symmetric state (all nodes being the same, with random initial keys for each other) leads to a highly asymmetric situation, with nodes coming to specialize in data that has closely related keys.

There are forces which tend to cause clustering (shared closeness data spreads throughout the network), and forces that tend to break up clusters (local caching of commonly used data). These forces will be different depending on how often data is used, so that seldom-used data will tend to be on just a few nodes which specialize in providing that data, and frequently used items will be spread widely throughout the network. This automatic mirroring counteracts the times when web traffic becomes overloaded, and due to a mature network's intelligent routing a network of size n should only require log(n) time to retrieve any given document. Freenet does not employ broadcast searches as used by Gnutella and other similar file sharing protocols.

One thing to keep in mind is that keys are hashes, hence there is no notion of semantic closeness when speaking of key closeness. Therefore there will be no correlation between key closeness and similar popularity of data as there might be if keys did exhibit some semantic meaning, thus avoiding bottlenecks caused by popular subjects.

There are two main varieties of keys in use on Freenet, the Content Hash Key (CHK) and the Signed Subspace Key (SSK).

A CHK is an SHA-1 hash of a document and thus a node can check that the document returned is correct by hashing it and checking the digest against the key. This key contains the meat of the data on freenet. It carries all the binary data building blocks for the content to be delivered to the client for reassembly and decryption. The CHK is unique by nature and provides tamperproof content. A hostile node altering the data under a CHK will immediately be detected by the next node or the client. CHKs also reduce the redundancy of data since the same data will have the same CHK.

SSKs are based on public-key cryptography. Currently Freenet uses the DSA system as its public key infrastructure. Documents inserted under SSKs are signed by the inserter, and this signature can be verified by every node to ensure that the data is not tampered with. SSKs can be used to establish a verifiable pseudonymous identity on Freenet, and allow for documents to be updated securely by the person who inserted them. A subtype of the SSK is the Keyword Signed Key, or KSK, in which the key pair is generated in a standard way from a simple human-readable string. Inserting a document using a KSK allows the document to be retrieved and decrypted if and only if the requester knows the human-readable string; this allows for more convenient (but less secure) URIs for users to refer to.

Scalability

A P2P network is said to be scalable if the performance of the network does not deteriorate even for very large network sizes. The scalability of Freenet is being evaluated, but similar architectures have been shown to scale logarithmically [1].

History

Freenet is an enhanced free software / open source software implementation of the system described by Ian Clarke's July 1999 paper "A distributed decentralized information storage and retrieval system", written while Clarke was a student at the University of Edinburgh. Shortly after the publication of this paper, Clarke and a small number of volunteers began work on what became Freenet. By March 2000 version 0.1 of Freenet was ready for release. Since March 2000 Freenet has been extensively reported on in the press, albeit primarily due to its implications for copyright rather than for its wider aim of freedom of communication.

Freenet has been developed via a collaborative, open source, methodology. Clarke, originally from Ireland, worked with individuals from many other countries, in a distributed manner utilizing the Internet. Not only is Freenet itself distributed, decentralized Internet software, but the method used to develop it is also distributed and decentralized.

Reports of Freenet's use in authoritarian nations is difficult to track due to the very nature of Freenet's goals. One group, Freenet-China, has translated the Freenet software to Chinese and is distributing it within China on CD and floppy disk.

According to Citeseer, Ian Clarke's "Freenet: A Distributed Anonymous Information Storage and Retrieval System" was the most cited computer science paper of 2000.

One analysis^[2] of Freenet files conducted in the year 2000 claimed that the top 3 types of files contained in Freenet were text (37%), audio (21%), and images (14%). 59% of all the text files were drug-related, 71% of all audio files were rock music, and 89% of all images were pornographic. It is important to note the fundamental design of Freenet makes accurate analysis of its content difficult. This analysis was done several years ago from within the United States, and the network has been vastly changed and expanded since it was published and many different types of content have been added.

Current development

Freenet is currently undergoing a major re-write incorporating a number of fundamental changes. Version 0.7 of Freenet aims to create a scalable darknet, where users only connect directly to other users they know and trust. The purpose of this change is to protect users who may be placed at risk simply by using the software, irrespective of what they are using it for. In the new model, users will choose who they connect to, and only those users will know that they are running the software. Previous darknets, such as WASTE, have been limited to relatively small disconnected networks. The core innovation in Freenet 0.7 will be to allow a globally scalable darknet, capable of supporting millions of users. This is made possible by the fact that human relationships tend to form small-world networks, a property that can be exploited to find short paths between any two people. The work is based on a speech given at DEF CON 13 by Clarke and Swedish mathematician Oskar Sandberg.

Other modifications include switching from TCP to UDP, which allows UDP hole punching along with faster transmission of messages between peers in the network. While previously Freenet only supported the insertion and retrieval of information, Freenet 0.7 will support new modes of usage including the real-time broadcast of messages. Features planned for later versions include anonymous "channels" to a particular node allowing for dynamic content, searching, and Onion Routing. Applications of this range from Internet Relay Chat to RSS-feeds.

It is anticipated that Freenet 0.7 will be released in 2006.

Controversy

The same technology which allows the oppressed to communicate with a large group, without either the publisher or the readers' identities being revealed, can also allow controversial information such as drug related information or child pornography to be made available to anyone. Freenet's founders point out that only with true anonymity comes true freedom of speech, and that what they view as the beneficial uses of Freenet outweigh its negative uses. Due to the nature of Freenet a typical user may unknowingly host this sort of information, which may hypothetically make them subject to severe civil and criminal penalties. Freenet attempts to prevent this through "plausible deniability", preventing the user himself from knowing what's on his own node and making it difficult to determine if a piece of information is in any given node without causing the distribution of that piece of information throughout the network to change in the process. No court cases have tested any of this to date.

Some anonymous friend-to-friend (F2F) networks do allow you to control what kind of files your friends exchange with your node in order to stop them from exchanging files you disapprove of but Freenet's "deniability" defence would not apply to users of these systems.

F2F prevents random people from proving that your IP address can effectively be used to get some controversial files. Freenet does not offer this protection because for efficiency reasons (path shortening) some random nodes are allowed to connect directly to your node, thus exchanging files faster, but thus knowing your IP and thus being able to prove that specific files can be obtained from your computer. However, due to Freenet's "plausible deniability" and the way in which Freenet redistributes files among nodes, one cannot prove that those files were placed there by the node owner or that the node owner knows what they are.

Project related criticism

Some users have criticized the development process used by the Freenet project, which they claim has lead to poor human resource management, many delays, and poor development decisions. They claim this has resulted in a program that is hard to get working for regular users and behaves unreliably. There are also many features they think should be in place after so many years of development, but are not. They claim the Freenet project is not open enough to outside influences from other projects or non-developer participation.

Related tools

Freesite Insertion Wizard (FIW)

FIW is one of the most popular of all the Freenet tools. It is currently the best maintained insertion tool for freesites. It is written in Java and provides both a terminal interface and a GUI. It supports the latest Freenet technologies such as containers as well.

Fishtools

Another extremely popular tool for inserting freesites. However, due to an argument, its main developer has broken off with the Freenet community, and it is no longer maintained. It is written in Python and it is still widely used and fairly up to date.

Freenet Utility for Queued Inserts and Downloads (FUQID)

This is a Windows-only tool (although reports say it works under WINE too), typically used to retrieve large splitfiles and to insert non-Freesite content such as binaries, audio, and archives. It is written in Delphi, and it is maintained sporadically. It is generally regarded as an excellent complement to either FIW or Fishtools.

FCPTools

The FCPTools are command-line driven programs for inserting and retrieving files with Freenet. Included separately is FCPLib, the Freenet Client Protocol Library. The FCPTools are linked against FCPLib and serve as nice examples for using the library in Freenet client programs.

FCPLib (Freenet Client Protocol Library) aims to be a platform independent but natively compiled set of C-based functions for storing and retrieving information to and from Freenet. There are routines for storing documents to Freenet from the local disk, and other routines for moving data in memory to and from Freenet.

Everything is released under the GNU General Public License.

FCPLib is now routinely compiled on the following platforms: Microsoft Windows (NT/2K/XP), Debian GNU/Linux, BSD, Solaris, and Mac OS X.

Freenet Tools

The Freenet Tools perform roughly the same tasks as FCPTools, however it does not include a client library for use in other projects. It is written in ANSI C, and runs under *nix.

Frost

Frost is a well-maintained and popular message board system for Freenet. It uses a web of trust concept as well as public and private keys to prevent excess spam. It is very popular in the area of file sharing and is the most widely used Freenet messaging application. For a long time many people preferred Freenet Message Board (FMB) but it has fallen into disuse. It is written in Java and therefore works on any platform where a JVM is available, such as Windows, Mac OS X, Linux and other operating systems.

Freenet Message Board (FMB)

The Freenet Message Board is the most branched message board software for Freenet ever coded. It is written in Java, and the original author does not maintain it. However, the community continues to provide several branches which are generally up to date. It works on both *nix and Windows.

See also

• Anonymous P2P
• Crypto-anarchism
• Cypherpunk
• distributed file system
• Entropy (anonymous data store)
• Friend-to-friend networks
• GNUnet
• I2P
• Tor (anonymous network)

External links

• The Freenet Project
• The Freenet Help Site
• China News on Freenet - Site in Chinese
• FreenetWiki - Wiki devoted to Freenet, suitable for technical discussions and content]
• ^ What's On Freenet? - An analysis of the types of files contained in Freenet
• Piracy and peer-to-peer - An online debate between Ian Clarke, Freenet's creator, and Matt Oppenheim, the RIAA's senior vice president of business and legal affairs
• DEF CON 13 darknet slides by Oskar Sandberg and Ian Clarke