An expert article by Ben Teitelbaum, Head of Engineering.
NewNode is a network that connects mobile users with each other and with content when there are disruptions to internet connectivity. It can provide a degree of connectivity even to phones that are completely offline. NewNode is made possible by several new technologies, but is also part of a long tradition of networking innovation in its use of an overlay network.
The Internet as an Overlay Network
Like all networks, overlay networks consist of a set of nodes connected by a set of links. In an overlay network, however, both are virtualized. Nodes are assigned new identifiers and links are logical connections between nodes that are implemented over one or more underlying networks.
Private corporate networks that unify distant branch offices on a single virtual private network (VPN) are implemented as overlay networks, as are peer-to-peer networks like BitTorrent and IPFS.
The Internet itself emerged as an overlay network. The dominant electronic network at the time was the telephone network, which since its beginnings in the 1870s, was circuit-switched. When a call was placed across this network, a circuit of copper was physically established between two phones. Initially, human operators connected patch cables on circuit boards to set up these connections and later the process was automated through switches triggered by dialing.
Point-to-point digital communication was possible since the 1930s, first through the Telex teleprinter network and later through fax machines and modems, which overlaid primitive digital communication across the analog public switched telephone network (PSTN). These connections were fragile. Any failure along the path of a call could cause the call to drop. They were also expensive, as they were available through the regulated, monopoly telephone company (“Ma Bell” in the United States).
The Internet was born of two founding ideas. First, that of a resilient packet-switched network that could route around failed equipment or links. And second, that of an internetwork (or network of networks) that allowed separately owned and administered networks to inter-operate over open protocols. These two ideas fundamentally changed the cost and scalability of networking, fueling decades of growth in the digital economy.
What allowed the Internet to get off the ground was that it was launched as an overlay network. The early Internet (including the ARPANET and the NSFNET) was layered over leased lines from the phone company. By harnessing the existing infrastructure of the phone network to get from point A to point B, the pioneers of the Internet were free to innovate at the nodes and at the networking and transport layers. Custom routing hardware was built and a flurry of experimentation occurred with protocols such as TCP/IP.
In time, the Internet was so successful that it came to dwarf the size of the terrestrial phone system and an inversion occurred: the phone system was redesigned to be layered over IP (Internet Protocol), even at its core.
How Did the Internet Take Over?
Using existing networks to bootstrap the Internet wasn’t just a technical decision—business and political factors also influenced this setup. Telcos (telephone companies) were politically powerful, regulated monopolies that sought to stifle innovation and competition to retain their market power.
For a hundred years the user experience remained essentially the same: the phone call. Calls could easily be tapped by law enforcement agencies and regularly were, sometimes illegally and abusively. Everything was metered and charged back to the customer. Long distance, and especially international, calls were expensive and rare. And, it was literally illegal to plug anything into your phone line other than an approved handset that you were compelled to rent from the phone company.
The Internet unleashed an unprecedented explosion of innovation, freedom, and competition that unseated telco monopolies. A best-efforts service model allowed IP to be implemented over any underlying network or link layer technology and an open protocol stack created a permissionless platform for innovation. Peer-to-peer applications like Napster, Skype, and BitTorrent could achieve huge scale quickly through distribution as free software. And, after a significant tussle with the law-enforcement and intelligence communities in the 1990s, consumer applications could use strong end-to-end encryption to secure communications and ensure privacy even against sophisticated and well-funded attacks. Any startup or sufficiently skilled individual could build a secure website or communications app without the need to ask permission from a central, controlling authority.
History Repeats Itself
For a time, it seemed as if the freedom of the Internet could not be contained. However, the tussle between distributed open communications and centralized control was not over.
Certain governments, ISPs, and other centralized institutions made heavy investments in technologies to control their users’ access to information and open communications. In parts of the Internet there are now sophisticated, high-performance firewalls that perform stateful deep packet inspection and high-speed traffic analysis. These firewalls can not only block access to banned content and websites, but also actively interfere with targeted apps by generating spoofed packets designed to confuse their protocols.
In response to this reassertion of central control, new overlays have begun to appear over existing internet infrastructure. Like the original internet pioneers who sidestepped central control by building a network of computers overlaid on the PSTN, modern-day internet innovators make use of increasingly powerful phones, computers, and cryptography to build more resilient new networks on top of the Internet itself. This is precisely what NewNode is doing.
NewNode: a Peer to Peer Overlay Network
NewNode is a P2P (peer to peer) overlay network, which means that every device is used as both an endpoint and a router. When a user device requests an online resource, NewNode first attempts to send the request through normal channels. If there is no IP access—whether due to censorship or a spotty internet connection—NewNode searches for a device in its network that does have access, and routes the request through that device. If a NewNode peer anywhere has access to the requested resource and can be reached, it will proxy the request and forward the content back to the requesting user. To find nodes, NewNode leverages the BitTorrent distributed hash table (DHT), a massive distributed data structure with tens of millions of users.
There are three products that make use of and grow the NewNode network: the NewNode messenger, a highly resilient messaging app; NewNode VPN, an app that seamlessly proxies access to web resources when they can’t be reached directly; and NewNode Kit, which can be used by app developers to give their apps and users the reliant connectivity of the NewNode network.
Products like NewNode continue historical trends by providing more resiliency and flexibility to the structure of the internet.