Implement server peering

.gitignore

@@ -1,23 +0,0 @@
-# Allowlisting gitignore template for GO projects prevents us
-# from adding various unwanted local files, such as generated
-# files, developer configurations or IDE-specific files etc.
-#
-# Recommended: Go.AllowList.gitignore
-
-# Ignore everything
-*
-
-# But not these files...
-!/.gitignore
-
-!*.go
-!go.sum
-!go.mod
-
-!README.md
-!LICENSE
-
-# !Makefile
-
-# ...even if they are in subdirectories
-!*/

README.md

@@ -1,23 +1,28 @@
 # Kela
 
-Kela is a new decentralized web protocol, combining the best ideas from ActivityPub, email, IPFS, BitTorrent, Nostr, Secure Scuttlebutt, Yggdrasil, AT, Spritely, and more.
+Kela is a new decentralized web protocol to make it easier for anyone to build and run decentralized web applications. It's basically a mishmash of ideas from ActivityPub, SMTP email, IPFS, BitTorrent, Nostr, the AT Protocol, and Spritely, but without the headaches of those existing protocols.
 
+## Motivation
 
-## Motivations
+One popular decentralized web protocol these days is ActivityPub, which is used by Mastodon and many other applications. ActivityPub is an incredibly flexible protocol, but it also has plenty of flaws. It doesn't have replicated, fault-tolerant storage so your data can be lost due to a server shutdown or ban. Also, usernames in ActivityPub include the domain name of your server, so you can't easily migrate your account from one server to another. Implementing ActivityPub is no fun either since it uses some complicated Semantic Web stuff.
 
-The web sucks. Today's web is dominated by a few gigantic walled gardens locking in billions of users, while tiny decentralized web projects fail to gain any traction. Recently, the ActivityPub federation protocol has seen some success with Mastodon, but it suffers from many problems. For instance, identity in ActivityPub is strongly tied to domain names, which complicates migrating accounts between servers and making ActivityPub vulnerable to sudden server shutdowns or bans. In addition, ActivityPub uses ActivityStreams 2.0 under the surface, which is both a JSON-LD modeling protocol and a great way to get lost in the Semantic Web rabbit hole of horrible complexity. In short, implementing ActivityPub isn't fun, and you don't even get a great federated network once you're done! Similarly, peer-to-peer networks like IPFS or Secure Scuttlebutt are also complicated, but we can fortunately simplify things by introducing hub nodes. Nostr shares many common ideas with Kela, but Kela takes them a step farther without introducing much more complexity.
+Nostr is another popular decentralized web protocol, except it doesn't really work. Instead of having a name resolution system, in Nostr you just try messaging a bunch of servers that might have information about a specific user and hope for the best. Also, Nostr is focused primarily on messaging and isn't suitable for building decentralized web applications.
 
+Purely peer-to-peer protocols like IPFS suffer from being way too complex and slow, so that's no good either.
 
-## Ideals
+## Design
 
-Kela strives for these ideals:
-- **Simple**: Many decentralized web protocols are horrendously complex (I'm looking at you, Urbit!). Kela is simple and can be taught to someone in 30 minutes.
-- **Powerful**: Kela is flexible and can handle all sorts of applications, as well as being agnostic to the underlying protocol (usually HTTP).
-- **Fast**: Kela uses aggressive caching and minimizes the number of hops between user-to-user connections when possible.
-- **Secure**: Strong cryptography ensures that all connections in Kela are end-to-end encrypted.
+Alright, let's solve all those problems listed above! Kela consists of three components, a name resolution system using a DHT, a messaging service, and a storage service.
 
+In Kela, each user has an ID, which is a public key. Each user is associated with one or more Kela servers, which store that user's data. To find out which servers a user is associated with, you can query the name resolution system. All Kela servers participate in the name resolution system and act as DHT nodes. Each server stores a complete list of all DHT nodes. When a new server joins the DHT, it tries to peer with an existing server in the DHT. Say server `example.com` would like to peer with `test.net`. `example.com` first sends a GET request to `test.net/peer?peer=example.com`. `test.net` replies with its list of DHT nodes. Once `example.com` receives this reply, it adds `test.net` to its list of DHT nodes and attempts to peer with all servers in the reply that it hasn't peered with yet. `test.net` now also tries to peer with the server that just contacted it, in this case `example.com`. Servers periodically go through their list of DHT nodes and remove nodes that are no longer online.
 
-## Identity
+DHT get/set TODO
+
+Messaging TODO
+
+Storage TODO
+
+Old stuff:
 
 Let's start where ActivityPub went wrong: identity. ActivityPub uses usernames plus the instance URL (for instance, `billiam@example.com`) as identifiers. This ties your identity strongly to a server, but it's actually not necessary. In Kela, each user is still associated with one (or more) servers, but public keys are identifiers. Each user has a public and private key, and when you want to find a user's server, you query a [DHT](https://en.wikipedia.org/wiki/Distributed_hash_table) formed by all servers. This returns a string containing their server's URL or some other address, signed with that user's private key to prevent tampering.
 

client/client.go

@@ -1,4 +1,4 @@
-package client
+package main
 
 import "fmt"
 

server/server.go

@@ -1,17 +1,115 @@
 package main
 
 import (
+	"crypto/sha256"
+	"encoding/hex"
+	"flag"
 	"fmt"
-	"html"
+	"io"
 	"log"
 	"net/http"
+	"sort"
+	"strings"
+	"sync"
 )
 
-func handler(w http.ResponseWriter, r *http.Request) {
-	fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
+var mu sync.Mutex
+var me string
+var hashToDomain map[string]string
+var peerHashes []string
+var kvstore map[string]string
+
+func sha256sum(s string) string {
+	// Get the sha256sum of string as a hex string
+	b := sha256.Sum256([]byte(s))
+	return hex.EncodeToString(b[:])
+}
+
+func addPeer(peer string) error {
+	// Try to peer with another server
+	peerHash := sha256sum(peer)
+	// Check if already peered
+	mu.Lock()
+	_, ok := hashToDomain[peerHash]
+	mu.Unlock()
+	if ok {
+		return nil
+	}
+	mu.Lock()
+	hashToDomain[peerHash] = peer
+	mu.Unlock()
+
+	// Try request to peer
+	log.Printf("%s trying to peer with %s", me, peer)
+	resp, err := http.Get(peer + "/peer?peer=" + me)
+	if err != nil {
+		// Request failed, delete peer
+		mu.Lock()
+		delete(hashToDomain, peerHash)
+		mu.Unlock()
+		return err
+	}
+
+	log.Printf("%s successfully peered with %s", me, peer)
+	mu.Lock()
+	peerHashes = append(peerHashes, peerHash)
+	sort.Sort(sort.StringSlice(peerHashes))
+	mu.Unlock()
+	// Read response body
+	defer resp.Body.Close()
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return err
+	}
+	// Try adding all peers of this peer
+	newPeers := strings.Split(string(body), "\n")
+	for _, newPeer := range newPeers[:len(newPeers)-1] {
+		go addPeer(newPeer)
+	}
+	return nil
+}
+
+func peerHandler(w http.ResponseWriter, r *http.Request) {
+	// Handle incoming peer requests
+	r.ParseForm()
+	peer := r.Form.Get("peer")
+	if peer == "" {
+		w.WriteHeader(http.StatusBadRequest)
+		return
+	}
+	go addPeer(peer)
+	for _, p := range hashToDomain {
+		fmt.Fprintf(w, "%s\n", p)
+	}
+}
+
+func getHandler(w http.ResponseWriter, r *http.Request) {
+
+}
+
+func setHandler(w http.ResponseWriter, r *http.Request) {
+
 }
 
 func main() {
-	http.HandleFunc("/", handler)
-	log.Fatal(http.ListenAndServe(":8080", nil))
+	bindAddr := flag.String("b", ":4200", "bind address")
+	domain := flag.String("d", "http://localhost:4200", "full domain name")
+	peer := flag.String("i", "", "initial peer")
+	flag.Parse()
+
+	log.Printf("Starting %s %s %s", *bindAddr, *domain, *peer)
+
+	// Record myself
+	me = *domain
+	peerHashes = append(peerHashes, sha256sum(me))
+	hashToDomain = map[string]string{peerHashes[0]: me}
+
+	if *peer != "" {
+		go addPeer(*peer)
+	}
+
+	http.HandleFunc("/peer", peerHandler)
+	http.HandleFunc("/get", getHandler)
+	http.HandleFunc("/set", setHandler)
+	log.Fatal(http.ListenAndServe(*bindAddr, nil))
 }

server/test.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+trap "kill 0" EXIT
+go build
+./server -b :4200 -d http://localhost:4200 &
+for i in $(seq 1 9)
+do
+    sleep 0.1
+	./server -b :420$i -d http://localhost:420$i -i http://localhost:420$((i-1)) &
+done
+wait