ok...Ill give a bit of background.
So, the current internet, is not actually the internet. Its a star network topology. This is highly inefficient....for every connected computer (i.e. your home laptop), if you want to connect to the person in the house next to you, or your friends down the road, your data travels thousands of miles (USA). There needs to be infrastructure between your laptop, and the data centre, and back to your friends house.
An edge computing network, takes a more direct route, as long as there is a "true" internet. So you hop along routers, or microwave repeaters, directly. The technology to support this has been round for the last two decades but due to the requirement of the big corps harvesting all sorts of metrics about you, people assume that the internet we have, is actually really great. Its only in the last few years where we have all realised that putting responsibility into the hands of big corps is a really, really bad idea. Given the energy market, current climate on the data protection issues, child safety, freedom of speech etc the list goes on...it's time we took back the internet....or should I say, replace the "internet".....
with the internet
About 20 years ago, I did a piece of work for a Charity to investigate the instant messenger apps, that were available back then. AOL, Microsoft, Yahoo, were sat round the table, at the Internet Select Security Committee, at the Home Office in the UK, where I was presenting my findings. We were looking at the feasibility of IM app providers implementing logging into their apps, to help protect children and all of the companies said it would be too expensive to implement. Lets be clear, this was more about the companies looking bad when the police find evidence and then the media getting hold of the logs and publishing the companies names. The negative media about that, ever since, has largely been absent from the news. Until recently. The costs involved in setting that up were millionths of percentages of their annual profits.
If we look at the take down of content from people with controversial views, albeit sometimes unpleasant, the very setup of the internet as it currently is allows people to hide. An edge computing platform requires that the content creators, host their own content. So, anything illegal is simply trivial to find. Someone's opinion that's unpopular? Well...Id rather have free speech and happily ignore the "idiots" knowing that the laws of a country can be enforced without a big corp having the capacity to control the content. The community rules a system like this, from the bottom up.
Currently 30% of the worlds energy is devoted to the internet. That's a big claim, but if you then add in the manufacture and energy required to make this equipment that figure quickly becomes very conservative. I'll let you do your own research on those numbers. Id be interested to hear from anyone who doesn't agree.
I've spent the last 20 years, working in and around engineering that works with big systems to drive application development. Often you might find an SME, will leverage external services...like AWS or Azure or another big cloud service provider. A big enterprise however, won't. The value in providing a system to a company that has no dependencies on an external service provider will not only protect that enterprise from corporate manoeuvres to cause problems for the other organisation but it also offers the organisation the protection from another service provider copying their core business offering. If you can offer a space for an enterprise to run their systems, which they control, and the hardware is distributed...they have no option but to either use it, or wait for their competitor to use it. The capacity of a system like this scales faster than the use. So 1 user, 1x speed. 10 users, 100x speed. That is a vary naïve simplification but for every user who joins, they offer more capacity than they use.
So let's deal with your questions....
Q: What do you believe is the use-case for heavy compute nodes distributed to homes?
A: This is a big question. And has many possibilities. if we look at the current blockchain tech, it's largely already past the point of being viable to do anything useful. The blockchain gets so large it becomes more of a storage and processing problem to become a node. If we jump 20 years into the future, a 15TB blockchain which has to be validated and then stored, distributed...suddenly looks rather clunky. Then if we look at the offerings the cloud service providers offer. A mini EC2 instance, is about the same as a single core of a Raspberry Pi...and with a Pi you get 8x more memory. At £20 per month, for an EC2 instance of the same size, if you could offer domestic customers access to an income stream like that, for every PI they host, they could earn far more per k/W of electric than the blockchain, if we use the cloud service providers costings as a basis. The side effects of that, mean for every PI, we don't need the air conditioning, space, server manufacture etc. The efficiency of this system is a dramatic advantage...plus you don't need the infrastructure to take the data 1000s of miles on a round trip.
We already have in part this happening right now. You all have a broadband router right? The only thing you'd need to do to jump onto the system I've been working on is to open a guest WIFI with SSID, username and password and allow people to connect to your PIs. And it uses all the same tech, encryption that already exists...and protects military equipment, so for the data encryption haters out there who say the system isn't secure...well...I'm kind of on the right side of the academic research and understanding there. There are quite a few people who argue that case.
Q: Why do you believe that a 100 node (or any size node) cluster is useful in an edge computing ? Very few software packages work well in clusters.
A: The internet is a cluster. It's just, a little bit disorganised to work well with a cluster, inside a cluster. However, if you asked Facebook for their code...it would run perfectly well on about 20 Pis. If you strip the advertising AI that is. In fact, if you build a system that has all of the components of the internet...so DNS, VPN, DHCP, Code mirrors, CI, CD... you can run and scale any open source project. This is what Amazon do. They leverage open source software, and scale that as a company. The bit they own is the glue, to make these services work on their infrastructure, inside their data centres. If you're talking about an SMEs code base, yes, it's not designed to work on a cluster, housed in a data centre. Which is kind of funny...the irony of it. That SME code tho, would work perfectly well on the system I'm building.
In short, the IT INDUSTRY...is designed to reinvent the wheel, over and over. Its an industry. If it didn't make any money...there wouldn't be the money to pay for the infrastructure and data centres needed to make it all work. We pay for this with our internet subscriptions and mobile phone contracts. We don't even think about it. If I was to turn around to you and say, ok, host 100Pis and we will pay you, and your internet bills, if you allow people to access a guest Wi-Fi, would you say no? If it was profitable and maintenance free? I've haven't met anyone so far who says no to that question. If you add in VOIP and say you can also stop paying your mobile phone bill as well. It's a no brainer.
It needs a critical mass to work out. There is a growing number of mesh networks now, mainly in the US where these network provide remote areas with internet access because the big telco's don't want to spend the money on scaling into those costly markets. This type of network is truer to the definition of the internet but lacks the processing capacity and systems to actually run software applications. Things are changing. A mobile phone has enough processing power to run an ecommerce site with 1000 connected users. If you joined this system, into their mesh network, they would become the silicon valley.
This tech, supersedes block chain in every way and is likely to be the future of the internet. The advantage of this sytem is even if someone copies it, the systems are compatible.
So, to finish the answer on this, if you have 100Pis, 400cores and 800GB of ram sitting at your disposal, then you stand to spent 10K, and earn let's say £2000 per month from it in an ideal scenario. And that would fit into the size of a large desktop printer. Block chain can't beat that for spend vs reward.
To get to that stage, we just need to provide financial incentive for node owners and a financial reduction for business that is already spending lots of money on their cloud systems.
Q: What software have you personally used that worked well in a cluster?
I partially answered that question above...but I've written a lot of systems designed around this tech. Its not about the software, it's about the open source software that's already available. Like I've said above, if you can run the infrastructure systems, to provide the deployments and name spacing required, the cluster is largely a transport mechanism...the software will run quite happily. The tricky bit admittedly, is getting the software to the node closest to the consumer. This is where I've spent a lot of time working towards that.
I'm also building a drag and drop platform which would allow your granny to build an invoicing system, or ecommerce site that you can run yourself and would be available on the current internet. If business can drive this deployment, with minimal outlay and it works with their existing systems...any place there is a business, corner shop, supermarket, there would be the capacity for that business to save money, and allow other nodes to join close by, and so on.
The saying is that through 7 people you know the world. If one person in 1000 joined this network, you'd probably have close to 100% coverage of the planet. If 1 in 7 joined, we could quadruple our current processing capacity and reduce the energy use of the planet by some number..those numbers are however close to impossible to predict.
I hope that answered your questions and maybe there might be more questions now than answers but thankyou for asking and I appreciate the internet...I mean interest
That was a ironic, convenient, lucky typo!
Facebook
Twitter
Instagram