I took this server, with four Raspberry Pis inside...
...and brought it a hundred miles away to my cousin's farm!
[cows mooing]
This is a Turing Pi 2 cluster, and it's hosting my website, live on the Internet, completely
off-grid!
And it's freezing cold out here.
Why are we doing this outside??
It's a bad idea.
But.
It just shows that I'm still connected to the Internet.
So I'm harnessing the power of the sun and using a 4G LTE modem to connect this Pi cluster
to the Internet!
I can literally pick up this battery and server and go anywhere without being bound by the
shackles of the grid!
But I had to solve a lotta problems to get here.
How do I keep it powered on?
How do I connect it to the Internet?
[zip up case] How can I get my website to run on it?
Well, buckle up, cuz we're gonna solve all those problems.
But first, the most important problem: how do I pay for all this?
Well, check out this video's sponsor, Ecoflow!
Ecoflow builds some of the best battery power stations on the market, from the Delta Pro
that could run your whole house, to the River mini, great for portable power wherever you
go.
Their power stations have some of the fastest charging times on the market, they're lightweight
and rugged, and they also sell a variety of solar panels to solve all your energy needs,
whether you're off-grid, or need backup power for your home!
You can find out more about Ecoflow in the link in this video's description.
So the biggest problem for my off-grid cluster is power.
Most servers you'd use in a data center eat up hundreds of watts all day long.
But even this server, a tiny little cluster of four Raspberry Pis, uses about 20 watts
when it's running.
That might not seem like much—heck there are even small solar panels that can put out
50 or 100 watts.
But if you want this thing to stay up 24-7... you're gonna need a battery.
And that's why I called up Ecoflow.
This battery might look like overkill, but if I'm gonna host my website off-grid, I need
it to stay up—even if there's a big storm or a few overcast days that block out the
sun.
And—speaking of overkill—Ecoflow also sent two of these, their massive 400 watt
portable solar panels.
Together they absorb eight hundred watts on a sunny day and charge up the battery in a
little over two hours!
They're not really overkill, though, because solar power's always a little funky—the
angle of the sun could be off, and even a slight haze will affect how much power they
put out.
Plus, well, there's nighttime, when there's no sunlight at all!
And right now, during winter, there are only a few hours when I can get full power.
So I've been testing this setup, and I can pull in enough power to run the cluster and
charge the battery, even on an overcast day.
And the battery has enough capacity to run the cluster for at least two days even if
the sun is completely blocked by storm clouds or hazy weather.
So I can run the cluster indefinitely, at least for a few years until the battery on
this thing wears out.
Power was a surprisingly easy problem to solve, but the next one turned out harder.
And that problem is Internet access.
If you wanna go truly portable, you don't have a lotta options.
Out here, on this farm, I asked my cousin Annie what kinda speeds she gets with her
DSL, and, well...
With DSL we are crawling, at 3.72 Megabits per second download speeds, and a whopping
0.36 Megabits per second upload speeds.
So not ideal for work.
Getting a new fiber or cable line laid out here would cost a ton of money, and there
are a lotta places on the planet where wired options just aren't possible.
Like what if you're driving on the road, and you wanna run your website from your car?
So wireless it is!
I could try to get satellite, but there are a couple problems: Satellite receivers aren't
usually portable.
Plus, the best option for upload speeds and latency—Starlink—isn't available here.
So I chose to go with 4G LTE.
In most parts of the world, you can get 4G or 5G wireless data, even if you're miles
away from civilization.
It's not the perfect solution for everyone, but it's good enough.
And performance isn't too bad.
Heck, in more densely populated areas, you might be able to get better upload speeds
through cell towers than you can through wired options!
I tested a 4G modem at my house, and could get 10-20 megabits up and down without a problem.
I also tested it out here on the farm, and got 10 megabits, so while it's not gonna
win records for speed, it should have enough to host my website, which uses 2 to 3 megabits
per second on a normal day.
But getting 4G working on the Pi cluster wasn't easy.
For 4G on the Turing Pi, I installed this Quectel 4G modem.
And that should be the end of the story, but no—it wouldn't even show up!
So I told the Turing Pi engineers about the problem, and they siad they'd get back to
me.
But I couldn't just wait and hope they found a fix!
I needed a backup, in case I couldn't get it working in time.
So I asked my Dad, and he handed me this, a five-hundred-dollar Cradlepoint.
It's basically the same thing as a MiFi wireless hotspot, but way more rugged.
I turned it on, and it worked, but then I found out I had two more problems to deal
with:
First, I couldn't get a static IP address, meaning people couldn't access my cluster
over the Internet.
And that's because of this thing called CG-NAT.
Think of it like calling someone on a phone.
With a normal phone number, I can call you from my phone, and you can call me back.
That's like a public IP address.
But with something like CG-NAT, my phone number is actually a 'private' number that my phone
company gives me, but it's only internal in their buildings.
I can call out to any public phone number, because the phone company routes my call.
But nobody can call me directly, because my phone number isn't actually a public phone
number.
That's probably the biggest problem with off-grid web hosting, but I'll get to that a bit later.
Another problem is how I network the four Raspberry Pis together.
Kubernetes—which I'll talk about in a bit—doesn't like IP addresses, or the phone numbers for
these things, changing all the time, so I had to program my own little network to make
sure Kubernetes didn't get confused.
And while I was going through all that testing with the Cradlepoint, Turing Pi actually got
back to me and said they had a fix for the onboard modem!
"All you have to do," they said, "is remove this little resistor."
Except, 'little' doesn't even begin to describe the tiny size of the resistor.
Look at it—it's like the size of a grain of sand!
But I got to work, using my tiny tweezers and a soldering iron, and luckily it wasn't
too bad.
I booted the Turing Pi back up with my 4G modem installed, and it worked, so I unplugged
the Cradlepoint and switched to the modem on the Turing Pi board, but now I had a third
problem!
With the Cradlepoint, it's an independent router.
It works with whatever computers you plug into it.
But using a modem on the Turing Pi board, that modem is connected directly to this first
Raspberry Pi here.
And it doesn't share that connection to the other Pis.
So I had to spend a few days figuring out a solution.
I set up the first Pi as a router, so it could share its Internet connection with the other
Pis.
I could've installed something like OpenWRT on this Pi, but then I couldn't join it to
the Kubernetes cluster, since OpenWRT can't run Kubernetes.
So I just set up some custom routing rules on the normal Pi OS install, and took care
of that problem.
Before I go any further, though, I have to give a big warning: running a Pi connected
to 4G 24-7 could eat up a lotta data.
And most 4G data plans aren't unlimited.
And they have huge overage charges.
So... make sure if you do something crazy like this, you know how much data you're using,
and choose an appropriate data plan.
I calculated that with my current traffic, my website would cost five thousand bucks
a month to run on 4G, so yeah... another problem to solve.
Networking is a bit crazy, but you know what else is crazy?
Relying on a Raspberry Pi to run a production website that gets a lot of traffic—or is
it?
A lot of people say Pis are flaky and can be unreliable.
But that's a wise sysadmin's secret: you should treat every single computer like it's flaky
and unreliable, whether it's a ten thousand dollar Dell PowerEdge server, or a thirty
dollar Raspberry Pi.
But how do you deal with that problem?
Well, don't rely on one single computer!
That's why I installed Kubernetes, or more specifically, K3s, on this cluster.
Kubernetes is a system that pools together
multiple computers, or 'nodes', and run applications
on 'em.
And the Turing Pi has four Raspberry Pis, or four 'nodes', that can all be pooled together
to run my website.
To make sure things run well even when one of the Pis breaks, I tell Kubernetes how to
split up my web server software so the website will 'self heal' if any of the Pis go offline.
Now, I know what you're thinking: what about that first Raspberry Pi?
Isn't it doing all the Internet routing with the 4G modem?
Well, yeah, with my setup, if the first Pi goes down, I'm toast.
So it's not perfect—there's this single point of failure.
But even there, I can work around temporary Internet outages, and I'll talk about how
I'm doing that later.
But applications themselves are the things that usually break, like Drupal or MariaDB.
When they blow up it's chaos.
So, Kubernetes takes care of that.
And you might think I'm crazy, running a website and database on microSD cards.
But don't worry, I'm not doing that.
I set up these two SSDs in a ZFS mirrored volume so either drive could break and all
my data would still be there.
Then I set up Kubernetes to store the database and all my website data on the SSDs using
Persistent Volumes and NFS.
You might be asking at this point "how did you do all that?"
Well, I can't get too deep in this video, but I did post a ton of guides going through
everything, from how I set up 4G, to how I set up ZFS on the Pis.
I even have entire free Kubernetes 101 series on YouTube—check that out by clicking the
card up there!
But we're still not done here.
Kubernetes keeps my software running, but what about the hardware?
How do I protect this Turing Pi board from the elements?
And how do I take it on the road without it getting banged up?
Well, I got this ITX rackmount case from MyElectronics
in the Netherlands, and I did a whole video
on racking up the Turing Pi inside it.
But then to protect that, I bought this soft 2U road case that's built for carrying around
sound equipment.
And knowing how roadies treat their gear, this bag can take a beating.
It's not weatherproof, but it fits the Turing Pi 2 cluster perfectly, and as long as I leave
the front and back unzipped, there's plenty of ventilation to keep it cool.
So now I can transport the cluster safely, and I use Kubernetes to keep things running
even if one of the Pis goes offline.
But how do I make sure that when you go and visit JeffGeerling.com, that request makes
it to this cluster?
I said earlier I had to deal with CG-NAT.
There's no easy way to access my cluster from the public Internet, so what's the fix?
There are a few options.
I could use a VPN, or a hosted service like Tailscale or CloudFlare Tunnels.
But I don't wanna rely on someone else's service.
I already host my website on a VPS from DigitalOcean.
It's just a boring old Ubuntu server, but it does have a static IP address like that
public phone number I mentioned earlier.
That means I can point the JeffGeerling.com domain name at it, and people can reach my
website.
But to make it so the website is actually served from my Pi cluster, I have to set up
the server as what's called a reverse proxy.
A reverse proxy accepts requests from the public Internet.
Then it distributes those requests to private servers running behind it.
And since I'm using Nginx on my server, I can also set it up to be a caching proxy.
That means I can save a ton of data usage by having the VPS handle all the traffic for
things like images and static web pages.
If I serve all the traffic over the 4G wireless connection, like all my site's pictures, CSS,
and Javascript, then I can serve about 1 or 2 pages per second, and the time per page
load is sometimes up in the 5 to 10 second range.
But if I configure Nginx to cache all the static content on the fast VPS, I can serve
over a hundred pages per second.
That means even if my website gets hit by a lotta traffic from Reddit or Hacker News,
it should stay up and running.
The only major risk is if a lotta people post comments at the same time, because all that
traffic has to go through to the cluster so the comments get saved in the site's database.
So caching is important.
Besides saving me thousands of dollars per month in data charges, this has a side benefit
that if the 4G connection goes down temporarily, the reverse proxy can keep serving my website
from its cache, at least for a while.
But how does my VPS know how to connect to the Pi cluster?
For that, I decided to set up a reverse tunnel, using SSH.
Basically, I have the main node with the 4G Internet connection set up to stay connected
to the VPS all day using a feature called 'TCP Forwarding'.
Using a little service I set up on the cluster, it sets up a persistent 'tunnel' through to
the VPS, and if the connection ever stops working, it'll re-connect automatically.
Using this, I can set up Nginx to proxy web requests to the Pi cluster like it was running
on the same server, just on a different port!
If this all sounds a little complicated—it is.
That's why this video's taken about 2 months to make.
Check out my blog posts if you want all the details.
So now I have my main VPS set up to accept requests for JeffGeerling.com.
When you click on one of the blog links in the description, the VPS routes that request
through Nginx and the reverse SSH tunnel, then the request will hit Kubernetes.
Kubernetes will route the request to an instance of Drupal running on the Pi cluster, then
Drupal sends back a web page, the VPS passes that back to you, and bingo!
You get to see my website, served up from anywhere in the world.
Of course, if someone else clicked the link in the last few minutes, it'll probably be
cached—in which case my VPS will skip sending the request all the way through to the Pi
cluster, and I'll be able to afford my mortgage this month.
I'm not gonna lie and say this is a better setup than hosting my website on my VPS.
It's slower, more prone to dropouts, and the whole setup is a lot more complicated, meaning
failures can be harder to troubleshoot.
But it is a fun learning experience.
And there are plenty of use cases for a resilient off-grid low-power cluster of computers that
make a lot more sense than web hosting.
Back in 2014, after reflecting on major disasters like Hurricane Katrina and the 2011 Tsunami
in Japan, there was a paper written on Wireless Networks for Disaster Relief.
It mentions the importance of having wireless communications systems in place after disaster
strikes, because emergency personnel can't rely on the existing infrastructure, which
might have been completely destroyed.
Especially today, with all our communications platforms tied to the Internet, having the
ability to run a local cluster of low-power servers managing applications like databases
and check-in systems can help first-responders in a bind.
And many existing solutions cost thousands of dollars.
If you could build a resilient 'edge cluster' like this one for a few hundred bucks, and
even have a spare, well, that makes a lotta sense to me.
Or what about things like music festivals?
They're often far from power and Internet connections, so having a mobile Kubernetes
cluster that can run ticketing and on-site organization software is critical, and helps
a lot if you only have sporadic Internet access.
And finally, it's nice to have options—especially as new, faster wireless communications become
mainstream, and maybe if you have unlimited data, you could conceivably do things like
host a website or a game server off-grid, living your best van life.
And before you go in the comments and ask why a cluster of Raspberry Pis, I'll answer
that too.
Sure, you could just run all this on a single NUC or an old laptop.
But then we're back to the sysadmin's dilemma.
How can you rely on just one physical server?
The answer is, you don't!
But it's true that Raspberry Pis aren't the best performing option.
But they're relatively cheap—when you can find them—and they're great for learning.
I learned a heck of a lot on this journey.
I learned how to power a remote Kubernetes cluster indefinitely, using solar power and
a battery.
I learned how to integrate 4G wireless Internet into a cluster of computers.
I learned how to proxy web traffic through a self healing reverse SSH tunnel, to get
past CG-NAT.
I also learned it's probably not a great idea to run my website like this, even if it works.
And like everything else I do, all the code I used, all the automation, all the instructions,
it's all open source, on GitHub.
Go check out the link in
the description.
Making this video was a lotta fun.
And a lotta work!
I wanted to thank Ecoflow one last time, for sponsoring this video, and Haarman Farms for
having me out to their place... and for the cows, for cooperating even though it was about
negative 10 degrees celcius, with 20 mile per hour wind gusts!
Check out their links below, and yeah... until next time, I'm Jeff Geerling!