Retrospective on hosting my blog inside an LTE modem, 4 years later

Exactly 4 years ago today, I wrote about hosting my blog on an LTE modem inside the PinePhone. To summarize, the Quectel LTE modem in the PinePhone is a fully functioning Linux system, and being fairly easy to root, I decided to host my blog on it and use USB networking to make it accessible to the world.

While a fun project, I had to abandon this experiment due to several issues, some more obvious than others.

The obvious

When I posted this article, I also submitted it on Hacker News. The post gained traction, and while the feedback was great (thank you!), the amount of traffic flowing to my poor little modem wasn't.

I wasn't keeping any statistics, but at one point, you could feel the site choking. I was using darkhttpd, which was supposed to be very minimal and lightweight with no extra bells and whistles, but rather surprisingly, the teeny processor on the modem was working overtime and you could feel the PinePhone getting hot. From my very basic analysis (that consisted of running top), it seemed like a lot of the processing was being eaten up by adbd. The way my little project worked was that I had set up port forwarding via adbd to actually access the site. I'm guessing this was being done in userland and wasn't exactly efficient.

The bandwidth was also an issue. My original post estimated the throughput at around 10Mb/s, but in practice, this seemed to be much lower. I'm not really sure what the reason for this is - it may have actually been bottlenecking at the CPU and this might've been reflected in my throughput, but I'm not sure why this didn't become apparent during testing with iperf.

Overall, I slapped a bandaid on both of these issues - I set up nginx with caching in front of the modem, which did seem a bit like cheating, but helped drastically in keeping the thing alive.

The less obvious

The physical link was a bit of a pain. Initially, I had set up the device to use USB networking and had it connected directly to my router's USB port. I use a static site generator, and when I had updated my blog, getting the new pages on the device was a bit awkward since I had to go through several layers - first the machine where I generated the site's content, through the router into the PinePhone, and finally getting the content from there to the modem using adb push. That wasn't too bad, and I had set up CI scripts which more-or-less automated this.

Another issue arose when I swapped out my router for a newer one. While this one did have an USB port, it wasn't possible to use that for USB networking with the firmware that came with the device. While this wouldn't usually be a problem and I would just jailbreak the router (which is 50% of what I write about on this blog) and enable USB networking, the router was a rental device and I didn't feel quite comfortable poking around in a device I didn't own.

This meant that I needed to use yet another device that the PinePhone would connect to and would act as an intermediary between the router and the PinePhone. Of course, if you weren't keen on torturing yourself, you would just host your blog on that device instead.

At one point, the PinePhone (or the modem?) started acting strangely. It would work fine for a few days, and then suddenly stop working entirely until I went to the PinePhone, turned the display off (I had it set to permanently on), and then back on again at which point the modem would spring back to life. At first this seemed like some sort of power management "feature" where the modem would go to sleep if not used, but no matter how I tried to disable power management, this would still happen. This was quite inconvenient, because it meant that my blog would randomly go down until I went and physically flipped it back on again, which wasn't always possible.

Sometimes simply disabling and enabling the display wouldn't even work, and I had to power cycle the entire phone.

Even with all of these issues in mind, I still kept hosting my blog on the modem, and had this set-up going for nearly two years.

And then the PinePhone died. After rebooting the device due to the modem being dead yet again, it simply did not come back up again. Nothing on the display, nothing over USB. I'm guessing something inside the phone died, because besides the blog, the phone wasn't being updated or used for anything else. I probably could've diagnosed what exactly had failed and maybe got it to live again, but I decided to let the poor thing rest, and moved my blog to a more sane hosting solution.

What I would do differently

If I had to repeat this experiment, I would probably strip out regular modem functionality from the modem entirely and use custom firmware on it designed solely for running the blog. I'd probably not keep my site's files as loose HTML files but bake them into the HTTP server binary directly - or just not unload them from memory after being loaded, just to reduce IO on the modem's flash.

This would also maybe let me bypass port forwarding with adbd, which seemed to have a lot of overhead. With these changes, maybe having nginx with caching in front of the device wouldn't have been necessary - perhaps just for SSL termination.

Since this was a low power device, it would've been interesting to experiment with alternative power sources for it, like wind or solar, and try and keep the device as off-grid as possible. Something similar has been done by Kris De Decker & co. at solar.lowtechmagazine.com.

The state of the blog now

If anyone's interested, the blog is currently hosted in a very conventional way. I decided to rewrite the blog using Hugo instead of Jekyll due to constant fights with Ruby and Bundler and chasing down different errors that randomly pop up every now and then. Publishing a blog post is as easy as pushing a markdown post to a git repo hosted on Forgejo, which runs a CI pipeline generating the posts and uploading them to BunnyCDN, which is what you're currently accessing. A far cry from having to push through several layers of stuff just to get my changes live.

Maybe I'll repeat this experiment with another device or set-up if I happen to get my hands on something interesting. Until then, I'll just enjoy the convenience of hosting my site with a minimal amount of hacks, workarounds, and headaches.