Start your own Internet Resiliency Club

bowshock.nl

579 points by todsacerdoti 15 days ago


alnwlsn - 15 days ago

I've found Meshtastic is simply not ready to be set up in an environment without internet, as I discovered when I brought some of the boards I bought with me on vacation to a rural area with more space to test them, but very limited internet.

The entirety of the meshtastic project is web first.

- To flash your boards, the suggested method is their "Web Flasher", and if you download the firmware source, it depends on PlatformIO (and the internet) to download and install the toolchains and flasher programs you need.

- The clients for meshtastic are available on the app stores, or as a web app at https://client.meshtastic.org/ None of these are offline. I did later learn the boards themselves host the web app, but they still have to be connected to an Wifi AP, you don't get it just by plugging the board into your computer.

- The docs are hosted at https://meshtastic.org/docs. "Download Docs" or "How to self host this project" are not topics described there or anywhere else. A technical person could figure this out, but this is seemingly not a primary concern.

I suppose this is the very point of this post, to get people to have it all set up beforehand, but not even having the docs as a PDF I can read offline? I learned about Meshcore too in this thread, but if I go to their site and the "getting started" guide is a Youtube video, then you're not ready for an emergency!

wao0uuno - 15 days ago

I tested meshtastic in a major european city with pretty much 100% mesh coverage and its real life performance was quite underwhelming. Often I would receive messages that I could not reply to because of differences in antenna gain and crappy mesh performance. Public chat was either completely dead or flooded with test messages. Everything was super slow because the mesh can’t actually scale that well and craps out with more than a 100 nodes. Even medium fast channel would clog up fast. I would never depend on meshtastic during an emergency because it barely works even when nobody is using it. I think a public wifi mesh would be more worthwhile. Older used wifi routers are pretty much free and in unlimited supply. They use very little power. Everyone already has a compatible client device on their pocket. Sure the mesh would fail during a total blackout but at least it would be useful for something when the power is up.

lljk_kennedy - 15 days ago

> One of my nightmares is waking up one morning and discovering that the power is out, the internet is down, my cell phone doesn’t work

I dunno.... as I get older, this sounds more and more idyllic

liotier - 15 days ago

Mesh networks are the foundation - they are essential to disaster resilience. Then what services to run over them ?

Real time chat: wild unsecure simplicity proven to run anywhere (IRC), bells & whistles with contemporary security (Matrix), some mesh native that almost no one knows ? What about post-disaster onboarding of actual users ?

Store & forward messaging: SMTP & friends may work nicely, but with actually distributed servers - in each local disaster POP. Also needs timeout and retry parameters to keeping stuff in queues practically forever.

Forums: anything better than ol' NNTP ? Other protocols merely adopted intermittent indirect connectivity - NNTP was born in it !

Is anything more sophisticated or more interactive realistic for actual disaster ?

An onboarding kit with clients for each major OS (à la AOL CDROM !) might be handy too, for snearkernet distribution over USB dongles.

lambdaone - 15 days ago

Mesh radio bandwidth is pretty poor. Firstly, you have to compete with many interferers (albeit this might get better if the power goes down), including other LoRa radios, but more to the point, long-distance connections consume bandwidth and aquire delay and delay variation at every intermediate hop. It might be reasonable to use it for text messaging, but with per-hop bandwidth ranging from 0.3 kbps to 27 kbps, which will get divided down further over shared multi-hop links it will be impractical to use it for anything else except perhaps very-low-bandwidth telephony over short distances or visiting minimalist text-only websites.

It might make more sense if augmented by fixed multi-megabit point-to-point microwave radio links to act as a backbone, with LoRa only functioning as an access network.

I'd be interested to hear what experiences people have had with doing this for real.

blueflow - 15 days ago

This article makes more sense if its coming from a city where only the large telco's are present.

Here Dresden (Germany), there are several volunteer organisations who laid wires through the city or have microwave-antennas (AG DSN, Bürgernetz, Freifunk), and there is a recently founded internet exchange run by volunteers (DD-IX). So as long as we have power, we got our own internet.

niczem - 14 days ago

Meshtastic is fun but limited—more of a radio chat app than real mesh infrastructure. If you're serious about decentralized comms, check out Reticulum: https://reticulum.network

It’s not limited to LoRa—Reticulum works over IP, serial, packet radio, or whatever you have. Delay-tolerant, multi-hop, encrypted, no servers needed. Still lots of work to do and apps to build, but the foundation is solid.

Great talk from EH22: https://media.ccc.de/v/eh22-97-eggceptional-meshnetworking

__MatrixMan__ - 15 days ago

I feel like the better path to resiliency is not persistent radio connections between hobbyists on other sides of the state but rather intermittent ones between people on opposite sides of the bus and an application layer that arranges for people who are heading that way anyhow to carry "internet" traffic on a filesystem in their pocket.

You just get a different type of threat landscape when each hop is also an opportunity to shake somebody's hand and attest that the holder of their private key is a real human. It creates a minimal trust layer you can then build on. You don't get that with a hardware address found drifting on the wind.

Both modes have some potential to attract harmful attention to network operators based on the behavior of their users, but to a very different degree. So far as I know nobody is kicking down meshtastic operators' doors looking to follow a transmission to its source, but I think that would change if the other modes of long range skulduggery were to fail.

The most resilient infrastructure would be one with no high value targets: one where each user is equally an operator.

nunobrito - 15 days ago

OK but kind of outdated and incomplete. Meshcore is largely competing with Meshtastic nowadays: https://meshcore.co.uk/

To remember: LoRa only permits small text messages. Don't even think about images, voice nor binary files (I mean it).

Another option is APRS using satellite connections through a cheap chinese walkie-talkie (Quangsheng UV-K5) for 20 euros to send text messages.

Fokamul - 15 days ago

1. Meshtastic / LoRa is just bad for communication, it has so many problems

2. In case of conflict, everyone who starts LoRa gets delivery of artillery shell/rocket on their position.

Just like in Ukraine, try to go there and start up stock firmware DJI drone there and see what happens :)

Same when using radios in UA, no.1 rule is to NOT use encrypted radios, I like this example the most, because it goes against common sense, why would you want to use unencrypted radios so enemies can see your whole communication.

Reason behind this is following, encrypted radio traffic is very interesting for enemy, so it means if someone using it, he must be someone important -> send shell, badabum.

estsauver - 15 days ago

* I would consider adding a T-1000 device to the recommended list of devices, it's about the size of a credit card and works very well to add Meshtastic to phones. https://www.seeedstudio.com/SenseCAP-Card-Tracker-T1000-E-fo... It's a lot easier (in my opinion) to get people to stash a radio and remember how to power it to Bluetooth then to get setup from 0 on a new device. I think I paid about 40 euros each when I bought a pair.

* I have a Starlink mini--in the event that there is ever a broadly disconnecting event I'd be happy to share access to it. I keep it pretty much exclusively for emergency use and occasional camping/rural holiday house vacations. You might want to consider one too? They're ~250 euros new, which for someone who's starting a club for anything seems like a plausible expense. I believe there's a chinese version in case you don't want to trust the whims and emotions of Musk. * https://kiwix.org/en/applications/ is pretty useful if you'd like to have an archive of technical information, wikipedia, stack exchange etc.

* I try and keep whatever feels like the smartest open weight LLMs at the time available so if something real bad ever happened it'd still be available. I might add that idea to your preparedness list too--I'd probably take LM Studio with Gemma 3 over another random engineer on the Meshtastic channel :)

* Would you share channel config details for your IRC community? I'm happy to join.

dansmith1919 - 15 days ago

Watch her 10-minute RIPE 90 talk and then listen to the first "question" for a short tutorial on how to behave like a prick: dude didn't even have a question, just wanted to let everyone know how much he knew about a somewhat related subject

Bender - 15 days ago

For grid-down data my preference would be laser instead of RF as laser is regulated by the FDA and not the FCC not that either would take interest. With laser one could send incredibly large amounts of data very fast. It's more manual setup but I would expect once set up it would be far more reliable, better for setting up mountain top repeaters and meshes. Laser is also better for data privacy encryption aside as the beam is directed to a target vs. omnidirectional broadcasting. During grid-down most people that would be using a mesh would be at static locations. One could then bridge in these RF omnidirectional devices into the mountain and home repeaters to prevent over-saturation.

Another nifty feature of a manually positioned laser is the automatic measurement of time domain. One could have an optional security feature to automatically disable the data-stream if the time domain of the laser changes in physical distance of more than {n} user-defined meters or centimeters to prevent MitM (Monster in the Middle) beam interception for the extra properly paranoid types.

There can be weather issues for laser but for that one could fall back to voice using any one of the hundreds of makes and models of HAM gear that can operate on and around 11 meters by moving a jumper or holding down two buttons when it is powered on. Illegal but only enforced by monthly example of someone impacting revenue generating sites. Voice changers and scramblers FTW. RF signature ignored. Don't use sloppy SDR's. In a grid down event TLA's will be busy with higher priority issues and will "look into it" eventually by which point the transceivers mysteriously vanish assuming one can even get the TLA to show up.

7373737373 - 15 days ago

I don't understand this fascination with networks that require special hardware to intermediate between end user nodes. Would be much nicer if things just ran, zero-click, via WiFi, on most common computers, netbooks and phones, pure p2p with automatic forwarding, no?

https://en.wikipedia.org/wiki/Wireless_ad_hoc_network

By requiring special hardware, and be it just some common router, or any sort of special technical skill, you are already excluding 99.99% of the world population...

pwndByDeath - 15 days ago

Its been on HN before but its worth a repeat https://reticulum.network/ Its got more optimization for a low bandwidth LoRa without the brute force stuff of meshtastic

1706213 - 15 days ago

I would use Reticulum instead of Meshtastic. https://reticulum.network

Pros: - it can actually scale past 20 devices - Forward secrecy encryption - Is designed to support multiple underlying transport systems such as TCP or LoRa - Announce based routing rather than flooding the entire network which is order of magnitudes faster Cons: - Not as many nodes as Meshtastic has - Python implementation with no C implementation (can be speed up with cython however)

aamederen - 15 days ago

For a more baby-steps approach to resiliency, one might start running software on less-virtualized computers, creating a small home-lab, running software on bare-metal hardware that you actually own.

b0a04gl - 15 days ago

but who has the gear, who keeps it charged, who actually shows up when the net goes dark. tech's the easy part. the hard part is getting 5 neighbors to agree on a channel, a meeting point, and a backup plan they’ll actually remember.

also, would be interesting to see people test these setups during a planned outage. like simulate a real failure for 24 hours and see what breaks. most systems look solid until you actually need them

esafak - 15 days ago

The problem with these things is that people have no urgency to prepare, by pro-actively improving software and documentation, or even simply installing them. They need to be something people get value out of even before disaster hits, by improving performance or decreasing costs, for example.

kcaseg - 15 days ago

saveitforparts has multiple videos on meshstatic, if you want to see it in action, it is super interesting, but not without flaws apparently

https://www.youtube.com/watch?v=8WRNTkbRuCI

https://www.youtube.com/watch?v=kdHB_5Z_CFE

Brendinooo - 15 days ago

This sort of thing is interesting but I guess I've always found it really hard to invest my limited time and money into prepper-type stuff.

Are there use cases for this sort of thing that could make it worthwhile even if doomsday doesn't arrive?

jvanderbot - 15 days ago

Just dropping a HAMWAN link for anyone who is interested in higher power / longer range links. https://hamwan.org/

mschuster91 - 15 days ago

> Initially I looked into ham radio, but it is just too expensive, difficult, and power-hungry to be practical.

Beg to disagree here. 30 dollars for a cheap-ass Quansheng will get you pretty far as long as a repeater is in reach (if it's Echolink capable, worldwide), and a bunch of repeaters for all kinds of modes are tied together not only via the Internet but also via AMPR / HamNet [1]. APRS and DMR capable devices are in the 200 dollar range.

For high bandwidth data communication it becomes a bit more involved - Ubiquiti hardware for example can be trivially software-modified to transmit on the amateur radio ranges, which is how that gear ends up powering a lot of HamNet stations. Sadly, unless there's a HamNet node on a nearby large structure you'll probably need to raise a tower large enough to achieve line-of-sight to the nearest HamNet node.

For people in reach of the QO-100 satellite (i.e. Europe, Africa, about half of Asia), there have been experiments to use that satellite not just as a repeater for voice and video, but also data [2].

[1] https://en.wikipedia.org/wiki/AMPRNet

[2] https://forum.amsat-dl.org/index.php?thread/4306-npr-vsat-ip...

specproc - 15 days ago

I read the title and the first few sentences as resilience _to_ rather than resilience _for_ the internet.

6510 - 15 days ago

Things would improve a lot if we added offline abilities to html documents. I think atm there is no way to guarantee a website stays in the cache(?)

Designing a system to decide when to keep something is tricky. Maybe each visit and each click should extend the expiration date and increase the storage for static documents. Say, 10 visits should be enough to buy 1 mb of permanent storage to be spend on however many pages it takes starting with the frequently visited pages then a manifest or the order of links on the front page then the first from each sub page etc

It should also be possible to have the browser manage updates rather than every man for himself with each website testing the connection, checking for updates and stitching things back together again. There are quite a few schemes it could follow, smaller requests would require more complicated backends. Different pages with different update frequencies.

I think the single star bookmark button could have 1-5 stars with 5 assigning somewhat generous data to the website and 3+ allowing a prompt for very large things.

Then, since I'm serving static content anyway I really couldn't care less how the user obtains the files. If there is a copy of the website on a network all you need is a public key or to trust the user (at the price of annoying prompts warning you on every page view and every request)

If it all works well enough HN could be a tiny website managing only active discussions. If you have the key and a working connection to some other users most of the archive could be there. The catching priority could change to the rarest pages.

lormayna - 15 days ago

I am into Meshtastic but the coverage, at least in Italy, is low and depends a lot on the position. If you are in a city, you can get many neighbors nodes, otherwise you need to be at high altitude, relying on other nodes or use a directional antenna.

Anyway, it's a nice hobby to learn a lot about solar powered systems and antennas/propagation.

I think that one of the best use cases for Meshtastic is to use it during protests, especially in authoritarian countries.

protocolture - 14 days ago

I was once doing a review of an ISP's emergency response documentation, and it was 1 page. All stakeholders were to report to the primary datacenter and begin restoring services. It only really accounted for a single risk (Outage with no physical hardware failure) and didnt account for the scope and presence of their network. When I flagged it with them, they added a second section for cyberattack that was also loosely defined.

I really love the idea its the proposal itself I find odd. "Internety people" isnt really a well defined list of stakeholders. And I am struggling to figure out what outages could be handled by a group of undefined meshtastic users, especially with the overall tone that warfare might be involved.

Really the work of network resilience begins before the bombs drop. And its basic stuff. Keep everything patched. Keep your physical infrastructure secure. Ensure your data is safe, and you implement security best practices. And thats a list of things that largely we know that ISP's dont do.

Once thats done, every netadmin needs reliable OOB access to their networks. Now, OOB access might cease to function in this scenario. In which case, you are looking at ensuring that you have reliable physical access to your network. If you need cell service to communicate with your OOB solution, and it fails, dont also rely on cellular access to hands on techs. I dont know if Meshtastic is a useful solution here either, but having pre drilled emergency response plans so that qualified staff show up to the right locations for briefings/network access are crucial. Generators within reach of technicians, you might not be able to buy a generator during an incident. Console cables and tools already distributed.

Of course this all costs money.

Ultimately if you have no power, your peers dont have power, and theres an unspecified amount of physical damage (like a datacenter getting bombed) theres not going to be much you can do on a timescale that makes meshtastic make sense.

If anything theres probably some room for a government solution. Propose a hardened communications channel for private sector network engineers. Make sure that access to government infrastructure comes with a requirement that you actually patch and secure your network. Begin any large scale internet infrastructure collapse by getting a list of available engineers and resources and working out the compensation later.

lormayna - 15 days ago

This module from SeedStudio is perfect for a small portable node. https://www.seeedstudio.com/Wio-SX1262-with-XIAO-ESP32S3-p-5...

There is also a version with Rak chip instead that ESP32, that is a lot less power hungry and it's perfect for a solar powered module.

bb88 - 15 days ago

Meshtastic isn't very good (at least before their 2.6 release) [3].

It's clear they didn't research any historical mesh network schemes (ALOHAnet [0] and other MANETs [1]) when writing it. And flood routing more or less kinda worked, but as it go popular, it stopped working reliably. There's a video from Jeff Geerling about it and he was generous I think when he called Meshtastic "Beta" [4].

Meshtastic a few years ago released a youtube video describing how it worked, and there wasn't anything about topology resolution, it was pretty much about signal strength and device type [2].

This caused an issue because anyone could create a router. And often they did. And when they did, this could break routing because the router is in the other direction of where the originator wanted the traffic could go.

They also prioritized letting the edges forward the message on. AFAICT they could only detect this by signal strength. So a badly performing node (bad antenna or maybe a node turned on in a basement) could get priority.

The last issue is congestion. Nodes can send telemetry, often rather quickly, but that could get flooded on the network. And with a hop count max of 7, it often will go where it's not wanted, wasting the network bandwidth -- as nobody really cares about one particular node's battery life.

So in a dense Meshtastic metro (I can see multiple sites) I couldn't reliably get a message to a friend in the same city. The lesson is that the hardware is better than the software at this point. And there's no use using it until they fix their software. [3]

[0] https://en.wikipedia.org/wiki/ALOHAnet

[1] https://en.wikipedia.org/wiki/Wireless_ad_hoc_network

[2] https://www.youtube.com/watch?v=7v6UbC5blJU

[3] I did some research on meshtastic and after finding and watching [2] I gave up on meshtastic, because it's clear that weren't super serious on routing algorithms, nor basic wikipedia reading. Version 2.6 maybe better, but there's a slew of nodes on 2.4 yet. And I don't want to bother with it anymore -- at least until they fix their reliability problems.

[4] https://www.youtube.com/watch?v=0A7A-CSd3e4

xondono - 15 days ago

Sounds like a reinvented HAM radio club to me

qwertox - 15 days ago

I've been thinking about an idea, that maybe it would be worthwhile for a city to create a wireless network where it uses rooftops for a mesh.

This WiFi offers a low-data-rate (<5-10 mbit/s) service to seniors for free or a very low fee (~3€/month), without service guarantees, but honest best-effort.

In the case when an internet problem arises, which affects the city's it-infrastructure, the city can switch to this WiFi to have their city-wide services still interconnected, while the seniors get kicked off of the network during this time.

Elaris - 15 days ago

I think we should focus on collaboration, not just individual action. Many times, we don't react until a problem occurs. This reminds us to be proactive, not reactive.

rubyfan - 15 days ago

I’m almost to the point of turning the internet off on purpose. The noise level is so high it’s almost not worth it anymore.

(but I get this is geared towards communication resilience)

crimsoneer - 15 days ago

As someone who has done a fair bit of playing with Meshtastic in the last few months, it's worth really managing expectations... it is in no way a replacement for any sort of internet. It's a way of sending very short text messages, with a system that is really quite flaky in any kind of built-up urban environment. Don't get me wrong, it's great fun, but there's a reason stuff like Ham radio is robust.

firesteelrain - 15 days ago

Why not something like WinLink which works over short and long distances using Ham Radio? It even has an email gateway.

Then, there is JS8Call, PSK, SSB, FM, etc

cameldrv - 15 days ago

One thing I’ve been very curious about along these lines is troposcatter systems. These, depending on the bandwidth, power, and antenna size available, should allow you to get tens to hundreds of megabits over up to hundreds of miles with moderate sized dishes. The military has some of these systems, but I haven’t seen too much ham activity with them.

- 15 days ago
[deleted]
swiftcoder - 15 days ago

On a local level, I feel like we can probably do better than just text messaging capabilities. Mesh network covering the village, with someone running mirrors of essential services in their basement (local email routing, wikipedia, etc)

nancyminusone - 15 days ago

>...all you hear is “Swan Lake” on repeat

Have I missed the meme on this one? What does this mean?

- 15 days ago
[deleted]
feiss - 15 days ago

This is fantastic. However, I only see the use case of messaging through the Meshtastic clients. Is there any other thing one can do over this setup, like Gopher or IRC?

1oooqooq - 15 days ago

people will try to design a plan to save humanity with Chinese radios, but won't sign up for a basic technician license. sigh

so many wrong assumptions on that article.

xpe - 15 days ago

One thought -- more of a question -- and I'm not the first person to ask it -- How can we design a 'smaller' kind of internet? One that is less data hungry, less commercial, but still sustainable (how?), more of a community vibe, with distributed governance, less enshittification. Think of it like a maintainable garden. One that still works without a lot of extra effort, such as constant browser 'innovations' or bandwidth upgrades. Something more like a hardscaped garden instead of a typical American-style monoculture grass lawn requiring nonstop interventions to make it look pristine. This would have many benefits, including resilience, redundancy, and archival. Yes, I realize I'm conflating layers and maybe even asking too much. But sometimes it feels good to dream. At the very least, it is a genuine question I can use to evaluate various proposals and ideas in this general area.

erremerre - 15 days ago

Wouldn't having something like this, you automatically became a target in case of the case of war?

amelius - 15 days ago

> a flood-forward mesh protocol

Is this scalable?

perlcommunity - 14 days ago

Valarie must be an AI powered roomba, don't fall for it fellow humans.

yannickdoteu - 15 days ago

any people around Leuven, Belgium that want to start a club?

- 15 days ago
[deleted]
- 15 days ago
[deleted]
whamlastxmas - 14 days ago

Really tiring to see Trump injected into everything. The accusation that Trump is going to cut off access to American software companies is fear mongering with no basis in reality

techgirl1637 - 15 days ago

[dead]

Mila-Cielo - 14 days ago

[dead]

abcbb - 15 days ago

[dead]

aaron695 - 15 days ago

[dead]

abcbb - 15 days ago

[dead]

adornKey - 15 days ago

[flagged]

Dachmwr - 12 days ago

[flagged]

charcircuit - 15 days ago

No mention of starlink? Even if the internet is entirely down locally your packets could be routed to the other side of Earth before making it to the internet.

Starlink is much simpler for the average consumer to setup than what this article suggests.

xvilka - 15 days ago

Another thing is to update mesh stack to more modern language, to improve security and resiliency - projects like B.A.T.M.A.N, Babel, OSLR, FRRouting, etc would largely benefit from being rewritten from pure C to language like Rust.