
I live in the middle of no where. The only ISP services I have access to is from a couple different WISPs and Starlink. The WISP’s have an edge in terms of pricing, being significantly cheaper than Starlink. Their download speeds aren’t too good being a quarter to half the download speed of Starlink but where they shine is that they are within spitting distance for upload. Because of that, there’s not too much reason to change providers.
Getting speedy fast internet was a dream…until recently when I started making headway in convincing a neighbor who (for some inexplicable reason) has fiber service to let me rent space for a point of presence. This took a while to do and in retrospect I think the reason was, he thought I just wanted to share his internet connection and not get my own.
So with that agreement in place, the hard stuff began.
What I thought would be a hard part was actually one of the easiest. I thought I’d have a tougher time talking with the fiber ISP and convincing them to sell me access for this. Turns out, they don’t really care as long as I’m not reselling the service.
What I thought was going to be the easy part, turned out to be the harder part!
Challenges
- Limited access of pop/remotely manage the pop
- Keep hardware to minimum at pop
- Only 1 ethernet to the home pole
- Graceful switchover
PoP Hardware
As I have limited access to the PoP and didn’t want to bother my neighbor when I needed to do a simple reboot or to talk to the radio for some reason. For that I needed some way to access the things from the driveway. To address this I decided to use a cheapo GL.inet Shadow that was laying around to give me a wifi AP for access. To be able to remotely power cycle things, I’m using Sonoff S31s flashed with Tasmota. These are super handy little things and I’m starting to use them all over the house for monitoring stuff or managing things like scent diffusors.
To segment power, management, and internet traffic, I’m using a Mikrotik CSS610-8G-2S+IN to VLAN off each network. If didn’t want a router onsite, at least not yet, because I wanted to keep that at home. One reason was because that’s something that can go really wrong and getting access to it physically could be problematic. The other reason is so I can have backup internet options without having to rely on the PtP radios to be working.
All the inside stuff sits nicely in a 6u wall mounted networking cabinet…except for the UPS and Shadow. I couldn’t get the UPS to fit with the switch in there and the Shadow is velcro’d to the outside to increase the wifi signal strength.
For radios I found a couple new in box Ubiquiti AF60-US’s that popped up on Ebay for a decent $200/each. I went with them because Linus Tech Tips did a video on the LR version with great success. I really wanted a 5Ghz backup link for those bad MN blizzards or summer rain and Wave-Pros were 3x the price. While the LR or Nano might have worked, they were still 1.5-2x the cost and the link was limited to 1.5Gbps (750/750 if maxing out the link).
Another radio I was seriously considering was the Tachyon Networks TNA-303x while using some old AirMAXs that were laying around as the backup link. I was really close to ordering some, but the AF60’s popped up and I couldn’t say no to the price.
I don’t off the top of my head know if other brands have it, but I like that Ubiquiti has a Bluetooth interface that can be talked with to do configuration on their hardware. This was really handy when I was in high places where I just needed to whip out my phone to do the aiming without having to have an entire network stack up and running first. (I had done the initial conf on the ground weeks before install.)
On the roof, my neighbor asked that I don’t put any holes in his roof. While putting a J-Pole on the side of the building might have worked, that both introduced some challenges for installation and it limited where I could put the radio. Instead I opted for a non-penetrating mast from EasyUp that would sit on his flat root. This made him really happy as it was something that could be removed easily and didn’t mar his building.
Home Front
My current ISPs radio is about 90m from the house and connected via a direct burial ethernet cable. While I could do a hard switchover by swapping radios, there’s no testing without interrupting the fam and not much going back easily if things go south. I’ve also put in some direct burial fiber that is really close to it so putting in another line wouldn’t be simple. The only feasible option left was to get my hands on some sort of outdoor switch that was managed and could do 802.3at PoE.
For this I settled on a TP-Link Omada SG2005P-PD along with a couple Ubiquiti INS-3AF-O-G to convert 802.3at to passive PoE for the existing radio and the AF60 I was planning on putting in. My only complaint against it is the webui is slow and it has high ping to itself.
Why TP-Link here and not Mikrotik’s new GEPR4? Because it doesn’t switch or that is, all traffic needs to go thru the uplink twice since each downstream port only can talk to the uplink port. While at the moment isn’t a problem since all the traffic will need to go into my home network anyway, it might not always be the case. I have a long term plan to add stuff to the Back 40 to monitor things like ditch water levels, watch wildlife, or see what’s going on beyond the tree line for weather. If I move the router to the PoP all this traffic would have to go thru the 1Gbps link twice if I used the GEPR4.
PtP Install
The fiber install was somewhere between 3 weeks out and 3 months, so I figured I’d best use that time to get the PtP up.
I started with the home end. I put the Omada in place, rewired things on the pole, and made sure the existing WISP still worked. It did and was like nothing ever happened. Putting the AF60 in place was a bit more of a challenge though. I should have know it when I bought it, but the AF60 doesn’t come with an aiming device! The AF60-LR apparently does, but not the AF60! I had to just eyeball the aim and hope I was close enough to at least get the 5Ghz radios to connect. This was really annoying that I couldn’t even find an ebay listing for a used aim tool.
For the PoP end, I ended up renting a scissor lift to haul the heavy things up to the roof. At first the plan was to just haul things one at a time up a ladder or pull them up with a rope, there was a problem though, I lacked a ladder that tall. Plus it seemed a bit dangerous keep going up and down all the time and with heavy things. A few trips up and down and all the hardware was up top.
To aim the radio, I took a UPS up too. As my luck had it, I had aimed the radios so well that I was able to get about a -65dBm 60GHz signal right out the gate! After some fiddling I was able to bring that down (up?) to the mid-50s. Bringing it closer to the expected strength required some fiddling at the home end afterwards. I might need to revisit the roof at some point, but I have a ~90% link potential so it’s not something I need to do soon.
From the Omada switch, I then had four vlans, one for each ISP, one for the far end power management, and the untagged/native for the ptp management. I went this route as it seems to be the best for simply connecting and troubleshooting the radio on the far end in case anything goes wrong.
This setup though causes the MTU to
Monitoring
As I’m not sure how well MN and this link distance fares for 60Ghz, I wanted to monitor stuff long term. For that I choose to put a Grafana and Prometheus setup in place. Besides the SNMP Exporter to monitor the radios, I also got the JSON Exporter too so I could monitor weather conditions and try to correlate that to signal strength and link speed. The JSON Exported took a bit of finagling to get working but between the logs and sparse documentation, it was working in short order.
Fiber Install Day
Everything went super well. I was really worried I had screwed something up, but the ISP techs got everything setup on their end. When they finally got the ONT provisioned, I saw my home router pick up a DHCP assigned address. Then I went home and changed the primary gateway to the fiber interface and IT WORKED!
The feels were like when my parents went from dialup to DSL.
So Far
So far this has been really interesting. We’ve had some really heavy rain days which have destroyed the 60GHz link. If my calculations are right, the rainfall was 2x that UISP design center’s precipitation calculator maximum. The 5GHz link though stayed up quite nicely and seemed to perform the best at 40MHz channel width. Speeds were even better than the old WISP service too. Not 1Gb speed, but still having ~75-100Mb was a nice thing to see.
A bit nitpicky, but you can’t run iperf on AF60s from the ssh/command line and get meaningful results. For me, I was getting about 1/3-1/4 the speeds the webui’s tests showed. You’ll need some device on the other end to do meaningful iperf tests. The jury is out if it’s a limitation of the radios (very well could be) or just a skill issue (also very likely).
The World of Tomorrow
For the future, I think I need to find radomes for the winter time. Posts on the Ubiquiti forums lean towards it being necessary for winter time snow. However, I’m not entirely certain now helpful it would be for one radio as it’s pointed away from the winter wind direction 99% of the time. It would suck to have to climb up on the roof during the winter time to deal with snow and ice accumulation though. So it might be a good idea to do it sometime before that. Ounce of prevention and all that.
Replacing the switch with a router at the PoP end to handle the ISP handoff and forwarding all traffic to the home router’s RFC1918 “WAN” address (for nat-pmp). Mostly this would be to make management of PoP assets easier and be able to access them thru a backup internet connection from home. Scratch that, I found out that I can have multiple devices on the WAN and each can request their own public IP thru DHCP. Plans are going to shift to put a just powerful enough router on the PoP end and use a VPN like Tailscale to manage things thru a backup internet connection at home or my cell.
Replacing the current masts with 10-20ft Rohn Self Supporting towers. For the PoP end, this would require some trenching and electrical work besides convincing the neighbor to be on board with it.
A backup internet. At the moment that’s my previous WISP. It’s kind of expensive for that purpose though. I’m thinking I might use my cell phone as a hotspot when I need to. The only issue I have with that at the moment is my office is in the basement and gets terrible signal. If I didn’t need my phone for MFA, I probably would just put it on an upper level. I’ve been reading up a bit on cell signal boosters which looks promising.
Gists
Tasmota
For flashing the Sonoff S31’s, I followed this guide by Aaron Cornelius. I highly recommend getting the clips he talks about. They made the task much much easier.
Here’s a template for setting up a S31s with Wi-Fi, auto power on after ~10s, & MQTT. Just replace the uppercase stuff with your stuff. This does all this conf in one go. If you’re not needing MQTT (I used it to connect it to my Home Assistant), it can be dropped.
The main things that make this work is the PowerOnState and PulseTime. The PowerOnState being set to 5 makes the default state of the switch to be on and the PulseTime sets the timeout for turning back on. I did things this way so that stuff should come back to a workable state if I turn the power off.
backlog ssid1 WIFISSID; password1 WPAPASSWORD; PulseTime 100; PowerOnState 5; WebPassword WEBUIPASSWORD; MqttHost MQTT.FQDN.EXAMPLE.COM; MqttUser MQTTUSER; MqttPassword MQTTPASSWORD; FriendlyName DEVICEFRIENDLYNAME; DeviceName DEVICEUNFRIENDLYNAME; module 41;
Prometheus SNMP Exporter
Importing MIBs into the SNMP Exporter is kind of a hassle. So I made this little script to combine my auths config with the MIB data compiled by the SNMP Generator. This does require the yq tool.
#!/bin/bash
docker run --rm \
-v "${PWD}/snmp-generator/generator.yml:/opt/generator.yml:ro" \
-v "${PWD}/snmp-generator/snmp.yml:/opt/snmp.yml" \
-v "${PWD}/mibs:/opt/mibs" \
prom/snmp-generator:latest \
generate --log.level=debug
#pick out one and preserve the tree.
yq eval-all '. as $item ireduce ({}; . * $item)' snmp-exporter/snmp.base.yml snmp-generator/snmp.yml > ./new_snmp.yml