But at some point, running Ansible over and over again to keep things up to date became both tedious and error probe. What happens when I want to run an Ansible playbook to automatically update Netbox with new IP address information from a VM? And I want to do it automatically.
Part of my 2024 goal was to make my homelab easier to manage: spinning up virtual machines, rolling out updates,
generally managing things should take less work. That includes managing the inventory of homelab machines, a mix of bare metal, virtual machines, or containers on a Proxmox cluster.
Things start with a source of truth, what I hope to use Netbox. Netbox contains APIs for managing inventories of IP addresses and prefixes, machines, virtual machines, etc.
Ever since I was able to share the Internet connection among the several computers we had at home 30+ years ago, I’ve always found it interesting the many ways computers can connect. First I learned about IPv4, then VPNs such as OpenVPN and Wireguard, and now IPv6.
Fast forward several years, and after running a IPv6-in-IPv4 tunnel thanks to Hurricane Electric, my ISP provided a delegation of IPv6 addresses which I could share among my machines. Several more years later, I wondered, how much IPv6 was I using instead of IPv4 now that all my internal machines had IPv6 addresses? I knew about ntop and how it provides visualiations for traffic analysis. Figuring out how to wire it up to my network, and explaining it, is the point of this post.
The problem: I keep getting connection refused when trying to view the consoles of VMs. Why?
Background: My home lab runs Proxmox on 3 physical machines (moon1, 2, and 3). Each Proxmox host runs several VMs. To access the Proxmox Web UI, keepalived+haproxy is configured to provide a single IP address for loadbalancing and TCP-connection proxying to the Proxmox UI via https://proxmox.mylocal.lan. From the Proxmox UI, when I click on Console for a particular VM, I’m greeted with the following error:
That feeling when you realize it’s been almost a year since your last post, and one of your new year’s goals was to write more.
So when I broke out the website repo to get back to writing, I found the workflows I use to deploy the site were broken.
The site’s repository is hosted on Github. I use (Firebase) for the actual serving. When the Github action attempted to push an update to Firebase, the action produced the following error:
With none of the free time I have, but with the desire to write more and keep that side of my brain fresh, I tried my hand at making my own Hugo theme.
Why? I have drifted between themes over the years, never quite liking anything enough to keep it. Sometimes the content area was too narrow, or the fonts didn’t seem easy enough on my eyes. With things like Bootstrap making the CSS side of things supposedly easy and lots of examples on Github, I set off.
As the title of this post says, I want to back up my Home Assistant (HA) configs to Git, so that in case of a disaster (if I I mistakenly delete the config folder), I can recover.
Note: Disaster recovery is my 2023 personal tech resolution. But that’s besides the point.
Howto
Create an SSH key to be used for commiting your cofigs inside the HA VM.
I did this by using the ‘Terminal’ plugin within HA and running the command
Ubuntu and Linux Mint use different values for their release codenames, even though they both use Ubuntu apt repositories. When trying to install docker using Docker’s instructions, the Linux Mint invocation failed because there is no jammy codename of Docker binaries.
Goal
Make an Ansible playbook like the following that works for both Ubuntu and Linux Mint that handles the release codename correctly.
I made the mistake of upgrading my home Kubernetes cluster, expecting that all the move-pods-around goodness would just work and there would be no downtime of anything.
A Kubernetes upgrade across a set of worker hosts downed the only instance of pi-hole which runs on my network. The upgrade for the worker node runniong pi-hole was stuck in between pods being down, and them being brought back up. The upgrade could not continue until the worker node was rebooted, and the upgrade restarted and successfully completed.
A long long time ago Verizon Fios did not provide native IPv6 connectivity to their IPv4 customers. Following a dslreports forum thread that started in 2018, it looked like it might never happen. But that thread came back to life in 2022 when IPv6 started rolling out to its footprint. This post is about how I set up IPv6 natively on Verizon Fios.
My network stack:
Verizon Fios with ethernet directly from ONT to my router/firewall.
OpenBSD 7.1 router/firewall already providing IPv4 services to my network
For some reason I wanted to try to run my own Git server at home. I’m always looking for ways to automate the infrastructure in my basement (VMs, Ansible, Kubernetes, etc) among other things, and having a centrally-located place for storage of configuration data would help me get one step closer to my goal.
I tried out Gitlab, but the resource requirements were super high, and I didn’t really need all the features. I just need a place to create repositories, add users/roles (with SSH keys), and display changes in a web browser without necessarily checking out the repository. And I wanted to run it on my Kubernetes cluster, for fun, profit, and generally playing around.
Way back in 2019 I wrote about how I moved from Wordpress to Hugo.
Publishing a new blog post involved two Git repositories, http://travis-ci.org, and lots of headaches (submodules, private and public repositories, etc). I always found dealing with the submodules a hassle. I never seemed to figure out a reproducible workflow of update repo, commit and push to Github, and finally updating Firebase. For all the playing around with submodules I’ve done, mastering them still escapes me.
Verizon Fios does not provide native dual-stack connectivity. To enable my connection for IPv6, I signed up for Hurricane Electric’s Tunnelbroker service. To make a long story short, whenever the IPv4 IP on my Fios connection cycled to a new address, I had to manually update my new IPv4 IP in a few places (Tunnel Broker website, /etc/hostname.gif0) and run a few commands.
Solution
Wrap ifstated with ping and a small shell script to automatically restart my IPv6 tunnel on the new IPv4 IP.
Now that I’m handing out IPv6 addresses to various VLANs on my network, I needed a way to see what percentage of my traffic was actually using IPv6. Enter pf, pflow, and ntop. pf is used by my router/firewall to process packets, and mark them for pflow processing. pflow is a psuedo device on the router/fw which exports pflow accounting data to a remote collector. ntop (and nprobe) is a collector and visualization application for digging into packet statistics. Below is the configuration to hook it all up. pf.conf snippet:
It all started with a Ubuntu Blog blog post about a slimmer Ubuntu server image. I play around with virtual machines at home, many based on Ubuntu’s full-size server ISO. It would take 20-25 minutes to spin up a new VM using some prebuilt preseed files I had constructed to automate user creation and SSH key copying. I knew there was a better way, and it turns out, using the pre-built Ubuntu minimal image (subsequently called cloud image), combined with cloud-init infrastructure, I was able to spin up Ubuntu Cloud Image minimal VMs in under 2 minutes.
I remember using he.net year ago for their IPv6 tunnels years ago, and have painful memories of configuring it, both on the router and to share to the subnets on my home LAN. Not this time. Years ago, I had Comcast Business Internet service, which along with providing a static IPv4 address, provided IPv6 connectivity. Not only just a single /128, but a whole /56 if you asked for it. After spending days/weeks configuring both dhcp client and servers for prefix delegation, and slaac/rtadvd to hand out addresses to my various LAN segments, I was in business. Flash forward to 2018, and I’m on a residential Verizon Fios connection, which provides a single dynamic IPv4 address, and no IPv6. Really, no IPv6 connectivity in 2018. Not here.Or here.Or here.Not here either. Your only option is to check out one of the IPv6 tunnel providers out there to wrap your IPv6 in IPv4 and go that direction.
Kubernetes has some incredible features, one of them being Ingress. Ingress can be described as a way to give external access to a Kubernetes-run service, typically over HTTP(S). This is useful when you run webapps (Grafana, Binder) in your Kubernetes cluster that need to be accessed by users across your network. Typically, Ingress integrates with automation provided by public cloud providers like GCP/GKE, AWS, Azure, Digital Ocean, etc where the external IP and routing is done for you. I’ve found bare-metal Ingress configuration examples on the web to be hand-wavy at best. So what happens when there are so many standards, but not sure which one to pick? You make your own. Below is how I configured my bare-metal Ingress on my CoreOS-based Kubernetes cluster to access Grafana.
Several months after my last post, and lots of code hacking, I can rebuild CoreOS-based bare-metal Kubernetes cluster in roughly 20 minutes. It only took ~1300 lines of Python following Kelsey Hightower’s Kubernetes the Hard Way instructions. Why? The challenge. But really, why? I like to hack on code at home, and spinning up a new VM for another Django or Golang app was pretty heavyweight, when all I needed was an easy way to push it out via container. And with various open source projects out on the web providing easy ways to run their code, running my own Kubernetes cluster seemed like a no-brainer. From github/jforman/virthelper: First we need a fleet of Kubernetes VM’s. This script builds 3 controllers (corea-controller{0,1,2}.domain.obfuscated.net) with static IPs starting at 10.10.0.125 to .127, and 5 worker nodes (corea-worker{0,1,2,4,5}.domain.obfuscated.net) beginning at 10.10.0.110. These VMs use CoreOS’s beta channel, each with 2GB of RAM and 50GB of Disk.
I finally got around to wiring Cat6 to my desktop machines at home, and ripped out those powerline network adapters. I ran a test if iperf between my desktop and my router before and after the upgrade to see how things fared. iperf results before:
desktop1:~$ iperf -f m -V -t 30 -c 10.10.0.1
------------------------------------------------------------
Client connecting to 10.10.0.1, TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[3] local 10.10.0.241 port 35262 connected with 10.10.0.1 port 5001
[ID] Interval Transfer Bandwidth
[3] 0.0-30.0 sec 510 MBytes 142 Mbits/sec
It all started when I began hearing about this container thing outside of work. I’ve been a Google SRE going on 6 years, but knowing that the way we do containers internally on Borg is probably not how the rest of the world does reliable, scalable, infrastructure. I was curious, how hard could it be to spin up a few containers and play around like I do at work? Little did I know, it would take two months, a few hours a few nights a week, to get the point where I was able to access a web service inside my home grown Kubernetes cluster. Below are the high level steps, scripts, and notes I kept during the process.
A while back, I wrote a post about setting up an L2TP/IPSec VPN on my home firewall/router. It required two daemons and a bunch of configuration that had hard coded IP addresses. While this solution used firmly-established practices (L2TP/IPSec), it felt too brittle. What happens when my dynamic IP address changes? Now I need to update config files, restart daemons, etc. There had to be a better way. Enter IKEv2. IKEv2 is a successor implementation to Internet Security Association and Key Management Protocol (ISAKMP)/Oakley, IKE version 1. One of the main reasons iked(8) is so great, is not having to use the accompanying ipsecctl binary to manage iked’s configuration. From OpenIKED Asia BSDCON 2013:
I was bored last weekend, so I configured a two-port LACP bonded trunk from my FreeBSD-running NAS connected to my HP Procurve switch. Why?
I could?
I had all these spare Ethernet ports on my NAS, and they seemed bored.
More seriously: high availability. One interface serving all my storage traffic just seemed ripe for failure. Imagine serving all your VMs over NFS to a VM server across the network over one NIC, and that one dies. Bad news bears.
I also wanted to set up VLANs on top of the trunk. Why? So if I wanted to add a network segment for my NAS on another Layer 3 domain, I don’t have to walk down to the basement to patch another cable.
On to the configuration. First, I configured the NAS box. Relevant /etc/rc.conf configuration:
I recently changed Internet providers from Comcast Business to Verizon Fios connection. As part of the Fios package, are TV Set Top Boxes (STB) which use coax for Video, and Internet via MOCA for the guide data. It made me curious, what kind of traffic were these things sending on the network? What would they be trying to access? And how hard would it be to DMZ these things off from the rest of my wired/wifi network given I have no idea what they are up to. Behold, a DMZ configuration Requirements:
For a while I had used Smokeping to generate pretty graphs of network latency between various hosts on my network. The downside with Smokeping was always getting it working. Did I configure my webserver just right? Did I remember to save the webserver configs so that the next time I set this up, things just worked? Did I install all the right Perl modules (and the right versions of each) so that Smokeping’s binary worked? Then there were the differences in operation depending on if I ran it on Linux, OpenBSD, or FreeBSD. There had to be a simpler solution. I’ve been dabbling in Go and Graphite as side projects at home for a while. Go was a language I’d been wanting to use more given its popularity where I work. Graphite was always this itch I scratched whenever I wanted to visualize machine and network statistics for the various machines on my network. I knew I could come up with a simple solution using these two pieces of tech. I wanted to start small. Smokeping provides graphs of minimum, maximium, average, and std deviation for round trip times, as well as packet loss. These are all statistics provided by the ping command line tool. Why couldn’t I just wrap ping in a Go binary, and send those data points off to Carbon for graphing in Graphite? I present the resultant Go binary and library. parallelping is a Go binary used to ping remote hosts, in parallel. If provided with a Carbon host and port, the data is shipped off to Carbon/Graphite. carbon-golang is a Go library used to take Carbon metrics and send them off to a Carbon Cache over TCP. I do admit I borrowed a lot of the logic from marpaia/graphite-golang, both because I couldn’t quite get that library to integrate as documented, but also because I wanted the learning experience of building my own Go-based TCP client. Both of these are my first non-trivial pieces of Go code. The more I spent time with Go the less I felt it’s barrier to entry was as high as anticipated (I’ve been mainly a Python person for many years). Further usage documentation for each bit of code can be found on their respective Github project pages, eventually. Enjoy!
It was as vanilla a setup as one can get, running on a $10/month Linode instance out of their datacenter in Atlanta. I never used the VM much other than for keeping what was an almost-completely static blog. I never had any issues with it. I just wanted to try something new.
No one likes to do repetitive OS installs. You know the kind, where you are just clicking through a bunch of prompts for username, password, and partitioning scheme as fast as you can to quickly get to the point where you can get some work done. This scenario happens to me every time OpenBSD releases a new errata. As my OS of choice for firewalls/routers, I use a fresh OS install as the baseline for building a -stable branch of install set files.
Problem Statement: While OpenVPN has served me well over the past few years both for site-to-site and road-warrior style VPN connections, it always bugged me that I had to hack a config file, juggle certificates, and use a custom client that isn’t part of the base OS to bring up the links. My Android phone has a built-in L2TP/IPSec VPN client. My Macbook Pro OS X 10.9 laptop has both an IPSec and L2TP VPN client GUI wrapped around racoon. I run OpenBSD as my firewall/router gateway at home. There must be a solution here.
I was hoping with my pastposts on this topic, I would have enough examples to just copy-and-paste along to configure my Gitolite+Nagios monitoring setup. Not so true. It looked like there were semi-colon’s missing in my past examples. After looking at the huge number of changes in Gitolite, I had to re-do everything. Not to mention I always wanted a better way to manage the hooks as opposed to editing them directly on the host. In short, my goal is still simple: be able to manage and verify Nagios configuration remotely via Git. Below is how I did it. For the third time.
I’ve had some free time and a desire to break stuff on my network at home. I wanted to fix my home network’s topology to more correctly split up my wired (DHCP), wireless (DHCP) and server (statically-configured) subnets. At a high level, I had to create a server subnet, create vlan’s on my layer-3 switch for each of those pervious subnets, then I had to move the network interfaces on my VM host around to only connect to the networks I wanted it to (wired and server).
In my initial post about unattended Ubuntu installs, I made the less-automated choice of hacking at the Ubuntu installation ISO and baking my preseed configuration right into the ISO. This proved to be incredibly inefficient and prevented a lot of the customization and quick-spin-up potential of what I interested in. In other words, if I wanted to spin up five identical VMs differing only by their hostname, was I really expected to bake five custom ISO’s whose preseed file only differed by their specification of the hostname?
In my day job, it’s all about automation. Automate what is repeatable, and move on to more interesting and not-yet-automated tasks. For a while, I’ve run a KVM/libvirt setup at home, running various iterations and distributions of Linux, OpenBSD and FreeBSD for various pet projects. Going through each distribution’s install procedure was getting old, requiring me to input the same parameters, set up the same users and passwords, over and over again. Given I use Ubuntu mostly as a VM guest, I dug into their preseed infrastructure, to be able to automate the installation and get me past the drudgery of adding another VM. Below are the steps and a bit of sample configuration that got me through the process.
After several years of mindlessly running Ubuntu on the desktop, I am attempting to dive (back) into running FreeBSD on the desktop. Considering that the majority of applications I use on the desktop are a browser (Firefox/Chrome), an ssh terminal, and Rhythmbox, how hard could this be?
Some of the hurdles
Given I still wanted to keep Ubuntu around and not redefine my default setup, I kept Grub2 as my bootloader on the MBR. I still needed a way to boot into FreeBSD at-will. I had installed FreeBSD on hd0a. Grub2 from Ubuntu makes finding the FreeBSD boot files incredibly easy:
A while back I blogged about how I hooked up Nagios and Git to run the Nagios preflight checks before restarting with a new checkin’s worth of configs. But the more I looked at how it all fit together, the more I knew it could be improved. A sed hack, expecting a certain pattern in the nagios.cfg? Bad bad bad. Most of the improvement revolves around Nagios’s ability to reference relative paths for its config files. Given the path of the ‘main’ nagios.cfg file, you can then reference directories that contain your services, hosts, and other custom commands, in relation to that main file. With this functionality I significantly improved the Git->Nagios pipeline.
We have all heard the same questions at one point in our careers, “Is the Internet down?” or “Getting to X site is slow.” You scramble to a browser to see if Google, ESPN or the NY Times websites are up. Then you fire up traceroute. In some cases, the pages might load slowly, in other cases not at all. These two situations are often downstream fallout of two connectivity issues: latency and packet loss. Latency is the time it takes for a packet to get from source to destination. The speed of light says the latency for one packet to get across the USA from New York to San Francisco is normally between 70-90ms [1]. Packet loss occurs when packets do not make it from their source to destination, being lost along the way. Many factors can contribute to packet loss, including overloaded routers and switches, service interruptions, and human error.
I started to write this post, explaining how I upgraded my home network setup with a dhcpd server, multiple dns servers communicating securely via tsig keys along with dynamic dns, but the post became unwieldy and would have been thousands of words. Instead, I’ll post some links and gotcha’s and hints on how to make it work a lot easier.
I was having issues trying to get Nagios to more easily query my APC UPS with the APC-provided MIB. It took me a while to figure out the right bits both on the file system and in my query to have the MIB ‘processed.’ I still don’t know how to add that MIB to the “automatically process me too if snmpwalk is run” piece of the puzzle.
But for what I have running a home, some notes for myself and others who ripped out enough hair already.
TL;DR: How to hand out DNS servers in different orders to different clients based upon MAC address.
Background: I was connected into my office’s VPN a few months ago and was noticed some very slow DNS resolution of host names back at the office. I would attempt to ssh into another host, and the connection would sit there for more than a few seconds before finally proceeding. This didn’t happen for just ssh, but also for making http requests. I dug into my resolv.conf locally and tried sending a few DNS queries via dig to the two DNS servers I was provided. The first one failed, the second one returned immediately with the correct response. I swapped the two entries and DNS resolution locally was back to where I would expect it, very fast. I alerted our IT group and the issue was fixed (the first DNS server had become hung, and needed a process restart).
For those who have spent time debugging their Comcast Internet connection, we all know the frustration of trying to explain to Comcast that something on their end is the problem. In this case, more data is better: latency history, ping times, traceroutes, etc. You can run Smokeping to monitor latency between your home connection and a remote Internet IP address for example. You can also print out traceroute examples and email them if you have an astute support contact. But if you want to monitor the data your cable modem is seeing, you need to look at the signal to noise ratio of your connection. This ratio refers to how much of your signal has been disturbed by noise on the physical line (Thanks Wikipedia). Newer cable modems will use multiple channels along the same line to increase your download and upload speed, and each channel can be disturbed independently.
At work we have a monitoring configuration workflow where our Nagios config files are parsed and generated before they are allowed to be ‘svn commit’ed. I know this verification has saved me many times when trying to add new hosts or services, since everything might not be ready for prime time. I wanted to see if I could recreate this scenario at home using Git hooks, if only for my own interest and curiosity.
Compared to most other home networks, mine is a bit more complicated. I admit networking has always been an interest of mine, so I run my own OpenBSD firewall/router/vpn-endpoint, which itself runs the ISC Dhcpd v3 and BIND. With these together, I am able to run dynamic DNS. But some background first.
Insert my 11-year old HP2100m printer that I have outfitted with an HP Jetdirect 610N 10/100 internal print server. This allows me to stick this printer on my network and print away, no intermediary required. What does this save? The printer is not tied to one computer’s parallel/USB port. I can shut off my main desktop, and still print over the network.