homelab Linux proxmox

Proxmox Cluster manual update

Recently ran into an issue where I added a node to my Proxmox cluster while a node was disconnected & off. That node (prox) caused the others to become unresponsive for a number of Proxmox things (it was missing from the cluster) upon boot.

Set node to local mode

The solution was to put the node that had been offline (called prox) into “local” mode. Thanks to Nicholas of Technicus / for the commands to do so:

sudo systemctl stop pve-cluster
sudo /usr/bin/pmxcfs -l

This allows editing of the all-important /etc/pve/corosync.conf file.

Manually update corosync.conf

I basically just had to copy over the config present on the two synchronized nodes to node prox, then reboot. This allowed node prox to join the cluster again and things started working fine.

Problem corosync.conf on node prox:

logging {
  debug: off
  to_syslog: yes

nodelist {
  node {
    name: prox
    nodeid: 1
    quorum_votes: 1
  node {
    name: prox-1u
    nodeid: 2
    quorum_votes: 3

quorum {
  provider: corosync_votequorum

totem {
  cluster_name: prox-cluster
  config_version: 4
  interface {
    linknumber: 0
  ip_version: ipv4-6
  secauth: on
  version: 2

Fancy new corosync.conf on nodes prox-1u and prox-m92p:

logging {
  debug: off
  to_syslog: yes

nodelist {
  node {
    name: prox
    nodeid: 1
    quorum_votes: 1
  node {
    name: prox-1u
    nodeid: 2
    quorum_votes: 3
  node {
    name: prox-m92p
    nodeid: 3
    quorum_votes: 1

quorum {
  provider: corosync_votequorum

totem {
  cluster_name: prox-cluster
  config_version: 5
  interface {
    linknumber: 0
  ip_version: ipv4-6
  secauth: on
  version: 2

The difference is that third node item as well as incrementing the config_version from 4 to 5. After I made those changes on node prox and rebooted, things worked fine.





SSH Key Tutorial

SSH Key Tutorial

This post will be a basic SSH key tutorial. In my Securing this WordPress blog from evil hackers! post, I recommended turning off password based SSH authentication and moving exclusively to SSH key based authentication. This post will go over the steps to set up SSH key authentication. The outline is:

  1. Generate SSH public/private key pair
  2. Transfer the public key to the host in question
    1. manual method (Windows)
    2. automatic method (Linux/Mac)
  3. Verify the SSH key authentication is functional.

At the end of this post is my user creation script which does all of this automatically.

I do go over this in a good amount of detail in my first ever YouTube video which you can view here if you’d prefer a video instead of text.

I will be generating keys on my new laptop and transferring them to the Raspberry Pi I set up in my second YouTube video (Austin’s Nerdy Things Ep 2 – Setting up a Raspberry Pi) –

Step 1 – Generate the SSH key pair

The first thing we need to do is generate the SSH key pair. On all major OSes these days, this command is included (Windows finally joined the other “real” OSes in 2018 with full SSH support out of the box). We will be using ED25519, which is advised for basically anything build/updated since 2014.

ssh-keygen -t ed25519

This will start the generation process. I hit enter when it prompts for the location (it defaults to C:/Users/<user>/.ssh/id_<type> which is fine). I also hit enter again at the passphrase prompt twice because I don’t want to use a passphrase. Using a passphrase increase security but it can’t be used for any automated processes because it’ll prompt every time for the passphrase. This is the full output from the process:

C:\Users\Austin>ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\Austin/.ssh/id_ed25519): [press enter here]
Enter passphrase (empty for no passphrase): [press enter here]
Enter same passphrase again: [press enter here]
Your identification has been saved in C:\Users\Austin/.ssh/id_ed25519.
Your public key has been saved in C:\Users\Austin/.ssh/
The key fingerprint is:
SHA256:t0FIIk<snip.......................snip>6Rx4 [email protected]
The key's randomart image is:
+--[ED25519 256]--+
|o++++++ .        |
|*  + * o .       |
|o.o o   . .      |
| . .     .       |
|..... o S o      |
| . + .  .o.o..   |
|  = . o.+.+=o..  |
|   +.o.  +a..o   |
| o+...    |

Step 2 – Transfer the SSH public key

For Windows, they included all the useful SSH utilities except one: ssh-copy-id. This is the easy way to transfer SSH keys. It is possible to transfer the key without using this utility. For this SSH key tutorial, I will show both.

Step 2a – Transfer SSH key (Windows)

First we need to get the contents of the SSH public key. This will be done with the little-known type command of Windows. In Step 1, the file was placed at C:\Users\Austin\.ssh\id_ed25519 so that’s where we will read:

C:\Users\Austin>type C:\Users\Austin\.ssh\
ssh-ed25519 AAAAC3NzaC1lZD2PT/VIK3KyYRliA0jgqmn7yzkVjuDiFK67Cio [email protected]

The contents of the file are the single line of the SSH public key (starting with the type of key “ssh-ed25519” and then continuing with the key contents “AAAAC3” and ending with the username and host the key was generated on “[email protected]”). This is the data we need to transfer to the new host.

I will first log onto the Raspberry Pi with the username/password combo from the default installation:

ssh [email protected]

I am now logged into the Pi:

Raspberry Pi Log In Prompt
Logged into the Raspberry Pi

Now on the target machine we need to create the user account and set up some initial things. We use sudo on all of these since root level permissions are required.

# create user austin with home directory (-m means create the home directory)
sudo useradd -m austin

# set the password for user austin
sudo passwd austin

# add user austin to sudo group
sudo usermod -aG sudo austin

# make the .ssh directory for user austin
sudo mkdir /home/austin/.ssh
sudo useradd
User created, password set, sudo added, and directory created

With those four commands run, the user is created, has a password, can run sudo, and has an empty .ssh directory ready for the keys.

Now we need to create the authorized_keys file:

sudo nano /home/austin/.ssh/authorized_keys

In this file, paste (by right-clicking) the single line from the id_ed25519 file:

nano authorized_keys
line pasted and ready to save

To save the file, press CTRL+O and a prompt will appear. Press enter again to save. Now to exit, press CTRL+X. There should be no “are you sure” prompt because the contents are saved.

Lastly, we need to change the owner of the .ssh directory and all of it’s contents to the new user (sudo means they were created as root):

sudo chown -R austin:austin /home/austin/.ssh

Now we can test. From the machine that generated the SSH key, SSH to the target with the new user specified:

ssh [email protected]

You should get in without having to type a password!

logged in with SSH key
Successfully logged in without typing a password (the SSH key worked)
Step 2b – Transfer SSH key (Linux)

This is much easier. The ssh-copy-id utility does what we just did in a single command.

# make sure you've already generated keys on your Linux/Mac machine
ssh-copy-id [email protected]
linux ssh-copy-id command
ssh-copy-id copied the keys and did everything for us in a single command

Microsoft – will you please add ssh-copy-id to default Windows installations?

Step 3 – Verify

We did this at the end of each of the other methods.

User creation script

This is the script I use on any new Linux virtual machine or container I spin up. It updates packages, installs a few things, sets the timezone, does all the user creation stuff and sets the SSH key. Lastly, it changes the shell to bash. Feel free to modify this for your own use. It is possible to copy the data from /etc/shadow so you don’t have to type in the password. I haven’t got that far yet.

apt update
apt upgrade -y 
apt install -y net-tools sudo htop sysstat git curl wget zsh
timedatectl set-timezone America/Denver
useradd -m austin
passwd austin
adduser austin sudo
usermod -aG sudo austin
usermod -aG adm austin
groups austin
mkdir /home/austin/.ssh
touch /home/austin/.ssh/authorized_keys
echo 'ssh-rsa AAAAB  <snip>  bhgqXzD [email protected]' >> /home/austin/.ssh/authorized_keys
chown -R austin:austin /home/austin/.ssh
usermod -s /bin/bash austin
Linux Python Weather

Python service to consume Ambient Weather API data

Python service to consume Ambient Weather API data

Continuing from the last post (Handling data from Ambient Weather WS-2902C to MQTT), we have a working script that reads the data coming from the Ambient Weather WS-2902 base station (Ambient Weather API) and sends it to a MQTT broker. In this post, we will turn that script into a Linux service that starts at boot and runs forever. This type of thing is perfect for Linux. Non-server Windows versions aren’t really appropriate for this since they reboot often with updates and such. If you want to run Windows Server, more power to you, you probably don’t need this guide. We will thus be focusing on Linux for consuming the Ambient Weather API. A Raspberry Pi is a perfect device for this (plenty of power, as big as a credit card, and less than $100).

Linux Services

Linux is made up of many individual components. Each component is designed to handle a single task (more or less). This differs from Windows where there are large executables/processes that handle many tasks. We will be taking advantage of Linux’s single task theory to add a new service that will run the Python script for ingesting and consuming the Ambient Weather API. In this instance for Austin’s Nerdy Things, the weather data is being provided by the Ambient Weather WS-2902C weather station.

Creating the service

For Ubuntu/Debian based distributions, service files live under /etc/systemd/system. Here is a list of services on the container I’m utilizing.

List command:

ls -1 /etc/systemd/system

Since this is a LXC container, there aren’t many services. On a standard Raspbian or Ubuntu full install, there will be 100+.

We will be creating a new service file using the nano text editor:

nano /etc/systemd/system/ambient-weather-api.service

In this file we need to define our service. The After lines mean don’t start this up until the listed services are running. The rest is pretty straight forward. I’m not 100% sure what the WantedBy line is for but it’s present in most of my service files. The contents of ambient-weather-api.service are as follows:

Description=Python script to ingest Ambient Weather API data

ExecStart=/usr/bin/python3 /srv/ambient-weather-api/
ExecStartPre=/bin/sleep 5
# Give the script some time to startup


Save the file. The service definition is looking for Python at /usr/bin/python3 and the python script at /srv/ambient-weather-api/ You will probably be fine with the Python executable, but be sure to mv or cp the file to /srv/ambient-weather-api/

We will need to reload the service definitions:

systemctl daemon-reload

Now we can start the service:

systemctl start ambient-weather-api.service

And verify it is running:

systemctl status ambient-weather-api.service
Ambient Weather API service status
Ambient Weather API service status

The above screenshot shows it is indeed running and active. It is still showing the print messages in the log as well, which we should disable by commenting out the lines by adding a # in front of the print line. In this case, it is coming from line 56 (the status check in the publish function).

#print(f"Sent {msg} to topic {topic}")





Home Assistant Weather

Handling data from Ambient Weather WS-2902C API to MQTT

Handling data from Ambient Weather WS-2902C API to MQTT

As I mentioned in my initial Ambient Weather WS-2902C post, there is a new feature that allows sending data to a custom server. I coded up a python script to take the sent data, and publish it to MQTT. This allows for super easy data ingestion with Home Assistant and other similar solutions. I should probably publish on GitHub but I’ll post here first.


This script is reliant on Paho-MQTT. EDIT: during the creation of the service to run this at boot, I discovered version 1.5.1 will throw errors. Use version 1.5.0. Install it with pip:

sudo pip install paho-mqtt==1.5.0

Create a python file and name it Paste in the following:

# Python script to decode Ambient Weather data (from WS-2902C and similar)
# and publish to MQTT.
# original author: Austin of
# publish date: 2021-03-20

# some resources I used include

from urllib.parse import urlparse, parse_qs
import paho.mqtt.client as mqtt
import time, os

# set MQTT vars
MQTT_BROKER_HOST  = os.getenv('MQTT_BROKER_HOST',"mqtt")
MQTT_BROKER_PORT  = int(os.getenv('MQTT_BROKER_PORT',1883))
MQTT_CLIENT_ID    = os.getenv('MQTT_CLIENT_ID',"ambient_weather_decode")
MQTT_USERNAME     = os.getenv('MQTT_USERNAME',"")
MQTT_PASSWORD     = os.getenv('MQTT_PASSWORD',"")

# looking to get resultant topic like weather/ws-2902c/[item]
MQTT_TOPIC_PREFIX = os.getenv('MQTT_TOPIC_PREFIX',"weather")
MQTT_TOPIC 		  = MQTT_TOPIC_PREFIX + "/ws-2902c"

# mostly copied + pasted from and some of my own MQTT scripts
def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print(f"connected to MQTT broker at {MQTT_BROKER_HOST}")
        print("Failed to connect, return code %d\n", rc)

def on_disconnect(client, userdata, flags, rc):
    print("disconnected from MQTT broker")

# set up mqtt client
client = mqtt.Client(client_id=MQTT_CLIENT_ID)
    print("Username and password set.")
client.will_set(MQTT_TOPIC_PREFIX+"/status", payload="Offline", qos=1, retain=True) # set LWT     
client.on_connect = on_connect # on connect callback
client.on_disconnect = on_disconnect # on disconnect callback

# connect to broker

def publish(client, topic, msg):
    result = client.publish(topic, msg)
    # result: [0, 1]
    status = result[0]

    # uncomment for debug. don't need all the success messages.
    if status == 0:
        #print(f"Sent {msg} to topic {topic}")
        print(f"Failed to send message to topic {topic}")

def application(environ, start_response):
    # construct a full URL from the request. HTTP_HOST is FQDN, PATH_INFO is everything after
    # the FQDN (i.e. /data/stationtype=AMBWeatherV4.2.9&&tempinf=71.1&humidityin=35)
    url = "http://" + environ["HTTP_HOST"] + environ["PATH_INFO"]

    # unsure why I need to parse twice. probably just need to do it once with variable url.
    parsed = urlparse(url)
    result = parse_qs(parsed.geturl())

    # send to our other function to deal with the results.
    # result is a dict 

    # we need to return a response. HTTP code 200 means everything is OK. other HTTP codes include 404 not found and such.
    start_response('200 OK', [('Content-Type', 'text/plain')])

    # the response doesn't actually need to contain anything
    response_body = ''

    # return the encoded bytes of the response_body. 
    # for python 2 (don't use python 2), the results don't need to be encoded
    return [response_body.encode()]

def handle_results(result):
    """ result is a dict. full list of variables include:
    stationtype: ['AMBWeatherV4.2.9'], PASSKEY: ['<station_mac_address>'], dateutc: ['2021-03-20 17:12:27'], tempinf: ['71.1'], humidityin: ['36'], baromrelin: ['29.693'],	baromabsin: ['24.549'],	tempf: ['58.8'], battout: ['1'], humidity: ['32'], winddir: ['215'],windspeedmph: ['0.0'],	windgustmph: ['0.0'], maxdailygust: ['3.4'], hourlyrainin: ['0.000'],	eventrainin: ['0.000'],	dailyrainin: ['0.000'],
    weeklyrainin: ['0.000'], monthlyrainin: ['0.000'], totalrainin: ['0.000'],	solarradiation: ['121.36'],
    uv: ['1'],batt_co2: ['1'] """

    # we're just going to publish everything. less coding.
    for key in result:
        # skip first item, which is basically a URL and MQTT doesn't like it. probably resulting from my bad url parsing.
        if 'http://' in key:

        #print(f"{key}: {result[key]}")
        # resultant topic is weather/ws-2902c/solarradiation
        specific_topic = MQTT_TOPIC + f"/{key}"

        # replace [' and '] with nothing. these come from the url parse
        msg = str(result[key]).replace("""['""", '').replace("""']""", '')
        #print(f"attempting to publish to {specific_topic} with message {msg}")
        publish(client, specific_topic, msg)

# this little guy runs a web server if this python file is called directly. if it isn't called directly, it won't run.
# Apache/Python WSGI will run the function 'application()' directly
# in theory, you don't need apache or any webserver. just run it right out of python. would need
# to improve error handling to ensure it run without interruption.
if __name__ == '__main__':
    from wsgiref.simple_server import make_server

    # probably shouldn't run on port 80 but that's what I specified in the ambient weather console
    httpd = make_server('', 80, application)
    print("Serving on http://localhost:80")


Execute with python3:


Watch the results populate in the console window:

PS C:\Users\Austin\source\repos\ambient-weather-decode> python3 c:\Users\Austin\source\repos\ambient-weather-decode\
connected to MQTT broker at mqtt
Serving on http://localhost:80
Sent E0:98:06:A3:42:65 to topic weather/ws-2902c/PASSKEY
Sent 2021-03-20 17:57:31 to topic weather/ws-2902c/dateutc
Sent 70.7 to topic weather/ws-2902c/tempinf
Sent 36 to topic weather/ws-2902c/humidityin
Sent 29.675 to topic weather/ws-2902c/baromrelin
Sent 24.531 to topic weather/ws-2902c/baromabsin
Sent 66.2 to topic weather/ws-2902c/tempf
Sent 1 to topic weather/ws-2902c/battout
Sent 26 to topic weather/ws-2902c/humidity
Sent 207 to topic weather/ws-2902c/winddir
Sent 0.2 to topic weather/ws-2902c/windspeedmph
Sent 1.1 to topic weather/ws-2902c/windgustmph
Sent 3.4 to topic weather/ws-2902c/maxdailygust
Sent 0.000 to topic weather/ws-2902c/hourlyrainin
Sent 0.000 to topic weather/ws-2902c/eventrainin
Sent 0.000 to topic weather/ws-2902c/dailyrainin
Sent 0.000 to topic weather/ws-2902c/weeklyrainin
Sent 0.000 to topic weather/ws-2902c/monthlyrainin
Sent 0.000 to topic weather/ws-2902c/totalrainin
Sent 697.92 to topic weather/ws-2902c/solarradiation
Sent 6 to topic weather/ws-2902c/uv
Sent 1 to topic weather/ws-2902c/batt_co2


Verify in MQTT by subscribing to topic ‘weather/#’. The # is a wildcard and will include all subtopics:

homeassistant ws-2902c mqtt messages
homeassistant ws-2902c mqtt messages


I am stoked the Ambient Weather WS-2902C makes it so easy to work the the data.

My next post will show how to turn this Python script into a persistent Linux service – Python service to consume Ambient Weather API data.

A further post will demonstrate incorporating these MQTT messages into Home Assistant sensors.


The Ambient Weather WS-2902C weather station!

The Ambient Weather WS-2902C

My new weather station came in the mail today. It is a Ambient Weather WS-2902C WiFi-Enabled Smart Weather Station. It was supposed to arrive tomorrow but I’m glad it showed up a day early, thanks Amazon! The newest firmware version also has a feature that blew me away – read on to find out what. I’m sure many others will appreciate it once they realize how powerful it is.

What I was looking for in a weather station

Before I bought a weather station, I decided on a few key requirements:

  • Basic weather data – outdoor temperature and humidity, wind speed and direction, and rain
  • The sensor station doesn’t need batteries replaced sooner than twice a year
  • A base station that shows the outdoor info with some additional inside info (inside temperature and humidity mostly)
  • The remote sensor station emits signals that could be decoded by rtl_433 so I could integrate it into my Home Automation/Smart Home system
  • Less than $200 (it is amazing how expensive some of these weather stations are! Looking at you, Davis Vantage Pro2, which is 3x the cost of the Ambient Weather WS-2902C as of writing this post)
Why I picked the WS-2902C

The Ambient Weather WS-2902C fit all of my requirements…. AND it has a few bonuses:

  • Additional sensors include: solar radiation (measured in watts per square meter) and pressure
  • Super capacitor charged by a small solar panel to really cut down on battery changes
  • Ability to interface with and send weather data to various weather services such as Weather Underground, Ambient Weather’s own service, and a few others
Initial thoughts

The sensor station is a nice little unit. I need to order a pole mount but it will be fine for a few days in my backyard. I am a but unsure why they put the solar panel on the north side of the solar sensor – it will surely block some sun. Here is what it looks like in my back yard (this snow is from the 4th largest snowstorm in Denver recorded history):

Ambient Weather WS-2902C testing
Ambient Weather WS-2902C sensor base in my backyard

The base station display looks crisp and the buttons are responsive and work as expected:

Ambient Weather WS-2902C base station
Ambient Weather WS-2902C base station in my kitchen

Nice bonus is the packaging and other packaging contents are almost entirely recyclable – nice work Ambient Weather!

Ambient Weather WS-2902C recyclable packaging
Ambient Weather WS-2902C recyclable packaging

Connecting it to my WiFi was straight-forward. The iOS app is showing 2 stars out of 5 in the Apple App Store. Not sure why – it worked fine for me.

The bonus feature

When loading up the app for the first time it listed firmware v4.2.8 as the latest. In the firmware release notes was an entry – Version 4.2.8 – adds path for custom server setup. This immediately caught my eye so I updated the firmware. Indeed the app did show an entry for a custom server that I immediately filled out even though I didn’t have hosts on my network named “weather”:

Ambient weather custom server
Ambient weather custom server screen

I had no idea what the format was but I guessed it was either an HTTP POST or GET. I created a quick LXC container in Proxmox named “weather”, installed apache2, and checked the access logs. Sure enough, it was hitting my web server! Further, it was valid, usable data! - - [19/Mar/2021:20:20:13 +0000] "GET /data/stationtype=AMBWeatherV4.2.9&PASSKEY=<MAC_ADDRESS>&dateutc=2021-03-19+20:20:12&tempinf=70.3&humidityin=29&baromrelin=29.900&baromabsin=24.756&tempf=62.8&battout=1&humidity=31&winddir=188&windspeedmph=1.1&windgustmph=3.4&maxdailygust=5.8&hourlyrainin=0.000&eventrainin=0.000&dailyrainin=0.000&weeklyrainin=0.000&monthlyrainin=0.000&totalrainin=0.000&solarradiation=622.94&uv=6&batt_co2=1 HTTP/1.1" 404 467 "-" "ESP8266" - - [19/Mar/2021:20:20:32 +0000] "GET /data/stationtype=AMBWeatherV4.2.9&PASSKEY=<MAC_ADDRESS>&dateutc=2021-03-19+20:20:31&tempinf=70.3&humidityin=29&baromrelin=29.894&baromabsin=24.750&tempf=62.8&battout=1&humidity=32&winddir=189&windspeedmph=2.5&windgustmph=3.4&maxdailygust=5.8&hourlyrainin=0.000&eventrainin=0.000&dailyrainin=0.000&weeklyrainin=0.000&monthlyrainin=0.000&totalrainin=0.000&solarradiation=620.73&uv=6&batt_co2=1 HTTP/1.1" 404 467 "-" "ESP8266"

This means I don’t even need to reconfigure my rtl_433 set up to add a listener for the Ambient Weather WS-2902C. It will send the data directly to my server! People have been poking around these for a while, trying to connect to the base stations (port 45000 is open on Telnet) which are ESP devices (likely ESP8266 or similar). Ambient Weather needs to promote this. This is a slam dunk for a ton of people in the Home Automation/weather nerd categories. This post is long enough, I’ll document how to parse the data out in a later post. Update: I have put together a python script to parse the data and send it to MQTT here – Handling data from Ambient Weather WS-2902C to MQTT

In summary

In summary, the Ambient Weather WS-2902C appears to check all my boxes for a weather station, be reasonable priced, and connects directly to my own server to post data without needing to decode the radio traffic. Initial score for this device is 10/10 (I was going to say 9/10 until I discovered the custom server feature. That easily adds +2 points).


Austin’s Nerdy Things is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to


Receiving aircraft ADS-B (position) signals – part 3 (antenna reposition)

Welcome back from Receiving aircraft ADS-B (position) signals – part 2!

Yesterday I moved the antenna up a couple feet from a “very temporary” position to a “less temporary” position. I still need to get it up on the roof. Either way, my reception and max range have increased by at least 20%. It is still in my garage, which is a terrible location, but at least is elevated.

New antenna placement

ADS-B antenna in garage
ADS-B antenna in garage (we need a shed for all that gardening stuff)

The antenna needs to be vertical for maximum reception and upside down works just as well as right side up!

New antenna placement results

I saw up to 82 aircraft being tracked this morning. You can see the big planes lining up for southerly landings at KDEN on three separate runways and a few planes taking off to the southeast.

piaware updated antenna KDEN focus
PiAware updated antenna KDEN focus

Here is a screenshot zoomed out. I’ve got quite a few position reports from >100 nm out.

piaware ADS-B zoomed out KDENProposed final placement of the FlightAware ADS-B antenna

Proposed placement of ADS-B antenna on chimney
Proposed placement of ADS-B antenna on chimney (1 is where it currently is inside the garage and 2 is desired final placement)

I have a new weather station coming this weekend so I’ll try to combine roof trips to install the weather station and the 1090 MHz Flightaware antenna at the same time. I’ll also need to drill a hole in the side of the house to run the cable. Or maybe I do it all solar powered and use WiFi 🤔? I also have a 25W solar panel arriving this weekend. Keeping it physically isolated from the house would eliminate some lightning risk. Decisions, decisions. Be on the lookout for a part 4 with the results from the new roof placement! Part 4 – moving the antenna to the roof, has produced great results! I’m now seeing 150+ planes at once and getting over 700 messages per second.

DIY Offgrid Solar

Planning my 600W DIY solar system with 6 kWh battery backup

This post is a work in progress! Update log here:
  • 2021-03-15 – initial post (private)
  • 2021-04-06 – fleshing out the background and requirements
  • 2021-04-29 – updated with parts ordered, reasoning for choices, and some more background for my DIY solar system with battery backup


I’ve always been interested in solar power. Being able to generate heat and electricity from the sun is just so cool on a fundamental level. When I was little, playing with magnifying glasses (read: setting things like plastics and mulch on fire) was always a good time. My mom got me a science book at one point that had a full letter sized (8.5×11″) fresnel lens.

large fresnel lens focused on pavement
The fresnel lens of my childhood wasn’t quite this big but but it sure seemed like it to my 10 year old brain

That fresnel lens upped my lighting things on fire game dramatically. Even since then I’ve wanted to harness large amounts of solar power. I’ve had 50-100W solar panels for a good portion of my adult life running fans and charging small deep cycle 12v batteries, and it is now time to move up to the big leagues. Read on to join my thought process for planning a large-ish system.


The requirements for my DIY solar system with battery backup aren’t too strict. I’m looking for the following:

  • Run my homelab for 5-10 minute until it can be powered off
  • Provide a couple hours (1-2) of space heating/cooling for comfort with plenty of battery left over
  • Run the refrigerators for 6-12 hours
  • Run the cable modem/router/WiFi for ~6-12 hours
  • Run the furnace as needed
  • Ability for a generator (to be purchased) to charge the batteries
  • Ability for grid power to charge the batteries
  • Ability for solar panels to charge the batteries
  • Less than $2000 total to get started with a system that can grow
  • Use my 2x300W solar panels I picked up off Craigslist for $100 each
Nice to haves
  • USB/RS-232/RS-485/Ethernet Interface to read status via Raspberry Pi or similar
  • Decent warranty (I don’t usually worry about warranties but this will be a decent chunk of change)
  • Not waiting another two months to ship from China (I may have already ordered the batteries. Ordered Feb 26 2021. Still waiting for even a tracking number as of April 29.)

Initial plan

If we add up all the electricity requirements, we end up with a couple to a few kWh (I am being intentionally vague here. I’ll post details with my next update.). This DIY solar system with battery backup is intended to grow with me – I’m not building a data center-sized system to start. As such, I have a tentative list of the basics:

  • 2x300W solar panels. They are Canadian Solar CSUN-something 36V nominal. Already have these.
  • 8x272Ah LiFePO4 batteries in series for 24V nominal. These will total out to 6.9 kWh of storage assuming full capacity. For $101 per cell shipped, this deal is hard to beat even if it is taking the slow boat from China. 6.9 kWh divided by $101 per cell is $116/kWh.
  • A 2.5ish kW inverter. Current choices are MPP Solar LV2424 (2.4 kW 24V with most of my requirements for ~$700) or the Growatt SPF 3000TL LVM (3.0 kW 24V with basically the same features as the MPP for ~$700. but there will be at least a month shipping delay).
  • A quality 8S BMS (expect to spend around $150 for this)

Solar panels

I get an urge to troll craigslist for solar panels (and NAS’s) every couple weeks and came across a post that had 300W solar panels in Loveland, CO. They were in great shape and they were $100 each. $0.33/W is a pretty good price for solar panels so I jumped on them. I didn’t really have a use but knew I would in the future. There is a slight “prepper” tendency I always have in the back of my mind so part of me was thinking I’d be able to use them to charge stuff in the event of an extended power outage. Since I bought them, we have had 3 power outage – one for 2 hours, one for 1 hour, and another for 15 minutes.

[insert pic of solar panels]


For batteries, there are a lot of good options. Some better than others. There are a few big decisions:

  • Battery chemistry
    • Lead acid – the traditional “car battery” type but deep cycle. Old tech, heavy, usable capacity is relatively little compared to the full rated capacity (generally recommended to not discharge deeper than 50%). Pretty good price in terms of watt-hours per dollar. Almost all inverters/chargers are designed around 12V/24V/48V as defined by the lead acid cell voltages.
    • Lithium-ion – new tech. Used in many electric car batteries – primarily Tesla. Lots of used cells available (often in bulk). Each cell is about 10Wh. This means many wire connections (500-1000) and soldering. Does not handle overcharging/discharging well. Can cause fires/explosions if handled improperly. No good solutions for 12V standard stuff. 7S (7 cells in series) can work for 24V. 13S works for 48V
    • Lithium polymer – very power dense. Not very energy density. Quite hazardous. That by itself is enough to write these off.
    • Lithium iron phosphate (LiFePO4) – new tech, decent tradeoff for all other aspects mentioned above. Used in electric buses in China (which is a source for cells). Very large capacity per cell (>200Ah), which means minimal wiring. Cell voltage is 3.2V, which matches up perfectly with traditional lead acid voltages (4S is 12V, 8S is 24V, 16S is 48V). Good cycle count/capacity curve (it takes many cycles to reduce capacity). I will be using LiFePO4 batteries in my system.
  • Battery bank voltage – requirements are for a 2.5kW inverter.
    • 12V – 200+ amps for a 2.5kW inverter. This would need large wires. Generally the amount of current at 12V throughout the system would be high. Ability to “start small” with only 4 LiFePO4 cells.
    • 24V – 100 amps for 2.5kW inverter. Much more reasonable. I will be using 24V for my system.
    • 48V – 50 amps for 2.5kW inverter. Even more reasonable but this requires greater up front investment to get enough batteries (16 cells for LiFePO4, meaning $2000+). Borders on what is considered “high voltage” for low voltage DC work (generally the cutoff is 50V).

Below is a table I created in Excel to help me make my decision. When I came across the group buy for the DIYSolar Michael Caro 272Ah cell group buy from China, I took 2 days to decide and ordered 8 cells. That was Feb 26, 2021. I still don’t even have a tracking number. I’ll probably cancel the order. Mid-April, 260Ah cells became available at They weren’t the cheapest in terms of watt-hours per dollar, but they were in Pennsylvania and would arrive to me in a predictable amount of time. With my yearly bonus and tax refund firmly in my bank account, I figured I could have two orders opened at once. I placed the order with BatteryHookup. It took 6 days for 8 cells to arrive. I still don’t have a tracking number for the group buy from China. I can afford to wait. Or I could cancel the China order and get 8 more cells on my door step a few days from now… decisions, decisions.

namelg chem 4p modulesnissan volt 8 packbasen 280ah lifepo4varicore 280ahmichael diysolarbatteryhookup 260ah cells
nom voltage (V)
rated cap (Ah)6064272280272260
cap remaining (%)70%70%100%95%100%100%
usable cap (Ah)4244.8272266272260
cell energy (wh)151.2340.48870.4851.2870.4832
cost ($/cell)2040116114.995101125
total cells1468888
nom bank V25.222.825.625.625.625.6
max bank V29.425.
min bank V22.419.222.422.422.422.4
total cells1688888
bank capacity211720436963681069636656
bank cost320320696919.968081000
shipping cost60542530088
total cost380374949919.968081088
bank $/kwh180183136135116163
prosmodular-ishmodular-ishnew, bignew, bignew, bigfast shipping, good capacity
consgood amount of hooking up, usedgood amount of hooking up, used, bad voltageslong shipping time (30-50 days)long shipping, grade Blong shippingexpensive


For the inverter, it really came down to two options:

  • MPP Solar LV2424 – 24V 2.4kW 120V (able to be stacked for split phase and/or more current) – this is what I picked
  • Growatt SPF 3000TL LVM – 24V 3.0kW 120V (able to be stacked for split phase and/or more current)

I posted a poll on DIYSolar asking for the popular opinion. Most said go with the Growatt (5 votes to 2 as of 4/29/2021). Will Prowse (solar genius) said they’re basically the same. Both batteries allow charging by utility, have solar MPPT chargers, and monitoring via serial.

I ordered the battery and knew it wouldn’t take long to arrive. The option for Growatt involved waiting 3-4 weeks for a container to arrive at the Port of Long Beach from China. The MPP option shipped from Utah (I am in Colorado – one state to the east). I picked MPP mostly based on shipping time. Also because 8S 100A BMSs are pretty common (which works well for 2.4kW because 100A * 24V = 2.4kW) which usually have a trip limit of around 110A. The next step up is usually 200A which is a correspondingly large increase in cost.

Battery Management System (BMS)

The BMS is there to protect the battery. It protects from a number of conditions – overcharge, overdischarge, overcurrent, cold temperatures, short circuit, and others. The main criteria here is 100A nominal (with overcurrent kicking in around 110A), 8S for 24V, with some sort of monitoring capability (serial, bluetooth, WiFi, etc). An active balancer would be good but that appears to be in the next higher price range. I ended up going with the JBD 8S 100A BMS for $80. One of the things that really caught my eye was this thread about monitoring – it appears these are really capable of putting out data.


With all the main materials/parts ordered, it is time to focus on how to construct the system. When it is all hooked up and ready to go, I will have a small DIY solar system with battery backup to power a few select loads in the house. The main components are:

  • 8x 260Ah prismatic LiFePO4 cells for a 24V nominal system with 6.6 kWh of storage
  • MPP LV2424 inverter for 2.4kW of 120V power with ability to charge from grid, solar, or generator as well as expand with more units in parallel
  • 2x300W solar panels to charge in case of long term outage
  • JBD 8S 100A battery management system
Blog Admin Linux

Securing this WordPress blog from evil hackers!

In my introduction post, I said I would write about topics in order of interest. Securing WordPress blogs from hackers isn’t exactly fun or interesting but it is very necessary in this day and age. Hackers are constantly probing sites on the internet for insecurities. They’re constantly trying to log into WordPress sites with easily guessed passwords (hint: don’t use ‘password’ as your password). Here are some hints on how to secure WordPress blogs from hackers.

If you prefer a video version, check out my first ever YouTube video (!) covering this same content here –

When I set this site up, the first 24 hours were pretty quiet. After that, the attacks started ramping up. I decided to take action and lock down access. There are three main things I did to secure this WordPress blog installation and VPS it is hosted on:

  1. Disable password-based SSH authentication for logins
  2. Install and enable Fail2Ban
  3. Install WordPress specific Fail2Ban filters

#1 – Disable password-based SSH authentication

Step 0 – Enable SSH Key Authentication

Before you disable password-based authentication, you need to enable SSH key based authentication. I have posted a SSH key tutorial here – SSH Key Tutorial.

Password-based SSH authentication

SSH stands for secure shell. It is how 99% of Linux/Unix servers on the public internet and private intranets are administered. There are two main methods of logging in with SSH: 1) password and 2) key. Password is pretty straight-forward and is what most people are familiar with. You have a username and password. If you enter the right password for the username, you get in. Hackers are constantly testing common usernames (root, admin, user, guest) with common passwords (password, password1, password123, test, etc.). Further – they aren’t testing just one combination of user/pass at a time, they keep trying passwords until they give up or are banned. I had my VPS for a few weeks before activating on it and here is a random sample starting a minute after midnight for about six minutes:
$sudo head -n 100 /var/log/auth.log.1

Feb 28 00:01:52 sshd[2265571]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=root
Feb 28 00:01:54 sshd[2265571]: Failed password for root from port 45182 ssh2
Feb 28 00:01:54 sshd[2265571]: Received disconnect from port 45182:11: Bye Bye [preauth]
Feb 28 00:01:54 sshd[2265571]: Disconnected from authenticating user root port 45182 [preauth]
Feb 28 00:04:59 sshd[2265587]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=root
Feb 28 00:05:02 sshd[2265587]: Failed password for root from port 53437 ssh2
Feb 28 00:05:04 sshd[2265587]: Connection closed by authenticating user root port 53437 [preauth]
Feb 28 00:06:06 sshd[2265591]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=root
Feb 28 00:06:07 sshd[2265591]: Failed password for root from port 37354 ssh2
Feb 28 00:06:08 sshd[2265591]: Received disconnect from port 37354:11: Bye Bye [preauth]
Feb 28 00:06:08 sshd[2265591]: Disconnected from authenticating user root port 37354 [preauth]
Feb 28 00:06:48 sshd[2265595]: Received disconnect from port 37056:11: [preauth]
Feb 28 00:06:48 sshd[2265595]: Disconnected from port 37056 [preauth]
Feb 28 00:06:56 sshd[2265589]: Connection reset by port 53318 [preauth]
Feb 28 00:08:00 sshd[2265597]: Received disconnect from port 61081:11: [preauth]
Feb 28 00:08:00 sshd[2265597]: Disconnected from authenticating user root port 61081 [preauth]

Each login attempt is 3-4 lines, so that’s 10 attempts in 6 minutes. Also notice the repeating IP addresses – tried 4 separate times to log in across 6 minutes!

Hackers try user/pass logins because they’re relatively easy. And they get lucky often enough it is worth it.

Key-based SSH authentication

The other method to logging in with SSH is via public/private key. How this works is you generate a public/private keypair. Then you put the contents of the public key on the server you want to log in to. When logging in, your SSH client says “hello, I am user austin and I have a key to login and here it is”! The public key that’s copied to the remote server looks like this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuzcK6yIyqJabWprjaZZI9mXpVaSewoGZROcYTf/iB6OvJklIYmM/j/YHPWq1fV30QcGPpUBwKFk8DrJNn5bIk3fow67TVC0Wr2tWy7DDweTNUpk7L01MBRhjLG2xpO9RU9F4hDyzAFI4NcrSOb6J9FL6ItrfQS/LZ7H3IrmBGIjp4OooQOhR4iw5KFEdgvNgs8rAaxSl2FziTRrxhISTzkQY0BUMBkUNjsJid4x3rTXJ9UyUDYwN2/WMfzf9aGJdRzPLIiNKsxbDeTzC3vd8TCfFOUJ+hmS8gSOY0vhLS/1wQp91jR10FF4d67z9FTwAyh+o6uKJfmvNpTXIhN [email protected]

And the private key (that should never be shared! this a throwaway key) looks like this:


As you might imagine, it’s a lot harder to guess that key than it is a password. In fact, cracking a 2048 bit key like the one above would take 300 trillion years with a quantum supercomputer (which doesn’t yet exist)! Source. The universe is 15 billion years old. That means it would require 300 trillion / 15 billion = 20,000 universe lifetimes to crack.

Before you disable password-authentication, you need to be 100% sure that key-based authentication is working or else you will lock yourself out of your server!

To disable password-based authentication, you need to edit /etc/ssh/sshd_config, find PasswordAuthentication and put no after it. If it is commented out (there is a # at the front of the line) delete the #. It will look like this when finished:

disable sshd password authentication
disable sshd password authentication

Then you need to restart the SSH daemon (service) for the change to take effect:sudo systemctl restart ssh.service. Now you password-based SSH authentication has been disabled!

My failed authentication attempts dropped dramatically after disabling password-based SSH authentication. Below is the same general timeframe from the morning of when this post was written:

Mar 13 00:00:24 sshd[108357]: Invalid user ftpuser from port 59060
Mar 13 00:00:24 sshd[108357]: Received disconnect from port 59060:11: Normal Shutdown, Thank you for playing [preauth]
Mar 13 00:00:24 sshd[108357]: Disconnected from invalid user ftpuser port 59060 [preauth]
Mar 13 00:03:09 sshd[108549]: Received disconnect from port 5402:11: disconnected by user
Mar 13 00:03:09 sshd[108549]: Disconnected from user austin port 5402
Mar 13 00:03:09 sshd[108438]: pam_unix(sshd:session): session closed for user austin
Mar 13 00:12:33 sshd[108934]: Invalid user postgres from port 46444
Mar 13 00:12:33 sshd[108934]: Received disconnect from port 46444:11: Normal Shutdown, Thank you for playing [preauth]
Mar 13 00:12:33 sshd[108934]: Disconnected from invalid user postgres port 46444 [preauth]
Mar 13 00:12:44 sshd[108941]: Received disconnect from port 11758:11: [preauth]
Mar 13 00:12:44 sshd[108941]: Disconnected from authenticating user root port 11758 [preauth]
Mar 13 00:17:40 sshd[109097]: Received disconnect from port 32827:11: [preauth]
Mar 13 00:17:40 sshd[109097]: Disconnected from authenticating user root port 32827 [preauth]
Mar 13 00:24:51 sshd[109322]: Invalid user postgres from port 33830
Mar 13 00:24:52 sshd[109322]: Received disconnect from port 33830:11: Normal Shutdown, Thank you for playing [preauth]
Mar 13 00:24:52 sshd[109322]: Disconnected from invalid user postgres port 33830 [preauth]

Most of these are just disconnects. The hackers see that my server is not accepting passwords and they just disconnect – they don’t even try to log in.

#2 – Install Fail2Ban

Fail2Ban is a helpful tool that monitors various logs and if it sees too many failed attempts, it will issue a ban on the offending IP address.

It is simple enough to install. First, update your package cache. On Ubuntu/Debian, this is done with apt:sudo apt update.
Then install fail2ban:sudo apt install -y fail2ban. This automatically enables Fail2ban so that it starts on boot. It has a bunch of out-of-the-box rules and will handle many services without any additional configuration. This is what my Fail2ban log looks like as of right now. This is all SSH bans. Notice that the duration is increasing for IP The default ban duration is 10 minutes and I have it configured to double (plus some randomness) every extra attempt.fail2ban log

#3 – Add WordPress specific Fail2ban jails and plugin

Attempts to log into WordPress look like normal web traffic in web logs. Failed logins aren’t recorded specifically. We can change that by adding a plugin to WordPress that writes to /var/log/auth.log for a number of activities. Fail2ban monitors /var/log/auth.log for failed logins so it can act appropriately. I am using WP-Fail2Ban-Redux which does exactly what it says and without any nonsense. To finish the install, I copied the files from wp-content/plugins/wp-fail2ban-redux/config/filters and /jail to my fail2ban filter.d/ and jail.d/ folders:

cp /var/www/wordpress/wp-content/plugins/wp-fail2ban-redux/config/filters/wordpress-hard.conf /etc/fail2ban/filter.d/wordpress-hard.conf
cp /var/www/wordpress/wp-content/plugins/wp-fail2ban-redux/config/filters/wordpress-soft.conf /etc/fail2ban/filter.d/wordpress-soft.conf
cp /var/www/wordpress/wp-content/plugins/wp-fail2ban-redux/config/jail/wordpress.conf /etc/fail2ban/jail.d/wordpress.conf

Restart fail2ban so the changes take effect:
sudo systemctl restart fail2ban
View all the bans in your log! Congrats, you’ve now applied some top notch security practices to your blog.

fail2ban wordpress bans
fail2ban wordpress bans


That is the entirety of #4.

#5 – To disable XMLRPC or not, that is the question

I haven’t disabled XML-RPC yet. XML-RPC is a way to programmatically interact with WordPress blogs. Hackers can use it to rapidly try user/password combinations and other things like that. Installing the WordPress specific Fail2Ban components will effectively ban offenders while still allowing access to the underlying services.

In conclusion

It isn’t too hard to make these three changes to secure your WordPress blog and doing so will increase the security drastically. If you would like assistance doing this on your site, please use the contact form to get in touch with me. Lastly, always keep your WordPress install up to date. Every so often, security researchers find holes in the base WordPress code. Automatic updates will prevent your site from being a target.