Categories
Blog Admin

My Fast WordPress Page Load Speeds – Part 1

How fast is fast enough?

People trust faster loading web sites than slower ones, all else being equal. There are many articles and studies saying so. It’s also better for search engine optimization (SEO). Personally, if a site takes more than 5 seconds to load, I wonder why and open the page source and start looking into it. I knew when I made austinsnerdythings.com that I wanted it to have a super fast WordPress page load speed.

Initial goal – less than 1 second for page load speed, and >90 for page speed tests

I decided on a initial target of loading in under a second. How did I achieve this target? It wasn’t scientific. Sites that load in 2 seconds or more are noticeable. For sites that load in under a second – it is hard to tell if the site loads in 0.6 seconds or 0.8. One second just felt like a good goal. I also wanted the 3 major page speed test sites to show a 90% or better.

Results – consistently fast WordPress page load speeds and 90+ on page speed tests

This site consistently loads in under a second. Here is a screenshot from the evening of writing this post showing a load time of 0.619 seconds. The server is located in New York City and I’m loading the page in Denver. That’s 50 milliseconds of ping by itself.

619 millisecond load time on my main PC

I also score 90+ on each of the big 3 page speed test sites – Google Page Speed Insights, GTMatrix, and Pingdom.

Here is the GTMatrix page speed result showing a strong 99% for performance. Note that this test was conducted from Vancouver, which is nearly 3000 miles away. That’s a lot of distance for the packets to travel.

99% for performance. Can’t argue with that

For Google Page Speed Insight, I score a perfect 100 for desktop load speed. Mobile is 90, not sure why they’re so different. I don’t think I need to worry about this anytime soon.

The elusive 100 for Google PageSpeed Insights
PageSpeed reporting 90 for mobile version? Must be some sort of artificial limit on performance or something.

For Pingdom, it is reporting I need to make less HTTP calls. I experimented with a couple different plugins and I think I need to go back to a different one for merging the site’s assets to reduce calls.

I’ve never seen E for a grade. Guessing it basically means F. I should improve on this.

It wasn’t particularly difficult to achieve the 90+ page speed scores and fast WordPress page load speeds. Read on to find out how I did this.

Austinsnerdythings.com stack

The stack driving this site is pretty standard. I use the following (ascending layer order)

  • Ramnode premium VPS ($12/month). I had a standard VPS but realized a faster CPU would provide better time to first byte and be just faster in general. It also has NVMe drives instead of SATA SSD, which further reduces latency.
  • Dual stack networking – IPv6 enabled as well as IPv4
  • MariaDB (MySQL drop in replacement) – zero tuning
  • PHP-FPM – interprets all dynamic requests, which is basically all the requests. zero tuning
  • NGINX webserver – this is the new hotness for webservers. It is event driven and runs fast. zero tuning
  • GZIP compression – much of the content can be compressed while being transferred which means faster load times
  • HTTP/2 – the newest generation of HTTP transfer protocol. not sure what’s faster about it than HTTP/1.1 but it is
  • Fast velocity minify plugin – combines javascript and CSS files into fewer entities which means faster load time
  • No extra plugins – don’t load up on plugins. some are really bad and will drastically increase page load times.
  • Lazy load pictures – no need to have pictures load until users get to them
  • Don’t embed videos – they just take forever to load up, which is a problem for page load speeds

Overall, this is a pretty standard stack. I haven’t done any manual tuning to any of the services/processes serving up my site. Starting with a fast VPS definitely helps get a fast WordPress page load speed. The rest just seemed logical to me. That doesn’t mean there isn’t room for improvement.

What’s next for an even faster WordPress page load speed

I have a development environment mocked up on my Proxmox virtual machine host that mimics almost exactly the production site (i.e. what served you this page). The page load speeds were roughly the same locally vs from NYC so it is a good comparison. I installed Varnish for caching and HAProxy for SSL termination (Varnish doesn’t do SSL/https). My homepage loaded consistently in a quarter of a second. The fastest I saw was 0.219 seconds. Sure Varnish and HAProxy are two more full-blown services to install and manage but is it worth it? I think it is. I’ll be migrating my production stack over to this 3 tier stack soon.

Further, I can add my content to a content delivery network (CDN), like Cloudflare, and have it cached there too for fast access anywhere they have a datacenter (they have like 200 datacenters across the planet). I did use Cloudflare CDN for a bit and didn’t like not seeing my statistics update in real-time so I backed off.

I am realizing this may be a good business opportunity – consistently super fast WordPress sites. Let me know if you’d be interested.

Check back for part 2 where I document the journey to the 3 tier stack.

Categories
DIY Offgrid Solar

DIY Solar System with battery backup update

Update

As of 5/1/2021, all of the main components of my DIY solar system with battery backup have arrived. I posted about the requirements, component select, and some fun with shipping from China in my initial solar post – Planning my 600W DIY solar system with 6 kWh battery backup. If you want some background of how we got here, head to that link then come on back.

Background

To recap, my DIY solar system with battery backup consists of a few main components:

  • 8x 260 amp hour prismatic LiFePO4 battery cells. They will be placed in series for a 24V nominal system with around 6.0 kilowatt hours of usable storage.
  • MPP LV2424 hybrid all-in-one inverter. This device handles converting direct current (DC, like a car battery) to alternating current (AC, like household outlets) and charging the batteries
  • 2x 310W Canadian solar panels. These will be wired in series for 72V maximum power point voltage.
  • 8S JBD 100A battery management system – to protect the batteries from a number of undesirable conditions

The other miscellaneous things that I need are: battery bus bars, wiring, ring terminals, and general connection things.

Materials arriving and resting battery voltages

We were on vacation when the batteries (and inverter) arrived so they sat for a few days before I got a chance to unbox them. The batteries were very well packaged and I can’t thank Battery Hookup enough for how fast they shipped after what I’ve been dealing with from China.

battery cells unpacked
battery cells unpacked

Upon unboxing, I made sure to record the resting voltage of each cell. Below were the resting voltages:

cellvoltage
13.341
23.345
33.435
43.339
53.351
63.376
73.343
83.363
min3.339
max3.435
avg3.362
delta0.096
Resting voltages of the 260 amp hour BYD LiFePO4 cells

Cell #2 had the lowest voltage and cell #3 the highest. This presented an easy chance to test out my bus bars for voltage equalization. I did not measure the cell internal resistance so I wasn’t really sure how much current would flow from cell 3 to 2 when shorted together so I did a quick estimate based on Ohm’s law (V=IR -> I=V/R). With an estimated internal resistance (IR) of 20 milliohms (I’ve had LiPo in this range after some degradation), and a voltage difference of 0.096V, that would mean a current of 0.096/0.020=4.8A. That wasn’t a huge number so I was comforable just connecting the cells with the bus bars. But first I wanted to actually measure with a multimeter.

I was expecting about 5A based on the calculation above. I’m not sure how I was 10x off on the estimate but after hooking up my multimeter, the equalization current was 0.6A. That was plenty low so I set my mind to balancing. But before that, I needed bus bars to connect all the cells in parallel.

Constructing copper bus bars

Bus bars are used to conduct high amounts of current in electrical applications. In essence, they are oversized, flat wires. I ordered 2x copper bars from McMaster Carr that are 1″x1/8″x3′. They arrived in two days with free shipping… Amazon is gaining competition. I got to work drilling holes.

bus bars ready for drilling
copper bus bars ready for drilling with marks spaced 2.5″ apart
drilling 2mm pilot hole
drilling 2mm pilot hole
drilling 6mm hole
drilling 6mm final hole
copper shards after drilling
copper shards after drilling
filing down rough edges
filing down rough edges

Heat shrinking the cells

With the bus bars ready, it was time to heat shrink the battery cells. Battery Hookup said they were uncovered and needed to be heat shrunk for every cell. Turns out 7 of 8 had a covering on them already. I heat shrunk them anyways for two reasons: 1) to further protect the cells from damage and short circuits and 2) so they look better. The cell on the left will be re-done with more heat shrink. I guestimated how much I needed and was short a few inches.

cells ready for heat shrink
cells ready for heat shrink
cells heat shrunk with BMS
cells heat shrunk with BMS on top

Balancing in 2 sets of 4 cells each

When making the bus bars, I put together a mental picture of what I needed to make and how many I needed. This worked fine for the final battery but I would need double for balancing. I had the full extra 3′ copper bar but it took forever to cut so I just decided to balance the cells in 2 groups of 4. Balancing is charging all cells in parallel as one low voltage, giant capacity battery to their rated voltage (3.65V for LiFePO4). Doing 4 cells at a time meant it was a 3.2V battery with a capacity of 1040 amp hours. The resting voltages were in the upper end of the voltage curve chart so it wouldn’t take super long. If the battery was fully depleted, it would take 104 hours at the 10 amps my bench power supply puts out.

So I got the bus bars hooked up and started balancing by setting my power supply to 3.65V and current to max.

battery cells hooked up in parallel
battery cells hooked up in parallel (4x cells in parallel means a single 3.2V 1000Ah “cell”)
Measurement discrepancies and voltage drop

The first thing I did when I started charging was take voltage measurements to make sure things were right. I noticed some irregularities.

The first irregularity was the fact that the bench power supply was over-reporting the voltage by 0.034V or so when comparing the display to the terminals. This isn’t a huge deal and is actually a pretty decent level of accuracy for a $40 bench power supply from Amazon.

verifying DC power supply output voltage
verifying DC power supply output voltage (set to 3.650V, actual output is 3.614V)

The next thing I noticed is that for a 10A power supply, it is putting out a lot less than 10A. So I measured the voltage at the bus bars.

measuring voltage at bus bars
measuring voltage at bus bars. this is a pretty big voltage drop through the charger wires. 3.616V at the terminals – 3.363V at the bus bars is 0.253V drop. At 3.962A, that is a resistance of (V=IR -> V/I=R) 0.063 ohms. the wires were warm.

By leaving the voltage set to 3.65V at the DC power supply, the voltage drop means I wouldn’t get anywhere the rated 10A. The actual drop would decrease proportionally as the battery voltage neared the terminal voltage.

Regardless, it only took a couple hours until the cells were topped off for each set of 4. They are currently resting. I will check their voltages again tomorrow morning. The amount the cells drop in voltage from where they left off indicates the strength of the cell, with larger drops meaning weaker cells.

balancing in progress
the very start of balancing the 2nd set of 4 cells. I do intend to rewrap cell 3 and wrap cell 1 when my next batch of shrinkwrap arrives.

Conclusion

This is where we will leave off for the day. The 8 cells have been balanced in 2 groups of 4 and are currently resting. After 24 hours I will recheck the voltages to see which cells are strong and which are weak. I need to buy more electrical tape to cover up the bus bar ends (I did start assembling the full battery but stopped because the potential for short-circuit was higher than I was comfortable with).

As of 5/17/2021, things are up and running! Check it out at DIY solar with battery backup – up and running!

Categories
Linux Raspberry Pi

Microsecond accurate NTP with a Raspberry Pi and PPS GPS

Introduction

Lots of acronyms in that title. If I expand them out, it says – “microsecond accurate network time protocol with a Raspberry Pi and pulse per second global positioning system [receiver]”. What it means is you can get super accurate timekeeping (1 microsecond = 0.000001 seconds) with a Raspberry Pi and a GPS receiver (the GT-U7 is less than $12) that spits out pulses every second. By following this guide, you will your very own Stratum 1 NTP server at home!

Why would you need time this accurate at home?

You don’t. There aren’t many applications for this level of timekeeping in general, and even fewer at home. But this blog is called Austin’s Nerdy Things so here we are. Using standard, default internet NTP these days will get your computers to within a 10-20 milliseconds of actual time (1 millisecond = 0.001 seconds). By default, Windows computers get time from time.windows.com. MacOS computers get time from time.apple.com. Linux devices get time from [entity].pool.ntp.org, like debian.pool.ntp.org. PPS gets you to the next SI prefix in terms of accuracy (milli -> micro), which means 1000x more accurate timekeeping.

YouTube video link

If you prefer a video version – https://www.youtube.com/watch?v=YfgX7qPeiqQ

Materials needed

Steps

0 – Update your Pi and install packages

This NTP guide assumes you have a Raspberry Pi ready to go. If you don’t, I have a 16 minute video on YouTube that goes through flashing the SD card and initial setup – https://youtu.be/u5dHvEYwr9M.

You should update your Pi to latest before basically any project. We will install some other packages as well. pps-tools help us check that the Pi is receiving PPS signals from the GPS module. We also need GPSd for the GPS decoding of both time and position. I use chrony instead of NTPd because it seems to sync faster than NTPd in most instances and also handles PPS without compiling from source (the default Raspbian NTP doesn’t do PPS) Installing chrony will remove ntpd.

sudo apt update
sudo apt upgrade
sudo rpi-update
sudo apt install pps-tools gpsd gpsd-clients python-gps chrony

1 – add GPIO and module info where needed

In /boot/config.txt, add ‘dtoverlay=pps-gpio,gpiopin=18’ to a new line. This is necessary for PPS. If you want to get the NMEA data from the serial line, you must also enable UART and set the initial baud rate.

sudo bash -c "echo '# the next 3 lines are for GPS PPS signals' >> /boot/config.txt"
sudo bash -c "echo 'dtoverlay=pps-gpio,gpiopin=18' >> /boot/config.txt"
sudo bash -c "echo 'enable_uart=1' >> /boot/config.txt"
sudo bash -c "echo 'init_uart_baud=9600' >> /boot/config.txt"

In /etc/modules, add ‘pps-gpio’ to a new line.

sudo bash -c "echo 'pps-gpio' >> /etc/modules"

Reboot

sudo reboot

2 – wire up the GPS module to the Pi

I used the Adafruit Ultimate GPS breakout. It has 9 pins but we are only interested in 5. There is also the Adafruit GPS hat which fits right on the Pi but that seems expensive for what it does (but it is significantly neater in terms of wiring).

Pin connections:

  1. GPS PPS to RPi pin 12 (GPIO 18)
  2. GPS VIN to RPi pin 2 or 4
  3. GPS GND to RPi pin 6
  4. GPS RX to RPi pin 8
  5. GPS TX to RPi pin 10
  6. see 2nd picture for a visual
Adafruit Ultimate GPS Breakout V3
We use 5 wires total. GPS PPS to pin 12 (GPIO 18), GPS VIN to pin 2 or 4, GPS GND to pin 6, GPS RX to pin 8, GPS TX to pin 10.
GPS with wires attached to the board (but not to the Pi) and the antenna. The antenna has a SMA connector, and there is another adapter that is SMA to u.fl to plug into the GPS board.

Now place your GPS antenna (if you have one) in a spot with a good view of the sky. If you don’t have an antenna, you might have to get a bit creative with how you locate your Raspberry Pi with attached GPS.

I honestly have my antenna in the basement (with only the kitchen and attic above) and I generally have 8-10 satellites locked all the time (11 as of writing). Guess that means the antenna works better than expected! Either way, better exposure to the sky will in theory work better. Pic:

My super awesome placement of the GPS antenna on top of the wood “cage” (?) I built to hold my 3d printer on top of my server rack. I guess this is a testimony for how well the GPS antenna works? It has 11 satellites locked on as of writing, with a HDOP of 0.88. The components in this rack (Brocade ICX6450-48P, 1U white box with Xeon 2678v3/96GB memory/2x480GB enterprise SSDs/4x4TB HDDs, Dell R710 with 4x4TB and other stuff) will be detailed in an upcoming post.

3 – check that PPS is working

First, check that the PPS module is loaded:

lsmod | grep pps

The output should be like:

[email protected]:~ $ lsmod | grep pps
pps_gpio               16384  0
pps_core               16384  1 pps_gpio

Second, check for the PPS pulses:

sudo ppstest /dev/pps0

The output should spit out a new line every second that looks something like this (your output will be a bit farther from x.000000 since it isn’t yet using the GPS PPS):

[email protected]:~ $ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1618799061.999999504, sequence: 882184 - clear  0.000000000, sequence: 0
source 0 - assert 1618799062.999999305, sequence: 882185 - clear  0.000000000, sequence: 0
source 0 - assert 1618799063.999997231, sequence: 882186 - clear  0.000000000, sequence: 0
source 0 - assert 1618799064.999996827, sequence: 882187 - clear  0.000000000, sequence: 0
source 0 - assert 1618799065.999995852, sequence: 882188 - clear  0.000000000, sequence: 0
^C

4 – change GPSd to start immediately upon boot

Edit /etc/default/gpsd and change GPSD_OPTIONS=”” to GPSD_OPTIONS=”-n” and change DEVICES=”” to DEVICES=”/dev/ttyS0 /dev/pps0″, then reboot. My full /etc/default/gpsd is below:

[email protected]:~ $ sudo cat /etc/default/gpsd
# Default settings for the gpsd init script and the hotplug wrapper.

# Start the gpsd daemon automatically at boot time
START_DAEMON="true"

# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"

# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group dialout.
DEVICES="/dev/ttyS0 /dev/pps0"

# Other options you want to pass to gpsd
GPSD_OPTIONS="-n"
sudo reboot

4.5 – check GPS for good measure

To ensure your GPS has a valid position, you can run gpsmon or cgps to check satellites and such. This check also ensures GPSd is functioning as expected. If your GPS doesn’t have a position solution, you won’t get a good time signal. If GPSd isn’t working, you won’t get any updates on the screen. The top portion will show the analyzed GPS data and the bottom portion will scroll by with the raw GPS sentences from the GPS module.

gpsmon showing 10 satellites used for the position with HDOP of 0.88. This indicates a good position solution which means the time signals should be good as well. The PPS of 0.000000684 indicates the Raspberry Pi is only 684 nanoseconds off of GPS satellite time.

5 – edit config files

For chrony, add these two lines to the /etc/chrony/chrony.conf file somewhere near the rest of the server lines:

refclock SHM 0 refid NMEA offset 0.200
refclock PPS /dev/pps0 refid PPS lock NMEA

My entire /etc/chrony/chrony.conf file looks like this:

###### below this line are custom config changes #######
server 10.98.1.1 iburst minpoll 3 maxpoll 5
server time-a-b.nist.gov iburst
server time-d-b.nist.gov
server utcnist.colorado.edu
server time.windows.com
server time.apple.com

# delay determined experimentally by setting noselect then monitoring for a few hours
# 0.325 means the NMEA time sentence arrives 325 milliseconds after the PPS pulse
# the delay adjusts it forward
refclock SHM 0 delay 0.325 refid NMEA
refclock PPS /dev/pps0 refid PPS

allow 10.98.1.0/24 # my home network
###### above this line are custom config changes #######

###### below this line is standard chrony stuff #######
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
#log tracking measurements statistics
logdir /var/log/chrony
maxupdateskew 100.0
hwclockfile /etc/adjtime
rtcsync
makestep 1 3

Restart chrony, wait a few minutes, and verify.

sudo systemctl restart chrony

5 – verify the NTP server is using the GPS PPS

Right after a chrony restart, the sources will look like this (shown by running ‘chronyc sources’)

[email protected]:~ $ chronyc sources
210 Number of sources = 9
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#? NMEA                          0   4     0     -     +0ns[   +0ns] +/-    0ns
#? PPS                           0   4     0     -     +0ns[   +0ns] +/-    0ns
^? pfsense.home.fluffnet.net     0   3     3     -     +0ns[   +0ns] +/-    0ns
^? time-a-b.nist.gov             1   6     3     1  -2615us[-2615us] +/- 8218us
^? time-d-b.nist.gov             1   6     1     3  -2495us[-2495us] +/- 7943us
^? india.colorado.edu            0   6     0     -     +0ns[   +0ns] +/-    0ns
^? 13.86.101.172                 3   6     1     4  -4866us[-4866us] +/-   43ms
^? usdal4-ntp-002.aaplimg.c>     1   6     1     4  -2143us[-2143us] +/-   13ms
^? time.cloudflare.com           3   6     1     3  -3747us[-3747us] +/- 9088us

The # means locally connected source of time. The question marks mean it is still determine the status of each source.

After a couple minutes, you can check again:

[email protected]:~ $ chronyc -n sources
210 Number of sources = 9
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#x NMEA                          0   4   377    23    -37ms[  -37ms] +/- 1638us
#* PPS                           0   4   377    25   -175ns[ -289ns] +/-  126ns
^? 10.98.1.1                     0   5   377     -     +0ns[   +0ns] +/-    0ns
^- 132.163.96.1                  1   6   177    22  -3046us[-3046us] +/- 8233us
^- 2610:20:6f96:96::4            1   6    17    28  -2524us[-2524us] +/- 7677us
^? 128.138.140.44                1   6     3    30  -3107us[-3107us] +/- 8460us
^- 13.86.101.172                 3   6    17    28  -8233us[-8233us] +/-   47ms
^- 17.253.2.253                  1   6    17    29  -3048us[-3048us] +/-   14ms
^- 2606:4700:f1::123             3   6    17    29  -3325us[-3325us] +/- 8488us

For the S column, * means the source that is active. + means it is considered a good source and would be used if the current one is determined to be bad or is unavailable. The x shown for the NMEA source means it is a “false ticker”, which means it isn’t being used. In our case that is fine because the PPS source is active and valid. Anything else generally means it won’t be used.

In this case, chrony is using the PPS signal. The value inside the brackets is how far off chrony is from the source. It is showing that we are 289 nanoseconds off of GPS PPS time. This is very, very close to atomic clock level accuracy. The last column (after the +/-) includes latency to the NTP source as well as how far out of sync chrony thinks it is (for example, the 17.253.2.253 server is 12.5 milliseconds away one-way via ping):

[email protected]:~ $ ping -c 5 17.253.2.253
PING 17.253.2.253 (17.253.2.253) 56(84) bytes of data.
64 bytes from 17.253.2.253: icmp_seq=1 ttl=54 time=25.2 ms
64 bytes from 17.253.2.253: icmp_seq=2 ttl=54 time=27.7 ms
64 bytes from 17.253.2.253: icmp_seq=3 ttl=54 time=23.8 ms
64 bytes from 17.253.2.253: icmp_seq=4 ttl=54 time=24.4 ms
64 bytes from 17.253.2.253: icmp_seq=5 ttl=54 time=23.4 ms

--- 17.253.2.253 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 23.403/24.954/27.780/1.547 ms

For a full list of information about how to interpret these results, check here – https://docs.fedoraproject.org/en-US/Fedora/18/html/System_Administrators_Guide/sect-Checking_if_chrony_is_synchronized.html

6 – results

A day after the bulk of writing this post, I turned on logging via the .conf file and restarted chrony. It settled to microsecond level accuracy in 57 seconds. For the offset column, the scientific notation e-7 means the results are in nanoseconds. This means that for the 19:54:35 line, the clock is -450.9 nanoseconds off of the PPS signal. There is that e-10 line in there which says that it is 831 picoseconds off PPS (I had to look up SI prefixes to make sure pico came after nano! also I doubt the Pi can actually keep track of time that closely.). After the initial sync, there is only 1 line in the below log that is actually at the microsecond level (the others are all better than microsecond) – the 20:00:59 entry, which shows the clock is -1.183 microseconds off.

Things that affect timekeeping on the Pi

Thought I’d toss in this section for completeness (i.e. thanks for all the good info Austin but how can I make this even better?). There are a few things that affect how well the time is kept on the Pi:

  • Ambient temperature around the Pi – if you plot the freq PPM against ambient temperature, there will be a clear trend. The more stable the ambient temp, the less variation in timekeeping.
  • Load on the Pi – similar to above. Highly variable loads will make the processor work harder and easier. Harder working processor means more heat. More heat means more variability. These crystals are physical devices after all.
  • GPS reception – they actually making timing GPS chips that prefer satellites directly overhead. They have better ability to filter out multipathing and such. In general, the better the GPS reception, the better the PPS signal.

Conclusion

After running through the steps in this guide, you should now have a functional Stratum 1 NTP server running on your Raspberry Pi with microsecond level accuracy provided by a PPS GPS. This system can obtain time in the absence of any external sources other than the GPS (for example, internet time servers), and then sync up with the extremely precise GPS PPS signal. Our NTP GPS PPS clock is now within a few microseconds of actual GPS time.

References

I read a ton on https://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html and other pages on that domain over the years which has been extremely helpful in getting GPS PPS NTP going for my setup. There is a lot of background info for how/why this stuff works and many useful graphics. Another source is https://wellnowweknow.com/index.php/2019/12/27/how-to-ntp-a-raspberry-pi-4-via-gps-and-pps/ for a short and sweet (maybe too short) set of commands.

Categories
ADS-B SDR

Receiving aircraft ADS-B (position) signals – part 4 (antenna up on roof)

Antenna up on roof

Coming from part 3, where I wanted to move the antenna, I finally got the antenna up on the roof. Our chimney was decommissioned by the previous owners and as far as I can tell, there isn’t brick under the siding (also why does our chimney have siding on it). So it is sitting a little lower than it should be but it is basically at the highest position of the roof. This has dramatically increased the ability to receive aircraft ADS-B signals.

Results

The results are pretty amazing. We’ve had bad weather for a week now but it’s going to be a clear day today. As of 9:12AM, my Raspberry Pi PiAware ADS-B signal receiver sees 116 aircraft, of which 103 are reporting positions. It is receiving 607 messages per second. The map looks like this:

116 aircraft signals received, 103 with position. farthest out is 190 NM.

You can see aircraft lining up to arrive into KDEN spaced out at regular intervals. It’s also picking up 3 planes on the ground at KBJC which is the closest airport to the antenna.

FlightAware has a cool radar type map that shows positions by compass direction and distance. The numbers speak for themselves.

Before

Max distance reported is in the 100-150 nm bucket (327 total reports)

After

That same 100-150 nm bucket now has 24k reports

Interesting features

While typing this post, the position count increased to 118. There are some interesting features I’m seeing – a survey plane over the Breckenridge area, a lot of planes on the ground at KBJC (not line of sight to my antenna), and even plane on the ground at KDEN (KDEN is definitely not line of sight to my antenna).

Survey grid being flown by N94S

Planes on the ground at KBJC

I see 4 Cessna/trainer type planes waiting for takeoff for 30R at KBJC. I’m even picking up a corporte jet type aircraft on the ground by the hangars (N4840W). None of this is line of sight to my antenna. There’s a chance the ADS-B signals are bouncing off buildings or something. I shouldn’t be seeing these.

The elevation profile to 30R at KBJC. Antenna on left, run up area on the right. Barely not LOS (line of sight).

Plane on the ground at KDEN

United UAL364 / N802UA (an Airbus A319) on the ground on runway 16L/34R heading south
Elevation profile to south end of 16L/34R at KDEN. Antenna on left, 16L/34R on right. Definitely not LOS. No idea how I’m picking up these signals. I see a plane on the ground at KDEN multiple times a day since moving the antenna.

151 planes!

I started this post around 9am on 4/18. Just before noon, there were 151 planes being tracked by my PiAware station! 773 messages per second. Notice that plane way out there over west central Nebraska – that’s probably 210 NM out!

Conclusion

Moving my FlightAware ADS-B antenna to the roof drastically increased the range and messages received. As a reminder from when I detailed the equipment in Post 2 – the antenna feeds a 1090MHz ADS-B filter, which in turn feeds the FlightAware Pro Stick. I don’t think I’ll make any other changes to the system other than put it on a battery with solar charger.

Categories
homelab Linux proxmox

Proxmox Cluster manual update

Recently ran into an issue where I added a node to my Proxmox cluster while a node was disconnected & off. That node (prox) caused the others to become unresponsive for a number of Proxmox things (it was missing from the cluster) upon boot.

Set node to local mode

The solution was to put the node that had been offline (called prox) into “local” mode. Thanks to Nicholas of Technicus / techblog.jeppson.org for the commands to do so:

sudo systemctl stop pve-cluster
sudo /usr/bin/pmxcfs -l

This allows editing of the all-important /etc/pve/corosync.conf file.

Manually update corosync.conf

I basically just had to copy over the config present on the two synchronized nodes to node prox, then reboot. This allowed node prox to join the cluster again and things started working fine.

Problem corosync.conf on node prox:

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: prox
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.98.1.14
  }
  node {
    name: prox-1u
    nodeid: 2
    quorum_votes: 3
    ring0_addr: 10.98.1.15
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: prox-cluster
  config_version: 4
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  secauth: on
  version: 2
}

Fancy new corosync.conf on nodes prox-1u and prox-m92p:

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: prox
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.98.1.14
  }
  node {
    name: prox-1u
    nodeid: 2
    quorum_votes: 3
    ring0_addr: 10.98.1.15
  }
  node {
    name: prox-m92p
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.98.1.92
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: prox-cluster
  config_version: 5
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  secauth: on
  version: 2
}

The difference is that third node item as well as incrementing the config_version from 4 to 5. After I made those changes on node prox and rebooted, things worked fine.

 

 

 

Categories
Linux

SSH Key Tutorial

SSH Key Tutorial

This post will be a basic SSH key tutorial. In my Securing this WordPress blog from evil hackers! post, I recommended turning off password based SSH authentication and moving exclusively to SSH key based authentication. This post will go over the steps to set up SSH key authentication. The outline is:

  1. Generate SSH public/private key pair
  2. Transfer the public key to the host in question
    1. manual method (Windows)
    2. automatic method (Linux/Mac)
  3. Verify the SSH key authentication is functional.

At the end of this post is my user creation script which does all of this automatically.

I do go over this in a good amount of detail in my first ever YouTube video which you can view here if you’d prefer a video instead of text.

I will be generating keys on my new laptop and transferring them to the Raspberry Pi I set up in my second YouTube video (Austin’s Nerdy Things Ep 2 – Setting up a Raspberry Pi) – https://youtu.be/u5dHvEYwr9M.

Step 1 – Generate the SSH key pair

The first thing we need to do is generate the SSH key pair. On all major OSes these days, this command is included (Windows finally joined the other “real” OSes in 2018 with full SSH support out of the box). We will be using ED25519, which is advised for basically anything build/updated since 2014.

ssh-keygen -t ed25519

This will start the generation process. I hit enter when it prompts for the location (it defaults to C:/Users/<user>/.ssh/id_<type> which is fine). I also hit enter again at the passphrase prompt twice because I don’t want to use a passphrase. Using a passphrase increase security but it can’t be used for any automated processes because it’ll prompt every time for the passphrase. This is the full output from the process:

C:\Users\Austin>ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\Austin/.ssh/id_ed25519): [press enter here]
Enter passphrase (empty for no passphrase): [press enter here]
Enter same passphrase again: [press enter here]
Your identification has been saved in C:\Users\Austin/.ssh/id_ed25519.
Your public key has been saved in C:\Users\Austin/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:t0FIIk<snip.......................snip>6Rx4 [email protected]
The key's randomart image is:
+--[ED25519 256]--+
|o++++++ .        |
|*  + * o .       |
|o.o o   . .      |
| . .     .       |
|..... o S o      |
| . + .  .o.o..   |
|  = . o.+.+=o..  |
|   +.o.  +a..o   |
|  ..ooo o+...    |
+----[SHA256]-----+

Step 2 – Transfer the SSH public key

For Windows, they included all the useful SSH utilities except one: ssh-copy-id. This is the easy way to transfer SSH keys. It is possible to transfer the key without using this utility. For this SSH key tutorial, I will show both.

Step 2a – Transfer SSH key (Windows)

First we need to get the contents of the SSH public key. This will be done with the little-known type command of Windows. In Step 1, the file was placed at C:\Users\Austin\.ssh\id_ed25519 so that’s where we will read:

C:\Users\Austin>type C:\Users\Austin\.ssh\id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZD2PT/VIK3KyYRliA0jgqmn7yzkVjuDiFK67Cio [email protected]

The contents of the file are the single line of the SSH public key (starting with the type of key “ssh-ed25519” and then continuing with the key contents “AAAAC3” and ending with the username and host the key was generated on “[email protected]”). This is the data we need to transfer to the new host.

I will first log onto the Raspberry Pi with the username/password combo from the default installation:

ssh [email protected]

I am now logged into the Pi:

Raspberry Pi Log In Prompt
Logged into the Raspberry Pi

Now on the target machine we need to create the user account and set up some initial things. We use sudo on all of these since root level permissions are required.

# create user austin with home directory (-m means create the home directory)
sudo useradd -m austin

# set the password for user austin
sudo passwd austin

# add user austin to sudo group
sudo usermod -aG sudo austin

# make the .ssh directory for user austin
sudo mkdir /home/austin/.ssh

sudo useradd
User created, password set, sudo added, and directory created

With those four commands run, the user is created, has a password, can run sudo, and has an empty .ssh directory ready for the keys.

Now we need to create the authorized_keys file:

sudo nano /home/austin/.ssh/authorized_keys

In this file, paste (by right-clicking) the single line from the id_ed25519 file:

nano authorized_keys
line pasted and ready to save

To save the file, press CTRL+O and a prompt will appear. Press enter again to save. Now to exit, press CTRL+X. There should be no “are you sure” prompt because the contents are saved.

Lastly, we need to change the owner of the .ssh directory and all of it’s contents to the new user (sudo means they were created as root):

sudo chown -R austin:austin /home/austin/.ssh

Now we can test. From the machine that generated the SSH key, SSH to the target with the new user specified:

ssh [email protected]

You should get in without having to type a password!

logged in with SSH key
Successfully logged in without typing a password (the SSH key worked)

Step 2b – Transfer SSH key (Linux)

This is much easier. The ssh-copy-id utility does what we just did in a single command.

# make sure you've already generated keys on your Linux/Mac machine
ssh-copy-id [email protected]

linux ssh-copy-id command
ssh-copy-id copied the keys and did everything for us in a single command

Microsoft – will you please add ssh-copy-id to default Windows installations?

Step 3 – Verify

We did this at the end of each of the other methods.

User creation script

This is the script I use on any new Linux virtual machine or container I spin up. It updates packages, installs a few things, sets the timezone, does all the user creation stuff and sets the SSH key. Lastly, it changes the shell to bash. Feel free to modify this for your own use. It is possible to copy the data from /etc/shadow so you don’t have to type in the password. I haven’t got that far yet.

apt update
apt upgrade -y 
apt install -y net-tools sudo htop sysstat git curl wget zsh
timedatectl set-timezone America/Denver
useradd -m austin
passwd austin
adduser austin sudo
usermod -aG sudo austin
usermod -aG adm austin
groups austin
mkdir /home/austin/.ssh
touch /home/austin/.ssh/authorized_keys
echo 'ssh-rsa AAAAB  <snip>  bhgqXzD [email protected]' >> /home/austin/.ssh/authorized_keys
chown -R austin:austin /home/austin/.ssh
usermod -s /bin/bash austin
Categories
Linux Python Weather

Python service to consume Ambient Weather API data

Python service to consume Ambient Weather API data

Continuing from the last post (Handling data from Ambient Weather WS-2902C to MQTT), we have a working script that reads the data coming from the Ambient Weather WS-2902 base station (Ambient Weather API) and sends it to a MQTT broker. In this post, we will turn that script into a Linux service that starts at boot and runs forever. This type of thing is perfect for Linux. Non-server Windows versions aren’t really appropriate for this since they reboot often with updates and such. If you want to run Windows Server, more power to you, you probably don’t need this guide. We will thus be focusing on Linux for consuming the Ambient Weather API. A Raspberry Pi is a perfect device for this (plenty of power, as big as a credit card, and less than $100).

Linux Services

Linux is made up of many individual components. Each component is designed to handle a single task (more or less). This differs from Windows where there are large executables/processes that handle many tasks. We will be taking advantage of Linux’s single task theory to add a new service that will run the Python script for ingesting and consuming the Ambient Weather API. In this instance for Austin’s Nerdy Things, the weather data is being provided by the Ambient Weather WS-2902C weather station.

Creating the service

For Ubuntu/Debian based distributions, service files live under /etc/systemd/system. Here is a list of services on the container I’m utilizing.

List command:

ls -1 /etc/systemd/system
dbus-org.freedesktop.resolve1.service
dbus-org.freedesktop.timesync1.service
default.target.wants
getty.target.wants
graphical.target.wants
multi-user.target.wants
socket.target.wants
sockets.target.wants
sshd.service
sys-kernel-debug.mount
sysinit.target.wants
syslog.service
timers.target.wants

Since this is a LXC container, there aren’t many services. On a standard Raspbian or Ubuntu full install, there will be 100+.

We will be creating a new service file using the nano text editor:

nano /etc/systemd/system/ambient-weather-api.service

In this file we need to define our service. The After lines mean don’t start this up until the listed services are running. The rest is pretty straight forward. I’m not 100% sure what the WantedBy line is for but it’s present in most of my service files. The contents of ambient-weather-api.service are as follows:

[Unit]
Description=Python script to ingest Ambient Weather API data
After=syslog.target
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/bin/python3 /srv/ambient-weather-api/main.py
ExecStartPre=/bin/sleep 5
Restart=always
RestartSec=5s
# Give the script some time to startup
TimeoutSec=10

[Install]
WantedBy=multi-user.target

Save the file. The service definition is looking for Python at /usr/bin/python3 and the python script at /srv/ambient-weather-api/main.py. You will probably be fine with the Python executable, but be sure to mv or cp the main.py file to /srv/ambient-weather-api/main.py.

We will need to reload the service definitions:

systemctl daemon-reload

Now we can start the service:

systemctl start ambient-weather-api.service

And verify it is running:

systemctl status ambient-weather-api.service

Ambient Weather API service status
Ambient Weather API service status

The above screenshot shows it is indeed running and active. It is still showing the print messages in the log as well, which we should disable by commenting out the lines by adding a # in front of the print line. In this case, it is coming from line 56 (the status check in the publish function).

#print(f"Sent {msg} to topic {topic}")

 

 

 

 

Categories
Home Assistant Weather

Handling data from Ambient Weather WS-2902C API to MQTT

Handling data from Ambient Weather WS-2902C API to MQTT

As I mentioned in my initial Ambient Weather WS-2902C post, there is a new feature that allows sending data to a custom server. I coded up a python script to take the sent data, and publish it to MQTT. This allows for super easy data ingestion with Home Assistant and other similar solutions. I should probably publish on GitHub but I’ll post here first.

Installation

This script is reliant on Paho-MQTT. EDIT: during the creation of the service to run this at boot, I discovered version 1.5.1 will throw errors. Use version 1.5.0. Install it with pip:

sudo pip install paho-mqtt==1.5.0

Create a python file and name it main.py. Paste in the following:

# Python script to decode Ambient Weather data (from WS-2902C and similar)
# and publish to MQTT.
# original author: Austin of austinsnerdythings.com
# publish date: 2021-03-20

# some resources I used include
#https://askubuntu.com/questions/29152/how-do-i-use-python-with-apache2
#https://www.toptal.com/python/pythons-wsgi-server-application-interface
#https://www.emqx.io/blog/how-to-use-mqtt-in-python

from urllib.parse import urlparse, parse_qs
import paho.mqtt.client as mqtt
import time, os

# set MQTT vars
MQTT_BROKER_HOST  = os.getenv('MQTT_BROKER_HOST',"mqtt")
MQTT_BROKER_PORT  = int(os.getenv('MQTT_BROKER_PORT',1883))
MQTT_CLIENT_ID    = os.getenv('MQTT_CLIENT_ID',"ambient_weather_decode")
MQTT_USERNAME     = os.getenv('MQTT_USERNAME',"")
MQTT_PASSWORD     = os.getenv('MQTT_PASSWORD',"")

# looking to get resultant topic like weather/ws-2902c/[item]
MQTT_TOPIC_PREFIX = os.getenv('MQTT_TOPIC_PREFIX',"weather")
MQTT_TOPIC 		  = MQTT_TOPIC_PREFIX + "/ws-2902c"

# mostly copied + pasted from https://www.emqx.io/blog/how-to-use-mqtt-in-python and some of my own MQTT scripts
def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print(f"connected to MQTT broker at {MQTT_BROKER_HOST}")
    else:
        print("Failed to connect, return code %d\n", rc)

def on_disconnect(client, userdata, flags, rc):
    print("disconnected from MQTT broker")

# set up mqtt client
client = mqtt.Client(client_id=MQTT_CLIENT_ID)
if MQTT_USERNAME and MQTT_PASSWORD:
    client.username_pw_set(MQTT_USERNAME,MQTT_PASSWORD)
    print("Username and password set.")
client.will_set(MQTT_TOPIC_PREFIX+"/status", payload="Offline", qos=1, retain=True) # set LWT     
client.on_connect = on_connect # on connect callback
client.on_disconnect = on_disconnect # on disconnect callback

# connect to broker
client.connect(MQTT_BROKER_HOST, port=MQTT_BROKER_PORT)
client.loop_start()

def publish(client, topic, msg):
    result = client.publish(topic, msg)
    # result: [0, 1]
    status = result[0]

    # uncomment for debug. don't need all the success messages.
    if status == 0:
        #print(f"Sent {msg} to topic {topic}")
        pass
    else:
        print(f"Failed to send message to topic {topic}")


def application(environ, start_response):
    # construct a full URL from the request. HTTP_HOST is FQDN, PATH_INFO is everything after
    # the FQDN (i.e. /data/stationtype=AMBWeatherV4.2.9&&tempinf=71.1&humidityin=35)
    url = "http://" + environ["HTTP_HOST"] + environ["PATH_INFO"]

    # unsure why I need to parse twice. probably just need to do it once with variable url.
    parsed = urlparse(url)
    result = parse_qs(parsed.geturl())

    # send to our other function to deal with the results.
    # result is a dict 
    handle_results(result)

    # we need to return a response. HTTP code 200 means everything is OK. other HTTP codes include 404 not found and such.
    start_response('200 OK', [('Content-Type', 'text/plain')])

    # the response doesn't actually need to contain anything
    response_body = ''

    # return the encoded bytes of the response_body. 
    # for python 2 (don't use python 2), the results don't need to be encoded
    return [response_body.encode()]


def handle_results(result):
    """ result is a dict. full list of variables include:
    stationtype: ['AMBWeatherV4.2.9'], PASSKEY: ['<station_mac_address>'], dateutc: ['2021-03-20 17:12:27'], tempinf: ['71.1'], humidityin: ['36'], baromrelin: ['29.693'],	baromabsin: ['24.549'],	tempf: ['58.8'], battout: ['1'], humidity: ['32'], winddir: ['215'],windspeedmph: ['0.0'],	windgustmph: ['0.0'], maxdailygust: ['3.4'], hourlyrainin: ['0.000'],	eventrainin: ['0.000'],	dailyrainin: ['0.000'],
    weeklyrainin: ['0.000'], monthlyrainin: ['0.000'], totalrainin: ['0.000'],	solarradiation: ['121.36'],
    uv: ['1'],batt_co2: ['1'] """

    # we're just going to publish everything. less coding.
    for key in result:
        # skip first item, which is basically a URL and MQTT doesn't like it. probably resulting from my bad url parsing.
        if 'http://' in key:
            continue

        #print(f"{key}: {result[key]}")
        # resultant topic is weather/ws-2902c/solarradiation
        specific_topic = MQTT_TOPIC + f"/{key}"

        # replace [' and '] with nothing. these come from the url parse
        msg = str(result[key]).replace("""['""", '').replace("""']""", '')
        #print(f"attempting to publish to {specific_topic} with message {msg}")
        publish(client, specific_topic, msg)

# this little guy runs a web server if this python file is called directly. if it isn't called directly, it won't run.
# Apache/Python WSGI will run the function 'application()' directly
# in theory, you don't need apache or any webserver. just run it right out of python. would need
# to improve error handling to ensure it run without interruption.
if __name__ == '__main__':
    from wsgiref.simple_server import make_server

    # probably shouldn't run on port 80 but that's what I specified in the ambient weather console
    httpd = make_server('', 80, application)
    print("Serving on http://localhost:80")

    httpd.serve_forever()

Execute with python3:

python3 main.py

Watch the results populate in the console window:

PS C:\Users\Austin\source\repos\ambient-weather-decode> python3 c:\Users\Austin\source\repos\ambient-weather-decode\main.py
connected to MQTT broker at mqtt
Serving on http://localhost:80
Sent E0:98:06:A3:42:65 to topic weather/ws-2902c/PASSKEY
Sent 2021-03-20 17:57:31 to topic weather/ws-2902c/dateutc
Sent 70.7 to topic weather/ws-2902c/tempinf
Sent 36 to topic weather/ws-2902c/humidityin
Sent 29.675 to topic weather/ws-2902c/baromrelin
Sent 24.531 to topic weather/ws-2902c/baromabsin
Sent 66.2 to topic weather/ws-2902c/tempf
Sent 1 to topic weather/ws-2902c/battout
Sent 26 to topic weather/ws-2902c/humidity
Sent 207 to topic weather/ws-2902c/winddir
Sent 0.2 to topic weather/ws-2902c/windspeedmph
Sent 1.1 to topic weather/ws-2902c/windgustmph
Sent 3.4 to topic weather/ws-2902c/maxdailygust
Sent 0.000 to topic weather/ws-2902c/hourlyrainin
Sent 0.000 to topic weather/ws-2902c/eventrainin
Sent 0.000 to topic weather/ws-2902c/dailyrainin
Sent 0.000 to topic weather/ws-2902c/weeklyrainin
Sent 0.000 to topic weather/ws-2902c/monthlyrainin
Sent 0.000 to topic weather/ws-2902c/totalrainin
Sent 697.92 to topic weather/ws-2902c/solarradiation
Sent 6 to topic weather/ws-2902c/uv
Sent 1 to topic weather/ws-2902c/batt_co2

Verification

Verify in MQTT by subscribing to topic ‘weather/#’. The # is a wildcard and will include all subtopics:

homeassistant ws-2902c mqtt messages
homeassistant ws-2902c mqtt messages

Conclusion

I am stoked the Ambient Weather WS-2902C makes it so easy to work the the data.

My next post will show how to turn this Python script into a persistent Linux service – Python service to consume Ambient Weather API data.

A further post will demonstrate incorporating these MQTT messages into Home Assistant sensors.