Categories
Blog Admin

Introduction / Table of Contents

The most recent post is the next post down!

I intend to use this site to document my journey down the path of nerdiness (past, present, and future). I’ve been learning over the years from various sites like what I hope this one becomes, and want to give back. I have a wide variety of topics I’d like to cover. At a minimum, posting about my activities will help me document what I learned to refer back in the future. I’ll also post about projects we do ourselves around the house instead of hiring professionals, saving big $$$$ in the process. Hope you enjoy the journey with me!

Below are some topic I plan on covering (I’ve already done something with every one of these and plan on documenting it):

  1. RTL-SDRs (receiving signals from your electric meter, ADS-B, general radio stuff)
  2. Virtual machines and my homelab setup
  3. Home automation / smart home (Home Assistant, Tasmota, Phillips Hue bulbs, automating various tasks throughout the house)
  4. My mini solar setup (2x300W panels) and not-so-mini battery backup (8x272Ah LiFePO4 batteries – should yield 7ish kWh of storage)
  5. Remote control aircraft running Arduplane with video downlink and two-way telemetry
  6. General computer stuff (building them, what I use mine for, Hyper-V)
  7. Home network (Ubiquiti setup, VLANs, wiring the house with CAT6, IP security cameras on Blue Iris)
  8. Formation of my LLC if anyone wants to hear about that
  9. The wheel options trading strategy
  10. Cryptocurrency (mining focus)
  11. SCADA (my day job)
  12. 3D printing
  13. Engine tuning (for my old WRX and new F-150)
  14. All the cool things you can do with a Raspberry pi and other SBCs
  15. Arduino/ESP32/ESP8266 automation devices
  16. My electric bikes
  17. Microsecond accurate Raspberry Pi NTP appliance using GPS pulse per second (PPS) timing signals
  18. DIY multi-zone sprinkler system install
  19. Drone survey of property
  20. Securing this WordPress site from hackers (Fail2Ban at both WordPress and system service level)
  21. Backing up WordPress sites
  22. General Linux tips/tricks
  23. VPNs (openvpn and wireguard)
Categories
MacOS proxmox Tutorials

How to install MacOS Monterey in a Proxmox 7 VM

I recently installed MacOS Monterey (12.1) in a Proxmox 7 virtual machine and made a YouTube video showing the process – https://youtu.be/HBAPscDD30M.

I followed the instructions on Nick Sherlock’s blog – https://www.nicksherlock.com/2021/10/installing-macos-12-monterey-on-proxmox-7/. He’s pretty good at instructions so I’ll just leave the link.

My main use for MacOS VM is to use Apple’s XCode for basic app development (Swift, SwiftUI, UIKit, React Native, etc.) in a fairly fast environment. I have a slower actual Mac for publishing and such but like the flexibility of working in a virtual environment within Proxmox.

I’m still thinking about writing a script to automate the deployment – keep checking back if you’re interested!

This post mostly serves as a link to the YouTube video for how to install MacOS Monterey in Proxmox.

Here’s a picture of the environment showing my Xeon e5-2678v3 as the processor in the MacOS desktop:

MacOS Monterey running in a Proxmox virtual machine for Xcode
MacOS Monterey running in a Proxmox virtual machine for Xcode
Categories
Home Assistant Home Automation

Using the Govee Bluetooth Thermometer with Home Assistant (Python and MQTT)

Introduction

Like many other Home Automation enthusiasts, I have been on the lookout for a cheap thermometer/hygrometer that has either WiFi or Bluetooth connectivity. I think I found the answer in the $12 USD Govee Bluetooth Digital Thermometer and Hygrometer. The fact that the device broadcasts the current temperature and humidity every 2 seconds via low energy Bluetooth (BLE) is the metaphorical icing on the cake. Here is a pic of the unit in our garage:

Govee bluetooth thermometer and hygrometer showing 56*F/24% in a garage

Specifications

This is a pretty basic device. It has a screen that shows the current temperature, humidity, and min/max values. It sends the current readings (along with battery health) every 2 seconds via low-energy bluetooth (BLE). The description on Amazon says it has a “Swiss-made smart hygrometer sensor”. Dunno if I believe that but for $12 it is good enough. The temperature is accurate to +/- 0.54F and humidity is +/- 3% RH. If you use the app, it is apparently possible to read the last 20 days or 2 years of data from the device (I haven’t used the app at all).

Enough with the boring stuff. Let’s get it connected to Home Assistant.

Reading the Govee bluetooth advertisements with Python

I found some sample code on tchen’s GitHub page (link) to help get me going in the right direction.

Without further ado, here is the code I’m using to read the data and publish via MQTT:

observe.py (updated 2022-01-04 with some logging improvements. default logging level is now WARNING, which disables printing every advertisement)

# basic govee bluetooth data reading code for model https://amzn.to/3z14BIi
# written/modified by Austin of austinsnerdythings.com 2021-12-27
# original source: https://gist.github.com/tchen/65d6b29a20dd1ef01b210538143c0bf4
import logging
import json
from time import sleep
from basic_mqtt import basic_mqtt
from bleson import get_provider, Observer

logging.basicConfig(
	format='%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',
	datefmt='%Y-%m-%d %H:%M:%S',
	level=logging.WARNING)

# I did write all the mqtt stuff. I kept it in a separate class
mqtt = basic_mqtt()
mqtt.connect()

# writing code on my windows computer, committing, pushing, and then 
# pulling the new code on the Raspberry Pi is a tedious "debug" process.
# there are quite a few errors that spit out from bleson, which as far
# as I can tell, isn't super polished. if we set the debug level to
# critical, most of those disappear.
logging.getLogger("bleson").setLevel(logging.CRITICAL)

# basic celsius to fahrenheit function
def c2f(val):
    return round(32 + 9*val/5, 2)

last = {}

# I didn't write this, but it takes the raw BT data and spits out the data of interest
def temp_hum(values, battery, address):
    global last
    values = int.from_bytes(values, 'big')
    if address not in last or last[address] != values:
        last[address] = values
        temp = float(values / 10000)
        hum = float((values % 1000) / 10)
        # this print looks like this:
        # 2021-12-27T11:22:17.040469 BDAddress('A4:C1:38:9F:1B:A9') Temp: 45.91 F  Humidity: 25.8 %  Battery: 100 %
        logging.info(f"decoded values: {address} Temp: {c2f(temp)} F  Humidity: {hum} %  Battery: {battery}")
        # this code originally just printed the data, but we need it to publish to mqtt.
        # added the return values to be used elsewhere
        return c2f(temp), hum, battery

def on_advertisement(advertisement):
    #print(advertisement)
    mfg_data = advertisement.mfg_data

    # there are lots of BLE advertisements flying around. only look at ones that have mfg_data
    if mfg_data is not None:
        #print(advertisement)
        # there are a few Govee models and the data is in different positions depending on which
        # the unit of interest isn't either of these hardcoded values, so they are skipped
        if advertisement.name == 'GVH5177_9835':
            address = advertisement.address
            temp_hum(mfg_data[4:7], mfg_data[7], address)
        elif advertisement.name == 'GVH5075_391D':
            address = advertisement.address
            temp_hum(mfg_data[3:6], mfg_data[6], address)
        elif advertisement.name != None:
            # this is where all of the advertisements for our unit of interest will be processed
            address = advertisement.address
            if 'GVH' in advertisement.name:
                #print(advertisement)
                temp_f, hum, battery = temp_hum(mfg_data[3:6], mfg_data[6], address)

                if temp_f > 180.0 or temp_f < -30.0:
                    return
                # as far as I can tell bleson doesn't have a string representation of the MAC address
                # address is of type BDAddress(). str(address) is BDAddress('A4:C1:38:9F:1B:A9')
                # substring 11:-2 is just the MAC address
                mac_addr = str(address)[11:-2]

                # construct dict with relevant info
                msg = {'temp_f':temp_f,
                        'hum':hum,
                        'batt':battery}
                
                # looks like this:
                # msg data: {'temp_f': 45.73, 'hum': 25.5, 'batt': 100}
                logging.info(f"MQTT msg data: {msg}")

                # turn into JSON for publishing
                json_string = json.dumps(msg, indent=4)

                # publish to topic separated by MAC address
                mqtt.publish(f"govee/{mac_addr}", json_string)

# base stuff from the original gist
logging.warning(f"initializing bluetooth")
adapter = get_provider().get_adapter()
observer = Observer(adapter)
observer.on_advertising_data = on_advertisement

logging.warning(f"starting observer")
observer.start()
logging.warning(f"listening for events and publishing to MQTT")
while True:
    # unsure about this loop and how much of a delay works
    sleep(1)
observer.stop()

And for the MQTT helper class (mqtt_helper.py):

# basic MQTT helper class. really needed to write one of these to simplify basic MQTT operations
# written/modified by Austin of austinsnerdythings.com 2021-12-27
# original source: https://gist.github.com/fisherds/f302b253cf7a11c2a0d814acd424b9bb
# filename is basic_mqtt.py
from paho.mqtt import client as mqtt_client
import logging
import datetime
logging.basicConfig(
	format='%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',
	datefmt='%Y-%m-%d %H:%M:%S',
	level=logging.INFO)


mqtt_host = "mqtt.home.fluffnet.net"
test_topic = "mqtt_test_topic"

# this is really not polished. it was a stream of consciousness project to pound
# something out to do basic MQTT publish stuff in a reusable fashion.
class basic_mqtt:
	def __init__(self):
		self.client = mqtt_client.Client()
		self.subscription_topic_name = None
		self.publish_topic_name = None
		self.callback = None
		self.host = mqtt_host
	
	def connect(self):
		self.client.on_connect = self.on_connect
		self.client.on_subscribe = self.on_subscribe
		self.client.on_message = self.on_message
		logging.info(f"connecting to MQTT broker at {mqtt_host}")
		self.client.connect(host=mqtt_host,keepalive=30)
		self.client.loop_start()

	def on_connect(self, client, userdata, flags, rc):
		print("Connected with result code "+str(rc))

		# Subscribing in on_connect() means that if we lose the connection and
		# reconnect then subscriptions will be renewed.
		#self.client.subscribe("$SYS/#")

	def on_message(self, client, userdata, msg):
		print(f"got message of topic {msg.topic} with payload {msg.payload}")

	def on_subscribe(self, client, userdata, mid, granted_qos):
		print("Subscribed: " + str(mid) + " " + str(granted_qos))

	def publish(self, topic, msg):
		self.client.publish(topic=topic, payload=msg)

	def subscribe_to_test_topic(self, topic=test_topic):
		self.client.subscribe(topic)

	def send_test_message(self, topic=test_topic):

		self.publish(topic=topic, msg=f"test message from python script at {datetime.datetime.now()}")

	def disconnect(self):
		self.client.disconnect()

	def loop(self):
		self.client.loop_forever()

if __name__ == "__main__":
	logging.info("running MQTT test")
	mqtt_helper = basic_mqtt()
	mqtt_helper.connect()
	mqtt_helper.subscribe_to_test_topic()
	mqtt_helper.send_test_message()
	mqtt_helper.disconnect()

Results

Running this script on a Raspberry Pi 3 shows the advertisements coming in as expected. The updates are very quick compared to the usual 16 second update interval for my Acurite stuff.

screenshot of Python code running to receive BLE advertisements from Govee Bluetooth Thermometer
screenshot of Python code running to receive BLE advertisements from Govee Bluetooth Thermometer

And running mosquitto_sub with the right arguments (mosquitto_sub -h mqtt -v -t “govee/#”) shows the MQTT messages are being published as expected:

mosquitto_sub showing our published MQTT messages with temperature/humidity data from the Govee sensor

Getting Govee MQTT data into Home Assistant

Lastly, we need to add a MQTT sensor to get the data importing into Home Assistant:

- platform: mqtt
  state_topic: "govee/A4:C1:38:9F:1B:A9"
  value_template: "{{ value_json.temp_f }}"
  name: "garage temp"
  unit_of_measurement: "F"

And from there you can do whatever you want with the collected data!

Home Assistant displaying data from Govee Bluetooth temperature and humidity sensor

Conclusion

This was a relatively quick post and code development. I really hate the cycle of developing on my Bluetooth-less Windows computer, committing the code, pushing to Git, pulling on the Pi, and running to “debug”. Thus, the code isn’t as good as it can be. I probably did 20-25 iterations before calling it good enough.

Regardless, I think this $12 Govee Bluetooth Thermometer and Hygrometer is a great little tool for collecting data around the house. You don’t need an SDR to get Acurite beacons, and you don’t need to spend a lot either. You just need a way to receive BLE advertisements (basically any Bluetooth-capable device can do this). There is even a 2 pack of just the sensors that I just discovered on Amazon for $24 – 2 pack of Govee bluetooth thermometer and hygrometer. I know there are other Bluetooth devices but they’re quite a bit more expensive.

Also, at least for me, these are available with same day shipping on Amazon. Scratch that instant gratification itch.

Same day shipping available on Amazon near Denver for the Govee sensors
Categories
Python XPlane

Coding a pitch/roll/altitude autopilot in X-Plane with Python

Introduction

Sorry it has taken me so long to write this post! The last post on the blog was October 19th – almost 6 weeks ago. Life happens. We have a 15 month old running around and she is a handful!

Anyways, back to the next topic – coding a pitch/roll (2 axis) autopilot in X-Plane with Python with altitude and heading hold. Today we will be adding the following:

  • Real-time graphing for 6 parameters
  • Additional method to grab data out of X-Plane
  • A normalize function to limit outputs to reasonable values
  • Altitude preselect and hold function

The full code will be at the end of this post.

Video Link

coming soon

Contents

  1. Adding PyQtGraph
  2. Developing a normalize function
  3. Initializing the data structures to hold the graph values
  4. Defining the PyQtGraph window and parameters
  5. Getting more data points out of X-Plane
  6. Feeding the graph data structures with data
  7. Adding altitude preselect and hold

1 – Adding PyQtGraph

Pip is my preferred tool to manage Python packages. It is easy and works well. I initially tried graphing with MatPlotLib but it took 200-300ms to update, which really dragged down the control loop to the point of being unusable. Instead, we will be using PyQtGraph. Install it with Pip:

pip install pyqtgraph

2 – Developing a normalize function

This task is pretty straightforward. There are a couple places where we want to pass values that need to be within a certain range. The first example is to the client.sendCTRL() method to set the control surfaces in X-Plane. The documentation states values are expected to be from -1 to 1. I have got some really weird results sending values outside that range (specifically for throttle, if you send something like 4, you can end up with 400% throttle which is wayyy more than the engines can actually output).

# this function takes an input and either passes it through or adjusts
# the input to fit within the specified max/min values
def normalize(value, min=-1, max=1):
	# if value = 700, and max = 20, return 20
	# if value = -200, and min = -20, return -20
	if (value > max):
		return max
	elif (value < min):
		return min
	else:
		return value

3 – Initializing the graphing data structures

We need a couple of arrays to store the data for our graphs. We need (desired data to plot) + 1 arrays. The +1 is the x-axis, which will just store values like 0,1,2,3,etc. The others will be the y-values. We haven’t added the altitude stuff yet, so you can add them but they won’t be used yet.

x_axis_counters = [] #0, 1, 2, 3, etc. just basic x-axis values used for plotting
roll_history = []
pitch_history = []
#altitude_history = []
roll_setpoint_history = []
pitch_setpoint_history = []
#altitude_setpoint_history = []
plot_array_max_length = 100 # how many data points to hold in our arrays and graph
i = 1 # initialize x_axis_counter

4 – Defining the PyQtGraph window and parameters

Working with PyQtGraph more or less means we’ll be working with a full blown GUI (just stripped down).

# first the base app needs to be instantiated
app = pg.mkQApp("python xplane autopilot monitor")

# now the window itself is defined and sized
win = pg.GraphicsLayoutWidget(show=True)
win.resize(1000,600) #pixels
win.setWindowTitle("XPlane autopilot system control")

# we have 3 subplots
p1 = win.addPlot(title="roll",row=0,col=0)
p2 = win.addPlot(title="pitch",row=1,col=0)
p3 = win.addPlot(title="altitude", row=2, col=0)

# show the y grid lines to make it easier to interpret the graphs
p1.showGrid(y=True)
p2.showGrid(y=True)
p3.showGrid(y=True)

5 – Getting more data points out of X-Plane

The initial .getPOSI() method that came in the example has worked well for us so far. But at this point we need more data that isn’t available in the .getPOSI() method. We will be utilizing a different method called .getDREFs() which is short for ‘get data references’. We will need to construct a list of data references we want to retrieve, pass that list to the method, and then parse the output. It is more granular than .getPOSI(). I haven’t evaluated the performance but I don’t think it is a problem.

The DREFs we want are for indicated airspeed, magnetic heading (.getPOSI() has true heading, not magnetic), an indicator to show if we are on the ground or not, and height as understood by the flight model. Thus, we can define our DREFs as follows:

DREFs = ["sim/cockpit2/gauges/indicators/airspeed_kts_pilot",
		"sim/cockpit2/gauges/indicators/heading_electric_deg_mag_pilot",
		"sim/flightmodel/failures/onground_any",
		"sim/flightmodel/misc/h_ind"]

And we can get the data with client.getDREFs(DREFs). The returned object is a 2d array. We need to parse out our values of interest. The full data gathering code looks like this:

posi = client.getPOSI();
ctrl = client.getCTRL();
multi_DREFs = client.getDREFs(DREFs)

current_roll = posi[4]
current_pitch = posi[3]
current_hdg = multi_DREFs[1][0]
current_altitude = multi_DREFs[3][0]
current_asi = multi_DREFs[0][0]
onground = multi_DREFs[2][0]

With those data points, we have everything we need to start plotting the state of our aircraft and monitoring for PID tuning.

6 – Feeding the real-time graphs with data

Next up is actually adding data to be plotted. There are two scenarios to consider when adding data to the arrays: 1) the arrays have not yet reached the limit we set earlier (100 points), and 2) they have. Case 1 is easy. We just append the current values to the arrays:

x_axis_counters.append(i)
roll_history.append(current_roll)
roll_setpoint_history.append(desired_roll)
pitch_history.append(current_pitch)
pitch_setpoint_history.append(pitch_PID.SetPoint)
altitude_history.append(current_altitude)
altitude_setpoint_history.append(desired_altitude)

The above code will work perfectly fine if you want the arrays to grow infinitely large over time. Ain’t nobody got time for that so we need to check how long the arrays are and delete data. We’ll check the length of the x-axis array as a proxy for all the others and use that to determine what to do. Typing this code that looks very similar over and over again means it’s probably time to abstract it into classes or something else. The more you type something over and over again, the larger indication you have that you need to so something about it. But for now we’ll leave it like this for ease of reading and comprehension.

# if we reach our data limit set point, evict old data and add new.
# this helps keep the graph clean and prevents it from growing infinitely
if(len(x_axis_counters) > plot_array_max_length):
	x_axis_counters.pop(0)
	roll_history.pop(0)
	roll_setpoint_history.pop(0)
	pitch_history.pop(0)
	pitch_setpoint_history.pop(0)
	altitude_history.pop(0)
	altitude_setpoint_history.pop(0)

	x_axis_counters.append(i)
	roll_history.append(current_roll)
	roll_setpoint_history.append(desired_roll)
	pitch_history.append(current_pitch)
	pitch_setpoint_history.append(pitch_PID.SetPoint)
	altitude_history.append(current_altitude)
	altitude_setpoint_history.append(desired_altitude)
# else, just add new. we are not yet at limit.
else:
	x_axis_counters.append(i)
	roll_history.append(current_roll)
	roll_setpoint_history.append(desired_roll)
	pitch_history.append(current_pitch)
	pitch_setpoint_history.append(pitch_PID.SetPoint)
	altitude_history.append(current_altitude)
	altitude_setpoint_history.append(desired_altitude)
i = i + 1

You will notice that there are quite a few entries for altitude. We haven’t done anything with that yet so just set desired_altitude to an integer somewhere in the code so it doesn’t error out.

To complete the graphing portion, we need to actually plot the data. The clear=true in the below lines clears out the plot so we’re not replotting on top of the old data. We also need to process events to actually draw the graph:

# process events means draw the graphs
pg.QtGui.QApplication.processEvents()

# arguments are x values, y values, options
# pen is a different line in the plot
p1.plot(x_axis_counters, roll_history, pen=0, clear=True)
p1.plot(x_axis_counters, roll_setpoint_history, pen=1)

p2.plot(x_axis_counters, pitch_history, pen=0,clear=True)
p2.plot(x_axis_counters, pitch_setpoint_history, pen=1)

p3.plot(x_axis_counters, altitude_history, pen=0,clear=True)
p3.plot(x_axis_counters, altitude_setpoint_history, pen=1)

You can now run the code to see your graph populating with data!

PyQtGraph plotting the aircraft’s roll/pitch and desired roll/pitch

7 – Adding altitude autopilot (preselect and hold)

Ok so now that we have eye candy with the real-time graphs, we can make our autopilot do something useful: go to a selected altitude and hold it.

We already have the roll and pitch PIDs functioning as desired. How do we couple the pitch PID to get to the desired altitude? One cannot directly control altitude. Altitude is controlled via a combination of pitch and airspeed (and time).

We will call the coupled PIDs an inner loop (pitch) and an outer loop (altitude). The outer loop runs and its output will feed the input of the inner loop. The altitude PID will be fed a desired altitude and current altitude. The output will then mostly be the error (desired altitude – current altitude) multiplied by our P setting. Of course I and D will have a say in the output but by and large it will be some proportion of the error.

Let’s start with defining the altitude PID and desired altitude:

altitude_PID = PID.PID(P, I, D)
desired_altitude = 8000
altitude_PID.SetPoint = desired_altitude

With those defined, we now move to the main loop. The outer loop needs to be updated first. From there, we will normalize the output from the altitude PID and use that to set the pitch PID. The pitch PID will also be normalized to keep values in a reasonable range:

# update outer loops first
altitude_PID.update(current_altitude)

# if alt=12000, setpoint = 10000, the error is 2000. if P=0.1, output will be 2000*0.1=200
pitch_PID.SetPoint = normalize(altitude_PID.output, min=-15, max=10)

# update PIDs
roll_PID.update(current_roll)
pitch_PID.update(current_pitch)

# update control outputs
new_ail_ctrl = normalize(roll_PID.output)
new_ele_ctrl = normalize(pitch_PID.output)

Now we just need to send those new control surface commands and we’ll be controlling the plane!

Outputting the control deflections should basically be the last part of the loop. We’ll put it right before the debug output:

# sending actual control values to XPlane
ctrl = [new_ele_ctrl, new_ail_ctrl, 0.0, -998] # ele, ail, rud, thr. -998 means don't change
client.sendCTRL(ctrl)

Full code of pitch_roll_autopilot_with_graphing.py

import sys
import xpc
import PID
from datetime import datetime, timedelta
import pyqtgraph as pg
from pyqtgraph.Qt import QtCore, QtGui
import time

def normalize(value, min=-1, max=1):
	# if value = 700, and max = 20, return 20
	# if value = -200, and min = -20, return -20
	if (value > max):
		return max
	elif (value < min):
		return min
	else:
		return value

update_interval = 0.050 # seconds, 0.05 = 20 Hz
start = datetime.now()
last_update = start

# defining the initial PID values
P = 0.1 # PID library default = 0.2
I = P/10 # default = 0
D = 0 # default = 0

# initializing PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)
altitude_PID = PID.PID(P, I, D)

# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2
desired_altitude = 8000

# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch
altitude_PID.SetPoint = desired_altitude

x_axis_counters = [] #0, 1, 2, 3, etc. just basic x-axis values used for plotting
roll_history = []
pitch_history = []
altitude_history = []
roll_setpoint_history = []
pitch_setpoint_history = []
altitude_setpoint_history = []
plot_array_max_length = 300 # how many data points to hold in our arrays and graph
i = 1 # initialize x_axis_counter

# first the base app needs to be instantiated
app = pg.mkQApp("python xplane autopilot monitor")

# now the window itself is defined and sized
win = pg.GraphicsLayoutWidget(show=True)
win.resize(1000,600) #pixels
win.setWindowTitle("XPlane autopilot system control")

# we have 3 subplots
p1 = win.addPlot(title="roll",row=0,col=0)
p2 = win.addPlot(title="pitch",row=1,col=0)
p3 = win.addPlot(title="altitude", row=2, col=0)

# show the y grid lines to make it easier to interpret the graphs
p1.showGrid(y=True)
p2.showGrid(y=True)
p3.showGrid(y=True)

DREFs = ["sim/cockpit2/gauges/indicators/airspeed_kts_pilot",
		"sim/cockpit2/gauges/indicators/heading_electric_deg_mag_pilot",
		"sim/flightmodel/failures/onground_any",
		"sim/flightmodel/misc/h_ind"]

def monitor():
	global i
	global last_update
	with xpc.XPlaneConnect() as client:
		while True:
			if (datetime.now() > last_update + timedelta(milliseconds = update_interval * 1000)):
				last_update = datetime.now()
				print(f"loop start - {datetime.now()}")

				posi = client.getPOSI();
				ctrl = client.getCTRL();
				multi_DREFs = client.getDREFs(DREFs)

				current_roll = posi[4]
				current_pitch = posi[3]
				current_hdg = multi_DREFs[1][0]
				current_altitude = multi_DREFs[3][0]
				current_asi = multi_DREFs[0][0]
				onground = multi_DREFs[2][0]

				# update the display
				pg.QtGui.QApplication.processEvents()

				# update outer loops first
				altitude_PID.update(current_altitude)

				# if alt=12000, setpoint = 10000, the error is 2000. if P=0.1, output will be 2000*0.1=200
				pitch_PID.SetPoint = normalize(altitude_PID.output, min=-15, max=10)

				# update PIDs
				roll_PID.update(current_roll)
				pitch_PID.update(current_pitch)

				# update control outputs
				new_ail_ctrl = normalize(roll_PID.output)
				new_ele_ctrl = normalize(pitch_PID.output)

				# if we reach our data limit set point, evict old data and add new.
				# this helps keep the graph clean and prevents it from growing infinitely
				if(len(x_axis_counters) > plot_array_max_length):
					x_axis_counters.pop(0)
					roll_history.pop(0)
					roll_setpoint_history.pop(0)
					pitch_history.pop(0)
					pitch_setpoint_history.pop(0)
					altitude_history.pop(0)
					altitude_setpoint_history.pop(0)

					x_axis_counters.append(i)
					roll_history.append(current_roll)
					roll_setpoint_history.append(desired_roll)
					pitch_history.append(current_pitch)
					pitch_setpoint_history.append(pitch_PID.SetPoint)
					altitude_history.append(0)
					altitude_setpoint_history.append(desired_altitude)
				# else, just add new. we are not yet at limit.
				else:
					x_axis_counters.append(i)
					roll_history.append(current_roll)
					roll_setpoint_history.append(desired_roll)
					pitch_history.append(current_pitch)
					pitch_setpoint_history.append(pitch_PID.SetPoint)
					altitude_history.append(0)
					altitude_setpoint_history.append(desired_altitude)
				i = i + 1

				p1.plot(x_axis_counters, roll_history, pen=0, clear=True)
				p1.plot(x_axis_counters, roll_setpoint_history, pen=1)

				p2.plot(x_axis_counters, pitch_history, pen=0,clear=True)
				p2.plot(x_axis_counters, pitch_setpoint_history, pen=1)

				p3.plot(x_axis_counters, altitude_history, pen=0,clear=True)
				p3.plot(x_axis_counters, altitude_setpoint_history, pen=1)

				# sending actual control values to XPlane
				ctrl = [new_ele_ctrl, new_ail_ctrl, 0.0, -998] # ele, ail, rud, thr. -998 means don't change
				client.sendCTRL(ctrl)

				output = f"current values --    roll: {current_roll: 0.3f},  pitch: {current_pitch: 0.3f}"
				output = output + "\n" + f"PID outputs    --    roll: {roll_PID.output: 0.3f},  pitch: {pitch_PID.output: 0.3f}"
				output = output + "\n"
				print(output)

if __name__ == "__main__":
	monitor()

Using the autopilot / Conclusion

To use the autopilot, fire up XPlane, hop in a small-ish plane (gross weight less than 10k lb), take off, climb 1000′, then execute the code. Your plane should bring roll to 0 pretty quick and start the climb/descent to the desired altitude.

X-Plane Python autopilot leveling off at 8000' with pitch/roll/altitude graphed in real-time
X-Plane Python autopilot leveling off at 8000′ with pitch/roll/altitude graphed in real-time
Categories
Python XPlane

Coding a wing leveler autopilot in X-Plane with Python

Introduction

Continuing from the first post, where we hooked up X-Plane to our Python code, we will build a wing leveler today. The first post was just about making sure we could get data into and out of X-Plane. Today will add a few features to our XPlane autopilot written in Python.

  • A control loop timer (we will be targeting a loop frequency of 10 Hz, or 10 updates per second)
  • Additional data feeds from X-Plane
  • Two PID controllers, one for roll, one for pitch
  • Some debugging output to keep track of what the PIDs are doing

The full code will be at the end of this post.

Video Link

Python Tutorial: code a wing leveler in X-Plane using PID loops

Contents

  1. Control loop timer/limiter
  2. Obtaining current pitch/roll values from X-Plane
  3. Initializing the PID controllers
  4. Feeding the PID controllers within the control loop
  5. Controlling the aircraft with the new PID output
  6. Monitoring the control loops via debug prints

1 – Control loop timer/limiter

Since we will be targeting a 10 Hz update rate, we need to develop a method to ensure the loop does not run more frequent than once every 100 milliseconds. We do not want the loop running uninhibited, because that will result in variable loop execution times and we like to keep those things constant. It could potentially execute thousands of times per second, which is entirely unnecessary. Most control loop algorithms run in the 10-100 Hz range (10-100 times per second). For reference, my RC plane running ArduPlane on Pixhawk uses 50 Hz as a standard update frequency.

To accomplish this task, we need to set up some timing variables.

First of all, add an import statment for datetime and timedelta:

from datetime import datetime, timedelta

Next, define the timing variables:

update_interval = 0.100 # this value is in seconds. 1/0.100 = 10 which is the update interval in Hz

# start is set to the time the line of code is executed, which is essential when the program started
start = datetime.now()

# last_update needs to be set to start for the first execution of the loop to successfully run.
# alternatively, we could've set start to something in the past.
last_update = start

# last update needs to be defined as a global within the monitor() function:
def monitor():
	global last_update

That handles the variables. Now we need to limit the loop execution. To do so requires wrapping all of the loop code into a new if statement that evaluates the time and only executes if the current time is 100 milliseconds greater than the last loop execution:

# loop is the "while True:" statement
while True:
	# this if statement is evaluated with every loop execution
	if (datetime.now() > last_update + timedelta(milliseconds = update_interval * 1000)):
		# when the if statement evaluates to true, the first thing we'll do is set the last update to the current time so the next iteration fires at the correct time
		last_update = datetime.now()

		# rest of the loop code goes here

2 – Obtaining current roll/pitch values from X-Plane

This task is pretty straightforward. In the first post, the monitorExample.py code included obtaining the position with posi = client.getPOSI(). There are 7 elements returned with that method, and two of them are roll and pitch.

Put the below after the .getPOSI() call.

current_roll = posi[4]
current_pitch = posi[3]

3 – Initializing the PID controllers

First you need to get the PID control file from the Ivmech (Ivmech Mechatronics Ltd.) GitHub page. Direct link PID.py file here. Put the file in the same working directory as everything else then import it with import PID at the top with the rest of the imports.

Then we can initialize the control instances. PID controllers are magic. But they do need to be set with smart values for the initial run. The default values with the PID library are P=0.2, I=0, D=0. This essentially means make the output be 20% of the error between the setpoint and the current value. For example, if the aircraft has a roll of 10 degrees to the left (-10), and the P=0.2, the output from the PID controller will be -2.

When setting PID values, it is almost always a good idea to start small and work your way up. If your gains are too high, you could get giant oscillations and other undesirable behaviors. In the YouTube video, I talk about the various types of PID controllers. They will basically always have a P (proportional) set. Many will also have an I (integral) value set, but not many use the D term (derivative).

Going with the “less is more” truism with PID controllers, I started with a P value of 0.1, and an I value of 0.01 (P/10). The I term (integral) is meant to take care of accumulated errors (which are usually long term errors). An example is your car’s cruise control going up a hill. If your cruise control is set to 65 mph, it will hold 65 no problem on flat roads. If you start going up a hill, the controller will slowly apply more throttle. With an I of 0, your car would never get to 65. It would probably stay around 62 or so. With the integrator going, the error will accumulate and boost the output (throttle) to get you back up to your desired 65. In the linked YouTube video, I show what happens when the I value is set to 0 and why it is necessary to correct long-term errors.

These values (P=0.1, I=0.01, D=0) turned out to work perfectly.

Place the following before the monitor function:

# defining the initial PID values
P = 0.1 # PID library default = 0.2
I = P/10 # default = 0
D = 0 # default = 0

# initializing both PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)

# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2

# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch

4 – Updating the PID controllers within the control loop

With the PIDs created, they will need to be updated with the new, current pitch and roll values with every loop execution.

Place the following after current_roll and current_pitch are assigned:

roll_PID.update(current_roll)
pitch_PID.update(current_pitch)

5 – Controlling the aircraft with the new PID output

Updating the PID controller instances will generate new outputs. We can use those outputs to set the control surfaces. You can place these two lines directly below the update lines:

new_ail_ctrl = roll_PID.output
new_ele_ctrl = pitch_PID.output

So we have new control surface values – now we need to actually move the control surfaces. This is accomplished by sending an array of 4 floats with .sendCTRL():

# ele, ail, rud, thr. -998 means don't set/change
ctrl = [new_ele_ctrl, new_ail_ctrl, 0.0, -998]
client.sendCTRL(ctrl)

6 – Monitoring the control loops via debug prints

The last bit to tie a bunch of this together is printing out values with every loop execution to ensure things are heading the right direction. We will turn these into graphs in the next post or two.

We will be concatenating strings because it’s easy and we aren’t working with enough strings for it to be a performance problem.

In newer Python versions (3.6+), placing ‘f’ before the quotes in a print statement (f””) means the string is interpolated. This means you can basically put code in the print statement, which makes creating print statements much easier and cleaner.

The first line will print out the current roll and pitch value (below). We are using current_roll and current_pitch interpolated. The colon, then blank space with 0.3f is a string formatter. It rounds the value to 3 decimal places and leaves space for a negative. It results in things being lined up quite nicely.

output = f"current values --    roll: {current_roll: 0.3f},  pitch: {current_pitch: 0.3f}"

The next code statement will add a new line to the previous output, and also add the PID outputs for reference:

output = output + "\n" + f"PID outputs    --    roll: {roll_PID.output: 0.3f},  pitch: {pitch_PID.output: 0.3f}"

The final line will just add another new line to keep each loop execution’s print statements grouped together:

output = output + "\n"

Finally, we print the output with print(output), which will look like this:

loop start - 2021-10-19 14:24:26.208945
current values --    roll:  0.000,  pitch:  1.994
PID outputs    --    roll: -0.000,  pitch:  0.053

Full code of pitch_roll_autopilot.py

import sys
import xpc
import PID
from datetime import datetime, timedelta

update_interval = 0.100 # seconds
start = datetime.now()
last_update = start

# defining the initial PID values
P = 0.1 # PID library default = 0.2
I = P/10 # default = 0
D = 0 # default = 0

# initializing both PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)

# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2

# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch

def monitor():
	global last_update
	with xpc.XPlaneConnect() as client:
		while True:
			if (datetime.now() > last_update + timedelta(milliseconds = update_interval * 1000)):
				last_update = datetime.now()
				print(f"loop start - {datetime.now()}")

				posi = client.getPOSI();
				ctrl = client.getCTRL();

				current_roll = posi[4]
				current_pitch = posi[3]

				roll_PID.update(current_roll)
				pitch_PID.update(current_pitch)

				new_ail_ctrl = roll_PID.output
				new_ele_ctrl = pitch_PID.output

				ctrl = [new_ele_ctrl, new_ail_ctrl, 0.0, -998] # ele, ail, rud, thr. -998 means don't change
				client.sendCTRL(ctrl)

				output = f"current values --    roll: {current_roll: 0.3f},  pitch: {current_pitch: 0.3f}"
				output = output + "\n" + f"PID outputs    --    roll: {roll_PID.output: 0.3f},  pitch: {pitch_PID.output: 0.3f}"
				output = output + "\n"
				print(output)

if __name__ == "__main__":
	monitor()

Using the autopilot / Conclusion

To use the autopilot, fire up XPlane, hop in a small-ish plane (gross weight less than 10k lb), take off, climb 1000′, then execute the code. Your plane should level off within a second or two in both axis.

Here is a screenshot of the output after running for a few seconds:

Screenshot showing current pitch and roll values along with their respective PID outputs

The linked YouTube video shows the aircraft snapping to the desired pitch and roll angles very quickly from a diving turn. The plane is righted to within 0.5 degrees of the setpoints within 2 seconds.

A note about directly setting the control surfaces

If you are familiar with PIDs and/or other parts of what I’ve discussed here, you’ll realize that we could be setting large values for the control surfaces (i.e. greater than 1 or less than -1). We will address that next post with a normalization function. It will quickly become a problem when a pitch of 100+ is commanded for altitude hold. I have found that XPlane will allow throttle values of more than 100% (I’ve seen as high as ~430%) if sent huge throttle values.

Categories
Python XPlane

Creating an autopilot in X-Plane using Python – part 1

Introduction

Today’s post will take us in a slightly different direction than the last few. Today’s post will be about hooking up some Python code to the X-Plane flight simulator to enable development of an autopilot using PID (proportional-integral-derivative) controllers. I’ve been a fan of flight simulators for quite some time (I distinctly remember getting Microsoft Flight Simulator 98 for my birthday when I was like 8 or 9) but have only recently started working with interfacing them to code. X-Plane is a well-known flight simulator developed by another Austin – Austin Meyer. It is regarded as having one of the best flight models and has tons of options for getting data into/out of the simulator. More than one FAA-certified simulator setups are running X-Plane as the primary driver software.

I got started thinking about writing some code for X-Plane while playing another game, Factorio. I drive a little plane or car in the game to get around my base and I just added a plug-in that “snaps” the vehicle to a heading, which makes it easier to go in straight lines. I thought – “hmm how hard could this be to duplicate in a flight sim?”. So here we are.

This post will get X-Plane hooked up to Python. The real programming will start with the next post.

Video Link

Contents

  1. Download and install X-Plane (I used X-Plane 10 because it uses less resources than X-Plane 11 and we don’t need the graphics/scenery to look super pretty to do coding. It also loads faster.)
  2. Download and install NASA’s XPlaneConnect X-Plane plug-in to X-Plane
  3. Verify the XPlaneConnect plug-in is active in X-Plane
  4. Download sample code from XPlaneConnect’s GitHub page
  5. Run the sample script to verify data is being transmitted from X-Plane via UDP to the XPlaneConnect code

1 – Download and install X-Plane 10 or X-Plane 11

I’ll leave this one up to you. X-Plane 10 is hard to find these days I just discovered. X-Plane 11 is available on Steam for $59.99 as of writing. I just tested and the plug-in/code works fine on X-Plane 11 (but the flight models are definitely different and will need different PID values). My screenshots might jump between the two versions but the content/message will be the same.

2 – Download and install NASA’s XPlaneConnect plug-in

NASA (yes, that NASA, the National Aeronautics and Space Administration) has wrote a bunch of code to interface with X-Plane. They have adapters for C, C++, Java, Matlab, and Python. They work with X-Plane 9, 10, and 11.

  1. Download the latest version from the XPlaneConnect GitHub releases page, 1.3 RC6 as of writing
  2. Open the .zip and place the contents in the [X-Plane directory]/Resources/plugins folder. There are few other folders already present in this directory. Mine looked like this after adding the XPlaneConnect folder:
Screenshot of X-Plane 10 plugins directory with XPlaneConnect folder added
Screenshot of X-Plane 11 plugins directory with XPlaneConnect folder added

3 – Verify XPlaneConnect is active in X-Plane

Now we’ll load up X-Plane and check the plug-ins to verify XPlaneConnect is enabled. Go to the top menu and select Plugins -> Plugin Admin. You should see X-Plane Connect checked in the enabled column:

Screenshot showing XPlaneConnect plug-in active in X-Plane 11
Screenshot showing XPlaneConnect plug-in active in X-Plane 10

4 – Download sample code from XPlaneConnect’s GitHub page

From the Python3 portion of the GitHub code, download xpc.py and monitorExample.py and stick them in your working directory (doesn’t matter where). For me, I just downloaded the entire git structure so the code is at C:\Users\Austin\source\repos\XPlaneConnect\Python3\src:

Screenshot showing xpc.py and monitorExample.py in my working directory

5 – Run sample code to verify data is making it from X-Plane to our code

With X-Plane running with a plane on a runway (or anywhere really), go ahead and run monitorExample.py! I will be using Visual Studio Code to program this XPlane Python autopilot stuff so that’s where I’ll run it from.

You will start seeing lines scroll by very fast with 6 pieces of information – latitude, longitude, elevation (in meters), and the control deflections for aileron, elevator, and rudder (normalized from -1 to 1, with 0 being centered). In the below screenshot, we see a lat/lon of 39.915, -105.128, with an elevation of 1719m. First one to tell me in the comments what runway that is wins internet points!

Screenshot showing Visual Studio Code running monitorExample.py in front of X-Plane 10 and the output scrolling by.

Conclusion

In this post, we have successfully downloaded the XPlaneConnect plug-in, and demonstrated that it can successfully interface with Python code in both X-Plane 10 and X-Plane 11.

Next up is to start controlling the plane with a basic wing leveler. As of writing this post, I have the following completely functional:

  • Pitch / roll hold at reasonable angles (-25 to 25)
  • Altitude set and hold
  • Heading set and hold
  • Airspeed set and hold
  • Navigate directly to a lat/lon point

See you at the next post! Next post – Coding a wing leveler autopilot in X-Plane with Python

Categories
NTP Raspberry Pi

How to update GPSd by building from source

Introduction

I was recently made aware of a bug in GPSd that will result in the time/date jumping backwards 1024 weeks, from October 16, 2021 to Sunday March 3, 2002 for versions 3.20, 3.21, and 3.22. GPSd version 3.22 is currently scheduled to ship with Debian Bullseye, which will be a problem. I use GPSd for my timekeeping interfaces between the source GPS and NTP/Chrony. GPSd version 3.17 is present in the current Raspberry Pi OS (Raspbian) images (based off Debian 9 – Stretch) as well.

Fortunately, it isn’t hard to update to your desired version!

Updating GPSd

The overview for updating GPSd is as follows:

  • Download the desired release with wget (look for >3.23)
  • Uncompress the archive
  • Use scons to build the new binaries
  • Use scons to install the new binaries

So with that out of the way, let’s get started. (The full install script is at the bottom if you just want to jump ahead to that).

You must first ensure you have the required packages to actually build GPSd from source:

sudo apt update
sudo apt install -y scons libncurses-dev python-dev pps-tools git-core asciidoctor python3-matplotlib build-essential manpages-dev pkg-config python3-distutils

Next, we will download the desired version of GPSd. In this case, we will be updating GPSd to version 3.23.1. A full list of the releases can be found here.

wget http://download.savannah.gnu.org/releases/gpsd/gpsd-3.23.1.tar.gz

Extract the files from the .tar.gz archive, and change to the created folder:

tar -xzf gpsd-3.23.1.tar.gz
cd gpsd-3.23.1/

Now we can build the binaries, which will take a few minutes to run:

sudo scons

# some sources say to do a config=force for scons, I found this wasn't necessary
# if you want to use this force argument, below is the required command
# sudo scons --config=force

Last up is to actually install the binaries:

sudo scons install

And with that, you should now have an updated GPSd running version 3.23.1! I rebooted for good measure with sudo reboot.

If you’re interested in a full script to do all this, check this out:

sudo apt update
sudo apt install -y scons libncurses-dev python-dev pps-tools git-core asciidoctor python3-matplotlib build-essential manpages-dev pkg-config python3-distutils
wget http://download.savannah.gnu.org/releases/gpsd/gpsd-3.23.1.tar.gz
tar -xzf gpsd-3.23.1.tar.gz
cd gpsd-3.23.1
sudo scons
sudo scons install
gpsd -V

Verifying you have the update for GPSd

gpsd -V
GPSd version 3.23.1 verified with the command ‘gpsd -V’

References/Sources

Categories
Linux NTP Raspberry Pi

Millisecond accurate Chrony NTP with a USB GPS for $12 USD

Introduction

Building off my last NTP post (Microsecond accurate NTP with a Raspberry Pi and PPS GPS), which required a $50-60 GPS device and a Raspberry Pi (also $40+), I have successfully tested something much cheaper, that is good enough, especially for initial PPS synchronization. Good enough, in this case, is defined as +/- 10 milliseconds, which can easily be achieved using a basic USB GPS device: GT-U7. Read on for instructions on how to set up the USB GPS as a Stratum 1 NTP time server.

YouTube Video Link

https://www.youtube.com/watch?v=DVtmDFpWkEs

Microsecond PPS time vs millisecond USB time

How accurate of time do you really need? The last post showed how to get all devices on a local area network (LAN) within 0.1 milliseconds of “real” time. Do you need you equipment to be that accurate to official atomic clock time (12:03:05.0001)? Didn’t think so. Do you care if every device is on the correct second compared to official/accurate time (12:03:05)? That’s a lot more reasonable. Using a u-blox USB GPS can get you to 0.01 seconds of official. The best part about this? The required USB GPS units are almost always less than $15 and you don’t need a Raspberry Pi.

Overview

This post will show how to add a u-blox USB GPS module to NTP as a driver or chrony (timekeeping daemon) as a reference clock (using GPSd shared memory for both) and verify the accuracy is within +/- 10 milliseconds.

Materials needed

  • USB u-blox GPS (VK-172 or GT-U7), GT-U7 preferred because it has a micro-USB plug to connect to your computer. It is important to note that both of these are u-blox modules, which has a binary data format as well as a high default baudrate (57600). These two properties allow for quick transmission of each GPS message from GPS to computer.
  • 15-30 minutes

Steps

1 – Update your host machine and install packages

This tutorial is for Linux. I use Ubuntu so we utilize Aptitude (apt) for package management:

sudo apt update
sudo apt upgrade
sudo rpi-update
sudo apt install gpsd gpsd-clients python-gps chrony

2 – Modify GPSd default startup settings

In /etc/default/gpsd, change the settings to the following:

# Start the gpsd daemon automatically at boot time
START_DAEMON="true"

# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"

# Devices gpsd should collect to at boot time.
# this could also be /dev/ttyUSB0, it is ACM0 on raspberry pi
DEVICES="/dev/ttyACM0"

# -n means start listening to GPS data without a specific listener
GPSD_OPTIONS="-n"

Reboot with sudo reboot.

3a – Chrony configuration (if using NTPd, go to 3b)

I took the default configuration, added my 10.98 servers, and more importantly, added a reference clock (refclock). Link to chrony documentation here. Arguments/parameters of this configuration file:

  • 10.98.1.198 is my microsecond accurate PPS NTP server
  • iburst means send a bunch of synchronization packets upon service start so accurate time can be determined much faster (usually a couple seconds)
  • maxpoll (and minpoll, which isn’t used in this config) is how many seconds to wait between polls, defined by 2^x where x is the number in the config. maxpoll 6 means don’t wait more than 2^6=64 seconds between polls
  • refclock is reference clock, and is the USB GPS source we are adding
    • ‘SHM 0’ means shared memory reference 0, which means it is checking with GPSd using shared memory to see what time the GPS is reporting
    • ‘refid NMEA’ means name this reference ‘NMEA’
    • ‘offset 0.000’ means don’t offset this clock source at all. We will change this later
    • ‘precision 1e-3’ means this reference is only accurate to 1e-3 (0.001) seconds, or 1 millisecond
    • ‘poll 3’ means poll this reference every 2^3 = 8 seconds
    • ‘noselect’ means don’t actually use this clock as a source. We will be measuring the delta to other known times to set the offset and make the source selectable.
[email protected]:~ $ sudo cat /etc/chrony/chrony.conf
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usuable directives.
#pool 2.debian.pool.ntp.org iburst
server 10.98.1.1 iburst maxpoll 6
server 10.98.1.198 iburst maxpoll 6
server 10.98.1.15 iburst

refclock SHM 0 refid NMEA offset 0.000 precision 1e-3 poll 3 noselect

Restart chrony with sudo systemctl restart chrony.

3b – NTP config

Similar to the chrony config, we need to add a reference clock (called a driver in NTP). For NTP, drivers are “servers” that start with an address of 127.127. The next two octets tell what kind of driver it is. The .28 driver is the shared memory driver, same theory as for chrony. For a full list of drivers, see the official NTP docs. To break down the server:

  • ‘server 127.127.28.0’ means use the .28 (SHM) driver
    • minpoll 4 maxpoll 4 means poll every 2^4=16 seconds
    • noselect means don’t use this for time. Similar to chrony, we will be measuring the offset to determine this value.
  • ‘fudge 127.127.28.0’ means we are going to change some properties of the listed driver
    • ‘time1 0.000’is the time offset calibration factor, in seconds
    • ‘stratum 2’ means list this source as a stratum 2 source (has to do with how close the source is to “true” time), listing it as 2 means other, higher stratum sources will be selected before this one will (assuming equal time quality)
    • ‘refid GPS’ means rename this source as ‘GPS’
server 127.127.28.0 minpoll 4 maxpoll 4 noselect
fudge 127.127.28.0 time1 0.000 stratum 2 refid GPS

Restart NTPd with sudo systemctl restart ntp.

4 – check time offset via gpsmon

Running gpsmon shows us general information about the GPS, including time offset. The output looks like the below screenshot. Of importance is the satellite count (on right, more is better, >5 is good enough for time), HDOP (horizontal dilution of precision) is a measure of how well the satellites can determine your position (lower is better, <2 works for basically all navigation purposes), and TOFF (time offset).

gpsmon showing time offset for a USB GPS

In this screenshot the TOFF is 0.081862027, which is 81.8 milliseconds off the host computer’s time. Watch this for a bit – it should hover pretty close to a certain value +/- 10ms. In my case, I’ve noticed that if there are 10 or less satellites locked on, it is around 77ms. If there are 11 or more, it is around 91ms (presumably due to more satellite information that needs to be transmitted).

5 – record statistics for a data-driven offset

If you are looking for a better offset value to put in the configuration file, we can turn on logging from either chrony or NTPd to record source information.

For chrony:

Edit /etc/chrony/chrony.conf and uncomment the line for which kinds of logging to turn on:

# Uncomment the following line to turn logging on.
log tracking measurements statistics

Then restart chrony (sudo systemctl restart chrony) and logs will start writing to /var/log/chrony (this location is defined a couple lines below the log line in chrony.conf):

[email protected]:~ $ ls /var/log/chrony
measurements.log  statistics.log  tracking.log

For NTPd (be sure to restart it after making any configuration changes):

[email protected] ~ % cat /etc/ntp.conf
# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

Wait a few minutes for some data to record (chrony synchronizes pretty quick compared to NTPd) and check the statistics file, filtered to our NMEA refid:

cat /var/log/chrony/statistics.log | grep NMEA

This spits out the lines that have NMEA present (the ones of interest for our USB GPS). To include the headers to show what each column is we can run

# chrony
cat /var/log/chrony/statistics.log | head -2; cat /var/log/chrony/statistics.log | grep NMEA

# ntp, there is no header info so we can omit that part of the command
cat /var/log/peerstats | grep 127.127.28.0
Screenshot showing chrony statistics for our NMEA USB GPS refclock

NTP stats don’t include header information. The column of interest is the one after the 9014 column. The columns are day, seconds past midnight, source, something, estimated offset, something, something, something. We can see the offset for this VK-172 USB GPS is somewhere around 76-77 milliseconds (0.076-0.077 seconds), which we can put in place of the 0.000 for the .28 driver for NTP and remove noselect.

[email protected] ~ % cat /var/log/ntpstats/peerstats | grep 127.127.28.0
59487 49648.536 127.127.28.0 9014 -0.078425007 0.000000000 7.938064614 0.000000060
59487 49664.536 127.127.28.0 9014 -0.079488544 0.000000000 3.938033388 0.001063537
59487 49680.536 127.127.28.0 9014 -0.079514781 0.000000000 1.938035682 0.000770810
59487 49696.536 127.127.28.0 9014 -0.079772284 0.000000000 0.938092429 0.000808697
59487 49712.536 127.127.28.0 9014 -0.079711708 0.000000000 0.438080791 0.000661032
59487 49728.536 127.127.28.0 9014 -0.075098563 0.000000000 0.188028843 0.004311958

So now we have some data showing the statistics of our NMEA USB GPS NTP source. We can copy and paste this into Excel, run data to columns, and graph the result and/or get the average to set the offset.

screenshot showing chrony/NTP statistics to determine offset

This graph is certainly suspicious (sine wave pattern and such) and if I wasn’t writing this blog post, I’d let data collect overnight to determine an offset. Since time is always of the essence, I will just take the average of the ‘est offset’ column (E), which is 7.64E-2, or 0.0763 seconds. Let’s pop this into the chrony.conf file and remove noselect:

refclock SHM 0 refid NMEA offset 0.0763 precision 1e-3 poll 3

For NTP:

Restart chrony again for the config file changes to take effect – sudo systemctl restart chrony.

6 – watch ‘chrony sources’ or ‘ntpq -pn’ to see if the USB GPS gets selected as the main time source

If you aren’t aware, Ubuntu/Debian/most Linux includes a utility to rerun a command every x seconds called watch. We can use this to watch chrony to see how it is interpreting each time source every 1 second:

# for chrony
watch -n 1 chronyc sources
watching chrony sources

In the above screenshot, we can see that chrony actually has the NMEA source selected as the primary source (denoted with the *). It has the Raspberry Pi PPS NTP GPS ready to takeover as the new primary (denoted with the +). All of the sources match quite closely (from +4749us to – 505us is around 5.2 milliseconds). The source “offset” is in the square brackets ([ and ]).

# for ntp
watch -n 1 ntpq -pn
NTP showing (via ntpq -pn) that the GPS source is 0.738 milliseconds off of the host clock. The ‘-‘ in front of the remote means this will not be selected as a valid time (presumably due to the high jitter compared to the other sources, probably also due to manually setting it to stratum 2).

7- is +/- five millseconds good enough?

For 99% of use cases, yes. You can stop here and your home network will be plenty accurate. If you want additional accuracy, you are in luck. This GPS module also outputs a PPS (pulse per second) signal! We can use this to get within 0.05 millseconds (0.00005 seconds) from official/atomic clock time.

Conclusion

In this post, we got a u-blox USB GPS set up and added it as a reference clock (refclock) to chrony and demonstrated it is clearly within 10 millisecond of official GPS time.

You could write a script to do all this for you! I should probably try this myself…

In the next post, we can add PPS signals from the GPS module to increase our time accuracy by 1000x (into the microsecond range).

A note on why having faster message transmission is better for timing

My current PPS NTP server uses chrony with NMEA messages transmitted over serial and the PPS signal fed into a GPIO pin. GPSd as a rule does minimum configuration of GPS devices. It typically defaults to 9600 baud for serial devices. A typical GPS message looks like this:

$GPGGA, 161229.487, 3723.2475, N, 12158.3416, W, 1, 07, 1.0, 9.0, M, , , , 0000*18

That message is 83 bytes long. At 9600 baud (9600 bits per second), that message takes 69.1 milliseconds to transmit. Each character/byte takes 0.833 milliseconds to transmit. That means that as the message length varies, the jitter will increase. GPS messages do vary in length, sometimes significantly, depending on what is being sent (i.e. the satellite information, $GPGSV sentences, is only transmitted every 5-10 seconds).

I opened gpsmon to get a sample of sentences – I did not notice this until now but it shows how many bytes each sentence is at the front of the sentence:

(35) $GPZDA,144410.000,30,09,2021,,*59
------------------- PPS offset: -0.000001297 ------
(83) $GPGGA,144411.000,3953.xxxx,N,10504.xxxx,W,2,6,1.19,1637.8,M,-20.9,M,0000,0000*5A
(54) $GPGSA,A,3,26,25,29,18,05,02,,,,,,,1.46,1.19,0.84*02
(71) $GPRMC,144411.000,A,3953.xxxx,N,10504.xxxx,W,2.80,39.98,300921,,,D*44
(35) $GPZDA,144411.000,30,09,2021,,*58
------------------- PPS offset: -0.000000883 ------
(83) $GPGGA,144412.000,3953.xxxx,N,10504.xxxx,W,2,7,1.11,1637.7,M,-20.9,M,0000,0000*52
(56) $GPGSA,A,3,20,26,25,29,18,05,02,,,,,,1.39,1.11,0.84*00
(70) $GPGSV,3,1,12,29,81,325,27,05,68,056,21,20,35,050,17,18,34,283,24*76
(66) $GPGSV,3,2,12,25,27,210,14,15,27,153,,13,25,117,,02,23,080,19*78
(59) $GPGSV,3,3,12,26,17,311,22,23,16,222,,12,11,184,,47,,,*42
------------------- PPS offset: -0.000000833 ------
(71) $GPRMC,144412.000,A,3953.xxxx,N,10504.xxxx,W,2.57,38.19,300921,,,D*48
(35) $GPZDA,144412.000,30,09,2021,,*5B
(83) $GPGGA,144413.000,3953.xxxx,N,10504.xxxx,W,2,7,1.11,1637.6,M,-20.9,M,0000,0000*52
(56) $GPGSA,A,3,20,26,25,29,18,05,02,,,,,,1.39,1.11,0.84*00
(71) $GPRMC,144413.000,A,3953.xxxx,N,10504.xxxx,W,2.60,36.39,300921,,,D*41
(35) $GPZDA,144413.000,30,09,2021,,*5A

These sentences range from 83 bytes to 35 bytes, a variation of (83 bytes -35 bytes)*0.833 milliseconds per byte = 39.984 milliseconds.

Compare to the u-blox binary UBX messages which seem to always be 60 bytes and transmitted at 57600 baud, which is 8.33 milliseconds to transmit the entire message.

UBX protocol messages (blanked out lines). I have no idea what part of the message is location, hopefully I got the right part blanked out.

The variance (jitter) is thus much lower and can be much more accurate as a NTP source. GPSd has no problem leaving u-blox modules at 57600 baud. This is why the USB GPS modules perform much more accurate for timekeeping than NMEA-based devices when using GPSd.

For basically every GPS module/chipset, it is possible to send it commands to enable/disable sentences (as well as increase the serial baud rate). In an ideal world for timekeeping, GPSd would disable every sentence except for time ($GPZDA), and bump up the baud rate to the highest supported level (115200, 230400, etc.). Unfortunately for us, GPSd’s default behavior is to just work with every GPS, which essentially means no configuring the GPS device.

Categories
homelab Kubernetes Linux proxmox Terraform

Deploying Kubernetes VMs in Proxmox with Terraform

Background

The last post covered how to deploy virtual machines in Proxmox with Terraform. This post shows the template for deploying 4 Kubernetes virtual machines in Proxmox using Terraform.

Youtube Video Link

Coming soon

Kubernetes Proxmox Terraform Template

Without further ado, below is the template I used to create my virtual machines. The main LAN network is 10.98.1.0/24, and the Kube internal network (on its own bridge) is 10.17.0.0/24.

This template creates a Kube server, two agents, and a storage server.

terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.7.4"
    }
  }
}

provider "proxmox" {
  pm_api_url = "https://prox-1u.home.fluffnet.net:8006/api2/json"
  pm_api_token_id = [secret]
  pm_api_token_secret = [secret]
  pm_tls_insecure = true
}

resource "proxmox_vm_qemu" "kube-server" {
  count = 1
  name = "kube-server-0${count.index + 1}"
  target_node = "prox-1u"

  clone = "ubuntu-2004-cloudinit-template"

  agent = 1
  os_type = "cloud-init"
  cores = 2
  sockets = 1
  cpu = "host"
  memory = 4096
  scsihw = "virtio-scsi-pci"
  bootdisk = "scsi0"

  disk {
    slot = 0
    size = "10G"
    type = "scsi"
    storage = "local-zfs"
    #storage_type = "zfspool"
    iothread = 1
  }

  network {
    model = "virtio"
    bridge = "vmbr0"
  }
  
  network {
    model = "virtio"
    bridge = "vmbr17"
  }

  lifecycle {
    ignore_changes = [
      network,
    ]
  }

  ipconfig0 = "ip=10.98.1.4${count.index + 1}/24,gw=10.98.1.1"
  ipconfig1 = "ip=10.17.0.4${count.index + 1}/24"
  sshkeys = <<EOF
  ${var.ssh_key}
  EOF
}

resource "proxmox_vm_qemu" "kube-agent" {
  count = 2
  name = "kube-agent-0${count.index + 1}"
  target_node = "prox-1u"

  clone = "ubuntu-2004-cloudinit-template"

  agent = 1
  os_type = "cloud-init"
  cores = 2
  sockets = 1
  cpu = "host"
  memory = 4096
  scsihw = "virtio-scsi-pci"
  bootdisk = "scsi0"

  disk {
    slot = 0
    size = "10G"
    type = "scsi"
    storage = "local-zfs"
    #storage_type = "zfspool"
    iothread = 1
  }

  network {
    model = "virtio"
    bridge = "vmbr0"
  }
  
  network {
    model = "virtio"
    bridge = "vmbr17"
  }

  lifecycle {
    ignore_changes = [
      network,
    ]
  }

  ipconfig0 = "ip=10.98.1.5${count.index + 1}/24,gw=10.98.1.1"
  ipconfig1 = "ip=10.17.0.5${count.index + 1}/24"
  sshkeys = <<EOF
  ${var.ssh_key}
  EOF
}

resource "proxmox_vm_qemu" "kube-storage" {
  count = 1
  name = "kube-storage-0${count.index + 1}"
  target_node = "prox-1u"

  clone = "ubuntu-2004-cloudinit-template"

  agent = 1
  os_type = "cloud-init"
  cores = 2
  sockets = 1
  cpu = "host"
  memory = 4096
  scsihw = "virtio-scsi-pci"
  bootdisk = "scsi0"

  disk {
    slot = 0
    size = "20G"
    type = "scsi"
    storage = "local-zfs"
    #storage_type = "zfspool"
    iothread = 1
  }

  network {
    model = "virtio"
    bridge = "vmbr0"
  }
  
  network {
    model = "virtio"
    bridge = "vmbr17"
  }

  lifecycle {
    ignore_changes = [
      network,
    ]
  }

  ipconfig0 = "ip=10.98.1.6${count.index + 1}/24,gw=10.98.1.1"
  ipconfig1 = "ip=10.17.0.6${count.index + 1}/24"
  sshkeys = <<EOF
  ${var.ssh_key}
  EOF
}

After running Terraform plan and apply, you should have 4 new VMs in your Proxmox cluster:

Proxmox showing 4 virtual machines ready for Kubernetes

Conclusion

You now have 4 VMs ready for Kubernetes installation. The next post will show how to install Kubernetes with Ansible.