Continuing from the first post, where we hooked up X-Plane to our Python code, we will build a wing leveler today. The first post was just about making sure we could get data into and out of X-Plane. Today will add a few features to our XPlane autopilot written in Python.
A control loop timer (we will be targeting a loop frequency of 10 Hz, or 10 updates per second)
Additional data feeds from X-Plane
Two PID controllers, one for roll, one for pitch
Some debugging output to keep track of what the PIDs are doing
Feeding the PID controllers within the control loop
Controlling the aircraft with the new PID output
Monitoring the control loops via debug prints
1 – Control loop timer/limiter
Since we will be targeting a 10 Hz update rate, we need to develop a method to ensure the loop does not run more frequent than once every 100 milliseconds. We do not want the loop running uninhibited, because that will result in variable loop execution times and we like to keep those things constant. It could potentially execute thousands of times per second, which is entirely unnecessary. Most control loop algorithms run in the 10-100 Hz range (10-100 times per second). For reference, my RC plane running ArduPlane on Pixhawk uses 50 Hz as a standard update frequency.
To accomplish this task, we need to set up some timing variables.
First of all, add an import statment for datetime and timedelta:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
from datetime import datetime, timedelta
from datetime import datetime, timedelta
from datetime import datetime, timedelta
Next, define the timing variables:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
update_interval = 0.100# this value is in seconds. 1/0.100 = 10 which is the update interval in Hz
# start is set to the time the line of code is executed, which is essential when the program started
start = datetime.now()
# last_update needs to be set to start for the first execution of the loop to successfully run.
# alternatively, we could've set start to something in the past.
last_update = start
# last update needs to be defined as a global within the monitor() function:
defmonitor():
global last_update
update_interval = 0.100 # this value is in seconds. 1/0.100 = 10 which is the update interval in Hz
# start is set to the time the line of code is executed, which is essential when the program started
start = datetime.now()
# last_update needs to be set to start for the first execution of the loop to successfully run.
# alternatively, we could've set start to something in the past.
last_update = start
# last update needs to be defined as a global within the monitor() function:
def monitor():
global last_update
update_interval = 0.100 # this value is in seconds. 1/0.100 = 10 which is the update interval in Hz
# start is set to the time the line of code is executed, which is essential when the program started
start = datetime.now()
# last_update needs to be set to start for the first execution of the loop to successfully run.
# alternatively, we could've set start to something in the past.
last_update = start
# last update needs to be defined as a global within the monitor() function:
def monitor():
global last_update
That handles the variables. Now we need to limit the loop execution. To do so requires wrapping all of the loop code into a new if statement that evaluates the time and only executes if the current time is 100 milliseconds greater than the last loop execution:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
# loop is the "while True:" statement
whileTrue:
# this if statement is evaluated with every loop execution
# when the if statement evaluates to true, the first thing we'll do is set the last update to the current time so the next iteration fires at the correct time
last_update = datetime.now()
# rest of the loop code goes here
# loop is the "while True:" statement
while True:
# this if statement is evaluated with every loop execution
if (datetime.now() > last_update + timedelta(milliseconds = update_interval * 1000)):
# when the if statement evaluates to true, the first thing we'll do is set the last update to the current time so the next iteration fires at the correct time
last_update = datetime.now()
# rest of the loop code goes here
# loop is the "while True:" statement
while True:
# this if statement is evaluated with every loop execution
if (datetime.now() > last_update + timedelta(milliseconds = update_interval * 1000)):
# when the if statement evaluates to true, the first thing we'll do is set the last update to the current time so the next iteration fires at the correct time
last_update = datetime.now()
# rest of the loop code goes here
2 – Obtaining current roll/pitch values from X-Plane
This task is pretty straightforward. In the first post, the monitorExample.py code included obtaining the position with posi = client.getPOSI(). There are 7 elements returned with that method, and two of them are roll and pitch.
Put the below after the .getPOSI() call.
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
current_roll = posi[4]
current_pitch = posi[3]
current_roll = posi[4]
current_pitch = posi[3]
current_roll = posi[4]
current_pitch = posi[3]
3 – Initializing the PID controllers
First you need to get the PID control file from the Ivmech (Ivmech Mechatronics Ltd.) GitHub page. Direct link PID.py file here. Put the file in the same working directory as everything else then import it with import PID at the top with the rest of the imports.
Then we can initialize the control instances. PID controllers are magic. But they do need to be set with smart values for the initial run. The default values with the PID library are P=0.2, I=0, D=0. This essentially means make the output be 20% of the error between the setpoint and the current value. For example, if the aircraft has a roll of 10 degrees to the left (-10), and the P=0.2, the output from the PID controller will be -2.
When setting PID values, it is almost always a good idea to start small and work your way up. If your gains are too high, you could get giant oscillations and other undesirable behaviors. In the YouTube video, I talk about the various types of PID controllers. They will basically always have a P (proportional) set. Many will also have an I (integral) value set, but not many use the D term (derivative).
Going with the “less is more” truism with PID controllers, I started with a P value of 0.1, and an I value of 0.01 (P/10). The I term (integral) is meant to take care of accumulated errors (which are usually long term errors). An example is your car’s cruise control going up a hill. If your cruise control is set to 65 mph, it will hold 65 no problem on flat roads. If you start going up a hill, the controller will slowly apply more throttle. With an I of 0, your car would never get to 65. It would probably stay around 62 or so. With the integrator going, the error will accumulate and boost the output (throttle) to get you back up to your desired 65. In the linked YouTube video, I show what happens when the I value is set to 0 and why it is necessary to correct long-term errors.
These values (P=0.1, I=0.01, D=0) turned out to work perfectly.
Place the following before the monitor function:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
# defining the initial PID values
P = 0.1# PID library default = 0.2
I = P/10# default = 0
D = 0# default = 0
# initializing both PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)
# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2
# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch
# defining the initial PID values
P = 0.1 # PID library default = 0.2
I = P/10 # default = 0
D = 0 # default = 0
# initializing both PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)
# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2
# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch
# defining the initial PID values
P = 0.1 # PID library default = 0.2
I = P/10 # default = 0
D = 0 # default = 0
# initializing both PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)
# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2
# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch
4 – Updating the PID controllers within the control loop
With the PIDs created, they will need to be updated with the new, current pitch and roll values with every loop execution.
Place the following after current_roll and current_pitch are assigned:
5 – Controlling the aircraft with the new PID output
Updating the PID controller instances will generate new outputs. We can use those outputs to set the control surfaces. You can place these two lines directly below the update lines:
So we have new control surface values – now we need to actually move the control surfaces. This is accomplished by sending an array of 4 floats with .sendCTRL():
The last bit to tie a bunch of this together is printing out values with every loop execution to ensure things are heading the right direction. We will turn these into graphs in the next post or two.
We will be concatenating strings because it’s easy and we aren’t working with enough strings for it to be a performance problem.
In newer Python versions (3.6+), placing ‘f’ before the quotes in a print statement (f””) means the string is interpolated. This means you can basically put code in the print statement, which makes creating print statements much easier and cleaner.
The first line will print out the current roll and pitch value (below). We are using current_roll and current_pitch interpolated. The colon, then blank space with 0.3f is a string formatter. It rounds the value to 3 decimal places and leaves space for a negative. It results in things being lined up quite nicely.
import sys
import xpc
import PID
from datetime import datetime, timedelta
update_interval = 0.100 # seconds
start = datetime.now()
last_update = start
# defining the initial PID values
P = 0.1 # PID library default = 0.2
I = P/10 # default = 0
D = 0 # default = 0
# initializing both PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)
# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2
# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch
def monitor():
global last_update
with xpc.XPlaneConnect() as client:
while True:
if (datetime.now() > last_update + timedelta(milliseconds = update_interval * 1000)):
last_update = datetime.now()
print(f"loop start - {datetime.now()}")
posi = client.getPOSI();
ctrl = client.getCTRL();
current_roll = posi[4]
current_pitch = posi[3]
roll_PID.update(current_roll)
pitch_PID.update(current_pitch)
new_ail_ctrl = roll_PID.output
new_ele_ctrl = pitch_PID.output
ctrl = [new_ele_ctrl, new_ail_ctrl, 0.0, -998] # ele, ail, rud, thr. -998 means don't change
client.sendCTRL(ctrl)
output = f"current values -- roll: {current_roll: 0.3f}, pitch: {current_pitch: 0.3f}"
output = output + "\n" + f"PID outputs -- roll: {roll_PID.output: 0.3f}, pitch: {pitch_PID.output: 0.3f}"
output = output + "\n"
print(output)
if __name__ == "__main__":
monitor()
import sys
import xpc
import PID
from datetime import datetime, timedelta
update_interval = 0.100 # seconds
start = datetime.now()
last_update = start
# defining the initial PID values
P = 0.1 # PID library default = 0.2
I = P/10 # default = 0
D = 0 # default = 0
# initializing both PID controllers
roll_PID = PID.PID(P, I, D)
pitch_PID = PID.PID(P, I, D)
# setting the desired values
# roll = 0 means wings level
# pitch = 2 means slightly nose up, which is required for level flight
desired_roll = 0
desired_pitch = 2
# setting the PID set points with our desired values
roll_PID.SetPoint = desired_roll
pitch_PID.SetPoint = desired_pitch
def monitor():
global last_update
with xpc.XPlaneConnect() as client:
while True:
if (datetime.now() > last_update + timedelta(milliseconds = update_interval * 1000)):
last_update = datetime.now()
print(f"loop start - {datetime.now()}")
posi = client.getPOSI();
ctrl = client.getCTRL();
current_roll = posi[4]
current_pitch = posi[3]
roll_PID.update(current_roll)
pitch_PID.update(current_pitch)
new_ail_ctrl = roll_PID.output
new_ele_ctrl = pitch_PID.output
ctrl = [new_ele_ctrl, new_ail_ctrl, 0.0, -998] # ele, ail, rud, thr. -998 means don't change
client.sendCTRL(ctrl)
output = f"current values -- roll: {current_roll: 0.3f}, pitch: {current_pitch: 0.3f}"
output = output + "\n" + f"PID outputs -- roll: {roll_PID.output: 0.3f}, pitch: {pitch_PID.output: 0.3f}"
output = output + "\n"
print(output)
if __name__ == "__main__":
monitor()
Using the autopilot / Conclusion
To use the autopilot, fire up XPlane, hop in a small-ish plane (gross weight less than 10k lb), take off, climb 1000′, then execute the code. Your plane should level off within a second or two in both axis.
Here is a screenshot of the output after running for a few seconds:
Screenshot showing current pitch and roll values along with their respective PID outputs
The linked YouTube video shows the aircraft snapping to the desired pitch and roll angles very quickly from a diving turn. The plane is righted to within 0.5 degrees of the setpoints within 2 seconds.
A note about directly setting the control surfaces
If you are familiar with PIDs and/or other parts of what I’ve discussed here, you’ll realize that we could be setting large values for the control surfaces (i.e. greater than 1 or less than -1). We will address that next post with a normalization function. It will quickly become a problem when a pitch of 100+ is commanded for altitude hold. I have found that XPlane will allow throttle values of more than 100% (I’ve seen as high as ~430%) if sent huge throttle values.
Screenshot showing our Python code running in front of X-Plane as we start development of a Python Autopilot for X-Plane
Introduction
Today’s post will take us in a slightly different direction than the last few. Today’s post will be about hooking up some Python code to the X-Plane flight simulator to enable development of an autopilot using PID (proportional-integral-derivative) controllers. I’ve been a fan of flight simulators for quite some time (I distinctly remember getting Microsoft Flight Simulator 98 for my birthday when I was like 8 or 9) but have only recently started working with interfacing them to code. X-Plane is a well-known flight simulator developed by another Austin – Austin Meyer. It is regarded as having one of the best flight models and has tons of options for getting data into/out of the simulator. More than one FAA-certified simulator setups are running X-Plane as the primary driver software.
I got started thinking about writing some code for X-Plane while playing another game, Factorio. I drive a little plane or car in the game to get around my base and I just added a plug-in that “snaps” the vehicle to a heading, which makes it easier to go in straight lines. I thought – “hmm how hard could this be to duplicate in a flight sim?”. So here we are.
This post will get X-Plane hooked up to Python. The real programming will start with the next post.
Download and install X-Plane (I used X-Plane 10 because it uses less resources than X-Plane 11 and we don’t need the graphics/scenery to look super pretty to do coding. It also loads faster.)
Download and install NASA’s XPlaneConnect X-Plane plug-in to X-Plane
Verify the XPlaneConnect plug-in is active in X-Plane
Download sample code from XPlaneConnect’s GitHub page
Run the sample script to verify data is being transmitted from X-Plane via UDP to the XPlaneConnect code
1 – Download and install X-Plane 10 or X-Plane 11
I’ll leave this one up to you. X-Plane 10 is hard to find these days I just discovered. X-Plane 11 is available on Steam for $59.99 as of writing. I just tested and the plug-in/code works fine on X-Plane 11 (but the flight models are definitely different and will need different PID values). My screenshots might jump between the two versions but the content/message will be the same.
2 – Download and install NASA’s XPlaneConnect plug-in
NASA (yes, that NASA, the National Aeronautics and Space Administration) has wrote a bunch of code to interface with X-Plane. They have adapters for C, C++, Java, Matlab, and Python. They work with X-Plane 9, 10, and 11.
Open the .zip and place the contents in the [X-Plane directory]/Resources/plugins folder. There are few other folders already present in this directory. Mine looked like this after adding the XPlaneConnect folder:
Screenshot of X-Plane 10 plugins directory with XPlaneConnect folder addedScreenshot of X-Plane 11 plugins directory with XPlaneConnect folder added
3 – Verify XPlaneConnect is active in X-Plane
Now we’ll load up X-Plane and check the plug-ins to verify XPlaneConnect is enabled. Go to the top menu and select Plugins -> Plugin Admin. You should see X-Plane Connect checked in the enabled column:
Screenshot showing XPlaneConnect plug-in active in X-Plane 11Screenshot showing XPlaneConnect plug-in active in X-Plane 10
4 – Download sample code from XPlaneConnect’s GitHub page
From the Python3 portion of the GitHub code, download xpc.py and monitorExample.py and stick them in your working directory (doesn’t matter where). For me, I just downloaded the entire git structure so the code is at C:\Users\Austin\source\repos\XPlaneConnect\Python3\src:
Screenshot showing xpc.py and monitorExample.py in my working directory
5 – Run sample code to verify data is making it from X-Plane to our code
With X-Plane running with a plane on a runway (or anywhere really), go ahead and run monitorExample.py! I will be using Visual Studio Code to program this XPlane Python autopilot stuff so that’s where I’ll run it from.
You will start seeing lines scroll by very fast with 6 pieces of information – latitude, longitude, elevation (in meters), and the control deflections for aileron, elevator, and rudder (normalized from -1 to 1, with 0 being centered). In the below screenshot, we see a lat/lon of 39.915, -105.128, with an elevation of 1719m. First one to tell me in the comments what runway that is wins internet points!
Screenshot showing Visual Studio Code running monitorExample.py in front of X-Plane 10 and the output scrolling by.
Conclusion
In this post, we have successfully downloaded the XPlaneConnect plug-in, and demonstrated that it can successfully interface with Python code in both X-Plane 10 and X-Plane 11.
Next up is to start controlling the plane with a basic wing leveler. As of writing this post, I have the following completely functional:
Pitch / roll hold at reasonable angles (-25 to 25)
I was recently made aware of a bug in GPSd that will result in the time/date jumping backwards 1024 weeks, from October 16, 2021 to Sunday March 3, 2002 for versions 3.20, 3.21, and 3.22. GPSd version 3.22 is currently scheduled to ship with Debian Bullseye, which will be a problem. I use GPSd for my timekeeping interfaces between the source GPS and NTP/Chrony. GPSd version 3.17 is present in the current Raspberry Pi OS (Raspbian) images (based off Debian 9 – Stretch) as well.
Fortunately, it isn’t hard to update to your desired version!
Updating GPSd
The overview for updating GPSd is as follows:
Download the desired release with wget (look for >3.23)
Uncompress the archive
Use scons to build the new binaries
Use scons to install the new binaries
So with that out of the way, let’s get started. (The full install script is at the bottom if you just want to jump ahead to that).
You must first ensure you have the required packages to actually build GPSd from source:
Next, we will download the desired version of GPSd. In this case, we will be updating GPSd to version 3.23.1. A full list of the releases can be found here.
Extract the files from the .tar.gz archive, and change to the created folder:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
tar -xzf gpsd-3.23.1.tar.gz
cd gpsd-3.23.1/
tar -xzf gpsd-3.23.1.tar.gz
cd gpsd-3.23.1/
tar -xzf gpsd-3.23.1.tar.gz
cd gpsd-3.23.1/
Now we can build the binaries, which will take a few minutes to run:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
sudo scons
# some sources say to do a config=force for scons, I found this wasn't necessary
# if you want to use this force argument, below is the required command
# sudo scons --config=force
sudo scons
# some sources say to do a config=force for scons, I found this wasn't necessary
# if you want to use this force argument, below is the required command
# sudo scons --config=force
sudo scons
# some sources say to do a config=force for scons, I found this wasn't necessary
# if you want to use this force argument, below is the required command
# sudo scons --config=force
Last up is to actually install the binaries:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
sudo scons install
sudo scons install
sudo scons install
And with that, you should now have an updated GPSd running version 3.23.1! I rebooted for good measure with sudo reboot.
If you’re interested in a full script to do all this, check this out:
Building off my last NTP post (Microsecond accurate NTP with a Raspberry Pi and PPS GPS), which required a $50-60 GPS device and a Raspberry Pi (also $40+), I have successfully tested something much cheaper, that is good enough, especially for initial PPS synchronization. Good enough, in this case, is defined as +/- 10 milliseconds, which can easily be achieved using a basic USB GPS device: GT-U7. Read on for instructions on how to set up the USB GPS as a Stratum 1 NTP time server.
How accurate of time do you really need? The last post showed how to get all devices on a local area network (LAN) within 0.1 milliseconds of “real” time. Do you need you equipment to be that accurate to official atomic clock time (12:03:05.0001)? Didn’t think so. Do you care if every device is on the correct second compared to official/accurate time (12:03:05)? That’s a lot more reasonable. Using a u-blox USB GPS can get you to 0.01 seconds of official. The best part about this? The required USB GPS units are almost always less than $15 and you don’t need a Raspberry Pi.
Overview
This post will show how to add a u-blox USB GPS module to NTP as a driver or chrony (timekeeping daemon) as a reference clock (using GPSd shared memory for both) and verify the accuracy is within +/- 10 milliseconds.
Materials needed
USB u-blox GPS (VK-172 or GT-U7), GT-U7 preferred because it has a micro-USB plug to connect to your computer. It is important to note that both of these are u-blox modules, which has a binary data format as well as a high default baudrate (57600). These two properties allow for quick transmission of each GPS message from GPS to computer.
15-30 minutes
Steps
1 – Update your host machine and install packages
This tutorial is for Linux. I use Ubuntu so we utilize Aptitude (apt) for package management:
In /etc/default/gpsd, change the settings to the following:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# this could also be /dev/ttyUSB0, it is ACM0 on raspberry pi
DEVICES="/dev/ttyACM0"
# -n means start listening to GPS data without a specific listener
GPSD_OPTIONS="-n"
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# this could also be /dev/ttyUSB0, it is ACM0 on raspberry pi
DEVICES="/dev/ttyACM0"
# -n means start listening to GPS data without a specific listener
GPSD_OPTIONS="-n"
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# this could also be /dev/ttyUSB0, it is ACM0 on raspberry pi
DEVICES="/dev/ttyACM0"
# -n means start listening to GPS data without a specific listener
GPSD_OPTIONS="-n"
Reboot with sudo reboot.
3a – Chrony configuration (if using NTPd, go to 3b)
I took the default configuration, added my 10.98 servers, and more importantly, added a reference clock (refclock). Link to chrony documentation here. Arguments/parameters of this configuration file:
iburst means send a bunch of synchronization packets upon service start so accurate time can be determined much faster (usually a couple seconds)
maxpoll (and minpoll, which isn’t used in this config) is how many seconds to wait between polls, defined by 2^x where x is the number in the config. maxpoll 6 means don’t wait more than 2^6=64 seconds between polls
refclock is reference clock, and is the USB GPS source we are adding
‘SHM 0’ means shared memory reference 0, which means it is checking with GPSd using shared memory to see what time the GPS is reporting
‘refid NMEA’ means name this reference ‘NMEA’
‘offset 0.000’ means don’t offset this clock source at all. We will change this later
‘precision 1e-3’ means this reference is only accurate to 1e-3 (0.001) seconds, or 1 millisecond
‘poll 3’ means poll this reference every 2^3 = 8 seconds
‘noselect’ means don’t actually use this clock as a source. We will be measuring the delta to other known times to set the offset and make the source selectable.
pi@raspberrypi:~ $ sudo cat /etc/chrony/chrony.conf
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usuable directives.
#pool 2.debian.pool.ntp.org iburst
server 10.98.1.1 iburst maxpoll 6
server 10.98.1.198 iburst maxpoll 6
server 10.98.1.15 iburst
refclock SHM 0 refid NMEA offset 0.000 precision 1e-3 poll 3 noselect
pi@raspberrypi:~ $ sudo cat /etc/chrony/chrony.conf
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usuable directives.
#pool 2.debian.pool.ntp.org iburst
server 10.98.1.1 iburst maxpoll 6
server 10.98.1.198 iburst maxpoll 6
server 10.98.1.15 iburst
refclock SHM 0 refid NMEA offset 0.000 precision 1e-3 poll 3 noselect
Restart chrony with sudo systemctl restart chrony.
3b – NTP config
Similar to the chrony config, we need to add a reference clock (called a driver in NTP). For NTP, drivers are “servers” that start with an address of 127.127. The next two octets tell what kind of driver it is. The .28 driver is the shared memory driver, same theory as for chrony. For a full list of drivers, see the official NTP docs. To break down the server:
‘server 127.127.28.0’ means use the .28 (SHM) driver
minpoll 4 maxpoll 4 means poll every 2^4=16 seconds
noselect means don’t use this for time. Similar to chrony, we will be measuring the offset to determine this value.
‘fudge 127.127.28.0’ means we are going to change some properties of the listed driver
‘time1 0.000’is the time offset calibration factor, in seconds
‘stratum 2’ means list this source as a stratum 2 source (has to do with how close the source is to “true” time), listing it as 2 means other, higher stratum sources will be selected before this one will (assuming equal time quality)
Running gpsmon shows us general information about the GPS, including time offset. The output looks like the below screenshot. Of importance is the satellite count (on right, more is better, >5 is good enough for time), HDOP (horizontal dilution of precision) is a measure of how well the satellites can determine your position (lower is better, <2 works for basically all navigation purposes), and TOFF (time offset).
gpsmon showing time offset for a USB GPS
In this screenshot the TOFF is 0.081862027, which is 81.8 milliseconds off the host computer’s time. Watch this for a bit – it should hover pretty close to a certain value +/- 10ms. In my case, I’ve noticed that if there are 10 or less satellites locked on, it is around 77ms. If there are 11 or more, it is around 91ms (presumably due to more satellite information that needs to be transmitted).
5 – record statistics for a data-driven offset
If you are looking for a better offset value to put in the configuration file, we can turn on logging from either chrony or NTPd to record source information.
For chrony:
Edit /etc/chrony/chrony.conf and uncomment the line for which kinds of logging to turn on:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
# Uncomment the following line to turn logging on.
log tracking measurements statistics
# Uncomment the following line to turn logging on.
log tracking measurements statistics
# Uncomment the following line to turn logging on.
log tracking measurements statistics
Then restart chrony (sudo systemctl restart chrony) and logs will start writing to /var/log/chrony (this location is defined a couple lines below the log line in chrony.conf):
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
pi@raspberrypi:~ $ ls /var/log/chrony
measurements.log statistics.log tracking.log
pi@raspberrypi:~ $ ls /var/log/chrony
measurements.log statistics.log tracking.log
pi@raspberrypi:~ $ ls /var/log/chrony
measurements.log statistics.log tracking.log
For NTPd (be sure to restart it after making any configuration changes):
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
austin@prox-3070~ % cat /etc/ntp.conf
# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
austin@prox-3070 ~ % cat /etc/ntp.conf
# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
austin@prox-3070 ~ % cat /etc/ntp.conf
# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
Wait a few minutes for some data to record (chrony synchronizes pretty quick compared to NTPd) and check the statistics file, filtered to our NMEA refid:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
cat /var/log/chrony/statistics.log| grep NMEA
cat /var/log/chrony/statistics.log | grep NMEA
cat /var/log/chrony/statistics.log | grep NMEA
This spits out the lines that have NMEA present (the ones of interest for our USB GPS). To include the headers to show what each column is we can run
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
# chrony
cat /var/log/chrony/statistics.log| head -2; cat /var/log/chrony/statistics.log| grep NMEA
# ntp, there is no header info so we can omit that part of the command
cat /var/log/peerstats | grep 127.127.28.0
# chrony
cat /var/log/chrony/statistics.log | head -2; cat /var/log/chrony/statistics.log | grep NMEA
# ntp, there is no header info so we can omit that part of the command
cat /var/log/peerstats | grep 127.127.28.0
# chrony
cat /var/log/chrony/statistics.log | head -2; cat /var/log/chrony/statistics.log | grep NMEA
# ntp, there is no header info so we can omit that part of the command
cat /var/log/peerstats | grep 127.127.28.0
Screenshot showing chrony statistics for our NMEA USB GPS refclock
NTP stats don’t include header information. The column of interest is the one after the 9014 column. The columns are day, seconds past midnight, source, something, estimated offset, something, something, something. We can see the offset for this VK-172 USB GPS is somewhere around 76-77 milliseconds (0.076-0.077 seconds), which we can put in place of the 0.000 for the .28 driver for NTP and remove noselect.
So now we have some data showing the statistics of our NMEA USB GPS NTP source. We can copy and paste this into Excel, run data to columns, and graph the result and/or get the average to set the offset.
screenshot showing chrony/NTP statistics to determine offset
This graph is certainly suspicious (sine wave pattern and such) and if I wasn’t writing this blog post, I’d let data collect overnight to determine an offset. Since time is always of the essence, I will just take the average of the ‘est offset’ column (E), which is 7.64E-2, or 0.0763 seconds. Let’s pop this into the chrony.conf file and remove noselect:
Restart chrony again for the config file changes to take effect – sudo systemctl restart chrony.
6 – watch ‘chrony sources’ or ‘ntpq -pn’ to see if the USB GPS gets selected as the main time source
If you aren’t aware, Ubuntu/Debian/most Linux includes a utility to rerun a command every x seconds called watch. We can use this to watch chrony to see how it is interpreting each time source every 1 second:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
# for chrony
watch -n 1 chronyc sources
# for chrony
watch -n 1 chronyc sources
# for chrony
watch -n 1 chronyc sources
watching chrony sources
In the above screenshot, we can see that chrony actually has the NMEA source selected as the primary source (denoted with the *). It has the Raspberry Pi PPS NTP GPS ready to takeover as the new primary (denoted with the +). All of the sources match quite closely (from +4749us to – 505us is around 5.2 milliseconds). The source “offset” is in the square brackets ([ and ]).
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
# for ntp
watch -n 1 ntpq -pn
# for ntp
watch -n 1 ntpq -pn
# for ntp
watch -n 1 ntpq -pn
NTP showing (via ntpq -pn) that the GPS source is 0.738 milliseconds off of the host clock. The ‘-‘ in front of the remote means this will not be selected as a valid time (presumably due to the high jitter compared to the other sources, probably also due to manually setting it to stratum 2).
7- is +/- five millseconds good enough?
For 99% of use cases, yes. You can stop here and your home network will be plenty accurate. If you want additional accuracy, you are in luck. This GPS module also outputs a PPS (pulse per second) signal! We can use this to get within 0.05 millseconds (0.00005 seconds) from official/atomic clock time.
Conclusion
In this post, we got a u-blox USB GPS set up and added it as a reference clock (refclock) to chrony and demonstrated it is clearly within 10 millisecond of official GPS time.
You could write a script to do all this for you! I should probably try this myself…
In the next post, we can add PPS signals from the GPS module to increase our time accuracy by 1000x (into the microsecond range).
A note on why having faster message transmission is better for timing
My current PPS NTP server uses chrony with NMEA messages transmitted over serial and the PPS signal fed into a GPIO pin. GPSd as a rule does minimum configuration of GPS devices. It typically defaults to 9600 baud for serial devices. A typical GPS message looks like this:
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
$GPGGA, 161229.487, 3723.2475, N, 12158.3416, W, 1, 07, 1.0, 9.0, M, , , , 0000*18
$GPGGA, 161229.487, 3723.2475, N, 12158.3416, W, 1, 07, 1.0, 9.0, M, , , , 0000*18
$GPGGA, 161229.487, 3723.2475, N, 12158.3416, W, 1, 07, 1.0, 9.0, M, , , , 0000*18
That message is 83 bytes long. At 9600 baud (9600 bits per second), that message takes 69.1 milliseconds to transmit. Each character/byte takes 0.833 milliseconds to transmit. That means that as the message length varies, the jitter will increase. GPS messages do vary in length, sometimes significantly, depending on what is being sent (i.e. the satellite information, $GPGSV sentences, is only transmitted every 5-10 seconds).
I opened gpsmon to get a sample of sentences – I did not notice this until now but it shows how many bytes each sentence is at the front of the sentence:
These sentences range from 83 bytes to 35 bytes, a variation of (83 bytes -35 bytes)*0.833 milliseconds per byte = 39.984 milliseconds.
Compare to the u-blox binary UBX messages which seem to always be 60 bytes and transmitted at 57600 baud, which is 8.33 milliseconds to transmit the entire message.
UBX protocol messages (blanked out lines). I have no idea what part of the message is location, hopefully I got the right part blanked out.
The variance (jitter) is thus much lower and can be much more accurate as a NTP source. GPSd has no problem leaving u-blox modules at 57600 baud. This is why the USB GPS modules perform much more accurate for timekeeping than NMEA-based devices when using GPSd.
For basically every GPS module/chipset, it is possible to send it commands to enable/disable sentences (as well as increase the serial baud rate). In an ideal world for timekeeping, GPSd would disable every sentence except for time ($GPZDA), and bump up the baud rate to the highest supported level (115200, 230400, etc.). Unfortunately for us, GPSd’s default behavior is to just work with every GPS, which essentially means no configuring the GPS device.
Without further ado, below is the template I used to create my virtual machines. The main LAN network is 10.98.1.0/24, and the Kube internal network (on its own bridge) is 10.17.0.0/24.
This template creates a Kube server, two agents, and a storage server.
Update 2022-04-26: bumped Telmate provider version to 2.9.8 from 2.7.4
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
terraform {
required_providers {
proxmox = {
source = "telmate/proxmox"
version = "2.9.8"
}
}
}
provider "proxmox"{
pm_api_url = "https://prox-1u.home.fluffnet.net:8006/api2/json"# change this to match your own proxmox
