For a little side project, I build uuid7.com almost entirely with the help of AI tools. Check it out!
I also built the stack with Docker Compose. I have resisted Docker for so long because it was such a paint for homelab type stuff. But I recently started needing to use it at work (we are migrating to AWS Kubernetes – yay! not) so I figured I’d give it a go.
With the assistance of ChatGPT, I put together a full Docker stack using Cloudflare tunnels (cloudflared), Nginx as the webserver, and Flask as the backend all in a couple hours. It works great!
That said, it is running on my main desktop at home to see if it’s a popular site so fingers crossed it holds up.
I have finally started actual work on a “Tinder for restaurants” that I’m calling MonchMatch. I’ve been tossing the idea around for a few months but have finally made some moves with outsourcing the design. I can get the app and backend going myself, but I have zero creative talent.
If you are or know anyone who wants to work on the design, please drop me a note.
If you’re interested in being notified when MonchMatch is released, head on over to the MonchMatch site and enter your email!
Sometimes I make a change to my Proxmox cluster configuration without all nodes in a healthy state (i.e. they are off). This isn’t a great habit to get into and sometimes results in troubleshooting.
Putting a quick post up so I can easily reference how to resolve corosync issues.
# stop corosync and pmxcfs on all nodes
$ systemctl stop corosync pve-cluster
# start pmxcfs in local mode on all nodes
$ pmxcfs -l
# put correct corosync config into local pmxcfs and corosync config dir (make sure to bump the 'config_version' inside the config file)
$ cp correct_corosync.conf /etc/pve/corosync.conf
$ cp correct_corosync.conf /etc/corosync/corosync.conf
# kill local pmxcfs
$ killall pmxcfs
# start corosync and pmxcfs again
$ systemctl start pve-cluster corosync
# check status
$ journalctl --since '-5min' -u pve-cluster -u corosync
$ pvecm status
Hey there, welcome back to Austin’s Nerdy Things. It’s been a while since my last post – life happens. I haven’t lost sight of the blog. Just needed to do some other things and finish up some projects before documenting stuff.
The MPP LV2424 inverter I’m using puts out a lot of data. Some of it is quite useful, some of it not so much. Regardless, I am capturing all of it in my InfluxDB database with Python. This allows me to chart it in Grafana, which I use for basic monitoring stuff around the house. This post will document getting data from the MPP inverter to Grafana.
Jumping ahead
Final product first. This is a screenshot showing my complete (work-in-progress) Grafana dashboard for my DIY hybrid solar setup.
Solar dashboard as seen in Grafana pulling data from InfluxDB
The screenshot shows the last 6 hours worth of data from my system. It was a cloudy and then stormy afternoon/evening here in Denver (we got 0.89 inches of rain since midnight as of typing this post!), so the solar production wasn’t great. The panels are as follows:
Temperatures – showing temperatures from 3 sources: the MPP LV2424 inverter heat sink, and both BMS temperature probes (one is on the BMS board itself, the other is a probe on the battery bank). The inverter has at least 2 temperature readings, maybe 3. They all basically show the same thing.
Solar input – shows solar voltage as seen by the solar charge controller as well as solar power (in watts) going through the charge controller.
Cell voltages – the voltage reading of each of my 8 battery bank cells as reported by the BMS. The green graph also shows the delta between max and min cells. They are still pretty balanced in the flat part of the discharge curve.
Total power – a mashup of what the inverter is putting out, what the solar is putting in, the difference between the two, and what the BMS is reporting. I’m still trying to figure out all the nuances here. There is definitely a discrepancy between what the inverter is putting out and what the solar is putting in when the batteries are fully charged. I believe the difference is the power required to keep the transformer energized (typically ranges from 30-60W).
DC voltages – as reported by the inverter and the BMS. The inverter is accurate to 0.1V, the BMS goes to 0.01V.
AC voltages – shows the input voltage from the grid and the output voltage from the inverter. These will match unless the inverter is disconnected from the grid.
Data table – miscellaneous information from the inverter that isn’t graphable
Output load – how much output I’m using compared to the inverter’s limit
Total Generation – how much total energy has been captured by the inverter/solar panels. This is limited because I’m not back feeding the grid.
Getting data out of the MPP Solar LV2424 inverter with Python to MQTT
I am using two cables to plug the inverter into a computer. The first is the serial cable that came with the inverter. The second is a simple USB to RS232 serial adapter plugged into a Dell Micro 3070.
The computer is running Proxmox, which is a virtual machine hypervisor. That doesn’t matter for this post, we can ignore it. Just pretend the USB cable is plugged directly into a computer running Ubuntu 20.04 Linux.
The main bit of software I’m using is published on GitHub by jblance under the name ‘mpp-solar’ – https://github.com/jblance/mpp-solar. This is a utility written in Python to communicate with MPP inverters.
There was a good bit of fun trying to figure out exactly what command I needed to run to get data out of the inverter as evidenced by the history command:
Trying to figure out the usage of the Python mpp-solar utility
In the end, what worked for me to get the data is the following. I believe the protocol (-P) will be different for different MPP Solar inverters:
And the results are below. The grid voltage reported is a bit low because the battery started charging from the grid a few minutes before this was run.
austin@mpp-linux:~/mpp-solar-python$ sudo mpp-solar -p /dev/ttyUSB0 -b 2400 -P PI18 --getstatus
Command: ET - Total Generated Energy query
------------------------------------------------------------
Parameter Value Unit
working_mode Hybrid mode(Line mode, Grid mode)
grid_voltage 111.2 V
grid_frequency 59.9 Hz
ac_output_voltage 111.2 V
ac_output_frequency 59.9 Hz
ac_output_apparent_power 155 VA
ac_output_active_power 139 W
output_load_percent 5 %
battery_voltage 27.1 V
battery_voltage_from_scc 0.0 V
battery_voltage_from_scc2 0.0 V
battery_discharge_current 0 A
battery_charging_current 19 A
battery_capacity 76 %
inverter_heat_sink_temperature 30 °C
mppt1_charger_temperature 0 °C
mppt2_charger_temperature 0 °C
pv1_input_power 0 W
pv2_input_power 0 W
pv1_input_voltage 0.0 V
pv2_input_voltage 0.0 V
setting_value_configuration_state Something changed
mppt1_charger_status abnormal
mppt2_charger_status abnormal
load_connection connect
battery_power_direction charge
dc/ac_power_direction AC-DC
line_power_direction input
local_parallel_id 0
total_generated_energy 91190 Wh
And to get the same data right into MQTT I am using the following:
The above command is being run as a cron job once a minute. The default baud rate for the inverter is 2400 bps (yes, bits per second), which is super slow so a full poll takes ~6 seconds. Kind of annoying in 2021 but not a huge problem. The cron entry for the command is this:
# this program feeds a systemd service to convert the outputted mqtt to influx points
* * * * * /usr/local/bin/mpp-solar -p /dev/ttyUSB0 -P PI18 --getstatus -o mqtt -q mqtt --mqtttopic mpp
So with that we have MPP inverter data going to MQTT.
Putting MPP data into InfluxDB from MQTT
Here we need another script written in… take a guess… Python! This Python basically just opens a connection to a MQTT broker and transmits any updates to InfluxDB. The full script is a bit more complicated and I actually stripped a lot out because my MQTT topic names didn’t fit the template the original author used. I have started using this framework in other places to do the MQTT to InfluxDB translation. I like things going to the intermediate MQTT so they can be picked up for easy viewing in Home Assistant. Original code from https://github.com/KHoos/mqtt-to-influxdb-forwarder. The original code seems like it was built to be more robust than what I’m using it for (read: I have no idea what half of it does) but it worked for my purposes.
You’ll need a simple text file with your InfluxDB password and then reference it in the arguments. If your password is ‘password’, the only contents of the file should be ‘password’. I added the isFloat() function to basically make sure that strings weren’t getting added to the numeric tables in InfluxDB. I honestly find the structure/layout of storing stuff in Influx quite confusing so I’m sure there’s a better way to do this.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
####################################
# originally found at/modified from https://github.com/KHoos/mqtt-to-influxdb-forwarder
####################################
# forwarder.py - forwards IoT sensor data from MQTT to InfluxDB
#
# Copyright (C) 2016 Michael Haas <[email protected]>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation,
# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
import argparse
import paho.mqtt.client as mqtt
from influxdb import InfluxDBClient
import json
import re
import logging
import sys
import requests.exceptions
class MessageStore(object):
def store_msg(self, node_name, measurement_name, value):
raise NotImplementedError()
class InfluxStore(MessageStore):
logger = logging.getLogger("forwarder.InfluxStore")
def __init__(self, host, port, username, password_file, database):
password = open(password_file).read().strip()
self.influx_client = InfluxDBClient(
host=host, port=port, username=username, password=password, database=database)
def store_msg(self, database, sensor, value):
influx_msg = {
'measurement': database,
'tags': {'sensor': sensor},
'fields': {'value' : value}
}
self.logger.debug("Writing InfluxDB point: %s", influx_msg)
try:
self.influx_client.write_points([influx_msg])
except requests.exceptions.ConnectionError as e:
self.logger.exception(e)
class MessageSource(object):
def register_store(self, store):
if not hasattr(self, '_stores'):
self._stores = []
self._stores.append(store)
@property
def stores(self):
# return copy
return list(self._stores)
def isFloat(str_val):
try:
float(str_val)
return True
except ValueError:
return False
def convertToFloat(str_val):
if isFloat(str_val):
fl_result = float(str_val)
return fl_result
else:
return str_val
class MQTTSource(MessageSource):
logger = logging.getLogger("forwarder.MQTTSource")
def __init__(self, host, port, node_names, stringify_values_for_measurements):
self.host = host
self.port = port
self.node_names = node_names
self.stringify = stringify_values_for_measurements
self._setup_handlers()
def _setup_handlers(self):
self.client = mqtt.Client()
def on_connect(client, userdata, flags, rc):
self.logger.info("Connected with result code %s", rc)
# subscribe to /node_name/wildcard
#for node_name in self.node_names:
# topic = "{node_name}/#".format(node_name=node_name)
topic = "get_status/status/#"
self.logger.info("Subscribing to topic %s", topic)
client.subscribe(topic)
def on_message(client, userdata, msg):
self.logger.debug("Received MQTT message for topic %s with payload %s", msg.topic, msg.payload)
list_of_topics = msg.topic.split('/')
measurement = list_of_topics[2]
if list_of_topics[len(list_of_topics)-1] == 'unit':
value = None
else:
value = msg.payload
decoded_value = value.decode('UTF-8')
if isFloat(decoded_value):
str_value = convertToFloat(decoded_value)
for store in self.stores:
store.store_msg("power_measurement",measurement,str_value)
else:
for store in self.stores:
store.store_msg("power_measurement_strings",measurement,decoded_value)
self.client.on_connect = on_connect
self.client.on_message = on_message
def start(self):
print(f"starting mqtt on host: {self.host} and port: {self.port}")
self.client.connect(self.host, self.port)
# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
self.client.loop_forever()
def main():
parser = argparse.ArgumentParser(
description='MQTT to InfluxDB bridge for IOT data.')
parser.add_argument('--mqtt-host', default="mqtt", help='MQTT host')
parser.add_argument('--mqtt-port', default=1883, help='MQTT port')
parser.add_argument('--influx-host', default="dashboard", help='InfluxDB host')
parser.add_argument('--influx-port', default=8086, help='InfluxDB port')
parser.add_argument('--influx-user', default="power", help='InfluxDB username')
parser.add_argument('--influx-pass', default="<I have a password here, unclear if the pass-file takes precedence>", help='InfluxDB password')
parser.add_argument('--influx-pass-file', default="/home/austin/mpp-solar-python/pass.file", help='InfluxDB password file')
parser.add_argument('--influx-db', default="power", help='InfluxDB database')
parser.add_argument('--node-name', default='get_status', help='Sensor node name', action="append")
parser.add_argument('--stringify-values-for-measurements', required=False, help='Force str() on measurements of the given name', action="append")
parser.add_argument('--verbose', help='Enable verbose output to stdout', default=False, action='store_true')
args = parser.parse_args()
if args.verbose:
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
else:
logging.basicConfig(stream=sys.stdout, level=logging.WARNING)
print("creating influxstore")
store = InfluxStore(host=args.influx_host, port=args.influx_port, username=args.influx_user, password_file=args.influx_pass_file, database=args.influx_db)
print("creating mqttsource")
source = MQTTSource(host=args.mqtt_host,
port=args.mqtt_port, node_names=args.node_name,
stringify_values_for_measurements=args.stringify_values_for_measurements)
print("registering store")
source.register_store(store)
print("start")
source.start()
if __name__ == '__main__':
main()
Running the MQTT to InfluxDB script as a system daemon
Next up, we need to run the MQTT to InfluxDB Python script as a daemon so it starts with the machine and runs in the background. If you haven’t noticed by now, this is the standard pattern for most of the stuff I do – either a cron job or daemon to get data and another daemon to put it where I want it. Sometimes they’re the same.
austin@mpp-linux:~$ cat /etc/systemd/system/mpp-solar.service
[Unit]
Description=MPP inverter data - MQTT to influx
After=multi-user.target
[Service]
User=austin
Type=simple
Restart=always
RestartSec=10
# data feeds this script from a root cronjob running every 60s
ExecStart=/usr/bin/python3 /home/austin/mpp-solar-python/main.py
[Install]
WantedBy=multi-user.target
Then activate it:
austin@mpp-linux:~$ sudo systemctl daemon-reload
austin@mpp-linux:~$ sudo systemctl enable mpp-solar.service
austin@mpp-linux:~$ sudo systemctl start mpp-solar.service
austin@mpp-linux:~$ sudo systemctl status mpp-solar.service
● mpp-solar.service - MPP inverter data - MQTT to influx
Loaded: loaded (/etc/systemd/system/mpp-solar.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-07-31 02:21:32 UTC; 1 day 13h ago
Main PID: 462825 (python3)
Tasks: 1 (limit: 1072)
Memory: 19.2M
CGroup: /system.slice/mpp-solar.service
└─462825 /usr/bin/python3 /home/austin/mpp-solar-python/main.py
Jul 31 02:21:32 mpp-linux systemd[1]: Started MPP inverter data - MQTT to influx.
All done
With that, you should now have data flowing into your InfluxDB instance from your MPP inverter via this Python script. This is exactly what I’m using for my LV2424 but it should work with others like the PIP LV2424 MSD, PIP-4048MS, IPS stuff, LV5048, and probably a lot of others.
Next Steps
Next post will cover designing the Grafana dashboard to show this data.
