Categories
Python

Controlling AsrockRack CPU & chassis fan speeds via ipmitool & PID loops

I have a 1U Datto NAS unit that I got for super cheap ($150 for 4x 3.5″ SAS3, D-1541, 4x32GB, 2400MHz, 2x 10GbaseT) that has worked quite well for me. The only downside, which is present among basically all 1U devices, is the noise.

During my research for how to control the tiny, high-RPM (like 8000+ RPM) fans, I stumbled across a thread on the FreeNAS forums – https://www.truenas.com/community/threads/script-to-control-fan-speed-in-response-to-hard-drive-temperatures.41294/. At the bottom of the post, there are a few other links to improvements. I ran the Perl logging scripts that made up the improvements a bit but I am no Perl expert so didn’t up implementing it.

I am not 100% sure of the default AsrockRack behavior but it seemed that if CPU temp >60C, both case and CPU fans would spike. My BlueIris instance sends videos over a couple times an hour, which would spike the fans, which would be annoying during my work from home weeks while I was in the basement, working.

The idea of using a PID loop to control fan speeds stuck with me though, and with the help of GitHub Copilot, I busted out a proof of concept in an hour or so during a particularly boring set of meetings. There is a very high probability this will work for Supermicro motherboards as well with only minor tweaks.

This is how well it works. Note the drop at the end is due to changing CPU setpoint from 57C to 55C. The temperature is very, very steady.

Screenshot of TrueNAS Core reporting page for CPU temp showing very constant CPU temperature due to PID fan control loop

Without further ado, below is the main script (you’ll also need PID.py, which I borrowed a few years ago for the Coding a pitch/roll/altitude autopilot in X-Plane with Python series of posts). It can be run via SSH for debugging purposes (it is no fun to edit python via nano over ssh on FreeBSD), or with native commands if it detects it is running on the target system.

import logging
import time
import PID
import datetime
import socket
import subprocess

logging.basicConfig(
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.INFO)

# don't care about debug/info level logging from either of these packages
loggers_to_set_to_warning = ['paramiko.transport', 'invoke']
for l in loggers_to_set_to_warning:
    logging.getLogger(l).setLevel(logging.WARNING)

user = "root"
password = r"password"
host = None # this is set via hostname detection below
DESIRED_CPU_TEMP = 55.0
DESIRED_MB_TEMP = 35.0
# HDD_TEMP_THRESHOLD = 44.0 # unused
MIN_FAN_PCT = 10.0
drives_to_monitor = ['da0', 'da1', 'da2', 'da3', 'nvme0','nvme1','nvme2']

# command to set fans via ipmitool
# ipmitool raw 0x3a 0x01 0x00 0x04 0x04 0x04 0x04 0x04 0x04 0x04
					     #cpu #fan #fan #fan #fan #fan #fan ????

BASE_RAW_IPMI = 'raw 0x3a 0x01'
INITIAL_STATE = [32,32,32,32,32,32,32,32] # all 32/64 = half speed
FAN_CURRENT_STATE = INITIAL_STATE

hostname = socket.gethostname()
if 'truenas' in hostname or hostname == 'truenas-datto.home.fluffnet.net':
    host = 'localhost'
    c = None
else:
    from fabric import Connection # importing here because freebsd 13 (or whatever truenas core 13 is based on lacks pip to install packages)
    host = "10.98.1.9"
    c = Connection(host, port=22, user=user, connect_kwargs={'password': password})

current_sensor_readings = {}
cpu_temp_sensor = "CPU Temp"
cpu_fan_sensor = "CPU_FAN1"
case_fans = ["FRNT_FAN2","FRNT_FAN3","FRNT_FAN4"]
mb_temp_sensor = "MB Temp"

def limiter(input_value, min_value, max_value):
    if input_value < min_value:
        return min_value
    elif input_value > max_value:
        return max_value
    else:
        return input_value
    
def set_fans_via_ipmi(connection):
    # raw_ipmi_cmd = construct_raw_ipmi_cmd() # not needed unless debug and remote
    # logging.info(raw_ipmi_cmd)
    if host == 'localhost':
        result = subprocess.run(['ipmitool', 'raw', '0x3a', '0x01',
                                 '0x'+FAN_CURRENT_STATE[0], 
                                 '0x'+FAN_CURRENT_STATE[1],
                                 '0x'+FAN_CURRENT_STATE[2],
                                 '0x'+FAN_CURRENT_STATE[3],
                                 '0x'+FAN_CURRENT_STATE[4],
                                 '0x'+FAN_CURRENT_STATE[5],
                                 '0x'+FAN_CURRENT_STATE[6],
                                 '0x'+FAN_CURRENT_STATE[7]], stdout=subprocess.PIPE)
    else:
        raw_ipmi_cmd = construct_raw_ipmi_cmd()
        result = connection.run('ipmitool ' + raw_ipmi_cmd, hide=True)
    #logging.info(result.stdout)

def scale_to_64ths(input_percent):
    result = input_percent / 100.0 * 64.0
    # prepend 0 to make it a hex value
    result_int = int(result)
    result_str = str(result_int)
    if len(result_str) == 1:
        result_str = '0' + result_str # turn a 0x1 into a 0x01
    return result_str

def adjust_cpu_fan_setpoint(hex_value_64ths):
    FAN_CURRENT_STATE[0] = hex_value_64ths

def adjust_case_fan_setpoint(hex_value_64ths):
    for i in range(len(FAN_CURRENT_STATE) - 1):
        FAN_CURRENT_STATE[i + 1] = hex_value_64ths

def construct_raw_ipmi_cmd():
    new_state = BASE_RAW_IPMI
    for i in range(len(FAN_CURRENT_STATE)):
        new_state = new_state + ' 0x' + str(FAN_CURRENT_STATE[i])
    return new_state

def populate_sensor_readings(sensor, value):
    current_sensor_readings[sensor] = value

def query_ipmitool(connection):
    if host == 'localhost':
        result = subprocess.run(['ipmitool', 'sensor'], stdout=subprocess.PIPE)
        result = result.stdout.decode('utf-8')
    else:
        result = connection.run('ipmitool sensor', hide=True).stdout
    for line in result.split('\n'):
        if line == '':
            break

        row_data = line.split('|')
        sensor_name = row_data[0].strip()
        sensor_value = row_data[1].strip()
        populate_sensor_readings(sensor_name, sensor_value)
        logging.debug(sensor_name + " = " + sensor_value)

def wait_until_top_of_second():
    # calculate time until next top of second
    sleep_seconds = 1 - (time.time() % 1)
    time.sleep(sleep_seconds)

def get_drive_temp(connection, drive):
    ###########################################
    # this is copilot generated, and untested #
    # not sure about row_data[0] stuff        #
    ###########################################
    if host == 'localhost':
        result = subprocess.run(['smartctl', '-A', '/dev/' + drive], stdout=subprocess.PIPE)
        result = result.stdout.decode('utf-8')
    else:
        result = connection.run('smartctl -A /dev/' + drive, hide=True).stdout
    for line in result.split('\n'):
        if line == '':
            break

        row_data = line.split()
        if len(row_data) < 10:
            continue
        if row_data[0] == '194':
            drive_temp = row_data[9]
            logging.info(drive + " = " + drive_temp)

def query_drive_temps(connection):
    for drive in drives_to_monitor:
        get_drive_temp(connection, drive)

# tune these values. the first one is the most important and basically is the multiplier for
# how much you want the fans to run in proportion to the actual-setpoint delta.
# example: if setpoint is 55 and actual is 59, the delta is 4, which is multiplied by 4 for
# 16 output, which if converted to 64ths would be 25% fan speed.
# the 2nd parameter is the integral, which is a cumulative error counter of sorts.
# the 3rd parameter is derivative, which should probably be set to 0 (if tuned correctly, it prevents over/undershoot)
cpu_pid = PID.PID(4.0, 2.5, 0.1)
cpu_pid.SetPoint = DESIRED_CPU_TEMP

mb_pid = PID.PID(2.5, 1.5, 0.1)
mb_pid.SetPoint = DESIRED_MB_TEMP

wait_until_top_of_second()

# set last_execution to now minus one minute to force first execution
last_execution = datetime.datetime.now() - datetime.timedelta(minutes=1)

while(True):
    if datetime.datetime.now().minute != last_execution.minute:
        # TODO: get drive temps
        logging.info("getting drive temps")

    query_ipmitool(c)
    cpu_temp = float(current_sensor_readings[cpu_temp_sensor])
    mb_temp = float(current_sensor_readings[mb_temp_sensor])

    cpu_pid.update(cpu_temp)
    mb_pid.update(mb_temp)
    
    logging.info(f'CPU: {cpu_temp:5.2f} MB: {mb_temp:5.2f} CPU PID: {cpu_pid.output:5.2f} MB PID: {mb_pid.output:5.2f}')
    
    # note negative multiplier!!
    cpu_fan_setpoint = scale_to_64ths(limiter(-1*cpu_pid.output,MIN_FAN_PCT,100))
    case_fan_setpoint = scale_to_64ths(limiter(-1*mb_pid.output,MIN_FAN_PCT,100))
    adjust_cpu_fan_setpoint(cpu_fan_setpoint)
    adjust_case_fan_setpoint(case_fan_setpoint)
    set_fans_via_ipmi(c)

    last_execution = datetime.datetime.now()
    wait_until_top_of_second()

As you can see, it is not quite complete. I still need to add the hard drive temp detection stuff to ramp case fans a bit if the drives get hot. Those NVMe drives sure get hot (especially the Intel P4800X I have in one of the PCIe slots – see Intel Optane P1600X & P4800X as ZFS SLOG/ZIL for details).

This is what the output looks like (keep in mind the -1 multiplier in the setpoint stuff!):

screenshot showing second-by-second output of the PID fan control loop keeping a very consistent 55C CPU temp

And here is a summary of the script provided by the ever helpful ChatGPT with some high-level summaries. I fed it the code and said “write a blog post about this”. I took out the intro paragraph but left the rest.

The Script Overview

This script leverages the PID controller – a control loop mechanism that calculates an “error” value as the difference between a measured process variable and a desired setpoint. It attempts to minimize the error by adjusting the process control inputs.

In this script, we are implementing a fan speed control system that reacts to temperature changes dynamically. Our desired setpoint is the optimal temperature we want to maintain for both the CPU (DESIRED_CPU_TEMP) and the motherboard (DESIRED_MB_TEMP).

Exploring the Script

The Python script begins by setting up the necessary libraries and logging. The logging library is used to log useful debug information, such as the current CPU temperature and fan speed, which can help you understand what’s happening in the script.

Next, we have a section where we define some constants, such as the desired temperatures and minimum fan speed percentage. It also defines a connection to the localhost or to a remote host, depending on the hostname.

It uses ipmitool – a utility for managing and configuring devices that support the Intelligent Platform Management Interface (IPMI) to control fan speeds.

The limiter() function ensures the fan speed remains within the predefined minimum and maximum thresholds. It’s important as it prevents the fan speed from reaching potentially harmful levels.

The script also includes several functions to set and adjust fan speeds, as well as to construct the appropriate ipmitool command. One thing to note is that the fan speeds are set using hexadecimal values, so there are functions to convert the desired fan speed percentages to hexadecimal.

A very useful function is query_ipmitool(). This function runs the ipmitool command, gets the current sensor readings, and stores them in the current_sensor_readings dictionary for further processing.

The script utilizes two PID controllers, cpu_pid for the CPU and mb_pid for the motherboard, with specific setpoints set to desired temperatures.

The core logic is inside the infinite loop at the end of the script. The loop constantly reads temperature sensor data and adjusts the fan speeds accordingly. The loop runs once every second, so it can respond quickly to changes in CPU and motherboard temperatures.

Conclusion

This script demonstrates a neat way of controlling fan speed in response to CPU and motherboard temperatures. It’s an effective approach to ensure that your system runs smoothly and without overheating, while minimizing noise.

Categories
AI Blog Admin

Using ChatGPT to fight spam on WordPress

Like all other WordPress blogs, this one attracts a good number of spam comments. I get usually 5-10 per day, but yesterday there were like 30. Almost all of them contain Cyrillic characters:

screenshot showing spam comments containing cyrillic characters

Since I specify that all comments are held until approved, that means I need to either approve or trash or spam every comment.

Enter ChatGPT

I use ChatGPT (specifically GPT 4) for a number of minor coding tasks. I find it helpful. It is not perfect. That doesn’t mean it isn’t useful. I decided to have it ponder this issue. I work with Python a lot at work and it’s typically my scripting language of choice. My initial request is as follows:

write a python script to log into a wordpress site as an admin, get the top 5 comments, see if there are any Cyrillic characters in them, and if there are, delete said comment

It was understandably unhappy about potentially being asked to “hack” a WordPress site, so I had to reassure it that I was the owner of said site:

yes, I have my own blog. I am the owner. I have the admin credentials. please proceed with how to delete those comments

It happily complied and spit out some very usable code:

chatgpt conversation around a python script to delete blog spam comments

After a bit more back and forth:

does this get comments in a pending state? I don't let them be published instantly because most of them are spam

I was informed there are 5 different comment states: approved, hold, spam, trash, unapproved.

perfect. can you please adjust the script to get the pending, unapproved, and hold comments. also make it top 20

It ran perfectly after copy + pasting the Python. Unfortunately I created an application password for my main login on this site and forgot to change the delete URL so it happily sent my application password and username to yourwebsite.com. After revoking that password and realizing there should be a base url:

please split out the site url (https://austinsnerdythings.com) from base_url for both retrieving the comments as well as deleting

I was left with a 100% functional script. This took 3-4 min of back and forth with ChatGPT 4.0. I definitely could’ve code this up myself with the basic structure in 15 minutes or so but I would’ve had to work out the json format for comments and all that. It is so much easier to just test out what ChatGPT provides and alter as necessary:

import requests
import json
import re

def has_cyrillic(text):
    return bool(re.search('[\u0400-\u04FF]', text))

site_url = "https://austinsnerdythings.com"
base_url = f"{site_url}/wp-json/wp/v2/comments?per_page=20&status="
statuses = ['pending', 'hold', 'unapproved']
auth = ('yourusername', 'yourpassword')

for status in statuses:
    url = base_url + status
    response = requests.get(url, auth=auth)
    comments = json.loads(response.text)

    cyrillic_comments = []

    for comment in comments:
        if has_cyrillic(comment['content']['rendered']):
            cyrillic_comments.append(comment)

    # delete comments with Cyrillic characters
    for comment in cyrillic_comments:
        delete_url = f"{site_url}/wp-json/wp/v2/comments/" + str(comment['id'])
        response = requests.delete(delete_url, auth=auth)
        if response.status_code == 200:
            print(f"Successfully deleted comment with id {comment['id']}")
        else:
            print(f"Failed to delete comment with id {comment['id']}. Response code: {response.status_code}")

Finishing touches

The other finishing touches I did were as follows:

  • Created a user specific for comment moderation. I used the ‘Members’ plugin to create a very limited role (only permissions granted are the necessary ones: Moderate Comments, Read, Edit Posts, Edit Others’ Posts, Edit Published Posts) and assigned said user to it. This greatly limits the potential for abuse if the account password falls into the wrong hands.
  • Copied the script to the web host running the blog
  • Set it to be executed hourly via crontab

Now I have a fully automated script that deletes any blog comments with any Cyrillic characters!

You may be asking yourself why I don’t use Akismet or Recaptcha or anything like that. I found the speed tradeoff to not be worthwhile. They definitely slowed down my site for minimal benefit. It only took a couple minutes a day to delete the spam comments. But now it takes no time because it’s automated!

Here’s the link to the full ChatGPT conversation:

https://chat.openai.com/share/aad6a095-9b90-42c5-b1ca-de2a18828ba2

Results

I created a spam comment and ran the script (after adding a print line to show the comment). Here’s the output:

And the web logs showing the 3 status being retrieved via GET and the DELETE for the single spam comment:

I am quite satisfied with this basic solution. It took me far longer to type up this blog post than it did to get the script working.

Categories
AI

Stable Diffusion Tutorial – Nvidia GPU Installation

Like most other internet-connected people, I have seen the increase in AI-generated content in recent months. ChatGPT is fun to use and I’m sure there are plenty of useful use cases for it but I’m not sure I have the imagination required to use it to it’s full potential. The AI art fad of a couple months ago was cool too. In the back of my mind, I kept thinking “where will AI take us in the next couple years”. I still don’t know the answer to that. The only “art” I am good at is pottery (thanks to high-school pottery class – I took 4 semesters of it and had a great time doing so, whole different story). But now I’m able to generate my own AI art thanks to a guide I found the other day on /g/. I am re-writing it here with screenshots and a bit more detail to try and make it more accessible to general users.

NOTE: You need a decent/recent Nvidia GPU to follow this guide. I have a RTX 2080 Super with 8GB of VRAM. There are low-memory workarounds but I haven’t tested them yet. An absolute limit is 2GB VRAM, and a GTX 7xx (Maxwell architecture) or newer GPU.

Stable Diffusion Tutorial Contents

  1. Installing Python 3.10
  2. Installing Git (the source control system)
  3. Clone the Automatic1111 web UI (this is the front-end for using the various models)
  4. Download models
  5. Adjust memory limits & enable listening outside of localhost
  6. First run
  7. Launching the web UI
  8. Generating Stable Diffusion images

Video version of this install guide

Coming soon. I always do the written guide first, then record based off the written guide. Hopefully by end of day (mountain time) Feb 24.

1 – Installing Python 3.10

This is relatively straight-forward. To check your Python version, go to a command line and enter

python --version

If you already have Python 3.10.x installed (as seen in the screenshot below), you’re good to go (minor version doesn’t matter).

Python 3.10 installed for Stable Diffusion

If not, go to the Python 3 download page and select the most recent 3.10 version. As of writing, the most recent is 3.10.10. Download the x64 installer and install. Ensure the “add python.exe to PATH” checkbox is checked. Adding python.exe to PATH means it can be called with only python at a command prompt instead of the full path, which is something like c:/users/whatever/somedirectory/moredirectories/3.10.10/python.exe.

Installing python and adding python.exe to PATH

2 – Installing Git (the source control system)

This is easier than Python – just install it – https://git-scm.com/downloads. Check for presence and version with git –version:

git installed and ready to go for Stable Diffusion

3 – Clone the Automatic1111 web UI (this is the front-end for using the various models)

With Git, clone means to download a copy of the code repository. When you clone a repo, a new directory is created in whatever directory the command is run in. Meaning that if you navigate to your desktop, and run git clone xyz, you will have a new folder on your desktop named xyz with the contents of the repository. To keep things simple, I am going to create a folder for all my Stable Diffusion stuff in the C:/ root named sd and then clone into that folder.

Open a command prompt and enter

cd c:\

Next create the sd folder and enter it:

mkdir sd
cd sd

Now clone the repository while in your sd folder:

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui

After the git clone completes, there will be a new directory called ‘stable-diffusion-webui’:

stable-diffusion-webui cloned and ready to download models

4 – Download models

“Models” are what actually generate the content based on provided prompts. Generally, you will want to use pre-trained models. Luckily, there are many ready to use. Training your own model is far beyond the scope of this basic installation tutorial. Training your own models generally also requires huge amounts of time crunching numbers on very powerful GPUs.

As of writing, Stable Diffusion 1.5 (SD 1.5) is the recommended model. It can be downloaded (note: this is a 7.5GB file) from huggingface here.

Take the downloaded file, and place it in the stable-diffusion-webui/models/Stable-diffusion directory and rename it to model.ckpt (it can be named anything you want but the web UI automatically attempts to load a model named ‘model.ckpt’ upon start). If you’re following along with the same directory structure as me, this file will end up at C:\sd\stable-diffusion-webui\models\Stable-diffusion\model.ckpt.

Another popular model is Deliberate. It can be downloaded (4.2GB) here. Put it in the same folder as the other model. No need to rename the 2nd (and other) models.

After downloading both models, the directory should look like this:

Stable Diffusion 1.5 (SD 1.5) and Deliberate_v11 models ready for use

5 – Adjust memory limits & enable listening outside of localhost (command line arguments)

Inside the main stable-diffusion-webui directory live a number of launcher files and helper files. Find webui-user.bat and edit it (.bat files can be right-clicked -> edit).

Add –medvram (two dashes) after the equals sign of COMMANDLINE_ARGS. If you also want the UI to listen on all IP addresses instead of just localhost (don’t do this unless you know what that means), also add –listen.

webui-user.bat after edits

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--listen --medvram

call webui.bat

6 – First run

The UI tool (developed by automatic1111) will automatically download a variety of requirements upon first launch. It will take a few minutes to complete. Double-click the webui-user.bat file we just edited. It calls a few .bat files and eventually launches a Python file. The .bat files are essentially glue to stick a bunch of stuff together for the main file.

The very first thing it does is creates a Python venv (virtual environment) to keep the Stable Diffusion packages separate from your other Python packages. Then it pip installs a bunch of packages related to cuda/pytorch/numpy/etc so Python can interact with your GPU.

webui-user.bat using pip to install necessary python packages like cuda

After everything is installed and ready to go, you will see a line that says: Running on local URL: http://127.0.0.1:7860. That means the Python web server UI is running on your own computer on port 7860 (if you added –listen to the launch args, it will show 0.0.0.0:7860, which means it is listening on all IP addresses and can be accessed by external machinse).

stable-diffusion-webui launched and ready to load

7 – Launching the web UI

With the web UI server running, it can be accessed via browser on the same computer running the Python at http://127.0.0.1:7860. That link should work for you if you click it.

Note that if the Python process closes for whatever reason (you close the command window, your computer reboots, etc), you need to double-click webui-user.bat to relaunch it and it needs to be running any time you want to access the web UI.

Automatic1111 stable diffusion web UI up and running

As seen in the screenshot, there are a ton of parameters/settings. I’ll highlight a few in the next section

8 – Generating Stable Diffusion images

This is the tricky part. The prompts make or break your generation. I am still learning. The prompt is where you enter what you want to see. Negative prompt is where you enter what you don’t want to see.

Let’s start simple, with cat in the prompt. Then click generate. A very reasonable-looking cat should soon appear (typically takes a couple seconds per image):

AI-generated cat with stable diffusion 1.5 with default settings

To highlight a few of the settings/sliders:

  • Stable diffusion checkpoint – model selector. Note that it’ll take a bit to load a new model (the multi-GB files need to be read in their entirety and ingested).
  • Prompt – what you want to see
  • Negative prompt – what you don’t want to see
  • Sampling method – various methods to sample new points
  • Sampling steps – how many iterations to use for image generation for a single image
  • Width – width of image to generate (in pixels). NOTE, you need a very powerful GPU with a ton of VRAM to go much higher than the default 512
  • Height – height of image to generate (in pixels). Same warning applies as width
  • Batch count – how many images to include in a batch generation
  • Batch size – haven’t used yet, presumably used to specify how many batches to generate
  • CFG Scale – this slider tells the models how specific they need to be for the prompt. Higher is more specific. Really high values (>12ish) start to get a bit abstract. Probably want to be in the range of 3-10 for this slider.
  • Seed – random number generator seed. -1 means use a new seed for every image.

Some thoughts on prompt/negative prompt

From my ~24 hours using this tool, it is very clear that prompt/negative prompts are what make or break your generation. I think that your ability as a pre-AI artist would come in handy here. I am no artist so I have a hard time putting what I want to see into words. Take example prompt: valley, fairytale treehouse village covered, matte painting, highly detailed, dynamic lighting, cinematic, realism, realistic, photo real, sunset, detailed, high contrast, denoised, centered. I would’ve said “fairytale treehouse” and stopped at that. Compare the two prompts below with the more detailed prompt directly below and the basic “fairytale treehouse” prompt after that:

AI-generated “fairytale treehouse” via stable diffusion. Prompt: valley, fairytale treehouse village covered, matte painting, highly detailed, dynamic lighting, cinematic, realism, realistic, photo real, sunset, detailed, high contrast, denoised, centered
AI-generated “fairytale treehouse” via stable diffusion. Prompt: fairytale treehouse

One of these looks perfectly in place for a fantasy story. The other you could very possibly see in person in a nearby forest.

Both positive and negative can get very long very quickly. Many of the AI-generated artifacts present over the last month or two can be eliminated with negative prompt entries.

Example negative prompt: ugly, deformed, malformed, lowres, mutant, mutated, disfigured, compressed, noise, artifacts, dithering, simple, watermark, text, font, signage, collage, pixel

I will not pretend to know what works well vs not. Google is your friend here. I believe that “prompt engineering” will be very important in AI’s future. Google is your friend here.

Conclusion

AI-generated content is here. It will not be going away. Even if it is outlawed, the code is out there. AI will be a huge part of our future, regardless of if you want it or not. As the saying goes – pandora’s box is now open.

I figured it was worth trying. The guide this is based off made it relatively easy for me (but I do have above-average computer skill), and I wanted to make it even easier. Hopefully you found this ‘how to set up stable diffusion’ guide easy to use as well. Please let me know in the comments section if you have any questions/comments/feedback – I check at least daily!

Resources

Huge shout out to whoever wrote the guide (“all anons”) at https://rentry.org/voldy. That is essentially where this entire guide came from.

Categories
Home Assistant Home Automation Python

Ultra efficient “air conditioner” (fans controlled with Home Assistant and Python) using cold outside air

Just getting this up as a draft now for a Reddit user.

In short, this Python script reads the temperatures of two different sensors (outside from an The Ambient Weather WS-2902C weather station and in our master bedroom with a Govee Bluetooth Thermometer), the temperature set point for a generic thermostat entity, does some logic, and turns a switch on or off, all with the Home Assistant API. The switch control two basic box fans that are set to blow air into our bedroom from outside. It runs every X minutes (currently set to 5). This method works great if nighttime temperatures drop below 70F before bedtime. We like the bedroom temp at 66F, so unless it gets below 70F by around 9PM, it probably won’t cool enough for us to be comfortable enough to fall asleep. My wife wakes up at 5:45am, me at 6:30am, pending what our 22 month old daughter thinks of that schedule, so we typically aim to be asleep by 10pm.

Today, 2022-05-14, was the day I got out the window AC. It will be in place for the rest of the summer.

Requirements:

  • A working Home Assistant installation (mine is Python venv install in a Ubuntu VM)
  • A bearer token authorization code for a/your Home Assistant user
  • A working MQTT installation
  • A switch controllable by Home Assistant
  • 1-2 box fans plugged into said switch controlled by Home Assistant

Here is what the Home Assistant control screen looks like. The buttons should be self explanatory. The generic thermostat entity doesn’t need to be on/active for this to work. It uses the set point for control purposes (set to 67.0F in the screenshot).

Home Assistant control screen for ultra efficient air conditioner system

I believe there is currently a logic bug with max cool not respecting the delta_temp variable. Other than that it works perfect. Below is a screenshot of the last 7 days showing the room cooling off nicely to the setpoint of 66F on nights 1-3 and 67F on nights 4-7. Switching a control device on and off every so often is a version of a bang-bang controller. If you look closely, you will notice that each cycle on and off results in a greater temperature drop, which is due to the colder outside air being blown in for the same duration regardless of delta T.

Master bedroom temperature with Home Assistant controlled fans blowing in cold air from outside. Setpoint was 66F for evening of 5/7-5/9 (first 3 nights) and 67F for the rest (next 4 nights).
Zoomed in view of the evening of 5/9 to the morning of 5/10. The temperature drops quickly to the setpoint of 66F and does not go much above. Not sure what the spike is right after 23:00. The outside temp starts at 65F for this same timeframe, dropping to 60 at 21:00 and 55 at 22:00, so a great night to use cold outside air for cooling (thus the “ultra efficient AC”.
import json
import datetime
import time
from dateutil import parser
from requests import get, post
import paho.mqtt.client as mqttClient
import logging

logging.basicConfig(
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.DEBUG)

loggers_to_set_to_warning = ['urllib3.connectionpool']
for l in loggers_to_set_to_warning:
    logging.getLogger(l).setLevel(logging.WARNING)

delta_temp = 3.0
mqtt_host = "mqtt.example.com"
mqtt_port = 1883

base_url = "http://ha.example.come:8123/"
states_url = base_url + "api/states/"
switch_url = base_url + "api/services/switch"
bearer_token = "ey...Vw"
full_bearer_token = "Bearer " + bearer_token
request_headers = {
    "Authorization": full_bearer_token,
    "content-type": "application/json"
}
endpoints = ["climate.masterbedfancooling",
             "sensor.real_outside_temp"]
fan_switch_entity_id = "switch.fan_switch"
states = {}
climate_topic = "climate/fan_control_state"
current_state = "off"
last_state = None
desired_seconds_to_sleep = 300
max_cool_outdoor_temp_limit = 66
desired_temp = None
outside_temp = None
current_temp = None
next_fan_action_time = datetime.datetime.now()


def set_fan_switch_state(state):
    new_fan_state = None
    if state == "on":
        new_fan_state = "on"
    elif state == "off":
        new_fan_state = "off"
    else:
        logging.warn("requested fan state unknown")
        return

    entity_info = {"entity_id": fan_switch_entity_id}
    full_url = switch_url + "/turn_" + new_fan_state
    response = post(full_url, headers=request_headers,
                    data=json.dumps(entity_info))
    if response.status_code != 200:
        logging.error(
            f"attempted to set fan state to {new_fan_state} but encountered error with status code: {response.status_code}")
    else:
        logging.info(f"successfully set fan state to {new_fan_state}")


def set_state_from_mqtt_message(message):
    global current_state, next_fan_action_time
    if message == "max_cool":
        current_state = "max_cool"
    elif message == "normal_cool":
        current_state = "normal_cool"
    elif message == "off":
        current_state = "off"
    elif message == "on":
        current_state = "on"
    else:
        logging.error(f"unable to determine state, setting to off")
        current_state = "off"
    logging.info(f"current_state set to: {current_state}")
    next_fan_action_time = datetime.datetime.now()


def connect_mqtt():
    # Set Connecting Client ID
    client = mqttClient.Client("python_window_fan_control")
    #client.username_pw_set(username, password)
    client.on_connect = on_connect
    client.connect(mqtt_host, mqtt_port)
    return client


def on_connect(client, userdata, flags, rc):
    if rc == 0:
        logging.info("Connected to MQTT Broker!")
    else:
        logging.info("Failed to connect, return code %d\n", rc)


def on_message(client, userdata, msg):
    logging.info(f"Received `{msg.payload.decode()}` from `{msg.topic}` topic")
    set_state_from_mqtt_message(msg.payload.decode())


def on_subscribe(client, userdata, mid, granted_qos):
    logging.info(f"subscribed to topic")


def set_fan_state(state):
    if state == "on":
        logging.info(f"setting fan state to on")
        set_fan_switch_state("on")
    elif state == "off":
        logging.info(f"setting fan state to off")
        set_fan_switch_state("off")


def get_and_set_temperatures():
    global desired_temp, current_temp, outside_temp
    logging.debug("executing loop")

    for endpoint in endpoints:
        full_url = states_url + endpoint
        response = get(full_url, headers=request_headers)
        parsed_json = json.loads(response.text)
        entity = parsed_json['entity_id']
        hvac_action = ""
        if endpoint == 'climate.masterbedfancooling':
            desired_temp = float(
                parsed_json['attributes']['temperature'])
            current_temp = float(
                parsed_json['attributes']['current_temperature'])
            hvac_action = parsed_json['attributes']['hvac_action']
        elif endpoint == 'sensor.real_outside_temp':
            outside_temp = float(parsed_json['state'])
        last_updated = parser.parse(parsed_json['last_updated'])
    logging.info(
        f"temps: current={current_temp}, desired={desired_temp}, outside={outside_temp}")
    if desired_temp == None or current_temp == None or outside_temp == None:
        logging.error(
            "one or more temps invalid, turning off switch and breaking execution")
        set_fan_state("off")


client = mqttClient.Client("window-fan-client")
client.on_connect = on_connect
client.on_message = on_message
client.on_subscribe = on_subscribe
client.connect(mqtt_host, mqtt_port)
client.subscribe(climate_topic)
client.loop_start()

while(True):
    # logging.info("loop")
    if datetime.datetime.now() > next_fan_action_time:
        logging.info("fan action time")
        if current_state == "off":
            new_fan_state = "off"
            logging.info(
                f"current_state is {current_state}, turning fan {new_fan_state}")
            set_fan_state("off")
        else:
            get_and_set_temperatures()

        if current_state == "max_cool":
            logging.info(
                f"current_state is {current_state}, call for max cooling")
            if outside_temp < max_cool_outdoor_temp_limit:
                logging.info(
                    f"able to max_cool with outside temp: {outside_temp}, lower than {max_cool_outdoor_temp_limit} ")
                set_fan_state("on")
            elif outside_temp < (current_temp - delta_temp):
                logging.info(
                    f"unable to max cool, but still can cool with outside: {outside_temp} and inside: {current_temp}")
                set_fan_state("on")
            else:
                logging.info("unable to cool at all, turning fan off")
                set_fan_state("off")
        elif current_state == "normal_cool":
            logging.info(
                f"current_state is {current_state}")
            if (current_temp > desired_temp):
                logging.info(
                    f"call for cooling. current: {current_temp}, desired: {desired_temp}")
                if (current_temp > (outside_temp - delta_temp)):
                    logging.info("can cool, turning fan on")
                    set_fan_state("on")
                else:
                    logging.info("can't cool, turning fan off")
                    set_fan_state("off")
            else:
                logging.info("no need for cooling, turning fans off")
                set_fan_state("off")
        elif current_state == "on":
            new_fan_state = "on"
            logging.info(
                f"current_state is {current_state}, turning fan {new_fan_state}")
            set_fan_state("on")

        last_state = current_state
        next_fan_action_time = datetime.datetime.now() \
            + datetime.timedelta(seconds=desired_seconds_to_sleep)
        logging.info(f"next fan action in {desired_seconds_to_sleep} seconds")
        logging.info("---------loop end-------------------")
    else:
        #logging.debug("not fan action time yet, sleeping 1s")
        time.sleep(0.25)
    # actual_seconds_to_sleep = desired_seconds_to_sleep - datetime.datetime.now().minute % desired_seconds_to_sleep
    # seconds_to_sleep = actual_minutes_to_sleep * 60.0
    # logging.info(f"sleeping {desired_seconds_to_sleep}s")
    # time.sleep(desired_seconds_to_sleep)
Categories
Home Assistant Home Automation

Using the Govee Bluetooth Thermometer with Home Assistant (Python and MQTT)

Introduction

Like many other Home Automation enthusiasts, I have been on the lookout for a cheap thermometer/hygrometer that has either WiFi or Bluetooth connectivity. I think I found the answer in the $12 USD Govee Bluetooth Digital Thermometer and Hygrometer. The fact that the device broadcasts the current temperature and humidity every 2 seconds via low energy Bluetooth (BLE) is the metaphorical icing on the cake. Here is a pic of the unit in our garage:

Govee bluetooth thermometer and hygrometer showing 56*F/24% in a garage

Specifications

This is a pretty basic device. It has a screen that shows the current temperature, humidity, and min/max values. It sends the current readings (along with battery health) every 2 seconds via low-energy bluetooth (BLE). The description on Amazon says it has a “Swiss-made smart hygrometer sensor”. Dunno if I believe that but for $12 it is good enough. The temperature is accurate to +/- 0.54F and humidity is +/- 3% RH. If you use the app, it is apparently possible to read the last 20 days or 2 years of data from the device (I haven’t used the app at all).

Enough with the boring stuff. Let’s get it connected to Home Assistant.

Reading the Govee bluetooth advertisements with Python

I found some sample code on tchen’s GitHub page (link) to help get me going in the right direction.

Without further ado, here is the code I’m using to read the data and publish via MQTT:

observe.py (updated 2022-01-04 with some logging improvements. default logging level is now WARNING, which disables printing every advertisement)

# basic govee bluetooth data reading code for model https://amzn.to/3z14BIi
# written/modified by Austin of austinsnerdythings.com 2021-12-27
# original source: https://gist.github.com/tchen/65d6b29a20dd1ef01b210538143c0bf4
import logging
import json
from time import sleep
from basic_mqtt import basic_mqtt
from bleson import get_provider, Observer

logging.basicConfig(
	format='%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',
	datefmt='%Y-%m-%d %H:%M:%S',
	level=logging.WARNING)

# I did write all the mqtt stuff. I kept it in a separate class
mqtt = basic_mqtt()
mqtt.connect()

# writing code on my windows computer, committing, pushing, and then 
# pulling the new code on the Raspberry Pi is a tedious "debug" process.
# there are quite a few errors that spit out from bleson, which as far
# as I can tell, isn't super polished. if we set the debug level to
# critical, most of those disappear.
logging.getLogger("bleson").setLevel(logging.CRITICAL)

# basic celsius to fahrenheit function
def c2f(val):
    return round(32 + 9*val/5, 2)

last = {}

# I didn't write this, but it takes the raw BT data and spits out the data of interest
def temp_hum(values, battery, address):
    global last
    #########################
    #
    # there is a fix for temperatures below freezing here - 
    # https://github.com/joshgordon/govee_ble_to_mqtt/blob/master/observe.py
    # I will adjust post code afternoon of 2022-11-28
    #
    #########################
    values = int.from_bytes(values, 'big')
    if address not in last or last[address] != values:
        last[address] = values
        temp = float(values / 10000)
        hum = float((values % 1000) / 10)
        # this print looks like this:
        # 2021-12-27T11:22:17.040469 BDAddress('A4:C1:38:9F:1B:A9') Temp: 45.91 F  Humidity: 25.8 %  Battery: 100 %
        logging.info(f"decoded values: {address} Temp: {c2f(temp)} F  Humidity: {hum} %  Battery: {battery}")
        # this code originally just printed the data, but we need it to publish to mqtt.
        # added the return values to be used elsewhere
        return c2f(temp), hum, battery

def on_advertisement(advertisement):
    #print(advertisement)
    mfg_data = advertisement.mfg_data

    # there are lots of BLE advertisements flying around. only look at ones that have mfg_data
    if mfg_data is not None:
        #print(advertisement)
        # there are a few Govee models and the data is in different positions depending on which
        # the unit of interest isn't either of these hardcoded values, so they are skipped
        if advertisement.name == 'GVH5177_9835':
            address = advertisement.address
            temp_hum(mfg_data[4:7], mfg_data[7], address)
        elif advertisement.name == 'GVH5075_391D':
            address = advertisement.address
            temp_hum(mfg_data[3:6], mfg_data[6], address)
        elif advertisement.name != None:
            # this is where all of the advertisements for our unit of interest will be processed
            address = advertisement.address
            if 'GVH' in advertisement.name:
                #print(advertisement)
                temp_f, hum, battery = temp_hum(mfg_data[3:6], mfg_data[6], address)

                if temp_f > 180.0 or temp_f < -30.0:
                    return
                # as far as I can tell bleson doesn't have a string representation of the MAC address
                # address is of type BDAddress(). str(address) is BDAddress('A4:C1:38:9F:1B:A9')
                # substring 11:-2 is just the MAC address
                mac_addr = str(address)[11:-2]

                # construct dict with relevant info
                msg = {'temp_f':temp_f,
                        'hum':hum,
                        'batt':battery}
                
                # looks like this:
                # msg data: {'temp_f': 45.73, 'hum': 25.5, 'batt': 100}
                logging.info(f"MQTT msg data: {msg}")

                # turn into JSON for publishing
                json_string = json.dumps(msg, indent=4)

                # publish to topic separated by MAC address
                mqtt.publish(f"govee/{mac_addr}", json_string)

# base stuff from the original gist
logging.warning(f"initializing bluetooth")
adapter = get_provider().get_adapter()
observer = Observer(adapter)
observer.on_advertising_data = on_advertisement

logging.warning(f"starting observer")
observer.start()
logging.warning(f"listening for events and publishing to MQTT")
while True:
    # unsure about this loop and how much of a delay works
    sleep(1)
observer.stop()

And for the MQTT helper class (mqtt_helper.py):

# basic MQTT helper class. really needed to write one of these to simplify basic MQTT operations
# written/modified by Austin of austinsnerdythings.com 2021-12-27
# original source: https://gist.github.com/fisherds/f302b253cf7a11c2a0d814acd424b9bb
# filename is basic_mqtt.py
from paho.mqtt import client as mqtt_client
import logging
import datetime
logging.basicConfig(
	format='%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',
	datefmt='%Y-%m-%d %H:%M:%S',
	level=logging.INFO)


mqtt_host = "mqtt.home.fluffnet.net"
test_topic = "mqtt_test_topic"

# this is really not polished. it was a stream of consciousness project to pound
# something out to do basic MQTT publish stuff in a reusable fashion.
class basic_mqtt:
	def __init__(self):
		self.client = mqtt_client.Client()
		self.subscription_topic_name = None
		self.publish_topic_name = None
		self.callback = None
		self.host = mqtt_host
	
	def connect(self):
		self.client.on_connect = self.on_connect
		self.client.on_subscribe = self.on_subscribe
		self.client.on_message = self.on_message
		logging.info(f"connecting to MQTT broker at {mqtt_host}")
		self.client.connect(host=mqtt_host,keepalive=30)
		self.client.loop_start()

	def on_connect(self, client, userdata, flags, rc):
		print("Connected with result code "+str(rc))

		# Subscribing in on_connect() means that if we lose the connection and
		# reconnect then subscriptions will be renewed.
		#self.client.subscribe("$SYS/#")

	def on_message(self, client, userdata, msg):
		print(f"got message of topic {msg.topic} with payload {msg.payload}")

	def on_subscribe(self, client, userdata, mid, granted_qos):
		print("Subscribed: " + str(mid) + " " + str(granted_qos))

	def publish(self, topic, msg):
		self.client.publish(topic=topic, payload=msg)

	def subscribe_to_test_topic(self, topic=test_topic):
		self.client.subscribe(topic)

	def send_test_message(self, topic=test_topic):

		self.publish(topic=topic, msg=f"test message from python script at {datetime.datetime.now()}")

	def disconnect(self):
		self.client.disconnect()

	def loop(self):
		self.client.loop_forever()

if __name__ == "__main__":
	logging.info("running MQTT test")
	mqtt_helper = basic_mqtt()
	mqtt_helper.connect()
	mqtt_helper.subscribe_to_test_topic()
	mqtt_helper.send_test_message()
	mqtt_helper.disconnect()

Results

Running this script on a Raspberry Pi 3 shows the advertisements coming in as expected. The updates are very quick compared to the usual 16 second update interval for my Acurite stuff.

screenshot of Python code running to receive BLE advertisements from Govee Bluetooth Thermometer
screenshot of Python code running to receive BLE advertisements from Govee Bluetooth Thermometer

And running mosquitto_sub with the right arguments (mosquitto_sub -h mqtt -v -t “govee/#”) shows the MQTT messages are being published as expected:

mosquitto_sub showing our published MQTT messages with temperature/humidity data from the Govee sensor

Getting Govee MQTT data into Home Assistant

Lastly, we need to add a MQTT sensor to get the data importing into Home Assistant:

- platform: mqtt
  state_topic: "govee/A4:C1:38:9F:1B:A9"
  value_template: "{{ value_json.temp_f }}"
  name: "garage temp"
  unit_of_measurement: "F"

And from there you can do whatever you want with the collected data!

Home Assistant displaying data from Govee Bluetooth temperature and humidity sensor

Conclusion

This was a relatively quick post and code development. I really hate the cycle of developing on my Bluetooth-less Windows computer, committing the code, pushing to Git, pulling on the Pi, and running to “debug”. Thus, the code isn’t as good as it can be. I probably did 20-25 iterations before calling it good enough.

Regardless, I think this $12 Govee Bluetooth Thermometer and Hygrometer is a great little tool for collecting data around the house. You don’t need an SDR to get Acurite beacons, and you don’t need to spend a lot either. You just need a way to receive BLE advertisements (basically any Bluetooth-capable device can do this). There is even a 2 pack of just the sensors that I just discovered on Amazon for $24 – 2 pack of Govee bluetooth thermometer and hygrometer. I know there are other Bluetooth devices but they’re quite a bit more expensive.

Disclosure: Some of the links on this post are Amazon affiliate links. This means that, at zero cost to you, I will earn an affiliate commission if you click through the link and finalize a purchase.

Also, at least for me, these are available with same day shipping on Amazon. Scratch that instant gratification itch.

Same day shipping available on Amazon near Denver for the Govee sensors