Bluetooth Environmental Monitoring

I love the idea of being able to get the status and history of temperatures and humidity around my house. You can use the data to infer certain things like when a shower is being taken, a door/window being left open, or when you’re heating a room you don’t need to. I can use this data to re-balance my forced-air system as well.

Please understand that I do not recommend the use of unencrypted wireless devices in any way, shape, or form for control systems. There are major safety vulnerabilities that can crop up from otherwise, well-written code. The data generated from these devices is used to help me make an informed decision about my environmental control. The data is not tied into a control loop at all.

The units that are available on the market are typically bluetooth and require an app to read the data. WiFi units send the data off to “the cloud” and likely log everything, but can be accessed by a phone/browser.

So what’s a good compromise between designing my own hardware and giving all of my information away to China? Enter Aaron Christophel!

I saw a post on Hackaday about Aaron Christophel modifying the firmware of $2 Xiaomi thermometers from Aliexpress. So I went ahead and bought 16 of them to scatter around the house.

The problem Aaron is trying to address is the fact that these thermometers obfuscate temperature and humidity data contained within their broadcast information. There is a way to pull this data out if you know the token and bind key (something the app does). But Aaron’s firmware takes out the need for bind keys and tokens and instead lets the thermometers broadcast the data as plain text so to speak. No app, and a simple python script can read the data.

Aaron wrote a really nice bootloader that uses google chrome and your computer’s bluetooth radio to flash new firmware on to the units. It’s a little slow taking about 1 minute to connect, activate, and re-flash the unit’s firmware.

Step 1: Preparation

Out of the 16 units I purchased, I only ended up deciding on locations for 13 units. Deciding on a location is the first step. Next you’ll transfer all of the locations to an excel spreadsheet as a column. Next door to the locations column you’ll put the mac address column. This will help you form an array later for matching up broadcast mac addresses with the rooms the devices are in.

Step 2: Re-Flashing Firmware

I’m condensing Aaron’s instructions here, but a full how-to exists on the project’s github page. First, download the ATC_Thermometer.bin file. This is the new compiled firmware that Aaron wrote. Okay now is when you use Aaron’s web-browser DFU tool. Select the .bin firmware file, connect to the thermometer, activate the thermometer, and then start flashing.

Keep an eye on the progress bar, and when it gets up around 90% watch the bottom right two digits on the thermometer. They will flash the last 3 bytes of the mac address in hex. Write these six characters down as you’ll need to pull the battery and put it back in to get it to display the mac again. If you have lot’s of devices it’s important to have the mac address written on the unit.

Before writing the address on the unit however, connect to the device via bluetooth and change the appropriate settings that make you happy. For example, I set my devices to display Fahrenheit, and to show the battery in the LCD. We do this step first to double check the mac address because the number 6 displays as the letter b and that can introduce problems if you read and wrote the address incorrectly.

Once you’re sure you’ve got the correct mac address for the correct device, go ahead and scribble down the address on the bottom of the unit. I chose the bottom because I could still read the address once they’re adhered to the wall. Add this mac address to your spreadsheet and now you have a relationship established between the room and the mac address.

Step 3: Parsing and Aggregating Broadcast Information

Tom Nardi has a good writeup exploring the software side of this project, including some example code to get you started reading data off of the sensors. I’ll post the full code below as I had to make several modifications to filter out redundant information his code would be saving. My script also saves the data to an array of verified sensors and then writes that to a file. This is important because the py-bluetooth-utils runs until keyboard interrupt, and the devices will all update their broadcast information out of sync. This isn’t 100% the best way to do this, but python really isn’t good at running continuously if your script is super bloated.

I wrote a script that runs on startup and simply writes all values broadcast by the thermometer to a 2Dx1 array and then writes that information to a file. This has been running stable for over a month now on a raspberry pi zero w.

Step 4: Publishing MQTT Information

If all you want to do is log this data to a CSV file then you’re pretty much done here, except for generating a CSV file. But if you’d like to get the information into a web service, I’d recommend using mqtt since it’s a fairly lightweight protocol and Home Assistant plays nice with mqtt. I wrote a second script which reads the output file written to by the BLE listener, and parses it into MQTT topics with data. A chronjob runs this second python script which publishes data once per minute (which is the refresh rate of the sensor).

Step 5: Receiving and plotting the sensor data in Home Assistant

If you just want the data aggregated in a place that you have control over and can plot then Home Assistant is a pretty good option. There’s also plugins available for HASS such as GRAFANA if you want more graphing options. I’m not going to go into Home Assistant setup but will outline how to install and configure the Mosquito MQTT Plugin and add the published data as a “sensor”

Step 6: Viewing and acting on Data

Since these sensors are not part of a control loop, we’ll use the information to make manual adjustments.

As you can see here,

My Movie Room (upstairs bedroom) temperature is about 6 degrees hotter than the rest of the house. While this doesn’t look so odd at first glance, I’ll say that the movie room is heated with electric baseboard heating. This means we’re spending more money heating a rarely-used room above what we consider a comfortable winter temperature. Clearly a point of cost savings.

Another thing you can do is determine if a room has proper ventilation. Below is a historical line graph of the relative humidity of the master bathroom.

See those spikes? Those indicate a shower was taken. The current relative humidity outside is 40%, so seeing the spikes return to “baseline” in a timely manner, indicates good air flow. Since our master bathroom doesn’t have a vent fan, this is an important variable to monitor to combat mold growth. From the looks of this, we don’t need to install a vent fan.

These sensors are marketed to last about a year on one coin cell. I’m sure their battery life could be extended by changing the broadcast interval in the firmware, however, planned sensor maintenance can be carried out as the sensors are broadcasting their battery level!

We can even determine a failure based on current time and the Last RX Time variable. If it’s last reported battery level was 50% and the last RX time was 2 days ago, the unit is probably dead or the scanning computer can’t pick it up anymore.

Closing Comments

So why all this hassle? Why not just use the xiaomi app and not hack the sensors? You’ve seen that this data can be used to infer more than just the temperature of a house. A company can gather some serious data about your behaviors all the way down to your showering habits, yikes!

Update: 2-21-21

I’ve started using Grafana and InfluxDB to store and graph my information. It works much better than the built-in graphing in Home Assistant.

Solidworks Musings: Design tables and bulk-design.

I’m what you might call a casual mechanical engineer. By trade, I’m an electrical engineer, but there’s something cathartic about designing parts in solidworks. I started off in high school and didn’t think much of it, but as I went through college I contracted out my skills to model up jewelry for clients. This jewelry was then 3D printed and cast into final products. Don’t worry, I actually was dumb enough to drop several thousand dollars on a “personal” copy of solidworks.

It was an enjoyable experience for both myself and my clients as we could hang out, drink scotch, and work through what exactly they were looking for. Eventually a design came up where they needed variations on a theme so to speak. Small changes in the design, but the overall concept was the same. Learning about design tables saved me so much time as a result. A guy can alter variables in an excel spreadsheet and solidworks will build derived configurations with these new variables.

Fast forward 5 years and I’m designing a part that helps me position my chopsaw back clamp at specific angles.

DEWALT Chop Saw, Quick-Change, 14-Inch, Old Model (D28715) - Cut Off Saw -  Amazon.com
You get increments of 10 degrees out of the scale built into this thing.

Getting my inspiration from machinist’s angle blocks I designed a part that fits in the track the front clamp rides on and butts up to the back clamp.

It’s not pretty, but it’s 3D printable.

Okay here’s the thing. I have the solidworks part reliably re-building from 0 degrees to 45 degrees. This means I can feasibly make any angle I’d like without throwing any weird solidworks errors. I don’t want to manually enter the angle I want, and then revise the text for all of the infinity of angles that exist between 0 and 45 degrees. I also want to generate a pallet of blocks that step from 0 to 45 degrees in increments of 5 degrees. I ALSO want the text on the side of the block to update with the angle the block was set to so I don’t have to go in to every derived configuration and update the text.

Here’s how we do this (keep in mind I’m running SW14, so your mileage may vary).

  1. Make your part how you want it, and rename your “primary value” of your smart dimensions that you want to change to something meaningful eg: “set_angle@Sketch3”
  2. File->Insert->Tables->Design Table…
  3. You should now be prompted with a list of primary values from your part. This is why you took the time to rename your primary value. Select “set_angle@Sketch3” and excel will automatically populate a new column with that name at the top and the value of that directly below.
  4. You can make as many new configurations as you’d like by adding another row. You will have to name them by entering the name in the column where “Defult” resides. I’d recommend something meaningful.
  5. This should successfully generate x number of spin-off parts with the values of smart dimensions changed appropriately.
  6. To get sketch text to update as a variable go to file->properties…->configuration specific
  7. Make a new property named “text”, type: text, value: 69420
  8. Go to your extruded cut or boss and edit the sketch text as so: $PRP:”text”
  9. Go back to your design table by right clicking on it and edit it. It should prompt you asking if you want to add the “text” property you just made.
  10. Now your angles and text values reside as variables in an excel table!

Use this to create as many variations as you want of your model. These derived configurations can then be loaded into an assembly with ease. This makes palleting up parts for 3D printing easy.

After inserting your part into an assembly you can right click and chose the derived configuration you want.
My “fleet” of angle guides.

I inserted a sketch of my build plate are into my assembly as a guide for how many I can fit on to the build plate at once. From here you can generate an STL from the assembly as one part to save you hassle from having to get things stacked up in Cura or whatever.

File->save as->.stl for the type->Options…->Check “Save all components of an assembly in a single file”

Import this STL into cura and now you have a pallet of different models you can print!

Fits Nice!
Having a little first-layer adhesion problem. Going to dial my printer in more before I do a 17 hour print of all of the other blocks.

Here’s an angle block in the saw. Works like a charm!

Now don’t go expecting to get decimal-place-precision out of your chopsaw. But these guides work way better than the scale on the saw itself and can give you reproducible cuts if you need to adjust every time you cut.

Sources:

https://centralinnovation.com/technical-resources/learn/add-sketch-text-to-design-table/

WoWLAN With a Raspberry Pi

Post In Progress!

Starting this blog post so I’m forced to follow through on my hare-brained idea.

Okay so picture this: You have a license to a brand-spanking-new copy of Solidworks 2014 sitting on your desktop computer. You live in the mountainous South West, it’s summer, and the idea of air conditioning never caught on in your town so you try to keep all of your electronic equipment off at all times unless it’s in use.

Finally! Relief! You find yourself a cool dark place to set up shop for the day and open your laptop so you can remote into your desktop back at home and do some designing on Solidworks! But wait. Your computer isn’t on, and home is a half hour away! Oh no! I wish there was some way to leverage this internet-of-bullshit to my advantage.

Well say hello to WoWLAN! Wake on Wireless LAN. This is the greatest idea that will never catch on. And unfortunately for me, it’s not available on my PC. Well, it is, but only from hibernate mode and windblows10 likes to wake up my PC from hibernation periodically so that option is OUT. Time for some hardware.

Step 1: Raspberry Pi

I’m purchasing a raspberry pi zero w for this project because they’re small and have wireless built in. Unfortunately they’re the slowest of the Pi’s so Raspbian isn’t going to work. Go ahead and install dietPi. It’s worked great for me and has low overhead.

Step 2: Software

Sudo apt-get install python3 and paho-mqtt on your Pi and download MQTT Dash on your smartphone. From here you can write a python script that connects to the MQTT test broker and you can start sending and receiving messages and debugging any issues that come along the way. I am leaving my code unpublished at the moment because it isn’t robust enough to deploy. But it’s in my best interest to fix it so it’ll be released eventually.

Step 3: More Hardware

It’s kind of a dick move to use a public MQTT broker (also insecure) to handle your web traffic so get a piece of hardware that is web-facing that you can install Mosquitto MQTT on. This can be a webserver you have or, in my case, an AWS instance. You get a free year of the t2.micro after that the t2.nano should run around $50/year and work just fine. If I ever figure out how to field Mosquitto on my lightsail instance it’ll save me from paying double for AWS services. I’ll update that here.

Be sure to use DuckDNS and your domain to adapt to your dynamic IP address. Also don’t forget to start Mosquitto MQTT on boot.

Step 4: Even more hardware

I started out with the idea of using a solid state relay to emulate a button press. Here’s my schematic:

And here’s my PCB.

Here’s My Bill of Materials:

Step 5: Integration

Build up your PCB, solder it on to the raspberry pi and install it on your PC.

Hey, you’re done!

Good Luck!

-Conrad

Amazon AWS Minecraft Server (With Remote Start of Course)

My wife and I set up a little rinky dink minecraft server on one of my Toshiba laptops. Unfortunately it’s starting to suffer from reliability issues. Instead of building a computer/server from scratch I decided to pay (or not) for the convenience of an AWS server. This should be cheaper than the upfront cost of a new computer + electricity + my time and effort setting up and maintaining it. Of course owning a server will be cheaper in the long run, but I simply do not want one in my house.

These servers have an hourly rate of $0-$0.22/hr. This project will outline my steps towards creating the minecraft server and implementing a method to save money when it’s not in use.

Requirements

  • Must host a minecraft server
    • Up to 7 players
  • Must shut down when not in use
  • Minimal effort in starting it back up
    • Needs to be accessible by all friends
  • Easily re-accessible

Plan to execute

  1. Establish an EC2 Instance
  2. Transfer over minecraft files
  3. Write Duckdns cronjob to update the IP address on startup
  4. Write minecraft server auto-start script
  5. Write a script that checks active connections on port 25565 (minecraft) and 22 (ssh) and shuts down the server if it’s not being talked to for over an hour
  6. Set up a webpage that lets users log in and turn on the server remotely

1) EC2 Instance

Starting an instance easily done by creating an amazon AWS account and following the tutorial below.

There is enough info here to get this thing running but we’ll have to deviate from the path to get it running on it’s own. The cool thing here is you can start in the free tier and if you’d like to upgrade the hardware, all you have to do is change the instance type.

2) The Transfer

Since I had a minecraft server already working, I figured I’d just delete the install on the AWS server and copy mine over. This was particularly difficult as I’m currently in Alaska and my computer is in New Mexico. It doesn’t like to stay alive longer than 20 or 30 minutes. What I did was zip up the entire minecraft server folder and transfer it as one chunk. Previous transfers failed mid-way through and I think a few files got corrupted. I got it transferred over thanks to the help of Jake and Melissa who kept rebooting my laptop!

Note for me for future scp transfers:

scp user@originalserver.com:/opt/zippedserver.zip /opt/destination_folder

3) DuckDNS Cronjob

This makes connecting easier as each time the server turns on, it gets a new ip address and having a name to connect to is a lot easier than copy-pasting the new ip address every time you start up the server.

I use DuckDNS because it’s free and has been working for me. Use whatever flavor you’d like for your dynamic DNS reroutes.

Here is the tutorial for “installing” DuckDNS as a linux cronjob.

4) Minecraft Auto Start

I had one of my nerd friends (Jake) get this working on my original rig. So this will be a challenge for me.

This tutorial seems to work just fine but I had to make a couple edits and I’ll clarify some of the code in it.

4.1) Writing a service

navigate to /etc/systemd/system/ and create a new file using your favorite text editor

sudo nano /etc/systemd/system/minecraft@.service 

I use nano, bite me. Also, insert the following text into the file:

Note and then omit the comments in parentheses.

[Unit]
Description=Minecraft Server: %i
After=network.target

[Service]
WorkingDirectory=/opt/minecraft/%i 

(THIS WORKING DIRECTORY IS IMPORTANT, PUT IT 1 FOLDER BACK FROM YOUR server.jar FOLDER)

User=minecraft (MAKE SURE YOU HAVE SET UP A USER WITH APPROPRIATE PRIVILEGES)
Group=minecraft (MINE IS DIFFERENT. DELETE COMMENTS IN PARENTHESES)

Restart=always

ExecStart=/usr/bin/screen -DmS mc-%i /usr/bin/java -Xmx2G -jar minecraft_server.jar nogui

ExecStop=/usr/bin/screen -p 0 -S mc-%i -X eval 'stuff "say SERVER SHUTTING DOWN IN 5 SECONDS. SAVING ALL MAPS..."\015'
ExecStop=/bin/sleep 5
ExecStop=/usr/bin/screen -p 0 -S mc-%i -X eval 'stuff "save-all"\015'
ExecStop=/usr/bin/screen -p 0 -S mc-%i -X eval 'stuff "stop"\015'


[Install]
WantedBy=multi-user.target

Start the service with this command.

sudo systemctl start minecraft@SERVERFOLDER

I then use the top command to ensure Java is running. Next you need to enable the service so it runs on boot.

sudo systemctl enable minecraft@SERVERFOLDER

This should work, but be sure to check the status as follows.

sudo systemctl status minecraft@SERVERFOLDER

If it looks good, go ahead and shut down the machine (this stops the instance on the EC2 control panel). When the instance is fully stopped, start it back up from the control panel and refresh your server list in minecraft. It takes a minute or two as I don’t think the dns changes propagate right away.

At this point, you have a fully functional minecraft server! If you have the urge to spend $2000/year on hosting, you’re all done! Pat yourself on the back. If you’re ready for a challenge continue on!

5) Auto Shutdown

This is really easy (in concept) as we don’t have to figure out how to talk to AWS, we can just write a script on the machine to turn itself off. I’m personally not worried about 25 cents, so the script will turn on only once an hour and check port 25565 and 22 for active connections.

First, let’s make a bash script called check4users.sh and put it in it’s own folder inside of your minecraft directory.

mkdir scripts
cd scripts
sudo nano check4users.sh

Throw a shebang on the first line of the file

#!/bin/bash

Close, and save the script. I know! We haven’t written anything yet! Let’s chmod this first so it can run!

sudo chmod +x check4users.sh

Okay now, let’s return to the script! Paste this code into check4users.sh

#!/bin/bash
#Checks for active connections on  port 22 (ssh) and 25565 (minecraft)
#If no connections are found, the script gracefully shuts down the
#Minecraft server (saving progress) and then shuts down the Service
#This will stop the EC2 instance and save money.

sshCons=$(netstat -anp | grep :22 | grep ESTABLISHED | wc -l)
mcCons=$(netstat -anp | grep :25565 | grep ESTABLISHED | wc -l)
echo "Active SSH Connections: $sshCons"
echo "Active Minecraft Connections: $mcCons"

if [ $((mcCons)) = 0 ]
then
        echo "We normally shutdown here, but let's check for SSH connections"
        if [[ $((sshCons)) = 0 ]]
        then
                echo "no ssh connections, shutting down server"
                sudo systemctl stop minecraft@christian
                sudo shutdown
        else
                echo "There are 1 or more active ssh connections, skip termination"
        fi
else
        echo "Somebody is playing minecraft, do nothing!"
fi

Next (again as root) add this script to your crontab.

crontab -e

It’s vim (ick) so press i to insert text and paste the following text.

0 */1 * * * /opt/minecraft/SCRIPTDIRECTORY/check4users.sh

Press esc to stop your text insertion. Then “:wq!” it’ll save and update your cronjobs.

This will run the check at the top of every hour. If you need to change this time, make it no less than every 10 minutes. I set it to every 5 and had some trouble catching the server before it shut itself off (remember, the server takes time to boot, and duckdns takes time to update).

One flaw in this script is that if you turn on the instance at, say, 7:58pm. The server could boot, run the chronjob at 8pm, see nobody connected, and shut itself down. I’ve only experienced this problem once so far, but I just don’t want to go down the rabbit hole of re-writing this.

6) Starting the server back up

Now that we have the server running minecraft on startup and then shutting down when nobody is playing it we need a way to start it remotely. I can do this by logging into amazon aws and clicking start instance.

However, I don’t want to answer every text message of “I want to play minecraft”. I also don’t want to share my amazon aws login information. So, let’s create a website that a person can log in to and start the server from there.

Requirements:

  1. A website (I think www.farnsworth.engineering/minecraft will do!)
  2. SQL Database
    1. Stores user information
    2. Stores some usage data from the AWS instance
  3. PHP Experience
    1. Sends a request through the AWS SDK to start, stop, and restart the server
    2. Handles login and account creation.
  4. XAMPP + Brackets
    1. This let’s you test out this code outside of production.

Rant: Long story short, I got this part working a long time ago. However, my hosting provider Arvixe has been dragging their feet for two weeks to solve a simple clock problem. It was literally faster for me to pack up everything and migrate it over to Amazon Lightsail than it was for Arvixe to update the system clock. End of Rant. Unfortunately my old blog got nuked…so here’s my first post.

Let’s get started!

6.1) Establishing a Database

Let’s make a database! In plesk it looks a lot like this:

Go ahead and name the database minecraft, related site: example.com, username: minecraft, generate the password (and copy it down). These will be your credentials for accessing the database later on. If you’re going to prototype this code with XAMPP you can use phpmyadmin to make the database. Do yourself a favor and don’t test your code in production.

Okay, now we need to create a table in phpmyadmin (webserver or local machine) called users. It’ll have 4 columns. “username, password, email, activated”.

6.2) PHP

This part has been bootstrapped off of a few other projects, but works fine. Download this zip file (minecraft_server.zip). Fair warning: the HTML is garbage. But this provides the PHP necessary to talk to AWS.

Speaking of talking to AWS, download the SDK here (don’t worry it’s from AWS).

READ THE DEVELOPER GUIDE.

Unzip both minecraft_server.zip and aws.zip into your webserver/local server.

Things you need to provide in server.php:

  1. Remember to edit the credentials you copied from your database into line 5.
  2. Line 15 of server.php should point to the location of the SDK file. It’s a little tricky to figure out at first, but if you turn on debugging, the error message should give you the basic directory structure of your hosting server.
  3. Line 25 and 26 need to be security credentials you generated from your user account.
  4. Line 30 is your particular EC2 Instance ID.

Extras:

  1. Remember the column you made in your database named “activated”? You’ll need to manually set it to value 1 from phpmyadmin. This keeps random yeahoos from making an account and starting your server. If you’re interested in a challenge you can have your server email you every time an account is created and you can click a link to approve/disapprove the account. I figured with just 7 players it wasn’t much of a hassle.

2. Server.php has the functions “stop” and “restart” in it. I omitted them as buttons from index.php to keep people from stopping the server while you’re playing. If you need to restart. You can log into your control panel yourself and do it. However, if you trust your players, you can add it in or put another column in your database as “administrator” or something like that and just give yourself permission. Worst case scenario, your players can log off and the server will shut down at the top of the hour.

From here you should have a working website that lets you create an account, log in, and turn on your AWS minecraft server!

Congrats!

Now write your own CSS and spiffy up the website!

UPDATE!

I’ve since Spiffied up the website with CSS (I bought a theme a while ago for another project) and recycled it into this.

The # of users is of course the number of rows I have in my users database. Easiest variable I’m importing here.

Runtime is calculated by incrementing a variable by 1 every time the check4users.sh script is ran. The result is sent via a post request to my webserver and that number overwrites the most current value of runtime in my database. This is a little janky I know. But it allows for asynchronous updating of the table without calling the Cost Explorer Service every hour.

Spent dollars is calculated using the AWS cost explorer sdk (included in the AWS folder). THIS CAN GET PRICEY. BE CAREFUL. The cost explorer charges your account $0.01 for every request. I have a separate php file that runs via crontab on the webserver once a month and updates a new table in my minecraft database. This keeps my users from charging me a penny every time they log in, to update a variable I already should know.

Saved dollars is simply the cost to run my instance every hour multiplied by the number of hours between when I started using this service and today’s date.

Thus far, I’ve saved $169 by using this service, and spent only $17!

$17 is actually an all encompassing amount as that includes costs for EC2+Lightsail+CostExplorer.