Battery Monitoring – You can never have too much data

I have been monitoring battery status with a pair of Thornwave battery monitors. They work OK, but because they do not log data. I have to be within Bluetooth range to check the batteries. There is no history, so I have to guess what happened when I’m not around.

Ideally I could continuously log data from the battery monitors & upload the data to something always available – like a cloud. There is one commercial app (RV Whisper) that does what I want, but it requires a subscription. I want to do this on the cheap.

Cheap is also the reason why I don’t simply replace my Thornwave monitors with something like Victron monitors & use the Victron control panel and cloud to log and display data. That will work – but for what it would cost to purchase I could buy another $950 lithium battery and quit worrying about power.

So plan ‘A’ is to use what I got and get to where I want without spending real money.

Reading a Thornwave

A bit of Google Bing searching found someone who figured out how to slurp up data from Thornwave monitors via Bluetooth and make it visible in another monitoring application. So the hard part is done.

The next question is how to platform whatever I build. I thought hard about using an old Android phone. Power consumption is low, but scripting language options are few and not very capable. Without scripting, even simple application programming is a really heavy lift. Raspberry Pi’s draw very little power, run from an ordinary USB port, have a plethora of scripting languages available, and are only $29.

I had a day to kill last week so I ran over to MicroCenter, bought a $29 Pi 3B+, installed the default operating system and learned enough Python to be dangerous. Of course I had to re-remember how to use git and github. That took longer than learning python.

By the end of the day I had a skeleton of an app that could connect to a Thornwave via Bluetooth and read a dozen of the available sensors and data elements. A couple more days of part-time hacking and I had a python script that can read a Thornwave and output data in a couple of useful formats (human readable, tab delimited, and JSON) tested and checked into github and running on a Pi as a cron job every 5 minutes. This was fun because it’s been many years since I’ve written anything & I got to exercise a few rusted out brain cells.

Sample outputs:

The data will by much more useful if I can access it anytime, anywhere. To do that I set up the Pi so that I can SSH to it from anywhere using a reverse SSH tunnel from a cloud-hosted Linux server that I use for other projects. The Pi automatically SSH’s into the Azure cloud hosted server using autossh and creates a reverse TCP tunnel from the Azure server back to itself on port 22 (SSH). To access the Pi, I SSH to the Azure server and tunnel an SSH session from the Azure server back down to the Pi. I don’t need to know the IP address of the Pi or be on the same network and I don’t need to open any firewall rules on the hotspot that I keep in the camper.

Graphs, Charts and Clouds

I want some sort of visualization of the data. For me, a visualization is far better than raw data. I tossed around a couple of ideas and started out with (1) use good ‘ol RRDtool to store and graph the data, and (2) build a simple Azure Logic App that presents a JSON API to the Internet that inserts the data into something useful (like an Excel spreadsheet) that can chart the data.

RRDtool

RRDTool was the easiest to set up. I’d been using it for a couple of decades, so after an RRD refresher I wrote a python script that takes the output from my Thornwave script and inserts it into an RRD database. From that database I can generate a variety of charts & output them as PNG’s.

Here are a couple of examples – one showing battery voltage for both batteries and one showing current.

You can see on the left side of the graph where the solar panels tried to charge the batteries Saturday during the day, the center of the graph where I charged the batteries using wall power Saturday evening, and towards the right where the battery voltage dropped overnight as the batteries were depleted. You can also see the impact of the refrigerator cycling. The current draw increases (more negative) and voltage temporarily drops while the compressor draws current.

When I was charting this data, the lithium and AGM batteries were tied together with a Schottky diode – something that I’ll write about someday.

Azure Logic App

Setting up a Logic App wasn’t too difficult either. I used the Microsoft Graph API to ‘discover’ the location of my OneDrive hosted Excel spreadsheet and used the Azure GUI interface to create a Logic App that accepted JSON data and added it to the end of an Excel table. I wrote a python script takes the data from the Thornwave script, formats it as JSON, and calls the Logic App API. The rows show up in Excel every time they are generated on the Raspberry Pi and the formulas and charts in the spreadsheet automatically update as the data is inserted.

I now have dynamically updated charts any time I open up the spreadsheet. This works from desktops, phones, tablets, whatever.

Here’s a sample Excel graph showing roughly the same data as the RRDtool charts above. The annotations are added by me.

Enter Blynk

Jason, in his Everlanders YouTube channel has taken the art of camping with technology to an extreme, with Raspberry Pi’s, Arduino boards, and custom circuit boards, software and electronics. As in ‘OK Google, turn on the fan’. I’m not to the point of using microcontrollers to manage the camper (yet), but I did observe that the user interface to his technology is the IoT platform ‘Blynk

Blynk is easy to use and for simple projects, free. So along with the RRD and Azure Logic App/Excel data stores and charts, I decided to see if Blynk would make life easier. I wrote a python script that uploads the data from the Thornwave script to Blynk and used the Blynk android app to build a real-time dashboard with widgets for battery status in Amp-hours, battery status as a percent of full, and a real-time chart of battery voltage and current for both battery banks.

Camper dashboard (as of today…). Shows Amp-hours consumed for the AGM and Lithium batteries (top -4.05Ah and -0,87Ah), percent of battery capacity (gauges – 98% and 99%), voltage both batteries (chart at center of dashboard – AGM Voltages between 13,5 & 13.7, Lithium between 13.6 and 14.5) ) and current (bottom – AGM = 7A to +29A, Lithium = 0-+15A).

Blynk stores whatever data I send it in their cloud and the app uses their cloud as its data source, so I have real-time status anytime my phone is on any network.

I’m not sure which of the three systems I’ll maintain and use. Blynk is the prettiest, but without an expensive subscription its capabilities are limited. RRDtool is free but clunky. Logic APP + Excel is pretty heavy weight.


Posted

in

,

by

Comments

12 responses to “Battery Monitoring – You can never have too much data”

  1. Fred

    This is an interesting project. I have put together two RPi (Zero & 3B) and both work fine. The RPiZ should draw less power which is desirable. One will stay connected in my camper and the other in a portable battery assembly. Getting the data to my phone is next. I am hoping to not use any cloud based services since it is not always online.

  2. I’ll abandon the Azure Logic App – it’s not doing anything that rrdTool and Blynk can’t already do.

    If I don’t have hotspot Internet, the rrdTool still runs and gathers data locally, but the other services simply have blank data. If I wanted to avoid cloud services, I could run a local Blynk server on the RPi and connect to it via local hotspot WiFi. I’m thinking of doing that.

    1. Fred Hillhouse

      I had a thought this morning. I often use a tablet to connect to another RPi with a VNC client. I could do the same here. I could install OpenOffice (or whatever the latest is called) which could import and chart the data.
      I will look closer at rrdTool and Blynk now that I am at this stage.
      What rate are you sampling the data?
      Thank you!

      1. For now, my samples are at 5 minutes. I can sample more, but I’m not sure if it’s a good idea to hit the thornwaves that often.

        I figured that if I wanted up to the second data the thornwave app would be appropriate.

      2. Fred Hillhouse

        I have currently two lines in my crontab file, 1 minute and 10 minutes. I haven’t actually tried the 1 minute yet.

        Do you think you might add your RRDtool work to Github as well?

        Thank you!

      3. Here’s my notes. I didn’t check the scripts in – they are pretty trivial.

        https://github.com/mkjanke/ThornwavePy/blob/master/RRDtool%20Notes.md

        In my implementation the cron job pipline also pipes into another script that uploads to Blynk, so reading Thornwave data, updating the RRDTool data files, and calling the Blynk API are all one pipline in one cron job per battery bank. The RRD charts are generated separately at longer intervals.

        I use rclone to copy the charts up to my OneDrive account so I can see them from any of my devices.

        RRDTool is very flexible – but the docs are not very well organized. I used it for 20 years at my day job running a large network, monitoring hundreds of network devices and measuring and creating thousands of charts every 5 minutes. It still leaves me flummoxed at times.

        –Mike

      4. Fred Hillhouse

        Thank you! I will study and hopefully get success. Perl, Python, scripting in general, is new to me.
        Best regards and happy trails!!

  3. Fred Hillhouse

    Your work and those ahead of you have been a great help!

    I added a new switches to your Python script to package the data into two different packets. It still needs a bit of work to clean it up. My current method works but it could be improved.

    Another application calls the script and transmits the data over RF (2M amateur radio). It ends up being added to a database which I can view online.

    Thank you!
    Best regards,
    Fred

    1. FYI – I modified the script to eliminate gatttool and use the bluepy library instead.

      https://github.com/mkjanke/ThornwavePy

      Gatttool was failing to connect fairly often. I’m hoping bluepy will be more reliable.

      I was licensed a long time ago, back when vacuum tube Drake gear was king of the hill. I just sold a whole bunch of 1970’s Drake HAM gear that had been sitting unused for 35 years. 🙂

  4. Chriss Koch

    Pretty nifty idea here. I’ve got an RV at a remote location and I’d like to keep tabs on its LiFePO4. Mine is already equipped with a Thornwave PowerMon-5S, and I already have a few unused RPi Zero W. So I thought “this should be pretty easy”. But, for some reason I am not having any luck. While using the verbose flag I get “Read failed”. However, bluetoothctl and gatttool seem to work just fine. I’ll keep poking at it, but any ideas are welcome!

  5. A couple things:
    Thornwave firmware newer than a certain version uses different BLE characteristics.
    Getting BLE and python to play nice together is a headache. I’ve tried different versions of python, of the various libraries, and of bluez. It’s still not clear to me which works the best.

    Right now I’m using newest versions of bluez and python, custom compiled. Seems to work most of the time.

    1. Chriss Koch

      Thanks for the quick response! On the Thornwave firmware, mine (PowerMon-5S) does have the newer version, and I manually verified that the BLE characteristics in my PowerMon-5S match those in your code. I also manually verified that your code properly unpacks the raw hex data from my PowerMon-5S.

      So, I think you’re right about my problem being something related to my python and/or bluez. It’s a 40 mile trip into the mountains to visit my RV–not really convenient for troubleshooting during the winter months when I’m not spending as much time up there. But I’ll get it figured out.

      My interest in this sort of came about while trying to figure out a way to keep my Starlink firmware up-to-date. At my remote camp site I have the original Starlink hardware and it’s a power hog, so I can’t afford (from a solar and battery capacity perspective) to leave it on all of the time. OK then, it was simple enough for me to hack together a solution to automatically power up the Starlink every other day for an hour. But as you know, these “simple” projects often develop a life of their own. Having the Starlink running every other day tempted me to use that connectivity (other than just for automatic firmware updates)…and at that point the monster had been awakened.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: