Category Archives: HEMS Project

The ‘Appiest Days of My Life

One of the consequences of integrating a smart home is the large number of different apps, web portals and potentially sources of APIs involved. The ones I use include:

TitleAppPortalAPIPurposeComment
BrightYNYReads and stores consumption from smart meter.No price data for my tariff due to smart meter limitations.
EveY /3NNEve’s alternative to Home for all HomeKit accessories with additional functionality for Eve’s own devices.I prefer this to Home for editing rules.
I use Eve products mostly for central heating control.
HomeY /3NNApple’s own app for the HomeKit smart home ecosystem.Need to refer to device manufacturers own apps (such as Eve or WIFIPLUG) for some configuration and data.
HEMSNYN My own web portal to view HEMS schedule and status via Apache web-server on Raspberry Pi.
MyImmersunYYY /1Control of ImmerSUN power diverter.Available API provides some measurement and status data as per main screen of the app.
PowervaultNYY /2Control of Powervault storage system.Available APIs provide some user scheduling and status capability.
OctoWatchdogY /3YYFuture cost, and historic costs and consumption (30 prior days) from Octopus (electricity supplier).APIs provided by Octopus.
App developed by an enthusiast using Octopus APIs.
Octopus’s own web portal provides historic consumption but does not pair this with cost. Monthly statements show graph of consumption and cost for each day.
WIFIPLUGYNYControl and measurements from own brand smart plugs.Plugs also appear in Home and Eve apps.
I use for dishwasher and washing machine.

Notes to table:

  1. APIs not officially released. Reverse-engineered by an enthusiast and available on line.
  2. APIs not officially released. Used as part of a sponsored trial when I first got the battery and re-used by myself with some manufacturer support.
  3. iOS only. Not available for Android.

Some of these apps have similarities:

  • Both Bright and OctoWatchdog show whole of house energy consumption (and potentially cost) derived from the smart meter. However they have differences too. A smart meter sits on two networks: (i) the Wide Area Network (WAN) via which the meter communicates with the energy supplier and (ii) the Home Area Network (HAN) which links the devices in the home (electricity meter, gas meter, CADs/IHD and gateway). Bright connects to the HAN via small piece of hardware called a Glow Stick Wi-Fi CAD and collects its own data in real time and stores its own records of energy consumption in the cloud; while OctoWatchdog involves no extra in-home hardware, and takes data a day in arrears from Octopus not storing anything in the cloud itself. Bright’s USP is the real time consumption and current day’s data (neither of which OctoWatchdog supports), while OctoWatchdog’s USP is the availability of electricity price which isn’t available from the meter.
  • Both Eve and Home interact with all devices in the whole HomeKit ecosystem. Eve is best for creating rules and has more ability to configure Eve’s own devices, while Home is best for sharing access with family members. WIFIPLUG’s app is more limited only interacting with their own devices, and thus cannot see Eve or other HomeKit devices.
  • Both MyImmersun and WIFIPLUG apps, and the Powervault portal, allow configuration of their own manufacturer devices. They all have, for example, timer capability and data logging. MyImmersun is better for giving a whole-of-home view showing solar panel output and net input to house (so provides a more comprehensive energy monitor), Powervault shows no solar panel output but does give a view of whole-of-home, while WIFIPLUG provides only a view of the energy consumption of devices plugged in to the WIFIPLUGs.

HEMS2 – improving the breed

It’s now around a year since I first started with my HEMS. Initially it was managing just the charging of my electric car around the cheapest electricity prices, but subsequently I added water heating, storage battery control and, most recently, dish washer and washing machine.

That original HEMS was built around a used Raspberry Pi that I bought cheaply from a colleague but, while its processing power was perfectly adequate for the task (which isn’t at all demanding), it did have some mechanical issues. Firstly, the mounting holes for accessory cards (known as HATs – HArdware on Top) moved between generations of Raspberry Pi and the holes in my Pi for the standoffs did not align with those in the newer accessory card. Secondly, I never managed to find a case with the space for the HAT and the mains cables that it switches. The former resulted in the electrical connector between the Pi and the HAT taking much of the mechanical load, and an occasional loss of connectivity to the HAT.

These mechanical issues are resolved by the new HEMS which uses a third generation Raspberry Pi allowing use of proper standoffs and a new case with the depth to enclose the HAT too.

Relay HAT on Raspberry Pi 3 (without standoffs fitted or insulation layer).
HEMS2 (the black box) replacing my original HEMS

As previously two cables connect to the HEMS – a black USB cable bringing 5V power and a grey multi core cable that brings a live mains feed to the relays with three switched lives returning to the adjacent junction box. The relays control my less intelligent loads – the ImmerSUN and the car charger – while other more intelligent loads are controlled via APIs via Wi-Fi and the cloud. The whole assembly continues to be mounted on a board on the side wall of the airing cupboard. When modification is required the assembly can be lifted off two screw heads and laid on the floor.

All the software on both generations of HEMS is the same except for the scripts that interact with the relays – either to set or read status. I assume that the pinouts on the Pi must have changed between generations as an identical HAT card moved from the ic0 to the ic1 bus requiring a single digit change to each interacting command.

Raspberry Pi images are licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license.
Attribution: Efa at English Wikipedia

Testing Times

So, over the last few days I’ve been adding control of my wet goods (dishwasher and washing machine) to my HEMS so that it can start washing loads at the optimum time – that is the times with the lowest projected electricity cost for a typical wash cycle.

Electricity prices for 18th and 19th December.

The above image shows the prices for December 18th and 19th as downloaded by the HEMS at 16:45 on the 18th after publication of prices for the 19th. The HEMS then reviews this pricing against the need for electricity to determine when to use electricity and, in the case of the battery when to discharge the battery. I currently have the HEMS configured to provide:

  • 3 hours of car charging
  • Not more than 2 hours of water heating (but only when electricity is cheaper than gas which can also heat water)
  • 6 hours of fixed battery charging.
  • Best start time for dishwasher and washing machine.
HEMS plan for 18/12 17:00 to 19/12 17:00.

I’ve described the operation of car charger, water heating, and battery previously; the new content here is the two final columns on each side for dishwasher and washing machine. The numbers in the columns are the estimated cost of a washing cycle if started at the beginning of the corresponding half-hour. The yellow colouring reflects the selected half-hour to start the appliance i.e. the one with the lowest estimated cycle cost. There’s also the option to set a threshold above which one is not prepared to pay to wash today which has resulted in the red box. If the whole day was red then the washing cycle would be deferred for consideration the next day.

History from Dishwasher WIFIPLUG

The extract from the WIFIPLUG history for the dishwasher shows the typical operating sequence:

  1. @22:19 I turn the WIFIPLUG on to enable the dishwasher to be programmed and the cycle started. The plug for the dishwasher is inconveniently at the back of a low kitchen cupboard so this was achieved via the WIFIPLUG app.
  2. @20:20 I turn the WIFIPLUG off via the app suspending the cycle in its first moments.
  3. @02:00 the HEMS turns the plug on remotely allowing the dishwasher cycle to continue.

The equivalents for the first two actions for the washing machine can more conveniently be achieved via the push button on the WIFIPLUG itself as the plug for the washing machine is above the counter.

Electricity consumption 19/12.

The above image shows the measured power consumption from my smart meter. Almost 9 kW is being drawn at times under the action of the HEMS when the electricity price is cheapest, but also zero at times when the price is highest. The washing machine contributes to the peak spike around 02:00 when both it and the car charger are enabled. The later spiking during the peak period is the electric oven cycling on and off under control of its thermostat as the battery isn’t powerful enough to run the oven, so the excess power is drawn from the mains.

Electricity consumption and price 19/12.

The screenshot above shows the half-hourly electricity consumption and costs from Octopus. It should be noted that this is half-hourly consumption in kWh whereas the prior chart was power so, for example, an average 6.6 kW of power consumption results in 3.3 kWh of energy consumption in a half-hour. That the blue line of price is almost the inverse of the red cost columns indicates the HEMS is doing its job to shift most energy use to when the energy price is lowest, and use the battery to offset demand when the price is highest.

Overall on the 19th I paid £1.07 for 21.776 kWh of electricity including standing charge which is independent of use. That’s 86 p for 21.776 kWh without the standing charge, or a weighted average of 3.95 p/kWh. That weighted average of 3.95 p/kWh compares to a range of 1.10 to 26.24 p/kWh during the day. According to UK power the average cost of electricity in the UK is 14.37 p/kWh so I paid 27.5% of the average UK price on December 19th.

The 5 Step Program

This post describes the evolution of my HEMS code to control my dumb wet goods (dishwasher and washing machine) using smart plugs.

The program for my HEMS works as described below. For clarity I’ve emboldened the new steps associated with the control of the wet goods:

  1. Download the cost data for Agile from Octopus. The API provides 48 hours of data, but I use only 24 hours at a time. I download at 16:45 to create a schedule from 17:00 today to 17:00 tomorrow. I use two fields only: the date-time stamp and the energy price inc-VAT.
  2. Calculate cycle cost. Reverse sort the unit cost data in descending time, and combine the energy price with the load profile for each non-interruptible device (dishwasher and washing machine) to estimate the cost of running a washing cycle starting on each half hour. Add as third and fourth fields to the data file.
  3. Establish start time for each non-interruptible load. For each appliance in turn, sort the cost data in ascending cycle cost. Enable the appliance for interval with the lowest cycle cost within an acceptable time window (typically the first row), and overwrite remaining instructions from the prior day. Repeat for other appliances.
  4. Establish on times for each interruptible load. For all interruptible loads (battery, car charger and immersion heater) sort the data in ascending unit cost. Replicate the unit cost column for each interruptible load. For each load enable for the required number of half-hourly intervals within the time window set by the user, and disable for higher cost half-hourly intervals.
  5. Prepare user screens. Sort data file by ascending time, split into first and second 12-hour periods, and present as two HTML files.
Shell script to create daily schedules and user screens.

The top level scheduling script which runs automatically at 16:45 each day is a shell script which calls a series of awk scripts to: (i) calculate cycle costs for wet goods, (ii) determine start time(s) for each wet-goods appliance, (iii) determine on/off times for interruptible loads.

Awk is a pattern-matching program for processing text files. Such text files may be thought of a series of records and fields in a textual database. Awk may seem an odd choice of scripting languages, but essentially the processing of a text file of 96 time and unit pairs to create HTML files of 48 times and cost combinations is a text file processing task. Along the way as each on or off decision is made a system call to the OS is made by awk to copy an on or off script to a time-stamped script (e.g. either washingmachineon.sh or washingmachineoff.sh is copied to washingmachine_0930.sh). An internal timer called cron runs each half-hour so, for example, at 09:30 it runs all the scripts with 0930 in their names which updates the status of each controlled device.

At the time of writing we have 5 HEMS-controlled devices:

  1. Battery storage – interruptible – the only device with 3 states (normal, charge-only, force charge).
  2. Car charger (EVSE) – interruptible – on/off.
  3. Dishwasher – non-interruptible- on only.
  4. Immersion heater – interruptible – on/off.
  5. Washing machine – non-interruptible – on only.

Characterising the loads

The loads that I already control via the HEMS (that is loads which can be interrupted like battery charging, car charging and water heating) are essentially constant, that is that they draw the same power regardless of progress. The battery charging does start to tail off at particularly high states of charge, but that effect is neglected here. However the non-interruptible loads like the dishwasher and washing machine are expected to have very variable loads as the cycle continues – largely because periods of water heating demand much more power than spraying water around, spinning the drum, or pumping water out.

I thus decided that I would measure the pattern of these variable loads and use that information as part of the decision when to run the load for lowest cost. Rather than tabulate the half-hourly energy price as I do for interruptible loads, I would instead estimate the energy price to complete a load for each half hour in which I might start the cycle. This process of measuring the load pattern during a cycle I have referred to as characterisation.

I characterised the loads using a plug-in energy meter reporting kWh which I read manually every half hour during a typical wash cycle to determine the blue lines above. I then calculated the energy usage in each half hour period as per the orange line. I did all of this half-hourly as that’s the interval after which the energy price changes and correspondingly the interval at which the HEMS updates its output controls.

The measured loads confirmed my suspicions regarding variation with heat inputs providing the largest loads – twice for the dishwasher as it both heats the water at the start and heats to dry the dishes at the end, while the washing machine also heats the water at the start but spins to dry the clothes at the end (which is more energy efficient). The effect of this is that the washing machine would generally be expected to start in the cheapest half hour as that’s when it uses most energy, while the optimum time for the dishwasher is more complex to determine.

Of course the actual loads for the wash cycle will vary from the prediction for various reasons including not only the ability to select different wash cycles and options but also the smartness of the device in assessing how full it is or how dirty the utensils are.

I also decided that I would not explicitly turn these loads off from the HEMS – only turn them on. Not turning the loads off allows for some uncertainty as to the length of a cycle depending on the actual cycle selected, options selected, loading of machine etc. Not turning the outlet off also provides for the user being able to do additional loads under manual control if at home later in the day.

Internally my HEMS creates a schedule of 48 half-hourly actions each day for each load. Typically each action is either ‘on’ or ‘off’ (the fixed battery is more complex). The actions for these devices will follow a similar pattern of 48 half-hourly actions, but typically only one of those is ‘on’ and the 47 actions that would normally be ‘off’ do nothing. There existence is largely for equivalence and re-use of code, but they also ensure that the previous day’s ‘on’ command is over-written as required if the start time for the new day is different to that for the prior day.

Choosing the smart plug

One of the enhancements to my HEMS that I’ve had in mind for some time is to control wet goods – that is dishwasher and washing machine. I had previously thought that this might have to wait until I replaced my current machines with smart equivalents, but recently discovered that both existing machines recover after a power outage and continue their cycles. This creates the opportunity to put the machines on smart plugs, to manually start washing cycles, but then immediately turn the smart plugs off, and then use the smart plug to resume the cycle at the optimum time as instructed by the HEMS.

I already have two Eve Energy smart plugs as part of my smart heating system using Apple HomeKit, but these aren’t ideal for this application as I really need an exposed API to allow the HEMS to have control. However I then came across WIFIPLUG which, not only has an exposed API, but also Apple HomeKit compatibility (and indeed compatibility with other smart home ecosystems).

The WIFIPLUG (unlike the Eve Energy) also features an integrated on/off button allowing the washing cycle to be paused using this button, or via the HomeKit app, before later resuming under HEMS control when the energy cost is lowest.

The WIFIPLUG also has its own app which allows timers to be set and energy consumption viewed (although not in great detail) giving the ability to control via Apple’s own Home app or via the WIFIPLUG app. The one thing that I hadn’t spotted first time around is that it’s preferred to add the unit to the WIFIPLUG app (which then automatically adds to Home) whereas if you add to Home directly then you lose the ability to connect to the WIFIPLUG app later.

I was sufficiently impressed to buy a second unit even before I’d integrated the first, and indeed I now have another two on order.

Better than free electricity

A couple of times last week our dynamic electricity price excelled itself by going negative so we were actually being paid to use electricity. This situation typically arises when the weather is unusually windy causing a surplus of renewable power. Then, rather than the wind turbines being turned off to eliminate excess generation, the market price drops to encourage more consumption. Such additional consumption at the cheapest times will be a combination of genuinely increased consumption (such as my own shift from gas water heating to electric) and shifting electricity consumption from more expensive times to cheaper times (such as my own electric car charging and static battery charging).

Electricity price and consumption for Monday 9th December

The electricity price dropped as low as -4.85 p/kWh between 3:30 and 4:00 AM, with an average consumption-weighted unit price of 0.62 p/kWh. The red line shows the electricity price in p/kWh on the left-hand scale, the blue shows the average consumption in this billing month, and the bars show today’s consumption driven by today’s prices. (The right hand cost column is missing the leading ‘-‘ symbol where appropriate.)

The increasing electricity consumption as the price falls is driven by automated control of loads driven by my HEMS. The HEMS controls fixed battery charging (and discharging), electric car charging, and water heating in response to electricity price.

Report on Agile Octopus last weekend from The Guardian.

You can learn more about Octopus Agile here and save yourself an extra £50 if you decide to switch.

Stepping up

One of the features of the way my HEMS works is the inverse relationship between electricity demand and electricity price – electricity demand increases as the price falls and falls as the price rises. This effect helps to contribute to my low average cost for electricity – 5.63 p/kWh ex-VAT most recently (Sep/Oct 2019). The effect exists because my different controlled loads (battery, car and water heating) each need different times to achieve full (approximately 8, 5 and 2 hours respectively).

Relationship between HEMS-controlled loads and electricity price.

This chart is illustrative only because day-to-day needs for the different loads may change, and also the electricity price varies half-hour-by-half-hour and day-by-day but typical recent behaviour is shown. The actual rules for each device may be summarised as follows:

  1. House baseload – support house from battery when cost greater than 10 p/kWh (limit adjusts automatically to reflect pricing on the day), otherwise take baseload from grid.
  2. Storage battery charging – charge from the grid in cheapest 5 hours over entire 24-hour period.
  3. Car charging – charge for a total period of so many hours within a window between arrival time at home and departure time the next day, and also any hours during day at equivalent price or lower.
  4. Water heating – charge for the cheapest 2 hours when electricity price falls below gas price (did not occur on day illustrated below).
Electricity consumption in kW versus time overnight.

These same loads may also be enabled in priority order when the solar panels are in surplus. Priority is:

  1. Baseload
  2. Storage battery charging
  3. Car charging
  4. Water heating.

HEMS in action

Yesterday provided a good example of my HEMS in action as the electricity price dropped quite low due to stormy weather conditions. Normally at this time of year the HEMS isn’t doing much with the storage battery as daytime solar output is enough to fully charge the battery, but yesterday low pricing was enough to automatically enable both battery charging and water heating overnight. Car charging was due to run anyway driven by the demand for an hour of charging, but battery charging and water heating was triggered by the low price rather than a needed to take power for a pre-defined period of time.

HEMS schedule 9th August

The screenshot above from my phone shows the HEMS’ plan for the the early hours of the 9th. The first price column shows one hour of car charging at the cheapest price. The second column shows half an hour of water heating as the electricity price has fallen below 3.5 p/kWh when it is assumed to be cheaper than gas. The third column shows four hours of battery charging when the electricity price is below 5 p/kWh.

Metered electricity consumption (HAN side) 9th August

The above image from the HAN side of my smart meter shows the energy consumption of the house varying through the night in response to these requests from the HEMS – battery charging at the widest point, car charging above that for an hour, and water heating above that for 30 minutes.

Half-hourly metered consumption (WAN side) and electricity price for 9th August

Finally this image shows the energy consumption versus price data for the same period shows how the action of the HEMS increases electricity demand as the price drops. Indeed on this day there was virtually no consumption at any other time.

For August 9th as a whole I paid 52 pence for 7.547 kWh of electricity. Taking off the 21 pence for the standing charge leaves 31 pence for the electricity kWhs alone, an average of 4.11 p/kWh.

HEMS Wiring Update #2

Today I’ve further refined the wiring of the relays on the HEMS. At the time that I’d originally wired it I didn’t have small enough flex, or indeed multi core, which created an unnecessary number of cables (one per used relay) of over large size (and thus difficult to insert into the terminals). During the week I acquired some smaller gauge multi core allowing me to wire all three relays with a single cable containing one live feed and three switched live returns.

Revised HEMS wiring with multi core to HEMS relay outputs

Of the 5 incoming / outgoing cables at the bottom (left to right):

  1. Incoming mains (live / neutral / earth) from mains plug
  2. Live and switched live to / from ImmerSUN output relay to activate car charger.
  3. Live and switched lives to / from HEMS to activate car charger and water heating.
    • Red – live to contacts
    • Green – switched live for car charger direct – charge in response to price
    • Black – switched live for car charger indirect via ImmerSUN relay output – enable proportional charge in response to surplus PV
    • White – switched live for water heating – heat in response to price
  4. Outgoing mains (switched live / neutral / earth) to RF solutions radio transmitter to activate car charger, and on the second cable clamp..
  5. Outgoing switched live and neutral to ImmerSUN Boost relay input to enable immersion heater.

The revised wiring diagram looks like this..

Revised HEMS wiring #2

All of this still leaves one unused relay on the HEMS (HAT #3) and one unused proportional output on the ImmerSUN (#2; available for future expansion.

Initially even my smallest boot lace ferrules would not fit into the terminals on the HAT. Fortunately, once the ferrules has been crimped around the new cables, and flattened by squeezing in pliers, then the ferrules could be persuaded into the terminals.