- This topic has 68 replies, 15 voices, and was last updated 4 years, 10 months ago by
.
-
Posts
-
@BinaryOS, did you take a look at this version of the OS Weather Service OS GitHub PR. It lets you send your local PWS data to a local version of the OS weather service without requiring WeeWX. It takes the PWS data and uses the same weather underground logic as before with the exception that it also replaces (Low-High)/2 with an average across all data points for a slightly more accurate calculation. I would suggest this is the simplest way of moving to a local solution if you have a PWS that generates WU format stream.
I do see some more advanced options that come available by introducing WeeWX into the equation. Namely, you could create a WeeWX Custom Report that calls the OpenSprinker API on an hourly basis to set the manual watering level. That way you wouldn’t need the OS weather service at all and could create whatever calculation method you wanted (e.g. ET based). Looks as though you need to know a bit about python to make this work but their architecture seems to support it. Have you looked into this ?
@paravis, I think the first option I mention above would work for a Meteobridge setup as I see that MB provides the ability to send data to other servers (RESTfull endpoints) in a format of your choosing. Add a post here if interested and I would be happy to help try and get it up and running.I guess this is really off-topic now as the OP was all about improving the OWM logic. I think this is challenging as OWM does not provide historic data unless the OS guys stump up c.$1000/month which would be a tall ask or radically change their software to build up history over time for every OS weather service user which probably violates OWM T&Cs. At the end of the day, this was down to WU changing their API / Business Model and I can understand if the OS guys want to avoid having to continually chase changes in 3rd Party interfaces. Weather service providers seem so volatile at this stage.
@binaryos, not quite sure how you generated the numbers so I may be misunderstanding your table…
But the drop off of watering level in the evening seen from the OS Weather Service when using OWM is interesting. OWM does not provide historic information so the logic used in the OS Weather Service is to use tomorrow’s forecast temps. This means that watering level may change during the day as new forecasts are provided. Also, there is no guarantee that the temperatures are what your PWS provided as the OWM call uses lat/long rather than a specific PWS id so you may be getting an average for, say, the city rather than your backyard.
This is very different to the logic that was used for WU. In that scenario the OS Weather Service used yesterday’s temperature and humidity which by definition would be a constant during the current day. So the WU logic provided a relatively stable watering level through out the course of the day. Only rainfall would result in a “within day” change in the watering level. In this scenario, if you have specified a PWS ID then the data used in the calculation is the data provided by the specific PWS and not some local average.
So the logic is very different, the OWM approach sets watering level based on how hot it is going to be in the general area whereas the WU approach sets watering level based on how hot it was yesterday at your PWS. For me, the WU approach was more intuitive and far less volatile than the current OWM approach but debateable as to which is preferred by the plants !
Binary this looks like an interesting way to do this, I too have a PWS but don’t have it setup fully yet but may look into this down the track if the OWM system keeps failing to report rain events incorrectly.
Can this weewx run on a NAS by any chance as that would almost be ideal for me to log local data seeming most of the weather services now see to have lost their way with greed.
The main issue with OWM integration to OS is this out of control rain reporting I still don’t think we have had an official response from any of the Devs around what is causing this and by many accounts here renders the OS automatic watering useless.
I haven’t had time to try and actually setup a comparison between the numbers the OS reports vs talking directly to the OWM api but I guess this would be a good start to see if the problem lies in the OS or OWM api.
Devs?????
@Peter, no I didn’t see that version. Agree it is a simple and neat solution. For myself, I’m finding weewx a better solution as I’ll now build my own historical data. WU has removed the “Custom” time span for data so I can no longer easily compare years or other custom spans :(. Who knows what else they will remove. Now I can record what I want in reports etc. I can also very easily change the report to the OS Weather service.
Yes I did look into having a script call OS to set watering rate, but in the end felt it was simpler/quicker to setup the existing system (Weather service – OS-Weather – OS) and modify. I did not loose any flexibility doing it this way as both can be easily modified. I’ve already got solar radiation reporting to OS-Weather for possible future ET based calculation.
Regarding OWM, Ah ok I see. That kinda makes sense if the forecast is swinging around.
Watering level values from http://weather.opensprinkler.com/weather1.py (OWM source) and local opensprinker-weather (weewx 3dayAvg source):
21:08 OWM 86%, WEEWX 100%
21:50 OWM 84%, WEEWX 100%
03:03 OWM 54%, WEEWX 100%
03:21 OWM 50%, WEEWX 100%
04:00 OWM 43%, WEEWX 100%
04:30 OWM 39%, WEEWX 100%
05:00 OWM 35%, WEEWX 100%
05:30 OWM 35%, WEEWX 100%
06:00 OWM 89%, WEEWX 100%
06:30 OWM 89%, WEEWX 100%
07:00 OWM 91%, WEEWX 100%
07:45 OWM 101%, WEEWX 100%
08:07 OWM 111%, WEEWX 100%Almost every site is forecasting Sunday/Monday 29°C-30°C (2°C-3°C warmer than yesterday) while OWM is forecasting 23°C-24°C. Regardless of broken forecast, the variance in OWM levels seem pretty messed up to me. This is without adding the rain issue into the mix.
@Craig, I did not do any research for running on NAS. Weewx is python so should be a good chance the NAS can do it. There is heaps of good info available, including importing the historical PWS data from existing online services or CSV files. 🙂
Once setup your then free to modify the calculation way beyond the inbuilt adjustments within OS.
We have datacenter services we could share. Is it possible to run one WeeWX and allow others to send data to a central server? And then use that for their weather data?
Or is this something that has to be local to the user’s situation?
To the original posters that were questioning the Openweathermap rain information, a problem with the OWM based calculation has been identified and the OS guys will be responding as soon as possible. Link is here .
This issue has been identified by Peter and he provided a fix for us which is now pushed out to production.
Thank you!
@samer thanks for that, so this is now fixed for the OWM service that was implemented? Or has something changed again?
The good news:
Looks like the opensprinkler data is agreeing with the openweathermap data.The bad news:
Open weather map says we have had 8mm of rain which I am pretty sure we haven’t had and my own weather station says zero precipitation last 24 hours. Understand thats more of an issue with open weather map.
Can confirm in Australia Weather App desktop matches closely Operweathermap for same location with slight 0.1 deg celcius rounding error for forecasts.
Switched on weather adjustment, skipping watering today, we’ve had a little rain 2.6mm since 18:43 hrs
Thanks everyone for their fine effort!
So just to add a bit to this conversation. The new approach using OWM (OWM graph lines) is very different to the old Weather Underground approach (PWS graph lines). I’ve attached a side by side comparison of the two approaches in the attached image. You can see that OWM is a pretty reasonable comparison to the current conditions reported by my PWS (as it happens I am publishing to OWM so your mileage may vary on this). But when you look at the top graph you can see very different Water Level calculations. The PWS line uses the old Weather Underground historical method using data from the last 48 hours. Whereas the OWM line uses forecast data for the next day to calculate the watering level.
Ignoring the regions where OWM water level is showing zero for the moment, you can see the PWS line is trending down over the last two days but the OWM line is trending up. That’s because in London we have been forecasting rain the last two days but are now heading into some fairer weather. So the PWS/WU water level says reduce watering as we have had some rain, whereas, the OWM approach says increase watering as we are predicting a dryer spell. This can appear a bit unintuitive, for example, you could now have the sprinklers coming on immediately after a heavy downpour if the forecast for the next day is sunny and dry. But then again WU approach had the lawns being watered immediately before the rain came along. So swings-and-roundabouts.
The periods where the OWM water level drops sharply to zero happens when OWN starts predicting rain in the next three hours. When that happens the logic says don’t water for an hour and check again later. This is somewhat analogous with the old Weather Underground approach of setting water level to zero when it is currently raining. Looking at the graph, my concern is that a “forecast of rain in the next three hours” isn’t quite as certain as “rain happened in the last hour” so this could be too sensitive and we might need to fine tune.
Be interested in peoples observations and thoughts on the logic. I’d also be interested to hear from those that experienced the water level reducing into the evening or rising during the morning. The OWM logic uses the next ten 3-hour forecasts to calculate the average forecast weather. Since this is 30 rather than 24 hours, I could see that this might bias the numbers a bit, For example, as you move into the evening, the preceding afternoon forecast (warmer) will roll off and be replaced with an overnight forecasts (cooler) from 30 hours out.
So the change from historical based calculations to forecast based calculations is different. Not sure which is better for the plants but I guess time will tell.
Attachments:
This is very interesting analysis. thanks a lot Peter.
I guess the major issue for the new approach is the next 3 hours temperature forecast. For those watering at night in places where there can be large differences between day and night temperatures, it is very likely that the watering level will be not sufficient. I think the ideal calculation should take into account:
– the last 24 hours averages (temperatures, humidity) to calculate the evaporation and watering level
– adjusted with the next 3 hours rain forecast
I think this would address all concerns.@Gilles , good question, my post wasn’t clear enough. I firstly wanted to show that OWM is a reasonable proxy for local weather conditions by providing a side-by-side comparison of what my PWS says at any point in time compared with what OWM think. In my case the two sources are pretty correlated. So that is good (but your mileage may vary). Secondly, I wanted to point out that the logic was very different i.e. using forecast rather than historical data and the effects of that on water level. The results can be counter-intuitive unless you understand what is going on. Not saying that one method is better than the other just pointing out the differences.
What I didn’t show was the stability of the forecast data used in the calculation. So the OWM logic uses an average of the next 30 hours worth of forecasts (i.e averaging over the next 10 x 3-hour forecasts) not just the next three hours. To show this more explicitly, the weather in London is getting better with temps going up over the last 4 days. Attached is the PWS temp at any point in time alongside the average temp for the next 30 hours. You can see that the later is fairly stable but trending upward so water level “shouldn’t” fluctuate too much between day and night but rather gradually increase in anticipation of warmer/dryer conditions. It is the 30 hour forecast (along with the humidity and rainfall equivalents) that is used in the new OWM based zimmerman calculation.
So Weather Underground changing their model has impacted a bunch of products/projects. Unfortunately, there is no de-facto source of historical weather data anymore so the OS guys have chosen to go with OWM but that only provides cheap access to forecast data. So historical data is not an option in the current setup. Realities of the whole internet cloud business model shaking out.
Attachments:
@Peter, thanks for clarifying the 30 hours vs. 3 hours. It definitely makes more sense and I feel more confident with the calculation. It has been raining quite a bit here in Paris lately. The watering has been at 0% for some time, so I couldn’t check if watering level fluctuates significantly during nights (which was a major issue before the last updates were introduced by Samer).
Thanks again@Peter Perfect analysis, explaining the differences between OS historical and forecasted weather data usage.
In my opinion only historical weather data is acceptable as OS Zimmerman Method input. Temperature forecasts may be acceptable and not be the problem. However, in regards to rain and especially the amount of rain, weather forecasts are often far away from reality. I can’t speak for all countries, but at least for Germany rain prediction is not always trustworthy.
I started last year with an OSBee and a selfmade IFTTT, apilio.io weather algorithm based on weather forecasts. My program often failed for garden watering as there was rain announced, which never happened during the ongoing day. This causes me to change to an OpenSprinkler 3.0 and the accompanied Zimmerman method. Based on historical WU data, forwarded by my own weather station, this worked and still works perfect for me.
I also believe that it is much preferable to have an own rain gauge. There is sometimes rain forecasted for the ongoing day, but there will be no remarkable rainfall at all at my backyard. With an own rain gauge and the use of historical weather data it is possible to avoid these uncertainties.
In regards to free access to historical weather data, there is a possibility to provide and collect weather data by an own weather station. This still works for me in conjunction with WU. However, I’m also depending on cloud services like Meteoware to forward the data from my Netatmo weather station to WU.
There are already some solutions like private weather servers or my ESP32 Node MCU program that controls the OS Watering via API commands discussed at this forum. Unfortunately, these solutions need some programming knowledge and additional HW efforts.The majority of OS users will still depend on the OWM weather adjustment feature provided at no extra costs by OpenSprinkler. I’m not sure if this will work out at the end for weather controlled garden watering. Forecasted weather data will always have uncertainties. With the current implementation it changes during night and day, which makes it difficult to perform sufficient early morning garden watering.
@franzstein , your last sentence interests me as I thought the might/day cycles had been materially resolved. Have you been seeing this in the last week ? And do you mean that you see a consistent reduction in water level as you enter evening and a consistent increase over the morning ? If you are happy to let me know your rough location (lat/long to one decimal ~ 10 miles and zimmerman parameters) then I can setup some OWM logging to see the impact.
@Peter I’m still using historical weather data from WU as a weather source forwarded by my Netatmo PWS to WU. My OpenSprinkler runs in Manual Mode and will be updated via the OS API interface. My ESP32 NodeMCU program does the Zimmerman calculation based on empirical weather data retrieved from WU. My base parameters are 70°F, 65% humidity and 0.0 inch precipitation. My geo coordinates are 49.47369, 10.94380. My WU station name is IFRTH65. The location name shown in OpenSprinkler is Burgfarnbach a suburb of Fuerth, Germany. This seems to be the location of the nearby OWM weather station.
I don’t know how the actual OWM based OS weather adjustments are working in detail. Reading forum comments I thought that there are still variations during night and day not caused by any rainfall.
It would be nice if you can provide more details or run a comparison.
@franzstein , I believe this is materially fixed and post #60204 a little bit earlier in this thread provides some related details.
Great news. I did not realize that the night and day variations are completely fixed. I still hesitate with the use of forecast weather data. However, time will show!
OpenWeatherMap is completely useless. Comparing OpenWeatherMap to the National Weather Service for my area, OpenWeatherMap is consistently forecasting temperatures 10-20 degrees F below the NWS forecast. And the NWS forecast has been quite accurate. As a consequence, my watering has been in single-digit percentage, while the temperature here has soared.
I’ve reported this discrepancy to both OpenThings.io (Samer) and to OpenWeatherMap (Pavel Zuykov) more than a month ago, but there’s no change–the forecast shown on my OpenSprinkler continues to be well below the NWS, WeatherUnderground, WeatherChannel, and just about any other source I can find.
What exactly does it take to get this fixed? Is there no alternative possible to OpenWeatherMap?
It’s based on the Zimmerman formula, perhaps try changing the temperature baseline setting to 50F (or 90F?)
https://openthings.freshdesk.com/support/solutions/articles/5000017312-using-weather-adjustments
There was a bug in the weather app for your phone so make sure you update that too.
The Zimmerman formula doesn’t change the forecast, and the forecast is clearly wrong. If the forecast were consistently off by some amount, then changing the baseline temp would help. But it’s not; the error varies significantly from day to day, and neither linearly nor proportionally to the correct forecast. The only thing consistent about it is that it’s consistently wrong.
Yea I think the bottom line (and what we are all trying to communicate in general) is that OWM is appealing on the outside, but total garbage for everything we expected it to do. Or just garbage in general.
If we base anything off of trash data, the best we will get is trash result. OWM data actually destroyed a noticeable chunk of our landscape … unfortunately … and we are now fixing it. And it sucks because it only took 6 weeks to destroy it with OWM. (Don’t overlook my negligence in not noticing fast enough.)
But looking at reality: humidity, temperatures and rain are variable. And plants and trees are obviously still alive today (humans, now that is a questionable existence, no idea how that’s continuing on).
So even with OWM’s horrible faults, and the fact that I would never use them for anything I value – and honestly wouldn’t wish them on my enemies – the plants and trees do recover, but something needs to change real quick.
Or they won’t recover next time.
We have spent probably $1K USD to fix what the OWM data screwed up. And that isn’t including my pain, time and anger dealing with it.
The OWM service has no legit support, we can’t even identify whether or not they are receiving our weather data, and the best I can say is “THANK GOODNESS FOR THE INDIVIDUALS WHO HAVE INVESTED THEIR TIME AND RESOURCES INTO ALLOWING US TO USE OUR PWS AGAIN.”
Cheers to them, including Peter, Ray and Samer, and the others who have contributed. I feel bad that OWM didn’t work the way it should have. But it has got to go.
Whatever the solution is, we are more than happy to donate datacenter resources to the cause. Email me at [email protected].
@Sturdley, any chance you can let me know your aprox location (lat/lon say one decimal place = 10 miles) and any nearby PWS that you have used in the past. I would like to log some data and check out the situation. I’ve posted elsewhere that OWM is OK for me (but then I’m in London only 5 miles from the OWM HQ). So fully understand that different locations will have different results. We cant do anything about the accuracy of OWM forecasts but we can tweak our Zimmerman algorithm a bit. In my view it is currently very sensitive to forecast rain and I’m working with the OpenSprinkler team to try and improve on this. I would like to log the data from your location and compare with a couple of other providers and see what the data can tell us.
@paravais, as I say above, OWM accuracy seems to vary significantly depending on location. I’m glad you have the local Weather Service up and running and that it seems to be meeting needs. I don’t think server resources are the problem here. It just seems that for some locations OWM doesn’t provide an accurate forecast. If you would be willing to post your approx. location also then I would be very keen to try and quantify the problem.
- You must be logged in to reply to this topic.