[Peter]

Jürgen, my apologies, you can only skip some of Step 5. You do need to create the “.env” file and you do need to set the HOST and PORT fields but you should set “WEATHER_PROVIDER=PWS” and ignore the OWM/DARKSKY_API settings. You can then manually run the command “npm start” to start the service and try Step 7 as a test. Note that since you specified PWS as the weather provider and we haven’t yet set up the PWS to send to the Synology then the test results will show a lot of “null” values.

[Peter]

Jürgen, great progress, if you just want to use your PWS then you don’t need to register with either OWM or DarkSky so you can skip Step 5.

[Peter]

Jurgen, great that you have node.js up and running. If you go back to the OpenSprinkler Weather Service github page above then you should see a section titled “Installing a Local Weather Service onto a Raspberry Pi” that leads to some very detailed command line instructions. You should follow Steps 2-5 to get a service up and running. You can then skip forward to Step 7 to test that the service has been installed correctly. However, from memory, Synology does not use “systemd” to make programmes start-up at reboot so you will need to use an alternative solution for Step 7. I believe Synology uses “upstart” but you may need to do some googling to get the sysntax/file location.

If we can get the local Weather Service working on the NAS then we can move on to the network side of things.

Let me know how you get on.

[Peter]

@Pat, the OS Weather Service has been extended to accept a WU format data stream. You need to host the Weather Service locally but then you can “point” your PWS at the service and have your local weather used by OS for calculating watering scale. There is a fair bit of documentation at the bottom of this [ LINK ]

Funnily enough, it was inspired by this blog post [ HERE ] 8)

[Peter]

Apologies if a duplicate but forum keeps ignoring my posts which I’m trying not to take personally 8)
=============================================================
Giles, so this is the way it is meant to work:

1) at start-up, the App chooses the units based on the settings in your browser
2) then when it has connected to your OS device, it looks at the location you have set in the Map and updates based on that information

So, it could be that it goes to Fahrenheit first as your browser reports, say, “US”, but then later it sees that your Map selection is somewhere in France and changes to Celsius.

Does this happen from both the phone App as well as a browser on your PC. If so then let me know the browser you use as we can try and diagnose the problem.

[Peter]

@jhaug40 , so Synology will run Node.JS and this [ LINK ] provides a good intro on how to set it up. Suggest installing the latest node.js package and then following the instructions on the link provided above to install the local Weather Service.

[Peter]

Jürgen, there is some good information at this link [CLICK]. But basically, we need to intercept the data that is going from your PWS to WU and send it to a locally running OS Weather Service instead.

I am note sure if the Synology NAS will run NodeJS. This is the underlying platform of the Weather Service and a key dependency. If it does then that would make a good host for the local Weather Service. If not then you could use a cheap raspberry pi (there are instructions in the link above that detail how to get Weather Service running on a raspberry pi).

The next step is to “intercept” the weather data and get it redirected to Synology. How to do this depends on your network setup. Some wifi routers let you setup an “intercept” but most telco provided routers do not. So let me know the make/model of your wifi router and we can go from there.

[Peter]

boondocks5, as per Roger’s comment, if you are able to run a local version of the Weather Service then you may be able to stream your PWS data directly to the Weather Service. I think your PWS supports WU format and the local Weather Service can accept that so you might not need to write any code for this. See here for some more information. Happy to help talk you through the process. I currently have my Ambient Weather WS-1400-IP doing this and avoid the cloud all together.

[Peter]

Pat, latest versions of OS/OSPi no longer supports WU api keys. The service has switched to OpenWeatherMap for which OpenThings provides the API key.

[Peter]

Hmm, this isn’t the first time I’ve seen this comment on the forums. It could be a caching issue given the difference between main and incognito.

Basically, the App checks the Browser locale at start-up to initialise an isMetric field. It uses the browser language setting to do this (i.e. “en-GB” says English Great Britain and therefore metric). It will then overwrite this default based on the Location you have set in the Map following a successful call to the Weather Service. So you could have a language set as “en-US” which puts you into Imperial but then subsequently see it change to Metric as the weather service identifies you in Germany.

I’m hoping you know how to use Chrome Developer Tools ? If you are able then could you open Chrome Developer Tools in both the General and the Incognito tabs and enter the following two lines into the Console. This will tell us the current situation and help bottom this out:

navigator.language

isMetric

Cheers.

[Peter]

Rondal, the first set of calculations look correct and match what I can calculate here. The precipitation is what is driving most of the impact. The scale,i.e. 0%, should match the Water Level presented in the UI via the App unless the OWM forecast has changed materially. This might occur if the precip forecast by OWM has increased in the last half hour. You can “force” the App to update the Water Level by going into the Map and slightly changing the location and then saving/submitting. This might make it easier to get two similarly timed calls (one via App and one direct to Weather Service) to compare.

Note that the Diagnostic page is not about miss-labeling. The actual values are not what is used in the Zimmerman calc. The values shown are the “current” temp, humidity etc rather than the forecast values. So using those to try any diagnose will not help. If you are going to go into the source then note that “master” is a good many commits beyond the live production service.

[Peter]

Sorry if duplicate posting, browser acting up.
———————————————-

@Rondal, so the change in service provided by Weather Underground has resulted in moving to a new weather service provider, OWM. This required a change to the zimmerman logic as OWM provides forecast data rather than historical data. So the Zimmerman calculation is now based on the average Temp and Humidity for the next 24 hours as well as the sum of forecast precipitation over the next 24 hours. This is different to WU. In essence, the new logic looks to ensure watering is sufficient for the forecast weather rather than replenishing based on yesterday’s weather.

But there are still some piece that need to be updated based on these changes and the Diagnostic popup in the App is one such area. Currently the Mean Temp, Humidity and Precip values are not those being used by the zimmerman calcualtion. Recognize that this causes some confusion and will be addressed soon. If you do want to see what the underlying inputs to the zimmerman calc are then you can call the service directly from a browser. The format is below but you need to update the loc to represent your lat,long and wto options to reflect what you have configures in the Zimmerman setup, i.e. replace the h,t,r with the % values and the bh,bt,bp with the baseline values (in imperial):

http://weather.opensprinkler.com/weather1.py?loc=51.5,-0.1&wto="h":100,"t":100,"r":100,"bh":70,"bt":59,"br":0

This should give a response as below where scale is the water level and rawData is the average temp and humidity as well as total precipitation over the next 24 hours. raining is an indicator of whether OWM believe it will rain in the next 3 hours.

&scale=20&rd=-1&tz=48&sunrise=268&sunset=1167&eip=3232235787&rawData={"h":47,"p":0,"t":54.4,"raining":0}

If you are seeing metric values in the UI when you expect to see imperial then make sure you have your location configured correctly in the Map page of the App as the locale is driven from that information (i.e. generally metric unless location is US and a few other countries).

The UI assets can be hosted locally and instructions are on the OS Support site here

[Peter]

@franzstein , I believe this is materially fixed and post #60204 a little bit earlier in this thread provides some related details.

[Peter]

@franzstein , your last sentence interests me as I thought the might/day cycles had been materially resolved. Have you been seeing this in the last week ? And do you mean that you see a consistent reduction in water level as you enter evening and a consistent increase over the morning ? If you are happy to let me know your rough location (lat/long to one decimal ~ 10 miles and zimmerman parameters) then I can setup some OWM logging to see the impact.

[Peter]

@Gilles , good question, my post wasn’t clear enough. I firstly wanted to show that OWM is a reasonable proxy for local weather conditions by providing a side-by-side comparison of what my PWS says at any point in time compared with what OWM think. In my case the two sources are pretty correlated. So that is good (but your mileage may vary). Secondly, I wanted to point out that the logic was very different i.e. using forecast rather than historical data and the effects of that on water level. The results can be counter-intuitive unless you understand what is going on. Not saying that one method is better than the other just pointing out the differences.

What I didn’t show was the stability of the forecast data used in the calculation. So the OWM logic uses an average of the next 30 hours worth of forecasts (i.e averaging over the next 10 x 3-hour forecasts) not just the next three hours. To show this more explicitly, the weather in London is getting better with temps going up over the last 4 days. Attached is the PWS temp at any point in time alongside the average temp for the next 30 hours. You can see that the later is fairly stable but trending upward so water level “shouldn’t” fluctuate too much between day and night but rather gradually increase in anticipation of warmer/dryer conditions. It is the 30 hour forecast (along with the humidity and rainfall equivalents) that is used in the new OWM based zimmerman calculation.

So Weather Underground changing their model has impacted a bunch of products/projects. Unfortunately, there is no de-facto source of historical weather data anymore so the OS guys have chosen to go with OWM but that only provides cheap access to forecast data. So historical data is not an option in the current setup. Realities of the whole internet cloud business model shaking out.

Attachments:

1. 

   forecast.jpg
   

[Peter]

@fennsen , you can follow the link here for progress GitHub. Note that there are some pre-requisities to be able to use this. Firstly, you need an “always on” server that can run OpenSprinkler Weather Service locally. You also need a PWS that generates Weather Underground format messages (or can build a man-in-the-middle solution). Happy to explore options via the github link above as I’m looking for a tester or two. Cheers.

[Peter]

So just to add a bit to this conversation. The new approach using OWM (OWM graph lines) is very different to the old Weather Underground approach (PWS graph lines). I’ve attached a side by side comparison of the two approaches in the attached image. You can see that OWM is a pretty reasonable comparison to the current conditions reported by my PWS (as it happens I am publishing to OWM so your mileage may vary on this). But when you look at the top graph you can see very different Water Level calculations. The PWS line uses the old Weather Underground historical method using data from the last 48 hours. Whereas the OWM line uses forecast data for the next day to calculate the watering level.

Ignoring the regions where OWM water level is showing zero for the moment, you can see the PWS line is trending down over the last two days but the OWM line is trending up. That’s because in London we have been forecasting rain the last two days but are now heading into some fairer weather. So the PWS/WU water level says reduce watering as we have had some rain, whereas, the OWM approach says increase watering as we are predicting a dryer spell. This can appear a bit unintuitive, for example, you could now have the sprinklers coming on immediately after a heavy downpour if the forecast for the next day is sunny and dry. But then again WU approach had the lawns being watered immediately before the rain came along. So swings-and-roundabouts.

The periods where the OWM water level drops sharply to zero happens when OWN starts predicting rain in the next three hours. When that happens the logic says don’t water for an hour and check again later. This is somewhat analogous with the old Weather Underground approach of setting water level to zero when it is currently raining. Looking at the graph, my concern is that a “forecast of rain in the next three hours” isn’t quite as certain as “rain happened in the last hour” so this could be too sensitive and we might need to fine tune.

Be interested in peoples observations and thoughts on the logic. I’d also be interested to hear from those that experienced the water level reducing into the evening or rising during the morning. The OWM logic uses the next ten 3-hour forecasts to calculate the average forecast weather. Since this is 30 rather than 24 hours, I could see that this might bias the numbers a bit, For example, as you move into the evening, the preceding afternoon forecast (warmer) will roll off and be replaced with an overnight forecasts (cooler) from 30 hours out.

So the change from historical based calculations to forecast based calculations is different. Not sure which is better for the plants but I guess time will tell.

Attachments:

1. 

   weather.jpg
   

[Peter]

To the original posters that were questioning the Openweathermap rain information, a problem with the OWM based calculation has been identified and the OS guys will be responding as soon as possible. Link is here .

[Peter]

@binaryos, not quite sure how you generated the numbers so I may be misunderstanding your table…

But the drop off of watering level in the evening seen from the OS Weather Service when using OWM is interesting. OWM does not provide historic information so the logic used in the OS Weather Service is to use tomorrow’s forecast temps. This means that watering level may change during the day as new forecasts are provided. Also, there is no guarantee that the temperatures are what your PWS provided as the OWM call uses lat/long rather than a specific PWS id so you may be getting an average for, say, the city rather than your backyard.

This is very different to the logic that was used for WU. In that scenario the OS Weather Service used yesterday’s temperature and humidity which by definition would be a constant during the current day. So the WU logic provided a relatively stable watering level through out the course of the day. Only rainfall would result in a “within day” change in the watering level. In this scenario, if you have specified a PWS ID then the data used in the calculation is the data provided by the specific PWS and not some local average.

So the logic is very different, the OWM approach sets watering level based on how hot it is going to be in the general area whereas the WU approach sets watering level based on how hot it was yesterday at your PWS. For me, the WU approach was more intuitive and far less volatile than the current OWM approach but debateable as to which is preferred by the plants !

[Peter]

@BinaryOS, did you take a look at this version of the OS Weather Service OS GitHub PR. It lets you send your local PWS data to a local version of the OS weather service without requiring WeeWX. It takes the PWS data and uses the same weather underground logic as before with the exception that it also replaces (Low-High)/2 with an average across all data points for a slightly more accurate calculation. I would suggest this is the simplest way of moving to a local solution if you have a PWS that generates WU format stream.

I do see some more advanced options that come available by introducing WeeWX into the equation. Namely, you could create a WeeWX Custom Report that calls the OpenSprinker API on an hourly basis to set the manual watering level. That way you wouldn’t need the OS weather service at all and could create whatever calculation method you wanted (e.g. ET based). Looks as though you need to know a bit about python to make this work but their architecture seems to support it. Have you looked into this ?

@paravis, I think the first option I mention above would work for a Meteobridge setup as I see that MB provides the ability to send data to other servers (RESTfull endpoints) in a format of your choosing. Add a post here if interested and I would be happy to help try and get it up and running.

I guess this is really off-topic now as the OP was all about improving the OWM logic. I think this is challenging as OWM does not provide historic data unless the OS guys stump up c.$1000/month which would be a tall ask or radically change their software to build up history over time for every OS weather service user which probably violates OWM T&Cs. At the end of the day, this was down to WU changing their API / Business Model and I can understand if the OS guys want to avoid having to continually chase changes in 3rd Party interfaces. Weather service providers seem so volatile at this stage.

[Peter]

@BinaryOS, I have done something similar. My Weather Station (Ambient) lets me re-direct the weather data to a server of my choice. Rather than using weewx as an intermediary, I modified a version of the OS Weather Service to accept a WU data stream directly. I run the modified weather service on my OSPi which keep everything local.

I’m not familiar with Weewx. How does it intercept the WU stream (DNS redirect ?). I’m interested in the comment about ET value. Does weewx calculate this from the WU data provided ?

[Peter]

@treb0r, I have upgraded to the latest versions of App and Firmware and can now replicate your first attachment error.

I can see here that the urlDecode() line has now been commented out of Firmware but not out of App. So you get the behavior you are seeing above (i.e. the %2C sequence is not being decoded back to a “,” delimiter). Looks like it was a conscious decision to remove the decoding but I’m not sure why. You have the option to edit stns.dat directly but, as you say, be wary of updating/saving.

Linking in github issue 70 that is tracking this.

[Peter]

@trebOr, I cant replicate the problem with the first attachment but I’m using slightly different versions to you so I will upgrade to the equivalents and test a bit more. Can you take a screen shot of the populated station dialog “before” you hit submit so that I can see what you are putting into each field. Can you then wait for 60 seconds and then re-open the dialog to see if the contents are correct. You need the pause as the App takes a while to reload data from Firmware.

I can replicate the second attachment and it looks like the App import functionality works differently to the station settings dialog. The import code does not uri encode the special station configuration data before sending across to Firmware. As a result, the Firmware parser sees the first “&” in the on command and thinks that’s the end of the station config. Hence you get a truncated On command and an undefined Off command. This is probably down to me as I think the HTTP Station is the only place where you need nested encoding. I’ll submit a fix into OpenSprinkler-App as soon as I get some time.

If you need an interim work-around, and are comfortable with linux command line, then you could edit the stns.dat file in the pi home directory to contain the right configuration. This is a binary file so you would need a HEX editor (sudo apt-get install hexedit). Let me know and I can tell you more on this.

[apols if duplicate post but forum disappeared the first response]

[Peter]

Interesting, I didn’t release the guys had flashed different versions of the firmware across the production units. This is useful as I can now see the specific fix made. They do not relate to how the SPIFFs works but rather how they publish large packets. From your comments above, I suspect you have a mix of both firmware.

Version 2.1.7(1) was the original firmware supporting the ESP8266 put out in August. This does not include the Button 2/3 ability to reset to AP mode nor the “packet fix”.
Version 2.1.7(2) was an updated firmware that was checked-in in December. This includes the new B2/3 selection, the “fix” and a new variable in the “http://os_ip/jc” command that provides last reset time “lupt”.

You should be able to confirm the firmware version of each unit via the Web App -> About.

If the one experiencing problems is the older 2.1.7(1) then I would swap it out for one of the newer firmware units and that might solve the problem for your main OS unit. You could still potentially experience the problem with the older firmware units once their log records grow but you may have bought yourself some time.

I would then get on to OpenSprinkler support for assistance in upgrading the other units or explore how to build the firmware directly from Github (not easy unless you are pretty technical).

[Peter]

I’m not a network expert either but I agree that connecting the mac to the extender should keep most things local so that does put a dent in the “flakey network” suspicion. It may still be worth ruling it out and it looks like you can get the OS to start-up in Access Point mode by pressing Button 2 while holding down Button 3. You should then be able to connect to the access point (the SSID is displayed on the OS LCD) from your phone’s wifi settings and use the phone’s browser to view the wifi config page (the IP to browse is also on the OS LCD). You can then enter the main wifi SSID and password, save and reboot. Please export and save the OS config before doing this!

So the Arduino and OSPi versions will reset themselves if they detect persistent network failures and/or if the software gets stuck (i.e. they have a watchdog timer). I know the ESP8266 has a watchdog timer so that could be triggering the reset if the code freezes for some reason.

I do wonder if your OS is in fact regularly resetting without being noticed, not just when querying log records. One thing I have setup is IFTTT integration where my OSPi will generate an SMS message whenever it reboots. This link shows how to setup and could be informative. Getting to the bottom of the barrel here, but a poor power supply could potentially cause a reset during heavy wifi activity. Are you using one provided with the OS or did you source independent. How many amps is it rated for?

[Author]

Posts