Exceeding Wunderground API Calls

[seockwig]

Yesterday I updated the firmware on my DIY 2.1 OS system to version 2.1.2.  Previously I was on v 2.1.0. After upgrading I was going through the setting because the restore didn’t work properly.  I noticed that my Wunderground Key was missing.  I logged into Wunderground grabbed my key and entered it into the system.  I ran a Weather Diagonstic and everything appeared to run normally.

However, later that evening I received the following email from Wundergound:

 

Hello,

Your wunderground API key (***********) exceeded its allotted usage today by making 501 calls in a day but the limit is 500.

We used one of your raindrops instead of disabling the key for the remainder of the day. You now have 26 remaining raindrops.
We check usage for 24-hour periods based on U.S. Eastern Time.

The plan you are registered for is Stratus – Developer, granting you 500 calls per day with 10 calls per minute limit. To upgrade your plan go to

Details: exceededcalls: calls: 501 (of 500)I received the same email early this morning as well.

 

I can’t seem to find a place to adjust the call interval.  I would think 500 calls/day would be more then enough.  Am I missing something?

 

Thanks!

[Ray]

500 calls/day should definitely be sufficient. The firmware is set to query Wunderground every 15 minutes so it’s 96 calls per day, well below the limit.

The only issue I can think of is when a query fails the firmware will keep sending a new query every few minutes, which can exceed the call limit. Can you tell me the location you set in OpenSprinkler, so I can check whether the issue is reproducible.

[seockwig]

As an FYI, I did get another email early this morning informing me that I’ve exceeded the call limit again.

 

The location is set to: Inver Grove Heights, MN.

 

Let me know if you need anything else.

[Ray]

I tried your location and cannot reproduce the issue. I am getting one call per 15 minutes, which is the correct call rate. Can you go to the homepage, click on ‘Current Status’, and check the Sunrise Time and Sunset Time to see if they are correct? If they aren’t, it’s likely that something, either your ISP or your router is blocking the query result, causing the controller to not be able to receive the result, and hence keep sending more queries.

[seockwig]

Sunrise time is 6:00 and Sunset is 18:00…and that is not correct for this time of year in MN.  I’ll dig into my router when I get a chance…but that may not be for a few days.

If you think of anything else let me know; otherwise I will post again after I’ve had a chance to look at the router.

[Ray]

OK, the sunrise and sunset times are clearly not being set (6:00 and 18:00 are the initial values). So this confirms my suspicion that something is blocking the result of the query. As an initial diagnose, you can open a browser and try the following url:
http://weather.opensprinkler.com/weather0.py?loc=Inver%20Grove%20Heights,%20MN
see what you get.

Another thing you can try is to use your zip code instead of your city name. See if that makes any difference.

[seockwig]

So this is the result I get when I goto the URL:

&scale=-1&tz=24&sunrise=469&sunset=1001&maxh=-1&minh=-1&meant=-500&pre=-1.000000&prec=-1.000000&hc=-1

 

I typed my zip code in to the location field and  hit the locate button..right now the location is set for icao:KSGS

I’ll see if I get another email message from Wunderground today and will let you know.

[seockwig]

I think it’s working now.  I checked the Current Status page and the Sunrise is now set for 7:49 and Sunset is set for 16:41.  Which seems about right.

[Ray]

OK, that means the controller is now getting the return data from the weather query. It’s still unclear to me what caused the problem initially. Anyways, let’s see if you will get further notifications from Wunderground.

[seockwig]

Well..something still isn’t right.  I received two of the notifications from Wunderground.  One yesterday afternoon and one early this morning stating that I’ve gone over the call limit.  I think I’m going to generate a new a key and see what happens.  I wouldn’t think my key has been “compromised”…but it’s an easy thing to check.

I suppose the next thing would be to do a factory reset on the controller and see what happens from that.

[Ray]

I checked the log file on our server and it’s pretty clear that your weather query is being sent every 30 seconds, which happens when the controller does not receive the last call result. That’s why I was asking if the sunrise/sunset time is correct, because if that’s not correct, the controller is clearly not receiving the call result, and hence repeatedly call again every 30 second. If the call is received correctly, the next call should be in 15 minutes.

We can change the firmware to reduce the call frequency upon failure. Unfortunately this doesn’t really fix the issue in your case because the root cause of issue is your controller is for some reason not receiving the call result, which is quite strange.

[Ray]

Forgot to ask: is your OpenSprinkler using the default HTTP port 80 or have you changed the HTTP port?

[seockwig]

I did change the key and I’ve been watching the stats on Wunderground and have been seeing the same thing.  The calls have been going up about 2/minute.

I did have my controller on a different port, but I changed it back to port 80 earlier this evening.  I just checked it again and rebooted it to make sure and everything looks alright.

I do have my network behind a firewall/UTM, and I’ve been going through the logs.  But I don’t see any issues with the firewall blocking any traffic destined for the controller.  I’m going to do some more testing with firewall just so I can rule that out completely or find a problem.

I don’t think you’ll need to change to firmware as right now this appears to be an isolated issue.

Thanks for all the assistance!

[seockwig]

Is there a time out for how long before another request is made from the controller?

I’m running a homebuilt Untangle UTM which in not only a firewall, but has modules for Webfiltering, Virus Scanning, Spyware Scanning; Application controller, IPS and ADBlocker.  Here is what I’ve found so far.  If I turn off all the modules but the actual firewall the Wunderground stat page goes up by 1 or 2 calls and then stops. Meaning to me the controller received its response and won’t call again for another 15 minutes. However if I turn on any additional module which scans the traffic; it doesn’t matter which one; I can watch the call counts go up by about 2/minute on Wunderground.  Once I turn the module off, the call counts level off.  From what I can tell the traffic isn’t being stopped by the UTM, but it’s possible the small delay introduced by scanning the traffic may be enough that the controller is thinking that it won’t get it’s response and then sends another request out.

Your thoughts?

I am going to leave everything off on the UTM except the actual firewall module to see what happens over the next day.  If you could check your logs again to see how many calls are coming in from controller; I’m thinking it will look correct now.  If that’s the case it is something to do the UTM, either a delay or it’s changing something in the traffic.

 

[Ray]

Yes, there is a time out. As I explained in the posts above: if a previous call is successfully received, the next call will be in 15 minutes; if a previous call is NOT successfully received within 30 seconds the next call will immediately issued. So as you can see, normally it should be every 15 minutes; but if it continues to fail it will be repeatedly called every 30 seconds.

I just thought about one possibility: because the server response may have been slow, and your additional network modules (which scan the incoming packets) can slow down it further, so if the overhead added up together exceeds 30 seconds every time, it will result in a call being issued every 30 seconds. I think this is the most feasible cause. For the server response part, we are already preparing to migrate the script to Amazon servers to reduce the load on our own server; also, I will modify the firmware to reduce the call frequency so that it won’t exceed the call limit even in the worst case.

[seockwig]

Your right,  you did explain the timeout.  It was just as I started making the connection with the scanning modules on the firewall the first thing that popped into mind was the possibility of a time out of a few milliseconds.  I know that scanning the traffic doesn’t take 30 seconds; if it did anything I did over the internet would be noticeably slower.  As an example when I checked the url you posted above the results came back in a second or two.  If the filter was slowing the traffic down that bad, the results would’ve taken considerably longer to show.

I have not received any emails from Wunderground today, and I would have by this point if I exceeded the call limit.  I just checked my Wunderground page and the usage looks about right for a call every 15 minutes.  So I believe that isolates the issue to scanning modules of the UTM, what that is I don’t know yet.  I’ll do some digging this evening to see what else I can figure out.

 

[seockwig]

Well I wish I could say I’ve got it figured out.  But I just can’t quite seem to nail this one down.  It is something with scanning of the traffic…but I don’t know what.  When I have any module on that scans the traffic the calls to Wunderground climb at a rate of 2/minute.  How ever as I watch the firewall logs it looks like everything is going through.   I turn off the modules and the calls drop to 1 every 15 minutes.

I used the query provided earlier and ran it with all the scanning modules off and then with just the webfilter on( I restarted my browser and cleared the web cache between tries as well.)  I received the same results both times and the response time was about 2 seconds.  I would think if the traffic was being blocked I wouldn’t get a result or if scanning was slowing down the response I would’ve seen a noticeable lag with the webfilter on.  The results on Wunderground were consistent as well.  With the modules off it was 1 call and done; with the webfilter on calls climbed.

When I get a chance I’ll post to the Untangle forums to see if anybody over there has an idea what may be happening.

For now, I’m going to shut the controller off as it isn’t being used right now.  I’ll turn it back on when I need to do some more testing.  Hopefully I can get it figured out before Spring rolls around as I would really love to use the weather feature of the controller.

[Ray]

OK, thanks for the update. Keep me posted about your findings. In the next firmware we will reduce the call frequency so that at least it won’t exceed the limit even in the worst case.

[seockwig]

Well, I think we have it worked out.  I’ve created a bypass rule on the firewall for the OpenSprinkler controller.  So none of the traffic coming from or going to the controller get’s scanned by the filters.  Still don’t know what the controller didn’t like about the scanned traffic, but it appears to be working.  If things change I’ll let you know and if I ever figure out  what was being changed in the traffic I’ll pass that along as well.

Can I make a suggestion?  I was on the Wunderground page to check my Call count….that required me refreshing the page every couple of minutes to see if the Call count changed.  Would it be possible to add a Date/Time stamp of Last Attempted Call and a Date/Time stamp of Last Successful Call.  If it’s possible, putting it on the Weather Diagnostics page I think would be the most logical place.  It would make it a lot easier to troubleshoot if that is possible.

Something like this:

Min Humidity   67%

Max Humidity   93%

Mean Temp   23F

Precip Yesterday   0.00″

Precip Today   0.00″

Current % Watering   0%

Last Call Attempt   1/7/2015  7:13PM

Last Successful Call   1/7/2015  7:13PM 

Thanks for all the help!

[Ray]

Ok, that’s pretty easy to do and we can add it to the next firmware update. Thanks for the suggestion.

[rhys]

Hi Ray,
Just curious if you’ve implemented the changes to “reduce the call frequency so that at least it won’t exceed the limit even in the worst case” in firmware version 2.1.2 as I’ve had similar emails from Wunderground, not many but a few???

[Ray]

Are you still getting the emails from Wunderground? Two days ago we found that our weather script is taking a significant amount of time to obtain sunrise/sunset time from openweathermap.org, so we immediately changed the script to use Python to directly calculate sunrise/sunset times. Although this is not directly related to Wunderground, if the script takes a long time to return, it can cause the controller to resend another request, and thus exceeding the call limit. I hopefully our fix to the weather script has eliminated the exceeding call limit problem.

[rhys]

Since your 21 Jan posting I haven’t had any more of those Wunderground notices, thank you…

[rongross]

I checked the log file on our server and it’s pretty clear that your weather query is being sent every 30 seconds.

Sorry to jump in here at the end of a topic (I am new to this product and want to run a version on RPi) but I see this quote from Ray and it makes me believe that the controller communicates back to home base (opensprinkler.com) to do some of it’s work.  I thought the idea was this was stand-alone and ‘runs programs on its own?’

[Samer]

@rongross The discussion in this thread is regarding automatic weather adjustments. There are only two ways a controller can reasonably do this: direct weather data via a weather station or online weather data from public weather stations. OpenSprinkler achieves automatic weather adjustments using the online method.

Therefore, this portion of the controller operation reaches out and grabs an adjusted watering level for the user selected method (eg. Zimmerman) and for the local weather at that time. Now keep in mind, the controller only polls for this value at intervals and stores the value locally meaning it will still have all the information locally to run without a connection. The default watering level is 100% and based on weather conditions the algorithm will scale between 0 to 200% which adjusts the duration of each stations run time.

I hope this helps!

[Author]

Posts