Caching oddities when using both the mobile app and browser interfaces

[Brian]

Hi folks,

I noticed a bit of an oddity when using both the mobile app for Android and the browser-based user interface that seems to be related to caching. This past week I added a few new programs via the mobile app after viewing the OS logs on my desktop’s Chrome browser (Version 42.0.2311.90). A few days later I checked the desktop’s browser previously opened session to view the new programs, but they were not listed. I then navigated to View Logs (via the grid on the lower right of the frame) and the most recent day’s activity was not shown (but the older activity that I had viewed earlier was). I then reloaded the current page (via the browser’s reload button) and both the latest logs and the new programs (after navigating to the programs section) were then visible.

Another separate issue I noticed when inspecting the browser’s error console is that the favicon hosted at http://ui.opensprinkler.com/img/favicon.ico is currently generating a 403 Forbidden error when loading (referenced on line 99 of home.js)

[Samer]

Hi Brian,

I specifically timestamp each request I make to the controller to avoid caching issues however Chrome has been caching some requests according to some users. I will re-examine this however I doubt I can do more than what I already am to prevent caching.

I opened the URL you provided and I got no error. Furthermore, when I load the app using the browser method, I don’t see the error in the console. If you open that URL yourself, do you see the icon or instead get an error?

Thanks!

Update: I am able to replicate this issue on some networks/devices and therefore am hunting down the issue. It appears to lie with CloudFlare, our CDN, since the server is never even seeing the request for those files much less giving out a 403 error.

[Brian]

Thanks Samer,

I had a quick look at the headers that are being returned by OS and noticed something that is likely contributing to the caching issue. It would seem that some of the standard cache control headers are not being set in the JSON responses that likely invokes some default caching behavior on some browsers. Currently OS 2.1.3 is setting “Pragma: no-cache”, in the response but nothing is set in the browser request. The HTTP 1.0 spec specifies either setting “Pragma: no-cache” in the request, or the “Expires” and “Last-Modified” dates in the past in the response so the data is not cached.

Since the OS is responding with HTTP 1.0, the kitchen sink “Cache-Control:max-age=0, no-cache, no-store, must-revalidate” response header for HTTP 1.1 will likely not work (unless going through a HTTP 1.0->1.1 proxy), but may be worth adding for those using a reverse HTTPS->HTTP proxy to protect their OS from snooping or MITM attacks.

[Samer]

Thank you for the detailed information and solution! I completely overlooked this aspect of the caching issue and have already merged the fix into the main code base.

Would you mind updating your code (git pull) and rebuilding to see if the cache issue is resolved?

I will likely go ahead and add the Pragma: no-cache to the request in the app/UI as well since older firmware’s will still have this issue.

Thank you!

[Brian]

Thanks Samer,

I currently don’t have an OS build environment set up to test the changes, but will see what I can do this evening.

 

 

[Samer]

Brian,

I found the issue regarding images not loading (weather and ajax loader). It was CloudFlare enabling “Hotlink Protection”. This has been removed and now the images are loading as expected. Let me know if you are seeing the same thing.

Thanks!

[Brian]

Thanks Samer,

I didn’t see a recurrence of the 403 error in my attempts at testing the new code this evening. Unfortunately I am having an error when attempting to upload the compiled code from the Arduino IDE (1.0.6) on my Yosemite 10.10.3 Mac. I’m an Arduino noob, so please excuse my ignorance if the answer to this is obvious.

I loosely followed the instructions here: https://opensprinkler.freshdesk.com/support/solutions/articles/5000165132-how-to-compile-opensprinkler-firmware however this doesn’t appear to have been updated to reflect the unified firmware yet. After installing the IDE, I grabbed the main sources from here: https://github.com/OpenSprinkler/OpenSprinklerGen2 and the aopensprinkler hardware description for my OS v2.3 from Ray’s repo here: https://github.com/rayshobby/opensprinkler. I then selected the Examples->OpenSprinklerGen2->mainArduino, selected Tools->Board->OpenSprinklerHW2.3, then selected my serial port (/dev/tty.Repleo-CH341-00001014, after first loading the drivers referenced on the firmware update instructions here: https://github.com/rayshobby/osFWUpdater/raw/master/Instructions.pdf). I then Verified/compiled the code without issue. Finally, I attempted to upload the compiled code from the IDE, but received the following (with verbose logging enabled):

/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/bin/avrdude -C/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/etc/avrdude.conf -v -v -v -v -patmega1284p -carduino -P/dev/tty.Repleo-CH341-00001014 -b115200 -D -Uflash:w:/var/folders/d1/h0z2v6cs2wxc1b96112l7qw9m603wb/T/build3006029794844066497.tmp/mainArduino.cpp.hex:i

 

avrdude: Version 5.11, compiled on Sep  2 2011 at 18:52:52

Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/

Copyright (c) 2007-2009 Joerg Wunsch

 

System wide configuration file is “/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/etc/avrdude.conf”

User configuration file is “/Users/brudy/.avrduderc”

User configuration file does not exist or is not a regular file, skipping

 

Using Port                    : /dev/tty.Repleo-CH341-00001014

Using Programmer              : arduino

Overriding Baud Rate          : 115200

avrdude: Send: 0 [30]   [20]

avrdude: Send: 0 [30]   [20]

avrdude: Send: 0 [30]   [20]

avrdude: ser_recv(): programmer is not responding

avrdude: stk500_recv(): programmer is not responding

 

After reaching 95% of the upload the OS restarted, but it doesn’t appear to be running the new code.

Any ideas?

[Samer]

Brian,

Sorry I thought you were on OSPi, apologizes. The new firmware should be released soon so if you want you could wait or maybe Ray can provide some assistance with compiling.

[Brian]

Thanks Samer,

This looks to be an issue with the CH341 USB serial drivers for Yosemite, as I experienced a few hangs/crashes during my many attempts to get the upload to work last night. I am currently setting up another build environment on a Linux box to see if I can work through this apparent driver issue.

[Brian]

Hi Samer,

I was finally able to test the caching changes in the 2.1.4 firmware, and unfortunately the problem persists. To test, I added a new program from the mobile interface, but was unable to see the new program on the browser until I reloaded the page.

Looking in Chrome’s debugger, I only see periodic calls to the jc and jo APIs, but only see a call to jp after reloading the page. I do, however, see the change to HTTP 1.1 and Cache-Control headers when the jp call is made. This suggests something else is going on as browser cached files typically appear in the debugger’s network timeline as such. This is likely something to do with the JavaScript handling this portion of the user interface, but I haven’t dug into it yet.

[Samer]

That is odd because if you look at the code that is saving a new program, the very end will send the new program to the controller, wait for a confirmation it’s done, request the updated program list from the controller, then tell you it’s done saving the program.

Code: https://github.com/OpenSprinkler/OpenSprinkler-App/blob/master/www/js/main.js#L7228-7236

I have selected the specific lines so you can see what I am referring to. You will notice the first call to sendToOS which just sends a request to OpenSprinkler. The done callback after calls updateControllerPrograms and it’s callback will change the page and show the message “Program added successfully”

Therefore, I am not sure what is causing the discrepancy you are seeing however I can say it is not common. At least this is the first I am hearing of the issue. Let us know if you find anything further or a way for me to replicate it. I am using Chrome on OS X and have never had this issue.

[Brian]

Hi Samer,

Perhaps I am not including sufficient detail to describe what I am observing.

The test environment consists of two client devices, the first being an Android phone running v1.4.2 of the OpenSprinkler mobile app, the second being a Chrome browser v42.0.2311.135 running on OSX Yosemite.

Test procedure:
1) Open the URL to the OpenSprinkler from the Chrome browser
2) On the Chrome browser, navigate to Edit Programs from the grid on the lower-right of the browser frame (the current set of programs are visible)
3) Launch the OpenSprinkler mobile app on the Android device
4) On the mobile app, navigate to Edit Programs (the current set of programs are visible)
5) On the mobile app, add a new program and save (the current set of programs, including the newly added program are visible)
6) On the Chrome browser, navigate to View Logs from the grid on the lower-right of the browser frame.
7) On the Chrome browser, navigate to Edit Programs from the grid on the lower-right of the browser frame (the previously viewed programs are visible, but the newly added program added from the mobile app is not visible)
8) On the Chrome browser, reload the current page (the current set of programs, including the newly added program are all visible)

[DaveC]

FWIW – I did a similar thing but I changed the program from the browser with the mobile app open. The mobile app did not see the change until I exited and restarted the app. I looked to see if there was refresh button on the mobile apps but there didn’t seem to be one.

[Samer]

Okay, now I understand. The issue is changing programs on one device and not seeing that reflected on a different device.

Yes this has nothing to do with caching and instead is a logical error. Basically, the apps functioning like its the only instance making changes to OpenSprinkler. I can easily add /jp call to the constant refresh and the only reason I don’t do it now is to avoid any HTTP round trips if not needed. Clearly this is needed so I will see what I can do.

Thanks for the detail!

Update: This will be addressed in the next update. My testing shows I can’t update program data at the same frequency as status information however I can do so less frequently without issues. Therefore, in the next app update (very soon hopefully), the problem will be addressed as the app will update the program and station information every 30 seconds. I could also update when you enter the programs page however that would require me waiting on the HTTP reply before loading the page which is not something I want to do. Hopefully this addresses the issue sufficiently. Changes here: https://github.com/OpenSprinkler/OpenSprinkler-App/commit/7e88d664fb686b90f93b5814f5c3b19eee613b84 (under main.js).

Long term, the fix will be reducing the HTTP requests which will be addressed in the next firmware. Eventually, a push instead of frequent pulls will resolve the issue 100%.

[DaveC]

Re: Long term push vs pull
This was an item I was going to ask about as I develop the automation interface. It would be nice if there were some push events rather than having to poll all the time. I’m not far along enough to know what I’d like, but my needs would probably be a subset of yours. I don’t intent to provide an interface for programming. I’m looking to provide status and simple actions like enable/disable operation and set rain delay. It would be great to have an idea of what you’re thinking when you’re getting close so I can plan for it. Very cool 🙂

[Author]

Posts