OpenSprinkler Forums Hardware Questions OpenSprinkler Pi (OSPi) Builduing a linux service that gets weather data

This topic contains 16 replies, has 0 voices, and was last updated by  Samer 6 years, 2 months ago.

Viewing 15 posts - 1 through 15 (of 18 total)
  • Author
    Posts
  • #22456

    aleatorvb
    Member

    Hello,
    I’m builduing a linux service that gets weather data, and i intend to run it on a linux machine, but it will be possible to run it on pi with minimal resource usage (10-12 mb of ram) and dependencies (php, memcached).

    What weather services are curently offering free api access? I have found
    – open weather
    – forecast io
    – world weather online
    – ham weather

    I plan to grab data from all these sources and keep a local copy in ram and do some weighted averages to get the current weather, and the next 24 hours important events (probability of rain, ice, sub-zero temperatures, etc). I plan to incluse this in a bigger linux service that can drive the open sprinkler operation.

    thank you for any help

    p.s. all the code will be released when it will be stable and working.


    Update: GitHub repository https://github.com/vladbabii/multiweather

    #23810

    Ray
    Keymaster

    wunderground is what a lot of people recommend. You do need to apply for an ID but you can get a free one if you don’t query it too frequently per day.

    #23811

    aleatorvb
    Member

    Progress update
    – 10 free services for current weather
    – out of the 10, 5 have forecast data
    – all services support a poll interval of at least 10 minutes (more than enough for private usage for a location)
    – configuration options for all apis
    – statistical analysis of current weather to produce a better result
    – output to display, file, memcache, mysql database

    Working on
    – statistics code to produce a better aproximations of forecasts

    Weather data has:
    – temperature
    – pressure
    – humidity
    – probability of rain or snow

    Not sure if i should also gather wind speed and direction.

    #23814

    Samer
    Keymaster

    I am planning on incorporating something like this into my web app and would love to know if you have made any more progress and subsequently planning on releasing your code. Would save me the trouble and the collaboration could only help.

    Thank you in advance, also the code for the app I am making is available at: https://github.com/salbahra/OpenSprinkler-Controller

    #23815

    aleatorvb
    Member

    Sample output for current weather


    weather_current
    timestamp - 1370932571
    values
    forecastio
    humidity - 0.76
    precip_intensity - 0.03
    precip_probability - 0.02
    pressure - 1009.86
    temperature - 20.82
    openweather
    humidity - 0.4
    pressure - 1011
    temperature - 29.497
    worldweatheronline
    humidity - 0.83
    pressure - 1009
    temperature - 20
    hamweather
    humidity - 0.83
    pressure - 1009
    temperature - 20
    wunderground
    humidity - 0.78
    pressure - 1010
    temperature - 21
    separated_values
    humidity
    0 - 0.76
    1 - 0.4
    2 - 0.83
    3 - 0.83
    4 - 0.78
    precip_intensity
    0 - 0.03
    precip_probability
    0 - 0.02
    pressure
    0 - 1009.86
    1 - 1011
    2 - 1009
    3 - 1009
    4 - 1010
    temperature
    0 - 20.82
    1 - 29.497
    2 - 20
    3 - 20
    4 - 21
    median
    humidity - 0.78
    precip_intensity - 0.03
    precip_probability - 0.02
    pressure - 1009.86
    temperature - 20.82
    averages
    humidity - 0.72
    precip_intensity - 0.03
    precip_probability - 0.02
    pressure - 1009.772
    temperature - 22.2634

    One should use $[ ]

    Sample output for forecasting


    weather_forecast
    timestamp - 1370932571
    values
    forecastio
    hourly
    2013061106
    humidity - 0.78
    precip_intensity - 0.826
    precip_probability - 0.87
    pressure - 1008.86
    temperature - 21.72
    2013061107
    humidity - 0.81
    precip_intensity - 0.846
    precip_probability - 0.89
    pressure - 1008.99
    temperature - 20.74
    2013061108
    humidity - 0.85
    precip_intensity - 0.848
    precip_probability - 0.92
    pressure - 1009.15
    temperature - 19.75
    2013061109
    humidity - 0.87
    precip_intensity - 0.832
    precip_probability - 0.93
    pressure - 1009.32
    temperature - 18.73
    2013061110
    humidity - 0.89
    precip_intensity - 0.811
    precip_probability - 0.95
    pressure - 1009.51
    temperature - 17.72
    2013061111
    humidity - 0.91
    precip_intensity - 0.775
    precip_probability - 0.95
    pressure - 1009.7
    temperature - 16.86
    2013061112
    humidity - 0.62
    precip_intensity - 0.363
    precip_probability - 0.87
    pressure - 1008.92
    temperature - 25.55
    2013061101
    humidity - 0.63
    precip_intensity - 0.465
    precip_probability - 0.92
    pressure - 1008.8
    temperature - 25.68
    2013061102
    humidity - 0.63
    precip_intensity - 0.567
    precip_probability - 0.91
    pressure - 1008.71
    temperature - 25.68
    2013061103
    humidity - 0.65
    precip_intensity - 0.655
    precip_probability - 0.88
    pressure - 1008.67
    temperature - 25.25
    2013061104
    humidity - 0.69
    precip_intensity - 0.726
    precip_probability - 0.88
    pressure - 1008.68
    temperature - 24.21
    2013061105
    humidity - 0.74
    precip_intensity - 0.785
    precip_probability - 0.87
    pressure - 1008.76
    temperature - 22.89
    2013061212
    humidity - 0.51
    precip_intensity - 0.089
    precip_probability - 0.09
    pressure - 1010.63
    temperature - 27.05
    2013061201
    humidity - 0.5
    precip_intensity - 0.084
    precip_probability - 0.07
    pressure - 1010.61
    temperature - 26.87
    2013061202
    humidity - 0.52
    precip_intensity - 0.073
    precip_probability - 0.06
    pressure - 1010.63
    temperature - 26.32
    2013061203
    humidity - 0.55
    precip_intensity - 0.073
    precip_probability - 0.1
    pressure - 1010.73
    temperature - 25.43
    2013061204
    humidity - 0.61
    precip_intensity - 0.094
    precip_probability - 0.22
    pressure - 1010.97
    temperature - 24.19
    2013061205
    humidity - 0.68
    precip_intensity - 0.125
    precip_probability - 0.39
    pressure - 1011.29
    temperature - 22.78
    2013061206
    humidity - 0.75
    precip_intensity - 0.15
    precip_probability - 0.51
    pressure - 1011.64
    temperature - 21.52
    2013061207
    humidity - 0.79
    precip_intensity - 0.162
    precip_probability - 0.54
    pressure - 1012.03
    temperature - 20.42
    2013061208
    humidity - 0.83
    precip_intensity - 0.168
    precip_probability - 0.53
    pressure - 1012.45
    temperature - 19.33
    2013061209
    humidity - 0.86
    precip_intensity - 0.167
    precip_probability - 0.49
    pressure - 1012.84
    temperature - 18.34
    2013061210
    humidity - 0.88
    precip_intensity - 0.152
    precip_probability - 0.44
    pressure - 1013.13
    temperature - 17.62
    2013061211
    humidity - 0.89
    precip_intensity - 0.132
    precip_probability - 0.36
    pressure - 1013.36
    temperature - 17.15
    2013061312
    humidity - 0.9
    precip_intensity - 0.131
    precip_probability - 0.33
    pressure - 1013.6
    temperature - 16.79
    2013061301
    humidity - 0.91
    precip_intensity - 0.165
    precip_probability - 0.35
    pressure - 1013.86
    temperature - 16.37
    2013061302
    humidity - 0.92
    precip_intensity - 0.216
    precip_probability - 0.4
    pressure - 1014.14
    temperature - 16.04
    2013061303
    humidity - 0.91
    precip_intensity - 0.248
    precip_probability - 0.44
    pressure - 1014.41
    temperature - 16.13
    2013061304
    humidity - 0.89
    precip_intensity - 0.244
    precip_probability - 0.43
    pressure - 1014.67
    temperature - 16.79
    2013061305
    humidity - 0.86
    precip_intensity - 0.222
    precip_probability - 0.41
    pressure - 1014.91
    temperature - 17.78
    2013061306
    humidity - 0.82
    precip_intensity - 0.208
    precip_probability - 0.39
    pressure - 1015.09
    temperature - 18.9
    20130610
    humidity - 0.74
    precipProbability -
    precip_intensity - 0.283
    pressure - 1009.39
    temperature -
    20130611
    humidity - 0.73
    precipProbability -
    precip_intensity - 0.229
    pressure - 1010.66
    temperature -
    20130612
    humidity - 0.77
    precipProbability -
    precip_intensity - 0.291
    pressure - 1014.67
    temperature -
    20130613
    humidity - 0.7
    precipProbability -
    precip_intensity - 0.105
    pressure - 1015.74
    temperature -
    20130614
    humidity - 0.72
    precipProbability -
    precip_intensity - 0.16
    pressure - 1014.96
    temperature -
    20130615
    humidity - 0.71
    precipProbability -
    precip_intensity - 0.043
    pressure - 1014.68
    temperature -
    20130616
    humidity - 0.67
    precipProbability -
    precip_intensity - 0.02
    pressure - 1014.82
    temperature -
    20130617
    humidity - 0.64
    precipProbability -
    precip_intensity - 0
    pressure - 1013.81
    temperature -
    daily
    openweather
    daily
    20130611
    pressure - 1008.96
    humidity - 0.99
    temperature - 29.141
    temperature_min - 28.9814
    temperature_max - 29.141
    20130612
    pressure - 1010.92
    humidity - 0.86
    temperature - 29.344
    temperature_min - 29.2699
    temperature_max - 29.344
    20130613
    pressure - 1014.94
    humidity - 0.67
    temperature - 29.3299
    temperature_min - 29.3299
    temperature_max - 29.3299
    20130614
    pressure - 1014.54
    humidity - 0.72
    temperature - 29.4487
    temperature_min - 29.4487
    temperature_max - 29.4487
    20130615
    pressure - 1014.85
    humidity - 0.81
    temperature - 29.3464
    temperature_min - 29.3464
    temperature_max - 29.3464
    20130616
    pressure - 1016.27
    humidity - 1
    temperature - 29.1586
    temperature_min - 29.1586
    temperature_max - 29.1586
    worldweatheronline
    hamweather
    wunderground
    daily
    20130611
    precip_probability - 0.8
    humidity - 0.69
    temperature_min - 15
    temperature_max - 27
    temperature - 21
    20130612
    precip_probability - 0.8
    humidity - 0.71
    temperature_min - 16
    temperature_max - 25
    temperature - 20.5
    20130613
    precip_probability - 0.8
    humidity - 0.79
    temperature_min - 16
    temperature_max - 25
    temperature - 20.5
    20130614
    precip_probability - 0.8
    humidity - 0.8
    temperature_min - 16
    temperature_max - 25
    temperature - 20.5
    separated_values
    20130611
    pressure
    0 - 1008.96
    humidity
    0 - 0.99
    1 - 0.69
    temperature
    0 - 29.141
    1 - 21
    temperature_min
    0 - 28.9814
    1 - 15
    temperature_max
    0 - 29.141
    1 - 27
    precip_probability
    0 - 0.8
    20130612
    pressure
    0 - 1010.92
    humidity
    0 - 0.86
    1 - 0.71
    temperature
    0 - 29.344
    1 - 20.5
    temperature_min
    0 - 29.2699
    1 - 16
    temperature_max
    0 - 29.344
    1 - 25
    precip_probability
    0 - 0.8
    20130613
    pressure
    0 - 1014.94
    humidity
    0 - 0.67
    1 - 0.79
    temperature
    0 - 29.3299
    1 - 20.5
    temperature_min
    0 - 29.3299
    1 - 16
    temperature_max
    0 - 29.3299
    1 - 25
    precip_probability
    0 - 0.8
    20130614
    pressure
    0 - 1014.54
    humidity
    0 - 0.72
    1 - 0.8
    temperature
    0 - 29.4487
    1 - 20.5
    temperature_min
    0 - 29.4487
    1 - 16
    temperature_max
    0 - 29.4487
    1 - 25
    precip_probability
    0 - 0.8
    20130615
    pressure
    0 - 1014.85
    humidity
    0 - 0.81
    temperature
    0 - 29.3464
    temperature_min
    0 - 29.3464
    temperature_max
    0 - 29.3464
    20130616
    pressure
    0 - 1016.27
    humidity
    0 - 1
    temperature
    0 - 29.1586
    temperature_min
    0 - 29.1586
    temperature_max
    0 - 29.1586
    median
    20130611
    pressure - 1008.96
    humidity - 0.84
    temperature - 25.0705
    temperature_min - 21.9907
    temperature_max - 28.0705
    precip_probability - 0.8
    20130612
    pressure - 1010.92
    humidity - 0.785
    temperature - 24.922
    temperature_min - 22.63495
    temperature_max - 27.172
    precip_probability - 0.8
    20130613
    pressure - 1014.94
    humidity - 0.73
    temperature - 24.91495
    temperature_min - 22.66495
    temperature_max - 27.16495
    precip_probability - 0.8
    20130614
    pressure - 1014.54
    humidity - 0.76
    temperature - 24.97435
    temperature_min - 22.72435
    temperature_max - 27.22435
    precip_probability - 0.8
    20130615
    pressure - 1014.85
    humidity - 0.81
    temperature - 29.3464
    temperature_min - 29.3464
    temperature_max - 29.3464
    20130616
    pressure - 1016.27
    humidity - 1
    temperature - 29.1586
    temperature_min - 29.1586
    temperature_max - 29.1586
    averages
    20130611
    pressure - 1008.96
    humidity - 0.84
    temperature - 25.0705
    temperature_min - 21.9907
    temperature_max - 28.0705
    precip_probability - 0.8
    20130612
    pressure - 1010.92
    humidity - 0.785
    temperature - 24.922
    temperature_min - 22.63495
    temperature_max - 27.172
    precip_probability - 0.8
    20130613
    pressure - 1014.94
    humidity - 0.73
    temperature - 24.91495
    temperature_min - 22.66495
    temperature_max - 27.16495
    precip_probability - 0.8
    20130614
    pressure - 1014.54
    humidity - 0.76
    temperature - 24.97435
    temperature_min - 22.72435
    temperature_max - 27.22435
    precip_probability - 0.8
    20130615
    pressure - 1014.85
    humidity - 0.81
    temperature - 29.3464
    temperature_min - 29.3464
    temperature_max - 29.3464
    20130616
    pressure - 1016.27
    humidity - 1
    temperature - 29.1586
    temperature_min - 29.1586
    temperature_max - 29.1586

    One should use $[ ][ ].

    I prefer using median to get a better value instead of average because it removes values that are way different than the others in the set.

    I am optimising the code right now. It has multiple output classes – mysql, file and echo right now but i want to add memcached support. Give me a day or two to clean up the code.

    I have done a number of services for pi and found a great option to be to store temporary data in memcache because it’s low footprint – a few megabytes of ram and very little cpu usage.

    What kind of storage would you want to use in your project?

    #23816

    aleatorvb
    Member

    This is the config file with the stable + good data + free services i found [keys have been replaced with random values]


    $weather_api_list=array(
    'forecastio' => array(
    // https://developer.forecast.io/
    'key' =>'3a3c60d8'
    ,'latitude' => '27.1556'
    ,'longitude' => '47.5914'
    ,'delay' => 10*60
    ,'use' => true
    )
    ,'openweather' => array(
    // two requests per update - current and forecast
    // http://openweathermap.org/api
    // use city name or latitude and longitude
    // city name will be used if all 3 fields are filled in
    'key' => 'ccf359c274'
    ,'cityname' => 'london,uk'
    ,'latitude' => '17.1556'
    ,'longitude' => '87.5914'
    ,'delay' => 10*60
    ,'use' => true
    )
    ,'worldweatheronline'=> array(
    // no forecast
    // http://developer.worldweatheronline.com/io-docs
    'key' => 'nh8r'
    ,'q' => 'London'
    ,'delay' => 10*60
    ,'use' => true
    )
    ,'hamweather' => array(
    // no forecast
    // http://www.hamweather.com/support/documentation/aeris/
    'location' => 'iasi,ro'
    ,'id' => 'zjS58'
    ,'secret' => 'pu874'
    ,'delay' => 10*60
    ,'use' => true
    )
    ,'wunderground' => array(
    // two requests per update - current and forecast
    'key' => 'f95'
    ,'location' => 'Iasi'
    ,'delay' => 10*60
    ,'use' => true
    )
    );

    // Delay between two checks, in seconds
    $main_delay=5;

    // Number of loops, use null for infinite or a number for limited run
    // $loops=1; or $loops=null;
    $loops=1;

    // Storage method
    // echo - displays on screen
    // file - writes to 'output.txt'
    $storage_class='echo';

    #23817

    aleatorvb
    Member

    I like the application you did.
    Could you separate the calls to the io sockets in a different class in your code?
    I’d like to create an alternate class to use remote curl calls to control a networked manual-mode sprinkler.
    If you would be willing to work on the sheduler front-end I will do the backend.

    #23818

    Samer
    Keymaster

    I honestly haven’t given it too much thought yet been focused on packaging up the web app as a whole right now. But, I was interested in having the PHP (which is already running on a crontab) to maybe poll the weather and at the very least displaying that to the user within the app. Of course, it would be very neat if the PHP could activate a rain delay if it noticed the weather forecast predicated heavy rains (similar to the function of a rain sensor).

    Right now, I am not using any storage for my app. It functions by grabbing the data from the OpenSprinkler. Also, I am not using a Raspberry Pi so I am not having to interact with the OpenSprinkler at a hardware level. With how efficient the logging system works I want to avoid tying a database to this setup to be honest with you. What I have in my head is simply checking at a set time (ex. 11PM) the weather for tomorrow. If it’s (and im making this up) >80% chance of rain automatically telling the OpenSprinkler to trigger a rain delay of 24 hours. Something simple. I was also thinking of adding a weather button on the main menu which just uses the location setting on the OpenSprinkler to grab weather data and display it in the iPhone. Of course, if your using an iPhone you can simply pull down your notification center and see the weather which is why that particular feature has remained a low priority.

    I also think the PHP could fire off an email if it triggers an automatic rain delay just to inform the user something happened (of course could be disabled).

    Thank you for the detailed post by the way! The research on which API’s is helpful as is the array structure’s and methods. I will update if I get some time to play with the weather frontend and let you know!

    #23819

    aleatorvb
    Member

    I’ve posted what i have right now on github –https://github.com/vladbabii/multiweather
    I’m getting the page with file_get_contents right now – i will add support for curl.

    #23820

    aleatorvb
    Member

    I have a storage class written for the pi. It uses memcache (with fallback to files/mysql) for reads and memcached / files / mysql for storing data, wichever is available. I created it for a webservice that monitors 16 nut-capable upses and storring data for the last few days. I will upload it to github if you need it.

    For what you need – to poll the weather service if it should trigger a rain delay – memcache is perfect. I will add to github a class you can just include in your project and just run the multiweather code as a service or cron.

    Any bug reports / features requests are welcome.

    #23821

    Samer
    Keymaster

    I am using file_get_contents but I set a global timeout. I was debating curl but decided that some installations might not have curl support (I don’t think it is part of the base PHP package) so I decided to stick with file_get_contents for now. I did refactor my code to call file_get_contents from my own function that way I can easily change how I wish to interface with the OS in the future and also add more error catching, etc.

    Thanks for the code post!

    #23822

    aleatorvb
    Member

    What I’d change in your code
    – a php file that runs as a service and polls the data from the OS and saves it to memcache
    – the webinterface polls the memcache information
    – save a serialized object in memcache and just deserialize it when you read the data
    This way a laggy network connections or other issues won’t trigger errors.
    All the exchanges happen in ram – fast ui updates.

    #23823

    Samer
    Keymaster

    Appreciate the suggestions! In my situation the OS is wired and in close proximity to my server thus the latency is almost nothing between the two. So far, I have noticed no problem in speed to be honest with you although I completely understand where you are coming from. I will consider using a database/cache but I really wanted to avoid it. Just to be clear my server already runs MySQL for other services so it would be no problem to use a database, personally. The way I see it, if the app is going to lag sending commands (waiting on the OS to send command) then it is probably okay to wait for it to load although for me personally the web app seems very responsive/fast.

    For my sake, have you tried the web app and if so are you experiencing lag? Would definitely help the direction I take the web app in. I will be honest though, I built the app for my father (Father’s Day is coming up soon after all) and he seems very happy with it (he even said it’s too responsive haha he was referring to the flip switches though) which is all that matters to me. But, if there is a problem I can fix I would be glad to!

    #23824

    aleatorvb
    Member

    I love the app, i did a few mock tests (don’t have the controller yet, but soon).
    I was planning on doing something like this too, but with scheduling and a few other options.
    Would it be ok with you if i fork the project to add more things to it?

    #23825

    Samer
    Keymaster

    Your absolutely right by the way, without a controller you hit the time out a lot. I will consider that in the future. I also added a feature in the installer to verify the IP to help prevent timeouts by accident for unsuspecting users.

Viewing 15 posts - 1 through 15 (of 18 total)

You must be logged in to reply to this topic.

OpenSprinkler Forums Hardware Questions OpenSprinkler Pi (OSPi) Builduing a linux service that gets weather data