Codeable info

Using PHP, YQL And RSS To Add Living Data To Your Site

Posted on by in WordPress Tutorials

Back in the 90’s, when the web was owned by coding cowboys and sites with nothing but a webcam shot of a fish tank updated every 3 minutes were the top attraction, I was working on a web application for the Insurance Industry. Yeah, I can smell your excitement from here! Property, Life AND Casualty! Yeah! OK I know that’s hot, but let’s settle down. One of the main drivers of daily use of the site was new information on the front page every day – stuff people in regulated industries needed to do their job. That information basically consisted of who did what to whom in the Insurance Industry. Sort of like People Magazine but with more money involved! Tiger Woods has nothing on AIG or Marsh & McLennan! “Improper steering of business?” mused Tiger, “that’s hot!”

Back then we bought our data from big vendors like Hoovers, Dun and Bradstreet, Wards and many others. Frequent news updates brought people to the site. At the time, it was hard to get a company’s financial reports, portfolio AND news all in one place with little effort. The value of our product came from the ease in which the analysts got the information. On your blog you probably have a specific topic you are covering (assuming it’s not an exercise in Narcissism and I am not judging here!) There is probably a great deal of available information fresh every day for your blog’s focus. Wouldn’t it be great if you could leverage that data.

Today there are APIs and data sources aplenty and many of them are free. Not just free, but useful information for every day. One of those APIs comes from the Yahoo Developer Network. Today I was thinking it would be nice to create and easy to use plugin to add arbitrary weather anywhere on your blog. Recently here on WPMods there was an article about adding shortcodes to your WordPress website. Why not put that to immediate and fun use! A simple and easy to do app would be a weather module. Something along the lines of a shortcode like [myweather loc=”busy corner, TN”] that gets replaced by weather data. That way if I am talking about Busy Corner, TN (where the population of cows and bison outnumber registered voters) I can also show the current weather data.

So my first question this morning was, where am I going to find weather data?

Yahoo Weather RSS Feed

URL: http://developer.yahoo.com/weather/

The weather RSS feed is dead simple to use. If you go to the url above you will see the documentation is really good and the base URL you would submit to get the information looks like this:

http://weather.yahooapis.com/forecastrss

Add to that “request parameters” and off you go. What parameters are there? There are actually only two:

w is for the WOEID.
u is for the units (Fahrenheit or Celsius).

Awesome, that must be it. We are done… oh wait, what the hell is WOEID? WOE is me I had to look it up. After a lot of poking and prodding I found that the WOEID is obtainable via an SQL like syntax. In fact, while this article started off in my mind to be about adding weather to your site, when I discovered the joy that is YQL it became about much more than that – it is about leveraging all the cool data Yahoo! has and making your site extra slick. In fact I was so amazed at the data I found I sent along the links to guys I used to work for who create classified intelligence platforms. Yahoo has created one really powerful platform that I think is the equal if not better than some of the systems I’d seen in cloak and dagger land. To be fair, their source of data comes from the same government sources, so Yahoo has only added depth and organization and a wonderful API – but that is the name of the game for developers. It is always the last mile of any platform; you can come up with great ideas but it takes a great API that taps into your data to really make it happen. If it doesn’t exist you have to create it. Luckily for us this does exist, is powerful and polished and ready for prime time.

Now we aren’t going to do anything really amazing in this tutorial. We just want to start slow and easy and get the weather. Even Jack Bauer needs to know the weather, right?

Whoa! What’s this WOEID of which you speak?

Where on Earth can we find information about this woeid thing?
Where On Earth indeed. There is an entire geospatial database just waiting for you to tap it. You need to understand YQL to do so but that won’t take long to explain. In a nutshell there is a gazetteer that relates latitude, longitude and name variations with an ID called WOEID. It is actually a pretty impressive system. It is based in part on the structure of the Global Name Server the National Geospatial-intelligence Agency (NGA) produces. But, unlike NGA which has no charter to cover the United States and its territories, this system also incorporates US data as well.

If you aren’t easily daunted, take a look at the YQL tables available for mining.

If you are easily daunted I suggest you think about kittens and rainbows. In the meanwhile, the first group that did click the link got to see some pretty intense stuff. It’s not as simple as the database that handles your address book. The wealth of data here can’t be understated – from the smallest identified refugee camp in Sudan to the largest cities in the world; this system has tons of information. It is hierarchical too so you can tell not only the WOEID of Busy Corner but that it is in Coffee County, Tennessee, United States. Postal Code 37355, centroid point: latitude 35.533150, longitude -86.161919, even the bounding box you’d need to have the whole town covered on your map. You probably are already thinking of the black ski-mask ops you’d like to plan using this data right now, but show some discipline and let’s get back to basics and determining the weather!

Ok, let’s learn some YQL and make this data our minion.

YQL? – Y not?

If you know SQL you know YQL. No joke – they are so similar I hesitate to waste your time even talking about it. If you don’t know SQL then the time for you to learn is now. I can’t teach you SQL in the next three paragraphs so bookmark this article and come back after your journey of a thousand steps has resulted in enlightenment. Namaste. Gesundheit!

The query to get the WOEID of Busy Corner, TN is no more complex than this:

select * from geo.places where text = 'busy corner, tn' [link leads to YQL console]

If you just want the WOEID without all of the other cool data then simply change the query to the not unexpected form of:

select woeid from geo.places where text = 'busy corner, tn'

Click one of the links above and check out the YQL console. As a web app I have to say it is impressive. The speed and ease of use is just fantastic. The example queries are instructive too. It gives you access to so much stuff you might forget to do anything with WordPress today and explore. I wouldn’t blame you, honestly. Just remember that it would be great if you could ALSO leverage that data on your website. If you are curious about what else you can do with YQL then start reading the YQL Guide for more details. Nested queries are just one of the many joys of this language.

Got my Query set, Got my Weather RSS, now what?

Now is when we make magic. You have just put centuries of information science to the test. Your code has tweaked millions of dollars’ worth of computers, fiber optics, vague cloud like things described variously as the Interwebz or The Matrix, had information encrypted, decrypted, compressed, uncompressed and transported from who knows where all just so you can show if it is raining or shining in someplace as obscure as Busy Corner, Tennessee. But now you want to know how to get it. You have two options, really. The first is to open up a pipe and read straight XMLv1.0. The other is to get it via json. For this article let’s just look the XML route.

Let’s look at the most basic method of getting the data. First, we did our homework and figured out what our WOEID is via the query. In a later article I will show how we can incorporate this part into our plugin such that we can search for an arbitrary location and disambiguate the result set to find the correct WOEID. For now, do it manually via the YQL console.

Our final, constructed RSS query is: http://weather.yahooapis.com/forecastrss?w=2372560&u=f

Now we need to do something on the server side to pull in that data. The code is rather straight forward.

load('http://weather.yahooapis.com/forecastrss?w=2372560&u=f');
  $arrWeather = array();
  foreach ($doc->getElementsByTagName('item') as $node) {
    $itemRSS = array ( 
      'title' => $node->getElementsByTagName('title')->item(0)->nodeValue,
      'desc' => $node->getElementsByTagName('description')->item(0)->nodeValue,
      'link' => $node->getElementsByTagName('link')->item(0)->nodeValue,
      'date' => $node->getElementsByTagName('pubDate')->item(0)->nodeValue
      );
    //array_push($arrWeather, $itemRSS);
    echo '<ul>';
    echo '<li>Title: ' .$itemRSS['title'] .'</li>';
    echo '<li>Description: ' .$itemRSS['desc'] .'</li>';
    echo '<li>Link: ' .$itemRSS['link'] .'</li>';
    echo '<li>Date: ' .$itemRSS['date'] .'</li>';
    echo '</ul>'; 
  }
?&gt;

That will produce all the information you need to show the weather for any given WOEID. This simple little bit of code makes it look as easy as it is. Remove the echo’s and uncomment the array_push if you want to incorporate this into a larger bit of code. You could create a theme where in the settings the WOEID is set to give a running feed of weather and news. Couple that with a timer to refresh every few minutes and you have a living page with automated data collection.

There is a lot more you can do with YQL. You can combine your search on locations right with your search for weather and construct something like this:

SELECT * FROM weather.forecast WHERE location IN (SELECT postal FROM geo.places WHERE text = 'busy corner, tn')

In the console this produces a link which you represents the query. That link can be edited with the location literal string of the query removed. You then replace it with any arbitrary text and load it like any DOMDocument. Now how cool is this?

Examining the XML result from this we see the nest of elements goes: query results channel item

Item looks the same as the RSS feed with a few additions. With some more work you can provide a much more significant block of weather data with YQL straight than you can with YQL and the Weather RSS feed. However it gets even better. There are many data reserves you can tap.

What do you think this query does?

SELECT * FROM music.video.SEARCH WHERE keyword="The Vandals"

or this?

SELECT * FROM LOCAL.SEARCH WHERE zip='37215' AND query='bar'

Really the only limits are the tables and data plus your imagination. For some APIs yahoo does want to you to sign up to get an API key. This is expected since information is powerful and they want to make sure they know you aren’t going to abuse it. You can sign up for the API key here:

https://developer.apps.yahoo.com/projects

Considering how much you get at no cost this is a nice trade. From what I’ve seen from others, there is no downside to this. So get coding. I’ve only slipped the door open a crack. It is up to you to take this information and run with it.

Enjoy!

Comments Closed

Codeable info