The following post is a portion of the D3 Tips and Tricks book which is free to download. To use this post in context, consider it with the others in the blog or just download the the book as a pdf / epub or mobi .
----------------------------------------------------------
Using the Yahoo Query Language (YQL) to get data.
I will use an example of looking for current weather information for Wellington New Zealand from the Weather Underground (wunderground).
One of the things that I find frustrating is difficulty getting data to play with. Often I will find myself thinking “It would be neat to see if this technique works” but then I spend a couple of hours looking for data that will be appropriate.
Another difficulty is that access to dynamic data is also problematic. D3.js really shines when displaying information that is changing and interactive. Finding data that is changing and which you can interact with is not easy.
Then, when you do come across a web site that has an interesting data set, accessing that data becomes programmatically difficult because of restrictions in accessing information that crosses domains. A solution to this problem (and there are a few) is to have a data set that is in a domain that permits Cross-Origin Resource Sharing (CORS). These are not as common as you might hope as it presents security concerns that need to be addressed.
One resource that I have come across is particularly useful for solving the problems outlined above is theYahoo! Developer Network which operates a service where you can write a query in an SQL type fashion that will return data in json (or others) format. This query can be formed into an http request so that you can simply paste a generated URL directly into your browsers URL bar and it will return the requested data. Better than that, because it supports CORS (and because d3.js can manage a CORS request without breaking a sweat) all you need to do is put the URL into you code (as a d3.json call for example) and you will be up and running.
In the ‘Examples’ chapter there will be several examples that use YQL queries to retrieve data as is is a great resource.
YQL by example
I will use an example of looking for current weather information for Wellington New Zealand from the Weather Underground (wunderground).
In this example, we’re interested in retrieving a JSON ‘blob’ that contains data on the weather conditions (temperature, wind speed / direction, humidity etc) for a specific location.
Start at the page for the console (http://developer.yahoo.com/yql/console/). On the left there is a range of data ‘topics that you can choose from. We will want to access one of the community tables, so select ‘Show Community Tables’ and scroll down the list until you find ‘wunderground’ and ‘wunderground.currentobservation’
 |
| Select Community Tables and wunderground.currentobservation |
This will bring up a query in the console that is looking for the current weather from San Francisco International airport.
 |
| Query for weather from San Francisco |
Start at the page for the console (http://developer.yahoo.com/yql/console/). On the left there is a range of data ‘topics that you can choose from. We will want to access one of the community tables, so select ‘Show Community Tables’ and scroll down the list until you find ‘wunderground’ and ‘wunderground.currentobservation’
For our example we want to be a bit more specific, so we replace the ‘SFO’ ‘location’ with ‘Wellington, New Zealand’.
If you then click on the ‘Test’ button the query will run and results will be presented in the box below it;
 |
| JSON weather from San Francisco |
If that looks like the results you were expecting, fantastic!
The next gem from YQL is the REST query that is automatically generated at the bottom of the page;
 |
| REST Query |
If you select this query and past it in your URL bar, you will have the JSON data returned into your browser.
At this point it might seem a bit confusing, and if you’re unfamiliar with how JSON based data sets are structured it will DEFINITELY be confusing (You will need to do some research to get up to speed there). My advice is to first understand a bit about JSON and then examine the output from the REST query in the developer console on your browser (this is a fairly long topic in itself, so I will not cover it here sorry). Additionally check out the ‘Examples’ chapter for real world usage.
If you are comfortable with understanding how the data can be encoded, it is then just a matter of including the REST query in your call when selecting data using d3.json and you will be away.
Please check out the examples I will include in the ‘Examples’ chapter for a deeper look at the use of the data with d3.js (This will also help with understanding the JSON structure).
The description above (and heaps of other stuff) is in the D3 Tips and Tricks book that can be downloaded for free (or donate if you really want to :-)).
This is incredible.
ReplyDelete