Getting the Data

We're going to jump forward a little bit here to the bit of the JavaScript code that loads the data for the graph.

I'm going to go out of the sequence of the code here, because if you know what the data is that you're using, it will make explaining some of the other functions that are coming up much easier.

So the piece that grabs the data is this bit.

d3.tsv("data/data.tsv", function(error, data) {
    data.forEach(function(d) {
        d.date = parseDate(d.date);
        d.close = +d.close;
    });

And in fact it's a combination of a few bits and another bit that isn't shown!, But let's take it one step at a time :-)

Ok... There's lots of different ways that we can get data into our web page to turn into graphics. And the way that you'll want to use will probably depend more on the format that it is in than the mechanism you want to use for importing.

For instance, if it's only a few points of data we could include the information directly in the JavaScript.

That would make it look something like;

var data = [
    {date:"1-May-12",close:”58.13”},
    {date:"30-Apr-12",close:”53.98”},
    {date:"27-Apr-12",close:”67.00”},
    {date:"26-Apr-12",close:”89.70”},
    {date:"25-Apr-12",close:”99.00”}
];

The format of the data shown above is called JSON (JavaScript Object Notation) and it's a great way to include data since it's easy for humans to read what's in there and it's easy for computers to parse the data out.

But if you've got a fair bit of data or if the data you want to include is dynamic and could be changing from one moment to the next, you'll want to load it from an external source. That's when we call on D3's 'Request' functions.

A 'Request' is a function that instructs the browser to reach out and grab some data from somewhere. It could be stored locally (on the web server) or somewhere out in the internet.

There are different types of requests depending on the type of data you want to ingest. Each type of data is formatted with different rules, so the different requests interpret those rules to make sure that the data is returned to the D3 processing in a format that it understands. You could therefore think of the different 'Requests' as translators and the different data formats as being foreign languages.

The different types of data that can be requested by D3 are;

• text - A plain old piece of text that can optionally be encoded in a particular way (see the D3 API).
• json - This is the afore mentioned JavaScript Object Notation.
• xml - Extensible Markup Language is a language that is widely used for encoding documents in a human readable forrm.
• html - HyperText Markup Language is the language used for displaying web pages.
• csv -  Comma Separated Values is a widely used format for storing data where plain text information is separated by (wait for it) commas.
• tsv - Tab Separated Values is a widely used format for storing data where plain text information is separated by a tab-stop character.

Details on these ingestion methods and the formats for the requests are explained well on the D3 Wiki page here - https://github.com/mbostock/d3/wiki/Requests. In this particular script we will look at the tsv request method.

Now, it is important to note here that this is not an exclusive list of what can be ingested. If you've got some funky data in a weird format, you can still get it in, but you will most likely need to stand up a small amount of code somewhere else in your page to do the conversion (we will look at this process when describing getting data from a MySQL database).

So, back to our request...

d3.tsv("data/data.tsv", function(error, data) {
    data.forEach(function(d) {
        d.date = parseDate(d.date);
        d.close = +d.close;
    });

The first line of that piece of code invokes the d3.tsv request (d3.tsv) and then the function is pointed to the data file that should be loaded ("data/data.tsv"). This is referred to as the 'url' (unique resource locator) of the file. In this case the file is stored locally, but the url could just as easily point to a file somewhere on the Internet.

The format of the data in the data.tsv file looks a bit like this;

date    close
1-May-12    58.13
30-Apr-12   53.98
27-Apr-12   67.00
26-Apr-12   89.70
25-Apr-12   99.00

(although the file is longer (about 26 data points)). The 'date' and the 'close' heading labels are separated by a tab as are each subsequent dates and numbers. Hence the 'tab separated values' :-).

The next part is part of the coolness of JavaScript. With the request made and the file requested the script is told to carry out a function on the data (which will now be called 'data').

function(error, data) {

Now, there are actually more things that get acted on as part of the function call, but the one we will consider here are the following lines;

data.forEach(function(d) {
        d.date = parseDate(d.date);
        d.close = +d.close;
    });

This block of code simply ensures that all the numeric values that are pulled out of the tsv file are set and formatted correctly. The first line sets the data variable that is being dealt with (called slightly confusingly 'data') and tells the block of code that for each group within the 'data' array it should carry out a function on them. That function is designated 'd'.

data.forEach(function(d) { 

The information in the array can be considered as being stored in rows with each row consisting of two values. One value for 'date' and another value for 'close'.

So the function is pulling out values of 'date' and 'close' one row at a time.

Each time it gets a value of 'data' and 'close it carries out the following operations;

d.date = parseDate(d.date);

This the specific value of date being looked at ( d.date) into a date format that D3 can process and do stuff with via a separate function 'parseDate'. Now, the 'parseDate' function is defined in a separate part of the script, and we will examine that later. So for the moment, just be satisfied that it takes the raw date information from the tsv file in a specific crow and converts it into a format that D3 can then process. That value is then re-saved in the same variable space.

The next line then sets the 'close' value to a numeric value (if it isn't already) using the '+' operator.

d.close = +d.close;

This appears to be good practice when the format of the number being pulled out of the data may not mean that it is automagically recognised as a number. This will ensure that it is.

So, at the end of that section of code, we have gone out and picked up a file with data in it of a particular type (tab separated values) and ensured that it is formatted in a way that the rest of the script can use it correctly.

Now, the astute amongst you will have noticed that in the first line of that block of code (d3.tsv("data/data.tsv", function(error, data) {) we opened a normal bracket ( ( ) and a curly bracket ( { ), but we never closed them. That's because they stay open until the very end of the file. That means that all those blocks that occur after the d3.tsv bit are referenced to the 'data' array. Or put another way, it uses 'data' to draw stuff!

But anyway, let's get back to figuring what the code is doing by jumping back to the end of the margins block.