Getting Started
with
LR Data Services

We notice you're not using Firefox. Some of the "Try It" features are only supported Firefox as they require a feature, E4X, that is present only in Mozilla's Javascript engine, which CouchDB uses. Please use Firefox if you wish to use "Try It" to it's fullest.

What Was Data Services?

Make it easier to get relevant data.
Get clean data.

In depth scrutiny of the LR data.
I.E. Schemas & Signatures validated, envelopes filtered.

Mostly external, complementary to a LR Node.

What Is LR Data Services Now?

A Way to Get the Data You Want

Only interested in one specific kind of data? Data Services aims to provide a way to extract the data that's relevant to you through simple customization.
It's A Design Pattern, With Some "Batteries Included".
- Follow simple conventions to identify discriminators within the LR documents
- Reuse or modify community-sourced libraries to aid in extracting data for your use case
- Install your data service code into a LR Node's CouchDB
- Access your custom solution via the extract HTTP service API installed on the node.
A Path Towards Making LR use Fewer Resources

We know LR can potentially hold mountains of data, and has a few data extraction services that utilize loads of storage space. This is a way to extract data in a very focused manner, which will result in substantial storage savings.

What Is a Discriminator?

Discriminators in Data Services are the characteristics of the data that you've identified that you want to capture.

This allows you to exclude data you do not know how to process or recognize as quality information.
For example, if you were only interested in ASN urls contained within the <conformsTo> element of XML metadata that uses a Dublin Core Terms schema, the identified ASN url becomes your discriminator.

What Can Data Services Do?

Following the conventions outlined, you will be able to extract data using:

Resource Locator by Discriminator
Resource Locator by start of Discriminator
Resource Locator by Timestamp
Discriminator by Resource Locator
Discriminator by start of Resource Locator
Discriminator by Timestamp
Discriminated Resource Locator by Timestamp
All Discriminated Resources

The Alignment to Standards Prototype Example

The prototype implementation can provide a Data Service that will allow use of the extract service to request resource data and aggregations where ASN's are Discriminators, `resource_locator`'s are Resource resource_locator, and `node_timestamp`'s are Timestamps, which will enable us to get:

`resource_locator` by ASN
`resource_locator` by start of ASN
`resource_locator` by `node_timestamp`
ASN by `resource_locator`
ASN by start of `resource_locator`
ASN by `node_timestamp`
ASN `resource_locator`s by `node_timestamp`
All ASN `resource_locator`s

Conventions

To follow the K.I.S.S principle, we're adopting a set of conventions to make adding new data services simple. These conventions are part of a CouchDB design documnent.

A field named dataservice so the extract service can locate the implementation. The field will contain a map that indicates the name and description of the data service. More details for this TBD!
Views must follow a specific naming convention.
View functions must emit specific keys according to the view they implement.
Timestamps must be represented as seconds from epoch.
List should be named respective of their output format. Included in the prototype is a reference sample for to-json.

{
    "_id": "_design/standards-alignment",
    "dataservice": { 
        "name": "Standards Alignment Data Service",
        "description": "This is where I would document how this data service works."
    },
    "views": { 
        "discriminator-by-resource": { 
            "map": "function (doc) { emit([doc.resource_locator, getDescriminator(doc), getEpochTimestamp(doc)], null); }",
           
        },
        "discriminator-by-resource-ts": { 
            "map": "function (doc) { emit([doc.resource_locator, getEpochTimestamp(doc), getDescriminator(doc)], null); }",
             
        },
        "resource-by-discriminator": { 
            "map": "function (doc) { emit([getDescriminator(doc), doc.resource_locator, getEpochTimestamp(doc)], null); }",
             
        },
        "resource-by-discriminator-ts": {
            "map": "function (doc) { emit([getDescriminator(doc), getEpochTimestamp(doc), doc.resource_locator], null); }",
             
        },
        "resource-by-ts": {
            "map": "function (doc) { emit([getEpochTimestamp(doc), doc.resource_locator], null); }",
             
        },
        "discriminator-by-ts": {
            "map": "function (doc) { emit([getEpochTimestamp(doc), getDescriminator(doc)], null); }",
             
        } 
    },
    "lists": { "to-json": "function(head, req) { ... }" }
}

How Map Functions Work

In LR, each record is JavaScript object, much like the one displayed below. The map function will process each individual object once for each view.

A Resource Data Document

{
    "_id": "c9966f4e59e04b168c733b19d7fcb4ad",
    "_rev": "1-a6aae49518aa1ec59f223d4631e9eef4",
    "keys": ["nanometer", "atom", "standards alignment", "Cells", "Environmental Issues", "ions", "magnetic field", "English-Language Arts", "California Academic Content Standards", "Equations and Inequalities", "Music", "Vocabulary & Spelling", "Physics", "Factoring", "delta", "predicatable text", "Botany and Plant Science", "California's Common Core Content Standards", "Algebra & Functions", "Earth & Space Science", "Grade 12", "Grade 11", "Grade Pre-K", "Chemistry", "Grade 10", "Biochemistry", "aurora borealis", "Mathematics", "natural resource", "watershed", "particle", "conservation", "electron", "charged particles", "Grade K", "water", "northern lights", "Life Sciences", "Structure and Function in Living Things", "Reading Comprehension", "nano", "Science", "water cycle", "Grade 9", "Visual Arts", "Literary Analysis", "Grade 8", "Grade 5", "Grade 6", "Grade 3", "Grade 1", "Grade 2", "high frequency"],
    "TOS": {
        "submission_attribution": "Copyright 2011 California Department of Education: CC-BY-3.0",
        "submission_TOS": "http://creativecommons.org/licenses/by/3.0/"
    },
    "payload_placement": "inline",
    "node_timestamp": "2012-02-28T17:01:59.202435Z",
    "create_timestamp": "2012-02-28T17:01:59.202435Z",
    "update_timestamp": "2012-02-28T17:01:59.202435Z",
    "active": true,
    "publishing_node": "3b7ea975ac0f4c2c89a62268be6cbb03",
    "resource_locator": "http://www.readwritethink.org/lessons/lesson_view.asp?id=131",
    "identity": {
        "signer": "Brokers of Expertise",
        "submitter": "Brokers of Expertise",
        "submitter_type": "agent"
    },
    "doc_type": "resource_data",
    "resource_data": {
        "activity": {
            "content": "A resource found at http://www.readwritethink.org/lessons/lesson_view.asp?id=131 was matched to the academic content standard with ID http://purl.org/ASN/resources/S10000FE by a member of the Brokers of Expertise Standards Matching Group on November 7, 2011",
            "related": [{
                "content": "Recognize that sentences in print are made up of separate words.",
                "id": "http://purl.org/ASN/resources/S10000FE",
                "objectType": "academic standard"
            }],
            "verb": {
                "action": "matched",
                "date": "2011-11-07",
                "context": {
                    "objectType": "LMS",
                    "id": "http://www.myboe.org/go/resource/7238",
                    "description": "Brokers of Expertise resource management page"
                }
            },
            "actor": {
                "url": "http://myboe.org/go/groups/standards_matchers",
                "displayName": "Brokers of Expertise Standards Matching Group",
                "objectType": "group"
            },
            "object": {
                "id": "http://www.readwritethink.org/lessons/lesson_view.asp?id=131"
            }
        }
    },
    "resource_data_type": "paradata",
    "payload_schema": ["LR Paradata 1.0"],
    "doc_version": "0.23.0",
    "doc_ID": "c9966f4e59e04b168c733b19d7fcb4ad"
}

How Map Functions Work: Try It

A map function takes takes one argument, the Resource Data Document. The function should perform the following tasks against the document:

Evaluate the document to determine if the document should be included in the index. Does it meet some set of criteria specific to your use case?
If the document should be included, define the structure of the keys and values in the index. Remember to follow the key conventions defined earlier.

Below is a sample implementation of a data services map function. Try editing and click run to learn what get's created in the index.

show sample resource data

function(doc) {

    /***
     * When using python CouchApp to build, !code is a macro to insert the contents of the
     * specified file when compiling into a design document.
     */
    // !code lib/alignment.js

    try {
        if (doc.doc_type == "resource_data" && doc.resource_data && doc.resource_locator && doc.node_timestamp) {
            var nodeTimestamp = convertDateToSeconds(doc);

            /****
             * For Parsing LR Paradata 1.0 with alignment information.        
             ****/
            var seen = {};
            var parserLR = function (objStr, verb) {
                for (re in ASNPatterns){
                    var stds = objStr.match(ASNPatterns[re]);
                    for (s in stds) {
                        if (!seen[s]) {
                            emit([doc.resource_locator, nodeTimestamp, [verb, stds[s]]], null);
                            seen[s] = 1;
                        }
                    }
                }
            };

            Alignment.parseLRParadata(doc.resource_data, parserLR);

            /****
             * For Parsing Dublin Core Terms <conformsTo> elements.        
             ****/ 
            var parserDct = function (conformsToText) {
                var seen = {};
                for (re in ASNPatterns){
                    var stds = conformsToText.match(ASNPatterns[re]);
                    for (s in stds) {
                        if (!seen[s]) {
                            emit([doc.resource_locator, nodeTimestamp, stds[s]], null);
                            seen[s] = 1;
                        }
                    }
                }
            };
            Alignment.parseDCT_ConformsTo(doc.resource_data, parserDct);
        }   
    
    } catch (error) {
            log("error: "+error);
    }
}

The Extract Service Output Specification

The basic JSON format for the extract service will is as specified in JSON Schema Internet Draft.

Data Service Response Wrapper

{
    "name": "data-service-json",
    "properties": {
        "documents": {
            "type": "array",
            "required": true,
            "items": {
                "$ref": "list-to-json"
            }

        }
    }
}

Data Service Result

{
    "name": "list-to-json",
    "properties": {
        "result_data": {
            "type": "object",
            "required": true,
            "properties": {
                "resource": {
                    "type": ["string", "null"],
                    "required": true,
                    "description": "The resource locator URL. This value may be null for DS results that do not yield a resource (i.e. ts-by-discriminator)"
                },
                "discriminator": {
                    "type": ["string", "number", "integer", "boolean", "object", "array", "null"],
                    "required": true,
                    "description": "The common identitifier that is related to the resource locator in accordance to the algorithim implemented by the data service. This value may be null for DS views that do not yield a discriminator (i.e. ts-by-resource)."
                },
                "timestamp": {
                    "type": "string",
                    "required": false
                    "description": "The ISO 8601 formatted date of the result. This should only be included in views that contain timestamps as part of the key."
                }
            }
            }
        },
        "supplemental_data": {
            "type": "object",
            "required": false,
            "description": "free form, additional data that the list function may choose to emit to aid harvesters"
        }.
        "resource_data": {
            "type": "array",
            "required": true,
            "items": {
                "type": ["string", "object"];
                "description": "elements will either be a list of doc_ID's or full resource_data documents."
            }
        }
    }
}

The Extract Service Example Output

This is an example of the data service output with doc_ID's being returned.

{
    "documents": [
    {
        "result_data": {
            /* 
                    Map data formatted any way you want. Could possibly be a paradata 'assertion' for those who 
                    want to crosswalk formats.
            */
            "resource": "http://www.readwritethink.org/lessons/lesson_view.asp?id=131",
            "discriminator": [
                "matched",
                "http://purl.org/ASN/resources/S10000FE"
            ]
        },
        "supplemental_data": {
            /*
                Optional that that can be dynamically extracted from documents from within the list function.
            */
        },
        "resource_data": [ /* doc ids of full docs */ 
            "c9966f4e59e04b168c733b19d7fcb4ad",
            "f024d8b2272941078ffd09cc8f21cf77"
        ]

    }

    ]
}

List Functions in Detail

List functions are responsible for formatting the response for the service output.

As the previous slides defined and displayed there are two basic parts to the response, the response wrapper, and the results.

Data Service Response Wrapper { "documents": [ /*** a list of data service results ***/ ] }

The list function will need to group each result in the "documents" property of the response wrapper.

Data Service Result

{
    "result_data": { 
        "resource": "http://example.com/my/resource",           //  the resource locator for this result
        "discriminator": {                                      /////
            "verb": "matched",                                  //  This is could also be a simple string, number, boolean, or array.
            "standard": "http://purl.org/ASN/resource/S000000"  //  Typically this is just the discriminator portion of your map key.
        },                                                      /////
        "timestamp": "2010-04-01T00:00:01Z"                     //  optional timestamp value, should only exist for views where ts is part of the key
     },
    "supplemental_data": { /*** insert optional custom list output here ***/ },
    "resource_data": [ /*** based upon request param, either full JSON doc, or just the doc_ID ***/ ]
}

Review of Prototype List Implementation

List functions group results and embed results into the Extract service's format for records in the documents list. Because the amount of data requiring processing could be large, we must try to always design list functions to buffer little and mostly stream.

function(head, req) {
  var util = require("lib/util").init();  // General utilities for processing data service views

  var info = util.GetViewInfo(req),       // Extracts some useful information from the request
      parser = util.RowParser(info.view), // The row parser knows how to take the view conventions and normalize the result into a 
                                          //    common data stru
      row = null,
      first_group = true,
      groups = {
        cur_group: null,
        prev_group: null
      },
      result_data = {},
      count = 0,
      seen = {};


  send('{"documents":[');               // start the extract template

  // iterate over the requested view.  Views will be returned in key collated order (sorted by key).
  while (row = getRow()) {
    count++;

    // because we want to be efficient and perform only a single query to get both doc_ID's or documents, we need to manage our
    // own groupings.  util.GroupIt takes most of the pain out of figuring out when groups have changed.
    groups = util.GroupIt(parser.group_length, groups, row);

    // when groups.changed is true, it means we have a new unique group to switch to.
    if (groups.changed) {
      if (first_group) {
        first_group = false;
      } else {
        ////
        //   If you have buffered supplemental data for the previous group, you should send it at this point here
        ////

        // close the previous group
        send("]},");
      }

      // define the result_data data structure using the parser.
      result_data = {
        resource: parser.resource(row),
        discriminator: parser.discriminator(row)
      };

      // check to see if this is a view that we should display the timestamp. Note the utility function to convert from the
      // stored integer format (number of seconds from epoch) into ISO Date.
      if (parser.showTimestamp){
        result_data.timestamp = util.secondsToISODate(parser.timestamp(row));
      }

      // send the first piece of result data
      send('{"result_data":' + JSON.stringify(result_data) + ',');

      // send the first resource data element as well
      send('"resource_data":[');
      send(JSON.stringify(parser.either(row)));

      ////
      //   If you were going to try and capture some supplemental data, this is most likely the location you'd want to buffer that.
      //   Remember to keep your buffer small!!! (Try grabbing only the first N things that match - first 5 titles.)
      ////

      // track the id's we've processed since it is possible for poorly designed views to include the same document multiple times
      // for a single group.
      var id = parser.id(row);
      if (id) {
        seen = {};
        seen[id] = 1;
      }

    } else {
      // Not a new group, so just process the next document - so either emit the doc_id or full doc.  parser.either handles most of that for you.

      // before emitting, make sure we haven't already done this document before.
      var id = parser.id(row);
      if (id && !seen[id]) {
        send(',' + JSON.stringify(parser.either(row)));
        seen[id] = 1;
      }

      ////
      //   If you were going to try and capture some supplemental data, this is most likely the location you'd want to buffer that.
      //   Remember to keep your buffer small!!! (Try grabbing only the first N things that match - first 5 titles.)
      ////
    }

  }
  // close the last group if there was one
  if (count > 0) {
    ////
    //   If you have buffered supplemental data for the previous group, you should send it at this point here
    ////

    send("]}");
  }
  send(']}');
}

Customize or Make your Own List Function

The prototype implementation should suffice as an adequate skeleton for making enhancing the prototype or building your own from scratch

Start out by copying one of the samples.
Don't forget any required files withing the lib folder. These helper functions have been abstracted to accomodate varied discriminator formats. In most cases unless you need supplemental_data, you can use the default to-json as is without modification.
Remember to stream as much as you can, this will help keep your system resources under control.

The Extract Service

Provides a common HTTP interface to access data services with simplified parameters.

What it does is create a bridge between the raw CouchDB view and list functions and query parameters, so you don't have to fully understand much of the complexity of CouchDB's query solution. The Extract service transforms your parameters into the equivalent query for CouchDB
If running your own node, and you understand CouchDB's API, nothing prevents you from using the views directly for doing queries not supported by the extract service.

The Extract Service HTTP Request Format

The Basics

GET /extract/<data service name>/<view name>[?<DS Query Params>]

Custom List functions (AKA Roll Your Own Output Format)

GET /extract/<data service name>/<view name>/format/<your function name>[?<DS Query Params>]

The DS Query Params

Parameter	Description
from	ISO 8601 formatted timestamp for start range.
until	ISO 8601 formatted timestamp for end range.
resource	The resource locator you wish to harvest data.
discriminator	The discriminator you wish to harvest data.
resource-starts-with	A partial resource locator you wish to harvest data that uses the specified value as a prefix. (i.e. `resource-starts-with=http://shodor.org` will return all resources from http://shodor.org.
discriminator-starts-with	The partial discriminator you wish to harvest data to be used for find the range that uses the specified value as a prefix.
ids_only	Presence of the value will cause the resource_data values to be a list of doc_ID's instead of full resource_data documents (default behavior)

Extract Service Parameter Matrix

Data services aims to allow you to narrow in on your data. So not all parameters work as expected with all data service views. Here is a breakdown of what parameters work together with specific view. Remember not to be concerned about the characteristics of the data returned, because the map functions have already taken care of that for you. If the data-service map functions only emit keys for documents that contain the word "exciting", that means anything this service returns will be at the very least, "exciting". The parameters just control how much to return.

View	Parameter Set	Description
discriminator-by-resource	resource	Get a list of discriminators for a specific resource locator.
discriminator-by-resource	resource-starts-with	Get a list of discriminators that where the resource locator starts with a specified prefix.
discriminator-by-resource-ts	resource, from, until	Get a list of discriminators for a specific resource locator between for a specified period of time.
discriminator-by-resource-ts	resource-starts-with	Get a list of discriminators that where the resource locator starts with a specified prefix, include the timestamp in the result
discriminator-by-ts	from, until	Get a list of discriminators for a specified time period.
resource-by-discriminator	discriminator	Get a list of resource locators for a specified discriminator.
resource-by-discriminator	discriminator-starts-with	Get a list of resource locators that start with the specified discriminator as a prefix.
resource-by-discriminator-ts	discriminator, from, until	Get a list of resource locators that for a specified discriminator for a specified period of time.
resource-by-discriminator-ts	discriminator-starts-with	Get a list of resource locators that start with the specified discriminator as a prefix. Timestamps are included in the output.
resource-by-ts	from, until	Get a list of resource locators for a specified period of time.

Extract Service API - Try It

Here are some example requests that you can try against a real data service install. Click on the line to populate the example into input box or edit by hand. Click Run to execute the request. This can take a bit of time, and cause your browser to complain. It's okay to click "Wait" or "Continue" if prompted by your browser until it completes.

GET /extract/standards-alignment-lr-paradata/resource-by-discriminator?ids_only&discriminator=["matched"]
GET /extract/standards-alignment-lr-paradata/resource-by-discriminator?ids_only&discriminator-starts-with=["matched","http://purl.org/ASN/resources/S"]
GET /extract/standards-alignment-dc-conformsTo/discriminator-by-resource?resource-starts-with=http://www.shodor.org/interactivate/activities/Advanced
GET /extract/standards-alignment-lr-paradata/resource-by-discriminator-ts?ids_only=true&discriminator=["matched","http://purl.org/ASN/resources/S1000132"]&from=2012-02-28T16:59:31Z&until=2012-02-28T16:59:31Z

GET

Next Steps

Check out the code on Github:
- https://github.com/LearningRegistry/LearningRegistry/tree/master/data_services
Ask Questions on the LR Dev Google Group:
- learningreg-dev@googlegroups.com
- http://groups.google.com/group/learningreg-dev

Getting StartedwithLR Data Services