Getting paged data from the Teamwork API using vue-resource and Promises

May 17, 2017

Teamwork Projects is an online project management app. Here at DesignHammer we are in the process of moving to Teamwork Projects. I am leading our transition and it’s a big job making sure everything goes smoothly. A key part of my job is identifying areas where standard Teamwork functionality doesn’t do exactly what we need and, if necessary, building tools to help meet that need.

Getting data from Teamwork

Our tools are built as a hosted Vue.js application. The app provides various reports and views which are not available in Teamwork itself but where the underlying data exists and is available via the Teamwork API.

Generating these reports typically requires fetching all of the available data for a given resource (e.g. all active projects, all time records in a given project, etc.). The Teamwork API provides these as REST-based URLs which are easy to fetch using vue-resource.

A basic request method in our Teamwork client looks like:

getAllProjects () {
  return Vue.http.get(TEAMWORK_ENDPOINT + '/projects.json')
  .then((data) => {
    return data.body.projects
  })
}

Vue.http.get() returns a Promise for the requested resource. We chain then() onto the request to pull out the project array from the response data. Code which calls the Teamwork client is therefore very simple:

TeamworkAPI.getAllProjects().then(projects => {
  console.log(projects)
})

However, some resources return a maximum number of records at one time (fetching all time records currently only returns 500 records per request). If there are more records in Teamwork for that query then the client needs to perform additional requests to get them all.

Client code can check the the response headers for information about how the data is broken up into pages. vue-resource provides these in headers.map in the response data. The main header we care about is X-Pages. If the value of this is 1 then we can just return the data from the first request; we got all of it.

If the value is greater than 1 then we need to perform additional requests. How we go about doing that in a clear and DRY way is an interesting question.

Supporting paged queries

Ideally, the code that is calling our Teamwork client shouldn’t have to care if the query is paged or not. Our UI is generally showing rolled up summaries of complete data sets, so we aren’t using paging for display. We can encapsulate all of the paging functionality within the Teamwork client as a reusable pagedGet() function and return full result sets from client methods.

pagedGet (endpoint, recordKey) {
  return Vue.http.get(endpoint)
  .then((data) => {
    if (data.headers.map['X-Pages'] === 1) {
      return data.body[recordKey]
    }

    // TODO: Handle multiple pages
  }
}

getAllProjects () {
  return pagedGet(TEAMWORK_ENDPOINT + '/projects.json', 'projects')
}

Teamwork returns result data in a data type-specific property within the body of the response. The calling code needs to know the name of this property. We pass that to pagedGet() as recordKey from the resource specific method. We also check the X-Pages header and if there’s only one page we can return the results without firing off any additional requests.

If there are additional pages of data and we know exactly how many pages there are in total, we can fire off requests for all of the remaining pages simultaneously and then combine the results as the requests complete. Once all of the results are combined we return them.

  pagedGet (endpoint, recordKey) {
    return Vue.http.get(endpoint)
    .then(data => {
      if (data.headers.map['X-Pages'] === 1) {
        return data.body[recordKey]
      }

      // Start with the records we have already fetched      
      var allRecords = data.body[recordKey]
      var promises = []      

      // Build an array of request promises, one for each additional page of data
      for (var i = 2; i  {
          return data.body[recordKey]
        }))
      }

      // Run all of the requests and combine the results
      return Promise.all(promises).then(results => {
        results.forEach(result => {
          allRecords = allRecords.concat(result)
        })

        return allRecords
      })      
    }
  }

We have added two parts to the pagedGet() method. First we loop through the remaining pages and add a new promise for the request to an array of promises. Vue.http.get accepts an options object as its second parameter. The params property can be an object with key-value pairs. Those key-value pairs are included in the request as GET parameters.

The second part executes all of the promises that were created simultaneously. Once all of them have been fulfilled we get results which is an array of result arrays. We get one result array for each page of data. All we need to do from there is combine them with our first page of data and return the full set.

The great part about this approach is that our calling code still looks exactly the same,

TeamworkAPI.getAllProjects().then(projects => {
  console.log(projects)
})

all of the resource-specific information is captured in resource-specific methods,

  getTimes () {
    return pagedGet(TEAMWORK_ENDPOINT + '/time_entries.json', 'time-entries')
  }

  getTimeEntriesByProject (projectId) {
    return pagedGet(TEAMWORK_ENDPOINT + '/projects/' + projectId + '/time_entries.json', 'time-entries')
  }

  getTasksByProject (projectId) {
    return pagedGet(TEAMWORK_ENDPOINT + '/projects/' + projectId + '/tasks.json', 'todo-items')
  }

and the request and paging logic is in a single reusable method, pagedGet().

Supporting additional parameters

Many resources in the Teamwork API support additional parameters beyond just page. We can adjust pagedGet() to support these alongside the paging functionality:

  pagedGet (endpoint, recordKey, params = {}) {
    return Vue.http.get(endpoint)
    .then(data => {
      if (data.headers.map['X-Pages'] === 1) {
        return data.body[recordKey]
      }

      // Start with the records we have already fetched      
      var allRecords = data.body[recordKey]
      var promises = []      

      // Build an array of request promises, one for each additional page of data
      for (var i = 2; i  {
          return data.body[recordKey]
        }))
      }

      // Run all of the requests and combine the results
      return Promise.all(promises).then(results => {
        results.forEach(result => {
          allRecords = allRecords.concat(result)
        })

        return allRecords
      })      
    }
  }

  getTimes (params) {
    return pagedGet(TEAMWORK_ENDPOINT + '/time_entries.json', 'time-entries', params)
  }

We allow the caller to pass a params object and use Object.assign to override the page property. We can then provide custom params in our calling code:

var params = {}

params.billableType = this.billableType

if (this.dateRange != null && this.dateRange.rangeType !== 'alldates') {
  params.fromDate = moment(this.dateRange.from).format(TeamworkAPI.DATE_FORMAT_SHORT)
  params.toDate = moment(this.dateRange.to).format(TeamworkAPI.DATE_FORMAT_SHORT)
}

TeamworkAPI.getTimes(params).then(filteredTimes => {
  console.log(filteredTimes)
}

Comments

Hey Jay,

This is a fantastic article! Thanks for sharing it and I love the style and breakdown of the steps.

One thing you mentioned was "Our UI is generally showing rolled up summaries of complete data sets, so we aren’t using paging for display"

We have started to add Stats and Totals calls to make getting aggregate data faster and easier so people don't have to get all the data just to show summaries. If you have specific areas where this may help you just drop me an email and I'd be happy to help.

Dan.

Thanks Dan, I appreciate the feedback!

To clarify, our view pages typically show show a hierarchy of times broken down by Client, Project, Milestone, Tasklist, and Task. At each level we show total estimated and actual time for all the items within a section. So, we still need the individual time records but after we fetch them we run a reduce operation to convert them form a flat list into a hierarchy with all of the times rolled up at each level.

I plan to write that approach up as part of this blog post series.