Debugging external host communication with UrlFetchApp

There is a large and growing number of Web APIs available out there, many of which can extend our ability to obtain information or automate workflows from our Google Apps Scripts. Using the URL Fetch Service is a little more challenging than the majority of the services provided by Google, and when things don’t work it can be significantly more difficult to figure out than those services.

We’re going to look at debugging techniques unique to the UrlFetchApp today. For people unfamiliar with HTTP’s arcane messages and responses, we’ll get an overview that should help unlock this powerful capability.

HTTP Overview

Everyone recognizes HTTP:// as the beginning of web addresses seen in browser address bars. That is the name of the client-server protocol that browsers (clients) use to request content from web sites (servers). It is also used for other forms of machine-to-machine communication, such as Web APIs. This is just a quick overview to set the context for our debugging, and is in no means exhaustive.

A client sends a request in the form of a simple text message with a specific format that specifies such things as:cdraw

  • The method or type of request (GET and POST are supported by UrlFetchApp)
  • The contentType of any message “payload”. Common types include application/x-www-form-urlencoded and application/json.
  • Any payload that goes with the message. This is how the content of a request is transferred to the server from the client, for the POST method. (A GET request passes query parameters as a string appended to the URL.)
  • Additional headers may be included as name / value pairs. If they are required by the API you are accessing, they should be described in that API’s documentation. Basic Authentication, for example, includes a header named “Authorization” with a value consisting of the encoded username and password.

The server sends back a response message. This message has a similar structure to the request, beginning with standard content:

  • A status is always present, and tells us whether the request was successful. Success is most commonly indicated by 200 OK, but some APIs may use other 2xx codes. All other codes indicate specific types of problems. A full list of Status Code Definitions is provided as part of the protocol standard.
  • When there has been an error, a text description of the error is returned in addition to the status. Helpful API developers provide specific guidance in these messages.
  • Optional returned content follows all headers.

GET and POST requests

As mentioned previously, Google Apps Script supports two HTTP request types, GET and POST. Fundamentally, a GET requests a response from a server, while a POST submits information to be processed. (That said, both can be used to submit information – but they do it differently.)

You’ve no doubt seen a GET request – that’s what a typical web page address is. Here’s an example of the URL for the monthly view of a Google Calendar.

Browser address bar

Here the base URL is https://www.google.com/calendar/render. The parameters for a GET request are passed to the server as part of the URL, which is often referred to as a query string. This is separated from the base URL by a question mark (?), and consists of Key – Value Pairs. Those pairs are expressed around an equals sign, e.g. key=value, and multiple parameters are separated by the ampersand (&). In this example, there is just one parameter, named “tab”:

tab=mc#main_7%7Cmonth

There are some recognizable characters in the value part of this pair; we see an octothorpe (#), an underline (_), and a percent sign (%). Here’s where things can be a bit tricky, because it turns out that not every character is recognized as legal within a URL – we call them reserved characters. To be included in a URL, a reserved character must be specially encoded, or “escaped”. In our value string, %7C is an escape sequence for the vertical bar (|). Before encoding, our value would have been mc#main_7|month. Modern browsers will take care of making these conversions for us when we type an address into them, but as a programmer, you are responsible for doing that job to enable your script to successfully communicate with other software. You can look up the appropriate escape values and convert reserved characters yourself, but there are built-in JavaScript utilities to take care of this for you.  Refer to the documentation for encodeURIComponent() and encodeURI().

Template functions

Here are a couple of template functions to help you develop your own scripts to communicate with external hosts. To help avoid problems with parameter encoding, these templates provide a consistent flow of control for both GET and POST methods, and offload the concern of encoding the GET query string to a utility function.

Debugging tips

When debugging external host communication with UrlFetchApp, there are a few tools available to you.

  1. It helps to muteHttpExceptions so you can view them. (In fact, you can simply write your code to handle them yourself, and only throw for exceptions you really don’t expect.) This is done by adding 'muteHttpExceptions' : true to your fetch parameters.With exceptions muted, fetch won’t throw an exception; instead it will pass failure response codes in the HTTPResponse. From there, you can extract the response code and content text for debugging (or automatic handling).Here’s a template that you can use to implement this suggestion:
    // Construct `fetch` params object
    var params = {
      'muteHttpExceptions' : true
    };
    
    var response = UrlFetchApp.fetch(url, params)
    
    // Check return code embedded in response.
    var rc = response.getResponseCode();
    var responseText = response.getContentText();
    if (rc !== 200) {
      // Log HTTP Error
      Logger.log("Response (%s) %s",
                 rc,
                 responseText );
      // Could throw an exception yourself, if appropriate
    }
    else {
      // Successful POST, handle response normally
      Logger.log( responseText );
    }
    
  1. We can examine the fetch options and parameters by using getRequest(). You could choose to log the result, but let’s assume that you will want to view it in the debugger instead. Add this line just above the existing fetch() call, and put a breakpoint on the fetch(). Once the breakpoint is hit, you will be able to look at the content of test to analyze the content of your proposed HTTP request.
    var test = UrlFetchApp.getRequest(url, options);
    
  2. Unexpected Error: This usually appears with the URL you’re trying to access appended to it, and is reported by the Google Apps Script server back to your script. When you see this, your HTTP request has not been successful, so you need to look first at the intended URL. Make sure that you did not incorrectly URI-encode the URL – this sometimes happens when you’re intending to encode in-line parameters for a GET request.

Common problems

A survey of questions posted about issues with external host communications identifies a few common problems. Keep these in mind when you’re trying to find defects in your own code!

  • Authentication: Watch for a 401 Unauthorized HTTP error code. Many APIs require user authentication to operate, and there are many ways that can be implemented by the remote server. Check the documentation for the API you’re accessing to ensure you’ve met their requirements. Be especially aware of encoding requirements; when in doubt, use encodeURIComponent() to encode any authorization token in the payload, and validate that you’ve got the right contentType set.
  • Cannot find API: You may see a 404 Not Found if you have part of the API URL wrong, or if the parameters for the API are not properly encoded. As with Authentication, double-check you encoding and contentType.
  • Successful fetch, but cannot access response object: All HTTP responses are text, although they often contain JSON-encoded objects. If you’re expecting an object response, ensure you’ve used JSON.parse(response) to convert the text into a regular JavaScript object.

Examples

Here are a few example questions from Stack Overflow that show web application APIs that you can manipulate via Google Apps Script’s UrlFetchApp. With the tips and tricks from today’s blog, you should be able to adapt any of these to your use, and debug any problems you find while doing so!

Conclusion

I hope this introduction helps ease you into the rich world of web APIs. There is far more to the topic of server-to-server communication than what we’ve covered in this post, of course. Get out there and explore!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s