Search

REST Connection Example - Finding Definitions for English Words

The Exercise Introduction

You have CSV file with English words. You want to produce a CSV with the original word and two extra columns:

the definition of the word
an example with the word

Sample Input CSV

Create a CSV file like below to use in this example:

word
vernier
yak
loganberry

Add this CSV file where it can be accessed from Hero Platform_. This example stores the CSV file in an S3 bucket.

Sample Output CSV

The end result is a CSV file with data similar to the following:

word,definition,example
vernier,"a small movable graduated scale for obtaining fractional parts of subdivisions on a fixed main scale of a barometer, sextant, or other measuring instrument.",
yak,a trivial or unduly prolonged conversation.,"one's time is filled with wining, dining, and yackety-yak"
yak,talk at length about trivial or boring subjects.,she wondered what he was yakking about
yak,"a large domesticated wild ox with shaggy hair, humped shoulders, and large horns, used in Tibet as a pack animal and for its milk, meat, and hide.",
loganberry,"an edible dull-red soft fruit, considered to be a hybrid of a raspberry and an American dewberry.",

Human readable it will look similar to the following:

word	definition	example
vernier	a small movable graduated scale for obtaining fractional parts of subdivisions on a fixed main scale of a barometer, sextant, or other measuring instrument.
yak	a trivial or unduly prolonged conversation.	one's time is filled with wining, dining, and yackety-yak
yak	talk at length about trivial or boring subjects.	she wondered what he was yakking about
yak	a large domesticated wild ox with shaggy hair, humped shoulders, and large horns, used in Tibet as a pack animal and for its milk, meat, and hide.
loganberry	an edible dull-red soft fruit, considered to be a hybrid of a raspberry and an American dewberry.

The Plan to Use and API to Find Definitions and Examples of Words

You will use:

a File System Input with CSV parser.
a REST Lookup function and OwlBot’s Rest API to get the the values for the definition and example columns.
a File System Output to write the new CSV file.

Hero Platform_ performs actions to:

send an HTTP request for each single word. (tuple)
parse the new tuple fields from the HTTP response JSON.

The OwlBot API

About the API

Visit their website and scroll to the developer section: https://owlbot.info/

They do not have much documentation other than a Curl example. Curl is a command line tool for making HTTP requests (among other features). People like to share Curl commands because it is quite readable and is in a ready-to execute form of an API call.

Curl is not in the scope of this tutorial. Here is how the API works explained in a few words:

You need to have a TOKEN for authorization
- Go to https://owlbot.info/
- Click the GET A TOKEN button.
- Enter your email address.
- Go to your email account and open the email from OwlBot. Your authorization information is included.
The token is sent in the HTTP request (configured in your Hero Platform_ Connection to OwlBot) as an HTTP header. The format is the following: Authorization: Token <Your Token Value>
Name Value
Authorization Token <Your Token Value>
In order to get the data for a word (e.g loganberry), you could send an HTTP GET request to https://owlbot.info/api/v4/dictionary/loganberry or the word yak the URL will be https://owlbot.info/api/v4/dictionary/yak

The response will be a JSON similar to the following:

{
  "definitions": [
    {
      "type": "noun",
      "definition": "an edible dull-red soft fruit, considered to be a hybrid of a raspberry and an American dewberry.",
      "example": null,
      "image_url": "https://media.owlbot.info/dictionary/images/ggggggggggggggggw.jpg.400x400_q85_box-0,0,204,204_crop_detail.jpg",
      "emoji": null
    }
  ],
  "word": "loganberry",
  "pronunciation": "ˈlōɡənˌberē"
}

definitions is an array in the JSON because a single world can have multiple meanings. An example is the word "spring" meaning both "the name of a season", "a source of water", or "to jump forward or upwards".
- This example will handle the case when a world has multiple meanings.
This example will also handle the case when the world is not in the dictionary.

Try the OwlBot API in Postman

Postman is a popular API client that makes it easy for technical users to test using documented APIs.

You can test out OwlBot's API in the free Postman desktop application (this example does not work in Postman's web app) or jump directly to example of using REST APIs in Hero Platform_.

It is good practice to try out the API from Postman (or using Curl from a command line) and get some experience with it before calling it from Hero Platform_:

See how the API behaves generally and in some edge cases.
Try out HTTP requests while you are reading the API documentation.
It is much easier to catch the pitfalls and the misunderstanding earlier than later when the Flow is being executed in Hero Platform_.

If you are new to Postman you can learn the basics from:

Remember to add authorization header (directly on the Headers tab of the request. (Not on the Authorization tab!)

https://learning.postman.com/docs/sending-requests/requests/#configuring-request-headers

Try loganberry:

GET https://owlbot.info/api/v4/dictionary/loganberry

Try spring:

GET https://owlbot.info/api/v4/dictionary/spring

Try a non-existing word kkkkk:

GET https://owlbot.info/api/v4/dictionary/kkkkk

Notice the response code (404 Not Found) and body message:

[
	{
		"message": "No definition :("
	}
]

You learned what the response for the OwlBot server is when a word is not found in their database. They could also have responded with an empty array. Both would be logical responses.

Try a real word but remove the authentication header:

Notice the response code (401 Unauthorized) and body message:

{
	"detail": "Authentication credentials were not provided."
}

This response from the server lets you know that it did not try and look up the word as authorization was required but not provided.

Adding the key/value (Authorization / Token <Your Token Value>) under the Headers tab and resending the same request should display the correct response.

Hero Platform_ Implementation Using the REST Lookup Function

Log into your Hero Platform_ application.

The first step is to create a Connection to where your data is coming from and then an Input so you can use that data in your Flow.

Create a Connection to where the data is located

In this example, the CSV file with the words to define are located in the File System, AWS S3.

Click Connections from the Hero Platform_ navigation menu and click Create New Connection.

Configure and save the Connection.

Create an Input for the data

Click Inputs from the Hero Platform_ navigation menu and click Create New Input.

Configure and save the Input.

Building the Flow

You have the Input data ready for a Flow.

Click Flows from the Hero Platform_ navigation menu and click Create New Flow.

Give the Flow a name.

Click Add Input and select the Input created above for this example.

After the Input has been added, click the Input box to preview the data.

The next step in example is using the REST Lookup function. In order to use this function you need a REST Connection.

Click Save from the Flow toolbar to save the progress of the Flow.

Create the REST Connection

After saving the Flow in progress, click Connections from the Hero Platform_ navigation menu and click Create New Connection.

Enter a name for the Connection and select REST at the connection type. (Not REST Endpoint)

Select No Authentication for the authentication type.

No Authentication is selected because the OwlBot API does not use the Basic/OAuth2/other authentication methods listed.

Enter https://owlbot.info/api/v4/ for the base URL.

Adding the “/“ on the end of the base URL is optional. Hero Platform_ automatically adds the "/" character if missing.
In this example the base URL https://owlbot.info/ or https://owlbot.info/api/ would also work. The concatenation of the base URL and the endpoint (from the REST Lookup function) should be the desired URL of the REST API Connection endpoint)

It is recommend to make the base URL as long as possible so the endpoint from the REST Lookup function fields are shorter and easier to read.

Double check that you are using https:// on the beginning of the URL (and not http).

Request Header:

Name: Authorization
Value: Token <Your Token Value> (this value is found in the email you received form OwlBot)

The value column acts like password field and are stored encrypted because an API token is sensitive data.
Other APIs can require adding authorization parameters into a cookie parameter or a query parameter.
When configuring other APIs, where an authentication type is selected (e.g. Oauth2), an authorization header is most likely not required.

Click Test Connection. If the REST Connection has been configured correctly, a Connection Successful message appears at the top of the screen.

The test method and test endpoint settings are for trying out the Connection settings. It’s not mandatory to test the connection before saving but it is recommended.
- Try testing the dictionary endpoint from the OwlAPI documentation.
  - Select GET at the test method.
  - Enter dictionary/owl for the test endpoint.
- It is not always possible to test the connection at this point.
It is recommended to always use a real endpoint with a matching method for testing the connection. Using only the base URL without an endpoint can result a false positive or false negative result.
Warning: Test connections makes a real HTTP request to the server. Never use endpoints that would change data on the server.

Save the REST Connection.

Create the REST Lookup

Now that the REST Connection has been created it is possible to create a REST Lookup function in the Flow.

Open your Flow again by clicking Flows from the Hero Platform_ navigation menu and selecting the Flow saved above.

Add a REST Lookup function by clicking the plus icon between the Input box and the Add Output box. Select REST Lookup.

Add an argument by clicking Add argument field. Select the field containing the word list from the sample data.

This argument fields data will be called as a variable parameter in the endpoint field.

Every argument should be added to the top of the form, that is used as parameter in the Endpoint, Header, Query, Cookie or Body

Select the REST Connection name.

Select GET for the method. (As specified in OwlBot's API documentation.)

Enter dictionary/${word} as the endpoint.

You can add a "/" at the beginning of the endpoint. If missing, Hero Platform_ automatically adds the "/".
${word} is a path parameter, it will be resolved from the actual tuple (the argument field data) before every HTTP request.
${word} (and all the parameters) should have an argument field on the top of the form that have exactly the same name.

Notice there isn’t a "/" on the end of the text field. Some endpoints needs an ending slash character, some endpoints requires not to have it, and some accept both.

OwlBot's API documentation says that the REST call is not required to have request headers, query/cookie parameters, or a body. These fields can be left blank.

Ensure you always configure Rest Lookup functions to satisfy the actual REST API's requirements.

Leave the keep unsuccessful responses box unmarked. See more.

Parameter escaping is about how body parameters are resolved. You can leave it as JSON.

OwlBot's API response is a JSON string. Select JSON as the Input format.

The JSON parser (Input format) can parse the response JSON to tuple fields.

Root path is discussed in the next section.

Mark the box to flatten the top level array.

Click the detect fields refresh button next to the fields mapping table.

A box pops up asking asks for a sample value for every argument field to create a valid HTTP request.
- Here there is one argument (word). Enter a sample value from the word list. (e.g. yak)

Field detection works from the response. Make sure you are using the right argument values for all the desired fields in the response with non-null values.
- In this example, you can check the detection with the word watch first and after with the word jaguar. For jaguar, OwlBot does not return a sample sentence so Hero Platform_ will not detect that field
Detect fields sends a valid HTTP request in order to be able to read the response and then to parse the field list.

IMPORTANT: Rename or remove word field in the field mappings table because there is already a word field in the tuple (from the Input). Two field cannot exist with the same name.
- In this example, change the word field in the mapping table to word2.

Click OK to save the REST Lookup function.

From the Flow Studio, click on the REST Lookup function box to display a preview of the data.

Dealing with a list in the response and defining the root path

In the image above, OwlBot’s Rest API is returning a list in the definitions field. To separate the elements in the list you could:

combine the definitions to single definition with string concatenation.
only use the first definition.
flatten the list and have as many output lines as the number of the meanings of the word.

This decision depends on the purpose of the Flow. This example wants all the definitions so you now need to flatten the list.

Right-click on the REST Lookup function box and select Edit.

Enter definitions for the value in the root path field.

The Rest Lookup function only parses the definitions section of the response JSON.
- Because the flatten top level array box is marked the and the root path has been set as definitions, Hero Platform_ flattens the the array from this point.

Click the detect fields refresh button next to the fields mapping table to update the mapped fields.

The fields mapping table updates to include the nested objects within the flattened definitions path.

Click OK to save the edit to the REST Lookup function.

In the Flow, click on the REST Lookup function box and view the preview data displayed below.

The desired fields (definition and example) are now being retrieved correctly.

Create an Output for the results

Now that you have the desired data being retried from the REST Lookup function in the Flow, you need to export that data. A Hero Platform_ Output is required.

For this example, the Output location is the same as the Input location, an AWS S3 File System.
- If you are exporting the data to the same location as your import data you have already created the required Connection.
- If you are exporting the data to a new location, a new Connection is required.

Click Save from the Flow toolbar to save the progress of the Flow.

Click Outputs from the Hero Platform_ navigation menu and click Create New Output.

Give the Output a name and select the Connection to the location where you want to export the data.

Configure the Output fields according to the Output location.

Click Add Field under the fields mapping table.

The fields this example wanted to find the word, definition, and example.

Add each of these field names under the Output's field mapping table.
The data type for each field is String.

Click OK to save the Output.

Open your Flow again by clicking Flows from the Hero Platform_ navigation menu and selecting the Flow saved above.

Click the Add Output button at the end of the Flow and select the Output created above.

Hero Platform_ tries to automatically match the fields from the Flow to the Output fields.

If the Input and Output fields match, click OK to save the Output in the Flow.
If the Input and Output fields do not match, select the matching Input fields from the corresponding drop-down list. When the Input and Output fields are correct, click OK to save the Output in the Flow.

Click Save on the Flow Studio toolbar to save the flow.

Congratulations, you have completed building a Flow using the REST Lookup function.

Click Run Now in the Flow Studio toolbar to execute the Flow automation.

Review the Results

If the Flow has run successfully, the CSV Output has been exported to the Output location.

Review the results of your Flow by navigating to the location system of the Output.

In this example, an AWS S3 File System.

Download/Open the CSV file and view the data.

The data should look similar to the following:

word,definition,example
vernier,"a small movable graduated scale for obtaining fractional parts of subdivisions on a fixed main scale of a barometer, sextant, or other measuring instrument.",
yak,a trivial or unduly prolonged conversation.,"one's time is filled with wining, dining, and yackety-yak"
yak,talk at length about trivial or boring subjects.,she wondered what he was yakking about
yak,"a large domesticated wild ox with shaggy hair, humped shoulders, and large horns, used in Tibet as a pack animal and for its milk, meat, and hide.",
loganberry,"an edible dull-red soft fruit, considered to be a hybrid of a raspberry and an American dewberry.",

Error Handling

With the current configuration, if the Input CSV contains an unknown word, like kkkkk, OwlBot’s Rest API will respond with status code 404 and the Flow will fail.

In such situations, If the desired behavior for the error is:

the Flow should fail in such a case:
- Do not change anything.
the Flow should skip that word:
- Make the Flow run in forgiving mode.
- or open the REST Lookup function configuration to mark the box to Keep unsuccessful responses, and and filter out the unsuccessful responses.
the Flow to have null in the definition and example fields in the exported file:
- open the REST Lookup function configuration and mark the box to Keep unsuccessful responses.

Example to keep a Flow from failing from unsuccessful API responses

This is an example how to keep a Flow from failing when the REST Lookup function encounters unsuccessful responses.

To simulate this, you can add a non existing word (e.g. kkkkk) to the Input word list CSV.

Open the Flows dashboard by clicking Flows in the Hero Platform_ navigation menu.

Open your Flow again to the Flow Studio.

Right-click the REST Lookup function box.

Mark the box Keep unsuccessful responses.

Click the detect fields refresh button by the fields mapping table to update the mapped fields.

Two new field appear, the (HTTP) response code and response error text.

Click OK to save.

From the Flow, click the REST Lookup function box to display the preview. (The fields type and response error text were removed from the field mapping to create a clear image.)

The line for kkkkk is present but the definition and example field values are null.
The response code is 200 (HTTP success status code) for all lines, except kkkkk is 404 (HTTP Not Found status code).

You can branch the Flow and add another Output for the unknown words (where status code is not 200).

Saving and running the Flow at this point should be successful and Ouput the data even where the API returns an unsuccessful response.