Getting Online Data

Instruction

Getting Online Data via an API

Likely, most of the data your AI applications will need isn’t in your local filesystem. It’ll probably be online, and you’ll need some way to retrieve it. Most of the time, you’ll do this through an API.

What is an API?

API is short for Application Programming Interface. The simplest way to describe APIs is by analogy. Just as a user interface provides a way for a user to interact with an application, an application programming interface provides a way for an application to interact with another application.

The formal definition of an API is “a set of defined tools and processes that allow software components to interact.” It’s an agreement that defines the services and data. The API describes what one application provides to another and how the former can request data and services from the latter.

A more practical way to think of an API is as a collection of functions that another application has made available to your application. Sometimes, the APIs are on your computer. These include Python’s built-in functions or the functions provided by any Python package that your application imports. The APIs covered in this lesson are online and accessed over the internet.

Two Ways to use Online APIs

There are generally two ways to use online APIs:

1. The “easy way: Many APIs provide Python libraries, often called SDKs or software development kits, that you can install, often using pip. These libraries let you use their functionality simply by calling a function they provide. The advantage of this approach is in its name: it makes calling an API easy.
2. The “hard way’: Some APIs expect you to communicate with them directly over the web by using HTTP (Hypertext Transfer Protocol). This protocol defines how communication takes place over the web. The advantage of this approach is that any programming language that can send messages using HTTP can access the API.

You’ll discover how to use libraries in some of our other AI modules, but in this lesson, you’ll concentrate on understanding the ‘hard way’.

The “hard way” isn’t a standard name for the method you’re about to cover. The proper name, which also is an acronym, is REST.

REST APIs

REST is short for REpresentational State Transfer, an architectural style most web services and APIs use. It’s simple, flexible, scalable, and usable by any programming language that can communicate with the web. It can also be hard to describe to someone new to it.

In the REST architectural style, applications communicating with each other take on one of two possible roles:

• Client: The application making a request.
• Server: The application that receives the request and provides a response.

In an upcoming demo, you’ll write a small app that communicates with a weather service to get the current weather in London. Your app will be the client, and the weather service will be the server.

In a REST API, clients make requests about resources. “Resource” is a catch-all term for any piece of information that you can access or manipulate through a web service provided by a server. A resource can be a user, an article or social media post, a product, or even a specific piece of data like a user profile picture. The resource that your weather app will access is the current weather for a given location.

Resources generally have two kinds of information associated with them:

• A uniform resource identifier (URI) that uniquely identifies the resource. REST APIs generally use URLs — uniform resource locators — a specific type of URI that uniquely identifies a resource and specifies where it can be found and how to access it. For the weather app you’ll build, the URI will be the URL for the weather service, which specifies where to find it (the “web address” in the URL) and how to access it (using the HTTP protocol).
• Attributes or properties, which are the data that make up the resource. In the weather app, these attributes or properties will be the current weather, temperature, humidity, and cloud cover. Most of the time, these attributes or properties will be contained within JSON data structures.

REST Client Requests

REST clients access or manipulate resources using the methods built into the HTTP protocol, which are requests that a client sends to a server. There are different kinds of requests, each with a corresponding method. The requests that are most important for communicating with an API are:

1. GET: Retrieves a resource or a list of resources from the server. It’s read-only; it can only retrieve information and can’t modify it. It’s often used to retrieve web pages and data from web APIs.
2. POST: Submits data to be processed by the server. It’s used to submit data from web page forms and create new resources, which often means adding data to a database or other store of data.
3. PUT: Updates an existing resource by replacing its data entirely with the new data included with the request.
4. PATCH: A less drastic version of PUT that modifies specific parts of an existing resource.
5. DELETE: Removes the specified resource from the server.

These HTTP methods are often referred to as HTTP verbs because they’re used to take actions on resources.

A Simple REST Example

Pretend that example.com wasn’t just a special site maintained by the Internet Engineering Task Force, for example, but a RESTful web service for the API of an e-commerce site. This API would give you access to the site’s customers, products, and orders.

Each customer, product, and order is a resource in this scenario. example.com’s web service would have dedicated directory paths, called collection endpoints, or more simply, endpoints:

• Customers would be accessed at the collection endpoint https://example.com/customers.
• Products would be accessed at the collection endpoint https://example.com/products.
• Orders would be accessed at the collection endpoint https://example.com/orders.

To get a list of customers, you would send a GET request to the collection endpoint for customers, https://example.com/customers. The example.com server would return a response with a list of customers, most likely in JSON format. Getting a list of products and orders would be similar, except that you would send the GET request to https://example.com/products or https://example.com/orders instead.

In a REST service, each resource has its own URL, called a resource endpoint. This URL is usually made of the collection endpoint to which it belongs, followed by something that uniquely identifies the resource, such as an ID number or string. Here are some resource endpoints that would be used in the example e-commerce site:

• A customer with the ID abc123 would be accessed at the resource endpoint https://example.com/customers/abc123.
• A product with the ID def456 would be accessed at the resource endpoint https://example.com/products/def456.
• An order with the ID ghi789 would be accessed at the resource endpoint https://example.com/orders/ghi789.

To access or manipulate a specific customer record, you would send the appropriate request to its resource:

• To view the customer record for the customer with ID, abc123, send a GET request. The customer’s resource endpoint would be, https://example.com/customers/abc123. The service’s response would include a status code. The status code would note if the GET request was successful. If so, the response would also contain the customer’s information, typically in JSON form.
• To add a new customer, you would send a POST request. The request would go to the collection endpoint, https://example.com/customers. In that request, you would include the required information for that customer: name, address, and so on. The service’s response would include a status code. The status code would show if the POST request was successful. If so, the response would also contain the new record.
• To completely update the information for the customer with ID, abc123, send a PUT request. The customer’s resource endpoint would be https://example.com/customers/abc123. In that request, you would include the required information for that customer — name, address, and so on. The service’s response would include a status code. The status code would show if the PUT request was successful. If so, it would also contain the updated record.
• Send a PATCH request to update specific parts of the information for the customer with ID, abc123. You can update data such as only their name and no other data fields. The request would go to the resource endpoint, https://example.com/customers/abc123. In that request, include only the customer information to update. In this case, include the customer’s name. The service’s response would include a status code. The status code would show if the PATCH request was successful. If so, it would also contain the customer’s updated data: their name.
• To delete the customer record for the customer with ID abc123, send a DELETE request. The customer’s resource endpoint would be https://example.com/customers/abc123. The service’s response would include a status code showing if the DELETE request was successful.

Path Parameters vs. Query Parameters

The resource endpoint URLs in the above example use the path parameters, where the ID that uniquely identifies the resource is included as part of the URL path. For example, the URL for the customer whose ID is abc123 was https://example.com/customers/abc123.

RESTful services often use path parameters where providing one parameter is enough to access a resource. In the demo, you’ll see this in action. You’ll provide the ID for a specific Star Wars character to get information about them.

Some RESTful services’ resource endpoint URLs follow an older style and use query parameters, where the ID is included as a parameter in a query string at the end of the URL. Under this style, the URL for the customer with ID abc123 would be something like https://example.com/customers?id=abc123.

RESTful services often use query parameters where you need to provide more than one parameter to access a resource. You’ll see this in action in the demo, where you’ll provide the latitude, longitude, and specific types of weather information for the location where you want the current weather.

REST Server Responses

The response provided by a server contains all kinds of information, including:

• An HTTP status code: A numerical code indicating the request was successfully fulfilled, or an error occurred on the client or server end. This is useful for determining whether getting, adding, updating, or deleting a resource actually happened.
• Headers: Metadata about the response, including a description of the type of data contained within it, such as whether it’s HTML or JSON.
• The response body: The actual data of the response. For example, if the request was for information about a specific customer, the response body would contain that customer’s data.

HTTP Status Codes

When your application receives a response from a server, it should first check the status code to determine whether it can proceed with the received data or take some error-correcting measure.

Status codes are three-digit numbers that are divided into five categories:

• 100 - 199: Informational: These usually are for the server to inform the client that their request has been received and that it’s in the middle of processing that request. This category of status code applies to long-running processes and is rarely seen when making API calls.
• 200 - 299: Success: There are different success status codes for different requests, but they generally mean that the server successfully received, understood, and processed the request.
• 300 - 399: Redirection: These codes tell the client to take additional steps to complete the request, usually because the resource has been moved to a different URL.
• 400 - 499: Client error: “This is your application’s fault.” If your application receives this status code, it means that there was an error with the request it sent.
• 500 - 599: Server error: “This is the server’s fault.” This usually indicates some issue with the server, such as the server application being unavailable, overloaded, or crashed.

For this lesson, the status codes you’re most interested in are the “2xx” and “4xx”.

For the “2xx” series, the codes that are most useful when calling APIs are:

• 200: OK: The server fulfilled the request.
  • For GET requests, the server could return the requested data for the resource.
  • For PUT and PATCH requests, the server could update the specified resource.
• 201: Created: This is specifically for POST requests. The server fulfilled the request, and a new resource was created.
• 204: No Content: The server fulfilled the request, but there is no content in the response body. This sounds like an error, but in the case of a DELETE request, it means that the resource was successfully deleted.

After making a request to an API, you would check the response to see if it contained the appropriate “success” status code: 200 for GET, PUT, and PATCH requests, 201 for POST requests, and 204 for DELETE requests. If it finds this kind of status code, it can get the data from the response and proceed with its operations.

For the “4xx” series, the codes that are most useful when calling APIs are:

• 400: Bad request: This server couldn’t fulfill the request because it couldn’t understand it. This indicates that the request had some bad syntax or was somehow malformed.
• 401: Unauthorized: This is the server saying, “I don’t know who you are, and I’m not letting you access this resource.” This means the client must provide valid login credentials for access, such as a username/password combination or an API key. You’ll see this on APIs that charge money for access or have limits on their usage.
• 403: Forbidden: This is the server saying, “I know who you are, and you’re not authorized to access this resource.” It confirms that the client is logged in with valid credentials. However, the client doesn’t have the necessary permissions or privileges to access the requested resource. One example is a standard user attempting to access a resource only available to administrators.
• 404: Not found: This is the status code known to every web user. It means that the server can’t find the requested resource, probably because the resource endpoint URL provided was invalid.
• 405: Not allowed: This indicates that the method in the request—GET, POST, PUT, PATCH, DELETE, and so on—isn’t allowed. For example, the server will return this error if the client tries to send a PUT request to a collection endpoint. The PUT request tries to write to resources meant to be read-only.

If your application finds a “4xx” or “5xx” status code in the response, it must take some error-handling measures. These are beyond the scope of this lesson. In the following demos, if an application finds one of these status codes, it’ll simply display an error message.

Enough theory — it’s time for practice! You’ll get a better feel for calling APIs in the following demos.