When we work with APIs, we typically work with a request-response pattern or a streaming pattern. However, another API pattern calls an out of process function initiating a long transaction.
I worked with a service that acted like a request-response API because it returned a JSON doc with metadata about the request. However, I would have to wait for the request to be fulfilled. For example, I would send a POST request with a JSON doc like this:
{ query: {
"queryFilter": {
"startDate": "2021-10-01",
"daysOfWeek": {
"U": false,
"M": true,
"T": true,
"W": true,
"R": true,
"F": true,
"S": false
},
"endDate": "2021-12-31",
"timeIntervals": [
"0500-0900",
"1700-2100"
]
},
"userEmail": "spara@example.com",
"userId":88AB
}
The API would return metadata about the request.
{
"requestId": 93434555,
"status": "Pending",
"userEmail": "spara@example.com",
"userId": 88AB,
"whenSubmitted": "2022-01-20T14:49Z"
}
At this point, I had two options. I could wait for a notification email with the download URL for the requested data or poll an endpoint to check the status. To poll the status endpoint, I would send a GET request with my userId and requestId, e.g.
If the request were completed, the service would return the download URL in the response.
{
"requestId": 93434555,
"status": "Completed Successfully",
"userEmail": "spara@example.com",
"userId": 88AB,
"whenSubmitted": "2022-01-20T14:49Z",
"downloadUrl": "https://longtransaction.com/88AB/93434555/data.csv.gz",
"urlExpirationDate": "2022-02-19T16:30Z"
}
I could then parse the response for the download URL.
Automating the process
Because I had many queries, I wrote a script to send the requests and a script to download the documents. Since the final output wasn't time-critical, I opted to wait for the email notifications to run the download script.
Here's an example requests script.
import json
import csv
import requests
durations = [{"startDate": "2019-01-01","endDate": "2019-03-31"}, ..., {"startDate": "2021-09-01","endDate": "2021-12-31"}]
query = { query: {
"queryFilter": {
"startDate": "",
"daysOfWeek": {
"U": false,
"M": true,
"T": true,
"W": true,
"R": true,
"F": true,
"S": false
},
"endDate": "",
"timeIntervals": [
"0500-0900",
"1700-2100"
]
},
"userEmail": "spara@example.com",
"userId":88AB
}
writer = csv.writer("request_ids.csv", w)
for d in durations:
start = d["startDate"]
end = d["endDate"]
new_query = query
new_query['queryFilter']['startDate']= start
new_query['queryFilter']['endDate']= end
response = requests.post(url, json=new_query)
r = response.json()
request_id = r["requestId"]
writer.writerow(request_id)
request_ids.close()
The script sends the requests to the service and writes the requestsIds to a CSV file. You can then use the requestIds to drive the download script.
import json
import requests
import csv
base_url = "https://longtransaction.com/88AB/"
def download_file(url):
local_filename = url.split('/')[-1]
with requests.get(url, stream=True) as r:
r.raise_for_status()
with open(local_filename, 'wb') as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
return local_filename
def get_download_url(request_id):
request_id = str(id)
response = requests.get(base_url+request_id)
doc = response.json()
download_url = doc["outputUrl"]
return download_url
with open("request_ids.csv") as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
for row in csv_reader:
id = str(row[0])
download_url= get_download_url(id)
download = download_file(download_url)
Summary
While this is far from an elegant solution, it does the job in the spirit of "get things done." REST and the request-response pattern have been implemented for many services; this was an interesting variation that adopted the request-response variation for a long transaction.
Top comments (0)