Handling Large Datasets (with Pagination)

The download API has been enhanced to support pagination for merchants needing to retrieve more than 100,000 records. This section outlines the process to fetch large datasets efficiently using the limit and last_charge_id parameters.

Process

Initial Request:

Set limit to 100,000 (the maximum allowed value).
Set last_charge_id to an empty string ("") for the first request. Send the request to the API endpoint (https://api.tap.company/v2/charges/download) with the required authorization token and payload (e.g., period, merchants, metadata, reference).

Response Headers:

has_more: Indicates if additional records are available (true or false).
page_size: The number of records per page (matches the limit, e.g., 100,000). number_of_pages: The total number of pages to iterate through to retrieve all records. total_count: The total number of available records (calculated as page_size * number_of_pages if has_more is false). last_charge_id: The ID of the last retrieved charge, used to fetch the next page.

Iteration Logic:

Loop through the API number_of_pages times.

For each iteration:

Call the API with limit set to 100,000 and last_charge_id set to the value from the previous response's last_charge_id.
Update last_charge_id with the new value from the response headers. Continue until has_more is false or all number_of_pages are processed.

Example Based on Response

From the provided API response:

has_more: true (indicating more records exist beyond the first page).
page_size: 100,000 (matches the limit set in the request). number_of_pages: 4 (total pages to fetch all records). total_count: Not explicitly provided but can be inferred as page_size * number_of_pages (e.g., 100,000 * 4 = 400,000 records, though has_more suggests additional pages may exist). last_charge_id: Not shown but will be included in the response headers for the next request.

Step-by-Step Implementation

First Request:

limit: 100,000
last_charge_id: ""

Use the following curl command, updating the payload as needed:

curl --request POST \
     --url https://api.tap.company/v2/charges/download \
     --header 'Authorization: Bearer sk_test_XKokBfNWv6FIYuTMg5sLPjhJ' \
     --header 'accept: text/plain' \
     --header 'content-type: application/json' \
     --data '
{
  "period": {
    "date": {
      "from": "1751457690000",
      "to": "1751901168712"
    },
    "type": "1"
  },
  "merchants": [
    "560208"
  ],
  "limit": 100000,
  "last_charge_id": ""
}
'

Process Response:

Extract last_charge_id from the headers.
Note number_of_pages (e.g., 4) and has_more (e.g., true).

Loop Through Pages:

• For each page up to number_of_pages:
• Call the API with limit: 100000 and last_charge_id from the previous response.
• Update last_charge_id with the new value from the response headers.
• Collect the records.

Completion:

• Stop when has_more is false or all number_of_pages are processed.
• Ensure all records are retrieved based on total_count (if provided) or page_size * number_of_pages.

Subsequent Request Example

For the next page:

curl --request POST \
     --url https://api.tap.company/v2/charges/download \
     --header 'Authorization: Bearer sk_test_XKokBfNWv6FIYuTMg5sLPjhJ' \
     --header 'accept: text/plain' \
     --header 'content-type: application/json' \
     --data '
{
  "period": {
    "date": {
      "from": "1751457690000",
      "to": "1751901168712"
    },
    "type": "1"
  },
  "merchants": [
    "560208"
  ],
  "limit": 100000,
  "last_charge_id": "<value_from_previous_response>"
}
'

(Replace <value_from_previous_response> with the actual last_charge_id from the response headers.)

Notes

📘

• Ensure limit and last_charge_id are included in the payload for all requests after the first.
• The total_count may need to be tracked across pages if not provided in the initial response, especially if has_more remains true beyond number_of_pages.
• Verify the payload structure with the API provider, as the example omits some fields (e.g., metadata, reference) that may be required.
• Test with a smaller dataset to validate the logic before handling large volumes.