How to Handle Pagination When Fetching External API Data in Odoo 19

Typically, a vast majority of external APIs are able to send data in chunks instead of sending an entire batch of information through a single transmission, and this process is referred to as pagination. Pagination is used by the API in order to cope with large amounts of data without stressing the server or the program sending requests. In the particular case of Odoo 19, connecting Odoo to an external device requires acquiring multiple items, customers, orders, and so forth, so there is no other way out but to utilize pagination.

Well-implemented pagination would imply that Odoo will be able to obtain the necessary information without sending repeated responses and without omitting anything. Pagination can be done via page numbers, offset, cursor or any other links that may be provided by the used API. Therefore the process should be grasped properly and applied correctly.

What Is Pagination in an API?

Pagination is a method used by APIs to split up a big amount of data into smaller sections. Instead of giving 10,000 records all at once, an API will send only 100 records in one request. After receiving this request, the client can send multiple requests until all records have been received.

E.g., a customer API from another company may return

{
    "data": [
        {"id": 101, "name": "Customer A"},
        {"id": 102, "name": "Customer B"}
    ],
    "page": 1,
    "total_pages": 50
}

In Odoo, the response should be processed, and after that, the next response should be requested.

The advantages of this method are numerous:

• A smaller amount of data results in a lower memory demand.
• The requests are simpler for the API.
• The whole amount of data can be shared gradually.
• Temporary failures do not require restarting the entire synchronization process.
• It is possible to control the number of data being processed at a time.

Why Is Pagination Important in Odoo Integrations?

It is possible that a straightforward API integration can work well when tested on a few tens of records but run into issues when the same integration is applied to thousands of records.

The moment an integration retrieves all the information at once, the response can be unreasonably large and would require even more time for processing. It is important to emphasize that some APIs may also have restrictions regarding the number of records returned in one request.

This is particularly crucial for the Odoo integration when it comes to synchronization of:

• customers and contacts;
• products and product variants;
• sales orders;
• invoices;
• inventory records;
• transactions from third-party apps.

Pagination makes it possible for the integration not to bother about the entire amount of data and get records in portions.

Common Pagination Methods

No common pagination format is present that every API follows. The execution of pagination on any API can be different depending upon the external services.

In page-based pagination, you need the page number along with page size. For example:

/api/customers?page=1&limit=100

/api/customers?page=2&limit=100

In offset-based pagination, the current offset of records is used as follows:

/api/customers?offset=0&limit=100

/api/customers?offset=100&limit=100

Cursor-based pagination uses the cursor received from the previous response as follows: 

/api/customers?limit=100

/api/customers?limit=100&cursor=eyJpZCI6MTAw...

Handling Page-Based Pagination in Odoo

Assuming that there are two parameters involved in the APIs, namely, page and limit. An Odoo function would still send requests for pages over and over whenever records are still being sent by the API.

import requests
page = 1
limit = 100
while True:
    response = requests.get(
        api_url,
        params={
            "page": page,
            "limit": limit,
        },
        headers=headers,
        timeout=30,
    )
    response.raise_for_status()
    data = response.json()
    records = data.get("data", [])
    if not records:
        break
    for item in records:
        self._process_external_record(item)
    page += 1

In this instance, Odoo is getting 100 records in each call. When the API sends an empty list, that means the loop will end.

The parameters will differ for each API; therefore, the developers must refer to the documentation for the API to identify the pagination that is required.

Using Response Metadata

Application program interfaces (APIs) can incorporate pagination information in the response itself. A practical example can be given below:

{
    "data": [],
    "pagination": {
        "current_page": 1,
        "total_pages": 10,
        "per_page": 100
    }
}

In such cases, Odoo can determine whether the synchronization process is completed by looking at total_pages.

page = 1
while True:
    response = requests.get(
        api_url,
        params={"page": page, "per_page": 100},
        timeout=30,
    )
    response.raise_for_status()
    result = response.json()
    for item in result.get("data", []):
        self._process_external_record(item)
    pagination = result.get("pagination", {})
    if page >= pagination.get("total_pages", page):
        break
    page += 1

This technique is very effective if the API is telling you the total number of pages.

Working with Cursor-Based Pagination

Cursor-based APIs function in a different manner. Instead of incrementing a page number, Odoo gets a cursor from the external API and sends it back in the next call.

Here's a simplified example:

cursor = None
while True:
    parameters = {"limit": 100}
    if cursor:
        parameters["cursor"] = cursor
    response = requests.get(
        api_url,
        params=parameters,
        headers=headers,
        timeout=30,
    )
    response.raise_for_status()
    result = response.json()
    for element in result.get("data", []):
        self._process_external_record(element)
    cursor = result.get("next_cursor")
    if not cursor:
        break

The key thing is that Odoo does not determine where to go next. The external API is the one providing the cursor, and Odoo simply uses it to make the next call.

Managing Errors During Paging

Synchronisation should not assume that each page will succeed. For example, the first five requests may succeed, but the sixth request may fail due to a temporary network problem.

A timeout and a check on the HTTP response prevent the integration from hanging indefinitely or continuing silently after a failed request.

response = requests.get(api_url, params=params, headers=headers, timeout=30)
response.raise_for_status()

Also, developers are able to add controlled retry logic for temporary failures in production integrations. But be careful about retrying, especially if the API operation can create or change data.

Logging the current page or cursor may also help in troubleshooting:

_logger.info("Retrieving customer page %s", page)

So if sync fails for some reason, you can look at the log and get a rough idea of where the problem occurred.

Preventing Duplicate or Lost Data 

Pagination is not simply moving to the next page but understanding what happens when there is a change in the remote data. An example of this would be creating a new record while waiting for the response between two pages of data requests. As a result, with offset pagination, there is a risk of missing some records or maybe displaying the record more than once depending on how the external system responds in regard to the order of the data.

The use of cursor-based pagination or stable sorting should be encouraged whenever it is possible to do so.

Odoo integration should always keep an external ID for all data that has been synchronized so that the integration can differentiate between an incoming record that has already been received and an incoming record that has to be created.

Best Practices of Pagination in Odoo 19

The methods of pagination that are successful have to take the operational specifics of the external API into account, as making any assumptions about the details of functionality of the offered services is not acceptable.

Helpful practices according to which correct pagination may be done include such ones as:

• Setting a reasonable page size that is small enough to avoid getting too many results,
• Always specifying timeout when sending a request,
• Handling HTTP errors properly,
• Keeping records of any synchronization information,
• Maintaining external IDs of synchronized documents,
• Avoiding any assumptions regarding the document count at the time of Synchronization,
• Using cursors, if available and suitable,
• Processing the documents in adequate amounts,
• Planning retries in detail to avoid possible duplicates.

Pagination is a fundamental element of developing successful Odoo 19 integrations. This is due to the large amounts of data being transferred across APIs. Understanding how to work with page and cursor pagination methods is needed for programmers to receive records in a gradual way without overusing Odoo server resources or API. Error handling should be implemented to complete the synchronization successfully. In addition, it is necessary to log the application-generated records, properly use timeout and avoid duplication.

To read more about Overview of API Integration in Odoo 19, refer to our blog, Overview of API Integration in Odoo 19.