> ## Documentation Index
> Fetch the complete documentation index at: https://product-discovery.developer.voyado.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scheduling

> How to schedule Voyado Elevate catalog imports, covering queueing, long-running imports, status polling, streaming, and error handling.

## Importing catalog data

Catalog data is imported using either the standard PUT or POST [HTTP method](https://en.wikipedia.org/wiki/HTTP#Request_methods) <Icon icon="external-link" size={12} /> with personal cluster credentials provided by Voyado. Valid catalog imports return a 204 OK response. Invalid imports return an [error](#error-handling) response with details describing the issues in the file. For more information about validation, see [Validation](/elevate/docs/integration/catalog-imports/validation). For details about the import request, see [Import](/elevate/docs/api/admin/v4/import/catalog).

## Scheduled imports

Once live, catalog data must be imported on a regular basis. This is typically handled by a program that [queues](#queueing-imports) and sends imports at regular intervals, either to fully [replace the catalog](/elevate/docs/integration/catalog-imports/catalog-imports#replace-catalog) or to [update the catalog](/elevate/docs/integration/catalog-imports/catalog-imports#update-catalog) incrementally.

### Replacing the catalog

Fully [replacing the catalog](/elevate/docs/integration/catalog-imports/catalog-imports#replace-catalog) involves sending an import containing all product data. This is recommended on a regular basis for ensuring data consistency.

* Small databases (fewer than 10,000 products) — Imports that replace the catalog can be used exclusively. In this case, imports are recommended to run once per hour.
* Large databases — Full catalog replacements are still recommended, though at a lower frequency, such as once per day, week, or month, depending on consistency needs and setup. These should be supplemented with smaller, more frequent imports that update specific parts of the catalog.

### Updating the catalog

Incrementally [updating the catalog](/elevate/docs/integration/catalog-imports/catalog-imports#update-catalog) can be done through various import [operations](/elevate/docs/api/admin/v4/import/catalog#operation). This allows both structural changes to product groups and modifications to individual variants, products, or product groups.

Updates are typically triggered by changes in ERP systems, such as price and stock updates, or by product updates from a PIM system. Examples of different types of updates can be found [here](/elevate/docs/integration/catalog-imports/import-examples).

### Queueing imports

<Note>
  Incoming imports will be rejected while another import is being processed. This means that software handling catalog imports must be set up to run imports one at a time.
</Note>

Since imports are processed one at a time and for various reasons could fail, all imports should ideally be sent to a queue that aggregates imports, handles retries, and [handles errors](#error-handling). A typical process for this is normally responsible for the following or similar tasks:

* Changes invoked by other systems are placed in a queue.
* Once every 5 minutes, empty the queue and, if there are any changes, aggregate them and send them as a single import.
* Log and notify an appropriate person to take action upon errors.
* Remove successfully imported changes from the queue.
* Save unsuccessful import files for easier debugging.

### Long-running imports

Depending on the size of the import to Elevate, it might take some time. Imports exceeding the maximum duration respond with `202 ACCEPTED` and return an import ID that can be used to query the status of the import. The default maximum duration allowed for blocking imports can be overridden by adding the `blockingTimeout` parameter to the request. The blocking timeout specifies the time to wait after the whole import body has been received. The latest status for an import will be kept for 7 days.

#### Example

The following example showcases the request flow for an import exceeding the `blockingTimeout`.

Import using `blockingTimeout` of 1 minute (=PT1M):

```bash icon="terminal" theme={null}
curl -i \
-X POST \
-H 'Api-Key: {api-key}' \
-H 'Accept-Encoding: gzip' \
-H 'Content-Type: application/xml' \
"https://{cluster-id}.elevate-api.cloud/api/admin/v3/import/data?blockingTimeout=PT1M&force=false&name=name" \
-T request-body.file
```

After the blocking timeout, the server hasn't yet processed the file. The server responds with `202 ACCEPTED` and the body:

```json icon="code" theme={null}
{
    "id": "5537431447739019570",
    "status": "IN_PROGRESS",
    "reason": ""
}
```

The previously returned import `id` can now be used to periodically poll the import status.

```bash icon="terminal" theme={null}
curl -i \
-X GET \
-H 'Api-Key: {api-key}' \
-H 'Accept-Encoding: gzip' \
"https://{cluster-id}.elevate-api.cloud/api/admin/v3/import/status?id=5537431447739019570"
```

The [import status](/elevate/docs/api/admin/v4/import/status) responds with `200 OK` and the status in the body:

```json icon="code" theme={null}
{
    "id": "5537431447739019570",
    "status": "SUCCESS",
    "reason": ""
}
```

#### Import statuses

| Status        | Description                                                                                     |
| ------------- | ----------------------------------------------------------------------------------------------- |
| `NOT_FOUND`   | No import found with the supplied ID.                                                           |
| `WAITING`     | The import is being received, unpackaged, stored, or waiting for a previous import to complete. |
| `IN_PROGRESS` | The import is currently being processed.                                                        |
| `SUCCESS`     | The import completed successfully, and the changes have been applied.                           |
| `FAILURE`     | The import failed, and a reason should be included.                                             |
| `ABORTED`     | The client disconnected while waiting for the import to be processed.                           |

#### Polling handling

When polling the status of an import, the result falls into three different groups.

* The import is being handled, keep polling. This occurs during `WAITING` and `IN_PROGRESS`.
* The import completed. This occurs during `SUCCESS` and `FAILURE`. Check the reason for more information.
* The import was not found. This occurs during `NOT_FOUND`.

Since `ABORTED` is raised only when the client closes the connection, it will not be returned in the `/v4/import/catalog` result, and neither will an import ID. This allows it to be omitted from the polling handling, however, it can be included. The only way to get the `ABORTED` status is to get the import ID from the logs in the apps.

### Translation imports

In Admin API v4, translations are designed to be handled independently from catalog imports for optimal performance. Translation entities can be imported periodically only when there are actual changes, similar to how product data changes are handled. This approach reduces data transfer and makes imports faster, especially when translations change less frequently than core entities.

**Separate periodic translation imports**

Handle translations independently by maintaining a separate translation feed or builder. Import translation entities periodically only when actual changes occur. This is the most efficient approach in terms of data transfer and import performance, and represents the ideal technical setup for v4.

**Include translations with every import**

Alternatively, translations can be included with every catalog import, which closely mirrors how translations worked in v3. While this approach is fully supported and provides simplicity, it results in larger payloads and slower imports due to the overhead.

### Streaming imports

Streaming catalog imports are supported. Streaming allows sending data in chunks without loading the entire payload into memory at once. This is useful for large catalog imports where memory constraints are a concern.

```java Java streaming example icon="java" expandable theme={null}
public static void main(String[] args) throws IOException {
    List<String> lines = List.of(
        "{some-catalog-operation-1}\n",
        "{some-catalog-operation-2}\n"
        // ...
    );

    HttpURLConnection connection =
        (HttpURLConnection) URI
            .create("https://{cluster-id}.elevate-api.cloud/api/admin/v4/import/catalog")
            .toURL()
            .openConnection();

    connection.setRequestMethod("PUT");
    connection.setChunkedStreamingMode(1 << 8);
    connection.setRequestProperty("Content-Type", "application/jsonlines");
    connection.setRequestProperty("Content-Encoding", "gzip");
    connection.setRequestProperty("Accept-Encoding", "gzip");
    connection.setRequestProperty("Api-Key", "{your-api-key}");
    connection.setDoOutput(true);

    try (var out = new OutputStreamWriter(
            new GZIPOutputStream(connection.getOutputStream()),
            java.nio.charset.StandardCharsets.UTF_8)) {

        for (String line : lines) {
            out.write(line);
            out.flush();
        }
    }

    connection.getResponseCode(); // wait for server response
    connection.disconnect();
}
```

## Error handling

All imports are atomic (either the entire import is processed or nothing will be imported), making it easier to recover from errors. An error can be caused either by the content being invalid or because of other factors, such as network errors. If an error is received, the import can be safely rerun once the connection is restored or the content of the catalog import has been resolved.

A detailed list of import response codes can be found in the specification for the [import endpoint](/elevate/docs/api/admin/v4/import/catalog#response).

If a retailer experiences persistent network errors, they should first contact their network administrator. If the problem seems to be on Voyado's end, please contact us.

<Tip>
  The catalog import software should handle temporary network errors without manual intervention, as this is usually done through retries.
</Tip>

### Force imports

<Warning>
  Do not override the built-in protection in scheduled imports. Overrides should only be made for isolated and manually triggered imports.
</Warning>

Protection from potentially harmful imports is built into Voyado Elevate. If an import looks dangerous, for example if it removes half of all products in a market, it will be rejected with an appropriate error message and code 400. To override the protection and force Elevate to accept such an import, a `force=true` argument can be added to the post request of the import.