Did you limit the scroll depth? You mentioned using bulk for initial data loading, but how did you handle incremental updates when the price or name of a product changed? How did you ensure Elasticsearch synchronization?

The scroll depth limit (pagination depth) is often used to prevent excessively long or resource-intensive queries when working with large volumes of data.

For initial data loading, it is indeed convenient to use bulk requests in Elasticsearch, which allow for quickly indexing large volumes.

For incremental updates (for example, when the price or name of a product changes), the following approaches are usually applied:

• Change tracking: store timestamps of the last update for each product.
• Fetching changed data: when updating, make a request to the data source with a filter by modification time to get only changed records.
• Updating in Elasticsearch: use the update API or re-index the changed documents.

To ensure synchronization of Elasticsearch with the data source:

• Use CDC (Change Data Capture) mechanisms or change logs to receive only current changes.
• Periodically perform full reindexing to eliminate desynchronization.
• Use message queues (e.g., Kafka) for guaranteed delivery of changes to the index.

Example of handling incremental updates:

// Get list of changed products since lastSync
changedProducts := getChangedProducts(lastSync)

// Form a bulk request for Elasticsearch
bulkRequest := elasticClient.Bulk()
for _, product := range changedProducts {
	req := elastic.NewBulkUpdateRequest().Index("products").Id(product.ID).Doc(product).DocAsUpsert(true)
	bulkRequest = bulkRequest.Add(req)
}

// Execute bulk request
_, err := bulkRequest.Do(ctx)
if err != nil {
	// handle error
}

// Update lastSync
lastSync = time.Now()

Thus, limiting scroll depth helps control load, bulk requests speed up indexing, and incremental updates along with proper synchronization ensure data relevance in Elasticsearch.