How did you handle deep pagination? If a user wants to see the 100th page of results, using standard from/size in Elasticsearch, it is very memory-intensive. Have you used Search After?

For handling deep pagination in Elasticsearch, it is indeed inefficient to use the from and size parameters with large from values, as this leads to high memory consumption and reduced performance.

Instead, the search_after mechanism is often used, which allows efficiently retrieving the next page of results by using the sort values of the last item on the previous page. This works like a cursor, helping to avoid skipping a large number of documents.

An example of using search_after in a query:

{
  "size": 10,
  "query": { "match_all": {} },
  "sort": [
    {"timestamp": "asc"},
    {"_id": "asc"}
  ],
  "search_after": ["2023-01-01T00:00:00", "some_id"]
}

In the Elasticsearch Go client, you can construct such a query by passing the search_after values from the last item of the previous page.

Thus, for viewing the 100th page, it is better to retrieve pages sequentially using search_after rather than doing from=990 and size=10.

If a random page without sequential traversal is needed, you can consider using the Scroll API or changing the UX to avoid deep jumps.