How to page the Iamazons3.Listobjects method?

Question

The method IAmazonS3.ListObjects that you can check out here limits the return by 1000 items per request but does not say if it is possible to make pagination, I have Buckets with more than 10 million files and I need to go through each of them. The description of the method in Portuguese is this:

Returns some or all (at most 1000) of the objects in a Bucket. You can use request parameters as a selection criterion to return a set of objects in a Bucket

Is it possible to complete this request? I haven’t found anything in Google or in the documentation.

My job is this:

static ListObjectsResponse GetBucketObjects(string bucketName)
{
  ListObjectsRequest request = new ListObjectsRequest();  
  request.BucketName = bucketName;
  request.MaxKeys = 1000; //Mesmo que eu coloque 90000000000, o sistema considera 1000 
  ListObjectsResponse response = client.ListObjects(request);
  return response;
}

Answer 1

Following the documentation of SDK for JavaScript, you have to know the NextMarker for the query to return the data after a given result.

Thus, it is not possible to consult by a specific page without having consulted the previous one.

Here’s an example in JS.:

var params = {
    Bucket: 'STRING_VALUE',
    MaxKeys: 500
}

var listObjects = function (params) {
    return new Promise((resolve, reject) => {
        s3.listObjects(params, function(err, data) {
            if (err) {
                reject(err);
            } else {
                resolve(data);
            }
        })
    })
}

async function GetBucketObjects () {
    var data = null;
    do
    {
        var data = await listObjects(params)
        // faça algo com data.Contents
        params.Marker = data.NextMarker
    } while (data.IsTruncated)
}

GetBucketObjects('')

Here is a sketch of the C# version (not tested)

var listResponse = default(ListObjectsResponse);
var listRequest = new ListObjectsRequest
{
    BucketName = "STRING_VALUE",
    MaxKeys = 500
};    

do
{

    listResponse = client.ListObjects(listRequest);
    // faça algo com listResponse.S3Objects
    listRequest.Marker = listResponse.NextMarker;
} while (listResponse.IsTruncated);