Multipart downloads in S3

As you can imagine, supporting omnichannel automation means shifting a lot of large files around.

One of the (not very well documented) limitations of the aws-sdk getObject method is the ~2GB limit.

This may rear it's ugly head in the form of this error:

RangeError [ERR_INVALID_OPT_VALUE]: The value \"2536463704\" is invalid for option \"size\"
    at Function.alloc (buffer.js:278:3)
    at Object.alloc (/path/to/node_modules/aws-sdk/lib/util.js:136:28)
    at Object.concat (/path/to/node_modules/aws-sdk/lib/util.js:175:28)
    at Request.HTTP_DONE (/path/to/node_modules/aws-sdk/lib/event_listeners.js:396:36)
    at Request.callListeners (/path/to/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/path/to/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/path/to/node_modules/aws-sdk/lib/request.js:683:14)
    at IncomingMessage.onEnd (/path/to/node_modules/aws-sdk/lib/event_listeners.js:306:26)
    at IncomingMessage.emit (events.js:203:15)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)

But don't fear, there's a relatively simple workaround involving '@aws/s3-managed-downloader'.

Firstly, install @aws/s3-managed-downloader using your preferred method.

npm install @aws/s3-managed-download

yarn add @aws/s3-managed-download

I always find it neater to work with promises over callbacks where possible, so you can extract the download logic to a helper method, like so:

  const downloadFile = async (s3: S3, bucketName: string, key: string, destination: string) => {
    // Instantiate ManagedDownloader
    const managedDownloader = new ManagedDownloader(s3)

    // Create a write stream to the provided destination file
    const writeStream = fs.createWriteStream(destination)

    // Use getObjectStream to download the file from S3 and resolve the promise once complete
    return new Promise((resolve, reject) =>
        managedDownloader
          .getObjectStream({
            Bucket: bucketName,
            Key: key,
          })
          .then((stream: Stream) => {
            stream.on('finish', resolve)
            stream.on('error', reject)

            stream.pipe(writeStream)
          }),
    )
  }

This method simply creates a download stream from the specified S3 bucket and key and pipes the output to the provided destination.

Once the stream has finished, the promise is resolved to signify the end of the download.

As most of the logic is now extracted away, calling it is as simple as:

  // Instantiate the S3 SDK
  const s3 = new S3()

  // Wait until the entire file has downloaded
  await downloadFile(s3, 'my-bucket', 'my-key', 'download.zip')

  // Process the downloaded file
  ...

Hopefully this is as useful to you as it is for us!

You can read more about the things we're working on here.