python - Open S3 object as a string with Boto3

ID : 20205

viewed : 46

Tags : pythonbotoboto3python

Top 5 Answer for python - Open S3 object as a string with Boto3

vote vote

100

read will return bytes. At least for Python 3, if you want to return a string, you have to decode using the right encoding:

import boto3  s3 = boto3.resource('s3')  obj = s3.Object(bucket, key) obj.get()['Body'].read().decode('utf-8')  
vote vote

82

I had a problem to read/parse the object from S3 because of .get() using Python 2.7 inside an AWS Lambda.

I added json to the example to show it became parsable :)

import boto3 import json  s3 = boto3.client('s3')  obj = s3.get_object(Bucket=bucket, Key=key) j = json.loads(obj['Body'].read()) 

NOTE (for python 2.7): My object is all ascii, so I don't need .decode('utf-8')

NOTE (for python 3.6+): We moved to python 3.6 and discovered that read() now returns bytes so if you want to get a string out of it, you must use:

j = json.loads(obj['Body'].read().decode('utf-8'))

vote vote

70

This isn't in the boto3 documentation. This worked for me:

object.get()["Body"].read() 

object being an s3 object: http://boto3.readthedocs.org/en/latest/reference/services/s3.html#object

vote vote

66

Python3 + Using boto3 API approach.

By using S3.Client.download_fileobj API and Python file-like object, S3 Object content can be retrieved to memory.

Since the retrieved content is bytes, in order to convert to str, it need to be decoded.

import io import boto3  client = boto3.client('s3') bytes_buffer = io.BytesIO() client.download_fileobj(Bucket=bucket_name, Key=object_key, Fileobj=bytes_buffer) byte_value = bytes_buffer.getvalue() str_value = byte_value.decode() #python3, default decoding is utf-8 
vote vote

56

If body contains a io.StringIO, you have to do like below:

object.get()['Body'].getvalue() 

Top 3 video Explaining python - Open S3 object as a string with Boto3

Related QUESTION?