Questions tagged [amazon-s3]
Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services. QUESTIONS MUST BE ABOUT PROGRAMMING. Questions about general S3 support, functionality, configuration, etc. are OFF-TOPIC.
50,814
questions
1090
votes
41
answers
871k
views
Downloading an entire S3 bucket?
I noticed that there does not seem to be an option to download an entire s3 bucket from the AWS Management Console.
Is there an easy way to grab everything in one of my buckets? I was thinking about ...
- 12.1k
407
votes
13
answers
197k
views
AWS EFS vs EBS vs S3 (differences & when to use?) [closed]
As per the title of this question, what are the practical differences between AWS EFS, EBS and S3?
My understanding of each:
S3 is a storage facility accessible any where
EBS is a device you can ...
- 6,029
371
votes
21
answers
728k
views
Listing contents of a bucket with boto3
How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")?
Doing the following:
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('some/path/')
returns:
s3.Bucket(name='...
- 94.1k
358
votes
25
answers
380k
views
check if a key exists in a bucket in s3 using boto3
I would like to know if a key exists in boto3. I can loop the bucket contents and check the key if it matches.
But that seems longer and an overkill. Boto3 official docs explicitly state how to do ...
- 5,914
338
votes
22
answers
403k
views
How to rename files and folder in Amazon S3?
Is there any function to rename files and folders in Amazon S3? Any related suggestions are also welcome.
- 3,562
334
votes
18
answers
177k
views
S3 Static Website Hosting Route All Paths to Index.html
I am using S3 to host a javascript app that will use HTML5 pushStates. The problem is if the user bookmarks any of the URLs, it will not resolve to anything. What I need is the ability to take all url ...
- 5,539
331
votes
24
answers
327k
views
AWS S3: The bucket you are attempting to access must be addressed using the specified endpoint
I am trying to delete uploaded image files with the AWS-SDK-Core Ruby Gem.
I have the following code:
require 'aws-sdk-core'
def pull_picture(picture)
Aws.config = {
:access_key_id =>...
- 3,587
325
votes
2
answers
181k
views
Make a bucket public in Amazon S3
How can I set a bucket in Amazon S3 so all the files are publicly read-only by default?
- 23.5k
315
votes
33
answers
388k
views
S3 - Access-Control-Allow-Origin Header
Did anyone manage to add Access-Control-Allow-Origin to the response headers?
What I need is something like this:
<img src="http://360assets.s3.amazonaws.com/tours/8b16734d-336c-48c7-95c4-...
- 3,920
312
votes
9
answers
129k
views
When to use Amazon Cloudfront or S3
Are there use cases that lend themselves better to Amazon cloudfront over s3 or the other way around? I'm trying to understand the difference between the 2 through examples.
- 3,597
296
votes
67
answers
555k
views
Amazon S3 - How to fix 'The request signature we calculated does not match the signature' error?
I have searched on the web for over two days now, and probably have looked through most of the online documented scenarios and workarounds, but nothing worked for me so far.
I am on AWS SDK for PHP V2....
- 5,749
268
votes
25
answers
448k
views
AccessDenied for ListObjects for S3 bucket when permissions are s3:*
I am getting:
An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
When I try to get folder from my S3 bucket.
Using this command:
aws s3 cp s3://bucket-name/...
- 3,419
266
votes
12
answers
271k
views
Setting up FTP on Amazon Cloud Server [closed]
I am trying to set up FTP on Amazon Cloud Server, but without luck.
I search over net and there is no concrete steps how to do it.
I found those commands to run:
$ yum install vsftpd
$ ec2-authorize ...
- 3,119
251
votes
18
answers
171k
views
S3 Bucket action doesn't apply to any resources
I'm following the instructions from this answer to generate the follow S3 bucket policy:
{
"Id": "Policy1495981680273",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "...
- 22.1k
243
votes
34
answers
274k
views
How can I tell how many objects I've stored in an S3 bucket?
Unless I'm missing something, it seems that none of the APIs I've looked at will tell you how many objects are in an <S3 bucket>/<folder>. Is there any way to get a count?
- 4,563
237
votes
3
answers
164k
views
How to rename AWS S3 Bucket
After all the tough work of migration etc, I just realised that I need to serve the content using CNAME (e.g media.abc.com). The bucket name needs to start with media.abc.com/S3/amazon.com to ensure ...
- 2,773
236
votes
8
answers
412k
views
Open S3 object as a string with Boto3
I'm aware that with Boto 2 it's possible to open an S3 object as a string with: get_contents_as_string()
Is there an equivalent function in boto3 ?
- 3,231
235
votes
6
answers
111k
views
Security of REST authentication schemes
Background:
I'm designing the authentication scheme for a REST web service. This doesn't "really" need to be secure (it's more of a personal project) but I want to make it as secure as possible as an ...
- 75k
233
votes
7
answers
312k
views
Downloading folders from aws s3, cp or sync?
If I want to download all the contents of a directory on S3 to my local PC, which command should I use cp or sync ?
Any help would be highly appreciated.
For example,
if I want to download all ...
- 2,607
229
votes
9
answers
304k
views
How to specify credentials when connecting to boto3 S3?
On boto I used to specify my credentials when connecting to S3 in such a way:
import boto
from boto.s3.connection import Key, S3Connection
S3 = S3Connection( settings.AWS_SERVER_PUBLIC_KEY, settings....
- 6,918
226
votes
7
answers
154k
views
How to make all Objects in AWS S3 bucket public by default?
I am using a PHP library to upload a file to my bucket. I have set the ACL to public-read-write and it works fine but the file is still private.
I found that if I change the Grantee to Everyone it ...
- 3,375
225
votes
4
answers
154k
views
How can I use wildcards to `cp` a group of files with the AWS CLI? [closed]
I'm having trouble using * in the AWS CLI to select a subset of files from a certain bucket.
Adding * to the path like this does not seem to work:
aws s3 cp s3://data/2016-08* .
- 12.1k
219
votes
23
answers
309k
views
How do you search an amazon s3 bucket?
I have a bucket with thousands of files in it. How can I search the bucket?
- 8,732
209
votes
22
answers
355k
views
Getting Access Denied when calling the PutObject operation with bucket-level permission
I followed the example on http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_examples.html#iam-policy-example-s3 for how to grant a user access to just one bucket.
I then tested the ...
- 12.5k
208
votes
14
answers
305k
views
Save Dataframe to csv directly to s3 Python
I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don't want to save the file locally before transferring it to s3. Is there any method like to_csv for writing ...
- 2,083
203
votes
8
answers
565k
views
How to write a file or data to an S3 object using boto3
In boto 2, you can write to an S3 object using these methods:
Key.set_contents_from_string()
Key.set_contents_from_file()
Key.set_contents_from_filename()
Key.set_contents_from_stream()
Is there a ...
- 11.5k
202
votes
31
answers
441k
views
Quick way to list all files in Amazon S3 bucket?
I have an amazon s3 bucket that has tens of thousands of filenames in it. What's the easiest way to get a text file that lists all the filenames in the bucket?
- 4,153
198
votes
10
answers
83k
views
AWS Error Message: A conflicting conditional operation is currently in progress against this resource
I'm getting this error intermittently.
I have a program that uses the java aws sdk and loads over the 10s of thousands of small files to s3. I see this error intermittently.
Could not find any ...
- 5,334
197
votes
6
answers
84k
views
Serving gzipped CSS and JavaScript from Amazon CloudFront via S3
I've been looking for ways of making my site load faster and one way that I'd like to explore is making greater use of Cloudfront.
Because Cloudfront was originally not designed as a custom-origin ...
- 3,495
190
votes
3
answers
147k
views
Technically what is the difference between s3n, s3a and s3?
I'm aware of the existence of https://wiki.apache.org/hadoop/AmazonS3 and the following words:
S3 Native FileSystem (URI scheme: s3n) A native filesystem for reading and writing regular files on S3. ...
- 18k
188
votes
10
answers
158k
views
Amazon S3 direct file upload from client browser - private key disclosure
I'm implementing a direct file upload from client machine to Amazon S3 via REST API using only JavaScript, without any server-side code. All works fine but one thing is worrying me...
When I send a ...
- 10.4k
187
votes
20
answers
401k
views
Amazon S3 exception: "The specified key does not exist"
I am using the AmazonS3Client in an Android app using a getObject request to download an image from my Amazon S3 bucket.
Currently, I am getting this exception:
com.amazonaws.services.s3.model....
- 1,991
184
votes
9
answers
146k
views
Nodejs AWS SDK S3 Generate Presigned URL
I am using the NodeJS AWS SDK to generate a presigned S3 URL. The docs give an example of generating a presigned URL.
Here is my exact code (with sensitive info omitted):
const AWS = require('aws-...
- 2,171
183
votes
21
answers
288k
views
AWS S3 CLI - Could not connect to the endpoint URL
$ aws s3 ls
Could not connect to the endpoint URL: "https://s3.us-east-1a.amazonaws.com/"
What could be the problem?
- 4,256
183
votes
4
answers
134k
views
Amazon S3 - HTTPS/SSL - Is it possible? [closed]
I saw a few other questions regarding this without any real answers or information (or so it appeared).
I have an image here:
http://furniture.retailcatalog.us/products/2061/6262u9665.jpg
Which is ...
- 21.9k
180
votes
20
answers
388k
views
Retrieving subfolders names in S3 bucket from b\Boto3
Using Boto3, I can access my AWS S3 bucket:
s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket-name')
Now, the bucket contains folder first-level, which itself contains several sub-folders named ...
- 9,546
179
votes
16
answers
245k
views
Amazon S3 console: download multiple files at once
When I log to my S3 console I am unable to download multiple selected files (the WebUI allows downloads only when one file is selected):
https://console.aws.amazon.com/s3
Is this something that can ...
- 9,420
176
votes
15
answers
292k
views
AWS CloudFront: Font from origin has been blocked from loading by Cross-Origin Resource Sharing policy
I'm receiving the following error on a couple of Chrome browsers but not all. Not sure entirely what the issue is at this point.
Font from origin https://ABCDEFG.cloudfront.net has been blocked from ...
- 4,082
175
votes
26
answers
179k
views
The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256
I get an error AWS::S3::Errors::InvalidRequest The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256. when I try upload file to S3 bucket in new Frankfurt region. ...
- 2,366
174
votes
19
answers
159k
views
How to upgrade AWS CLI to the latest version?
I recently noticed that I am running an old version of AWS CLI that is lacking some functionality I need:
$aws --version
aws-cli/1.2.9 Python/3.4.3 Linux/3.13.0-85-generic
How can I upgrade to the ...
- 8,264
172
votes
14
answers
230k
views
How to delete files recursively from an S3 bucket
I have the following folder structure in S3. Is there a way to recursively remove all files under a certain folder (say foo/bar1 or foo or foo/bar2/1 ..)
foo/bar1/1/..
foo/bar1/2/..
foo/bar1/3/..
...
- 25.8k
170
votes
11
answers
129k
views
Set cache-control for entire S3 bucket automatically (using bucket policies?)
I need to set cache-control headers for an entire s3 bucket, both existing and future files and was hoping to do it in a bucket policy.
I know I can edit the existing ones and I know how to specify ...
- 1,819
168
votes
13
answers
199k
views
Amazon S3 boto - how to create a folder?
How can I create a folder under a bucket using boto library for Amazon s3?
I followed the manual, and created the keys with permission, metadata etc, but no where in the boto's documentation it ...
- 4,330
167
votes
22
answers
279k
views
how to delete files from amazon s3 bucket?
I need to write code in python that will delete the required file from an Amazon s3 bucket. I am able to connect to the Amazon s3 bucket, and also to save files, but how can I delete a file?
- 2,899
165
votes
17
answers
424k
views
How to upload a file to directory in S3 bucket using boto
I want to copy a file in s3 bucket using python.
Ex : I have bucket name = test. And in the bucket, I have 2 folders name "dump" & "input". Now I want to copy a file from local directory to S3 "...
- 1,705
164
votes
30
answers
166k
views
S3 Error: The difference between the request time and the current time is too large
I have error The difference between the request time and the current time is too large when call method amazons3.ListObjects
ListObjectsRequest request = new ListObjectsRequest() {
BucketName = ...
- 8,165
164
votes
20
answers
96k
views
react router doesn't work in aws s3 bucket
I deployed my React website build/ folder into an AWS S3 bucket.
If I go to www.mywebsite.com, it works and if I click on a tag to go to Project and About pages, it leads me to the right page.
...
- 6,753
163
votes
11
answers
82k
views
Amazon S3 Change file download name
I have files stored on S3 with a GUID as the key name.
I am using a pre signed URL to download as per S3 REST API
I store the original file name in my own Database. When a user clicks to download a ...
- 19.5k
163
votes
8
answers
278k
views
FTP/SFTP access to an Amazon S3 Bucket [closed]
Is there a way to connect to an Amazon S3 bucket with FTP or SFTP rather than the built-in Amazon file transfer interface in the AWS console? Seems odd that this isn't a readily available option.
- 2,962
162
votes
14
answers
171k
views
Pipe a stream to s3.upload()
I'm currently making use of a node.js plugin called s3-upload-stream to stream very large files to Amazon S3. It uses the multipart API and for the most part it works very well.
However, this module ...
- 116k