Questions tagged [amazon-s3]

Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services.

0
votes
1answer
4 views

Reset hadoop aws keys to upload to another s3 bucket under different username

Sorry for horrible question title but here is my scenario I have a pyspark databricks notebook in which I am loading other notebooks. One of this notebooks is setting some redshift configuration for ...
0
votes
0answers
5 views

Signature calculation fails for s3 bucket upload using curl.

I am struggling to upload object on s3 bucket using curl and openssl(which I have to use as required). Please find below my script for upload. #!/bin/bash -e # Upload a file to AWS S3. ...
0
votes
0answers
10 views

How to upload an image file to AWS S3, after changing the orientation using Javascript?

I'm using the library Direct Upload to S3 - Signature V4 to upload photos. However, I have a requirement to find and correct the orientation of photos, before these files are uploaded to S3. I have ...
0
votes
0answers
12 views

Upload File as JSON to S3 using Java SDK causes error [duplicate]

I'm trying to upload files A string to S3 in JSON format. The uploads work fine, but they upload as text. When I download them they have the extension .txt... Here is that code: try { AmazonS3 ...
0
votes
0answers
9 views

Reading S3 file content fails using Chalice Framework AWS

I am trying to read contents of S3 file that I wrote to earlier. When I read from the local boto3 script I can see the content of the file but when I run the code on Lambda using Chalice I get "...
0
votes
1answer
12 views

how to set text/plain content-type in api gateway

I have an api gateway that is pointed to s3 and it reads the document and sends back the contents, about 5 paragraphs. It works great, however the content-type on the response is binary\octet-stream. ...
0
votes
0answers
6 views

aws glue s3 target - creating partitions of 20 files

By default when I read two three records also from dynamo DB source and target is s3, even then its creating 20 partition files in target and only two are having data and other are empty files. How ...
-1
votes
1answer
36 views

AWS S3 File Download from the client-side

I am currently trying to download the file from the s3 bucket using a button from the front-end. How is it possible to do this? I don't have any idea on how to start this thing. I have tried ...
0
votes
0answers
13 views

Storing Application State in MongoDB vs S3 as JSON

I am making a meteor app using Mobx-State-tree , graphql and Mongo DB. In the app, there is a section called courses which has chapters which in turn has topics. The Course Section state is ...
0
votes
1answer
18 views

How can I access data in s3 bucket from one account to process the data using redshift in another account?

There is huge data in one of my AWS account in S3 Bucket and I want to process the data in some another AWS account using Redshift, I want to save the cost of data transfer and storage since I already ...
-1
votes
0answers
7 views

How to move data to HBase from s3 parquet format

I have parquet files in s3 (Rows : ~600mn ,columns: 10[Mix of Array and Strings]), to make this queryable I want to move this to HBase Tables. I am using Amazon AWS. TIA.
0
votes
0answers
18 views

Unload a table from redshift to S3 in parquet format without python script

I Found that we can use spectrify python module to convert a parquet format but i want to know which command will unload a table to S3 location in parquet format. one more thing i found that we can ...
0
votes
0answers
9 views

S3 Bucket Policy for Amplify Front End

I've got a web front end with AWS-Amplify handling authentication through Cognito. Now I'm trying to set up an S3 bucket to let users upload media files to, which works alright when I don't bother ...
0
votes
1answer
16 views

post json directly to amazon s3 bucket from ionic/angular front end

I am struggling to figure out how, or if possible, I can post/put data into an aws s3 bucket from an angular/ionic application... I have a mobile application that has two surveys, one for before ...
0
votes
1answer
42 views

How to read CSV file from S3

I have the following code: package main import ( "encoding/csv" "fmt" "io/ioutil" "path" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" "...
1
vote
0answers
24 views

AWS Lambda access local resource or storage C#

How can I store and access a file in Lambda using C# I used the tmp folder that is available for the lambda but I get an error of could not load file or assembly. How can I solve the error? I used the ...
-1
votes
0answers
17 views

AWS s3 Uploading files on localhost, but not working in the real URL

function setUploadAWS($ORIGEM, $DESTINO, $DIR) { $BUCKET = 'some-bucket'; $ACCESS_KEY = "some-access-key"; $SECRET_KEY = "some-secret-key"; $credentials = new Aws\Credentials\...
0
votes
0answers
10 views

Start AWS S3 Multipart Upload Java Before Whole File Complete

The problem that I am having is that I am generating a large JSON Object to upload to S3 which is causing memory issues. When I attempt to translate the JSON object to a string so that I can get ...
1
vote
1answer
19 views

Incremenrtal load s3 folder files

What is easy way to apply the incremental load in s3 folder files using python? Date is taken from the "filename_180828_152153" like this! I have tried insert the all filenames and dates to the table....
0
votes
0answers
7 views

Data Masking on huge CSV files stored in AWS S3

I have huge csv files of size ~15GB in aws s3 (s3://bucket1/rawFile.csv). Lets say if the schema looks like this: cust_id, account_num, paid_date, cust_f_name 1001, 1234567890, 01/01/2001, Jonathan ...
0
votes
1answer
10 views

Cannot access S3 bucket from WildFly running in Docker

I am trying to configure WildFly using the docker image jboss/wildfly:10.1.0.Final to run in domain mode. I am using docker for macos 8.06.1-ce using aufs storage. I followed the instructions in this ...
0
votes
1answer
9 views

How can I have Snakemake confirm remote files without using {output}

Thanks for continuing to put so much effort into Snakemake! snakemake --no-shared-fs --default-remote-provider S3 --default-remote-prefix mybucket hellos3 So this does not work for me(...
0
votes
1answer
18 views

Can I alias an external table in redshift to remove the schema name?

We want to migrate tables to Spectrum, which requires defining an external schema create external schema spectrum from data catalog database 'spectrumdb' iam_role 'my_iam_role' create external ...
0
votes
1answer
24 views

AccessDenied when calling the CreateMultipartUpload operation in Django using django-storages and boto3

I want to use django-storages to store my model files in Amazon S3 but I get Access Denied error. I have granted the user almost all S3 permission PutObject, ListBucketMultipartUploads, ...
1
vote
0answers
14 views

Upload file S3 bucket using PHP [on hold]

I'm new to AWS and I'm trying to upload video and image files to an s3 bucket using PHP. I've seen several ways of doing this. I just downloaded the AWS PHP SDK with Composer but I have a few ...
0
votes
0answers
3 views

Using tags prefixed with aws: in IAM policies

I have a use case where I'm creating a set of resources such as EC2 instances and S3 buckets using CloudFormation. I see that Cloudformation adds tags to both EC2 instances and S3 buckets with the tag ...
0
votes
0answers
14 views

Unable to connect with S3 and Spark Locally

Below is my code : I am trying to access s3 files from spark locally. But getting error : Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: s3n://...
2
votes
0answers
23 views

Django celery and channels example

I have a Django app and need to generate files that can take up to a minute, so I pass that off to a background worker. Currently, the process works as follows. I post to the server that replies with ...
0
votes
1answer
8 views

How to setup email notifications for AWS operational issues

Yesterday our infrastructure started throwing lots of connection errors. We stated debugging and the more we looked, the more perplexing the issue appeared to be; until someone noticed the bell icon (...
0
votes
0answers
5 views

Is it possible to copy data from from Amazon S3 objects in Requester Pays Buckets using Azure Data Factory V2?

In Azure Data Factory exist connection to Amason S3 where I can set AccessKey and SecretKey. After I create Linked Service and try to use dataset in copy activity I get 403 error when I try to get ...
0
votes
0answers
2 views

Is it possible to take S3 snapshots of OpenEBS Jiva live volumes? If so, how?

How can I take S3 snapshot of OpenEBS jiva volume which is being accessed regularly? Is there any pause on volume while taking the snapshot? Do you have a way to do this one?
0
votes
0answers
17 views

Image is getting corrupted after uploading it from lambda to S3 using serverless framework and connect-multiparty module

When i am uploading image from local its working fine i am able to view the image from browser but when i upload it from lambda using serverless framework the image is displayed as empty white square ...
0
votes
1answer
11 views

Django: Issue with a Video file uploader to S3 bucket using Boto3

I want to upload a video file to AWS S3 bucket using Boto3. I've already created a bucket named 'django-test' and given the required permissions. I am using Django and working on Windows 10 machine. ...
1
vote
0answers
17 views

How to fetch array of images from amazon s3 ? NodeJS

This code snippet generates image url of AWS s3 by giving Image name as input.I have an array of images names to generate so now I am looping this code.But it is taking too much time to generate url, ...
0
votes
0answers
15 views

No 'Access-Control-Allow-Origin' header is presented on the requested resource on aws s3

I tried all the available options in google to configure the CORS in the bucket property of s3. but still when the website loads it does not load completely and inspecting the page throws 'Failed to ...
1
vote
1answer
13 views

Why is this bucket policy being shown as public

I am relatively new to AWS. So recently, we creates an AWS S3 bucket through the AWS console. Then we set the following policy on the bucket: { "Version": "2008-10-17", "Statement": [ ...
0
votes
1answer
9 views

Delete old files based on date extracted from filename by custom rules

I am trying to find best way to automatically delete my old SQL backup files on s3 based on following rules: keep all backups for last 7 day keep last backup of each day for last 6 month keep last ...
0
votes
0answers
12 views

No such file or directory found on amazon ec2 server python

This code is running in background service and working perfectly on localhost and uploaded the backup file successfully on Amazon S3 server but when this code deployed on ec2 amazon server its give ...
0
votes
0answers
11 views

Laravel 5 how to add prefix to S3 file storage config?

In the config/filesytems.php, the default settings is like this 's3' => [ 'driver' => 's3', 'key' => env('AWS_ACCESS_KEY_ID'), 'secret' => env('AWS_SECRET_ACCESS_KEY'), '...
0
votes
0answers
7 views

How would I define the aws cognito federated Identity policy for Unauthenticated users to access my s3 bucket?

How do I scope down the AWS IdentityPoolId access(through policy document) such that no Unauth requests are fulfilled for any other site other than mine(intended). I am using the Identity Pool Id ...
0
votes
0answers
14 views

s3 _$folder$ object causing issues in “MSCK REPAIR TABLE” on EMR 5.11.2 and above

EMR will produce _$folder$ object for each folder stored in S3, it's inevitable (Avoid creation of _$folder$ keys in S3 with hadoop (EMR)) . For example, under "day=09" directory, EMR stores the ...
0
votes
1answer
22 views

AWS S3 : Is there any way to count rows of uploaded file?

We'd like to check the row count of uploaded files on AWS S3, if the uploading task is sure to be end correctly. Now we're simply sending exported files by Windows AWS CLI command as follows, after ...
0
votes
0answers
18 views

Upload to AWS only if query is successful

How would one go about interacting with a 3rd party API while querying the database in the same request? For example, I'm using Multer/Multer-S3 to upload files to AWS, then upon successful upload I'...
0
votes
1answer
17 views

AWS operation prevents other operations

Using the management console I am currently changing the storage class for some folders in one bucket (Standard-IA) and while that operation is happening I am unable to cut and paste in another bucket....
0
votes
1answer
19 views

file created in s3 bucket but file is empty

while uploading a file from the browser. filename is created in the s3 bucket but the content of the file is empty. @app.route("/", methods=['GET', 'POST']) def index(): if request....
0
votes
0answers
9 views

Is there code sample available to read parquet file from S3 bucket by using IAM role (default client)

I have a java code that reads Parquet file. When I modidfy Path of that code as shown below to read S3 file, it is not working The statement s3:\\file1.parquet works only if I provide access key ...
0
votes
0answers
26 views

Where does SparkSession get AWS credentials from? SparkSession or HadoopConfiguration? [duplicate]

I am trying to read data from was s3 into a spark dataset in Java, code running on IntelliJ. I have added AWS keys in Spark Session config as mentioned in the code below but I am still getting the ...
0
votes
1answer
19 views

AWS S3 returning differing “Content-Range” compared to “Range” request header specification

We send the "Range" Header with our HTTP Get Requests to S3 for resuming failed downloads of our ~3GB application that we host there. A strange behavior has occurred. While at "lower" completion ...
0
votes
1answer
16 views

Access denied S3 bucket behind CloudFront using alternate domain name and custom SSL

Currently we are serving files from S3 using presigned urls, however, some places block S3 urls, so we are attempting to redirect this to a custom URL. We have set up a CloudFront distribution to ...
0
votes
1answer
15 views

S3 Cloud Custodian

I am writing a cloud custodian policy to find the s3 buckets which are not encrypted on creation- send an email to slack for 5 days- then encrypt on the 6th day. I have figured out the component on ...