What does “Stage Skipped” mean in Apache Spark web UI?
August 9, 2021Why is Namenode stuck in safe mode and how to fix it?
August 13, 2021Downloading an entire bucket or folder inside a bucket is quite straightforward with AWS CLI. Install AWS CLI from here if you don’t have it already.
Copy an entire folder from S3
Use the cp command to download a folder inside a bucket from S3 to local. recursive option will download all files and folders if you have a recursive folder/file structure.
aws s3 cp s3://BUCKETNAME/PATH/TO/FOLDER /home/hirw/s3-folder --recursive
Copy an entire bucket from S3
Use the sync command to download an entire bucket from s3 to local. If /home/hirw/s3-bucket is empty then all files from s3 bucket will be copied to s3-bucket directory. If you have same files in /home/hirw/s3-bucket directory then the contents in the directory will be overwritten with the contents from s3://BUCKETNAME
aws s3 sync s3://BUCKETNAME /home/hirw/s3-bucket
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.
