HDFS copy/move commands

  • Copying files from HDFS to Local file system:
    • hadoop fs -get <source hdfs location> <destination local location>
  • Copying files from Local file system to HDFS:
    • hadoop fs -put <source local location> <destination hdfs location>
  • Copying from HDFS to S3 and Vice versa:
    • hadoop distcp -Dfs.s3a.awsAccessKeyId=<<access_key>> -Dfs.s3a.server-side-encryption-algorithm=AES256 -Dmapreduce.map.memory.mb=6192  -Dmapred.job.queue.name=<<queue>> -Dmapred.reduce.tasks=20 -overwrite <<hdfs://>> <<s3a://>>
    • (access_key, encryption, memory, queue, tasts, overwrite are optional)
  • Copying inside HDFS locations:
    • hadoop fs -cp <source hdfs loc> <dest hdfs loc>
  • Moving inside HDFS locations:
    • hadoop fs -mv <source hdfs loc> <dest hdfs loc>

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s