HDFS Space quota

Get space quota:

hadoop fs -count -q -h -v <<hdfs://...>>

Set Space quota:

hadoop fs -setSpaceQuota <<quota in bytes>> <<hdfs://...>>

Else goto Cloudera Manager->HDFS->File Browser->select the dir->Edit Space quota

Advertisements

HDFS copy/move commands

  • Copying files from HDFS to Local file system:
    • hadoop fs -get <source hdfs location> <destination local location>
  • Copying files from Local file system to HDFS:
    • hadoop fs -put <source local location> <destination hdfs location>
  • Copying from HDFS to S3 and Vice versa:
    • hadoop distcp -Dfs.s3a.awsAccessKeyId=<<access_key>> -Dfs.s3a.server-side-encryption-algorithm=AES256 -Dmapreduce.map.memory.mb=6192  -Dmapred.job.queue.name=<<queue>> -Dmapred.reduce.tasks=20 -overwrite <<hdfs://>> <<s3a://>>
    • (access_key, encryption, memory, queue, tasts, overwrite are optional)
  • Copying inside HDFS locations:
    • hadoop fs -cp <source hdfs loc> <dest hdfs loc>
  • Moving inside HDFS locations:
    • hadoop fs -mv <source hdfs loc> <dest hdfs loc>