`-put`  and `-copyFromLocal` are documented as identical, while most examples use the verbose variant -copyFromLocal. Why?

Same thing for `-get` and `-copyToLocal`

Difference between hadoop fs -put and hadoop fs -copyFromLocal

I&#39;m currently designing an architecture for a web-based application that should also provide some kind of image storage. Users will be able to upload photos as one of the key feature of the service. Also viewing these images will be one of the primary usages (via web). 

However, I&#39;m not sure how to realize such a scalable image storage component in my application. I already thought about different solutions but due to missing experiences, I look forward to hear your suggestions. Aside from the images, also meta data must besaved.
Here are my initial thoughts:

1. Use a (distributed) filesystem like HDFS and prepare dedicated webservers as &quot;filesystem clients&quot; in order to save uploaded images and service requests. Image meta data are saved in a additional database including the filepath information for each image. 

2. Use a BigTable-oriented system like HBase on top of HDFS and save images and meta data together. Again, webservers bridge image uploads and requests. 

3. Use a completly schemaless database like CouchDB for storing both images and metadata. Additionally, use the database itself for upload and delievery by using the HTTP-based RESTful API. (Additional question: CouchDB does save blobs via Base64. Can it however return data in form of image/jpeg etc.)?  


Scalable Image Storage

I was using Hadoop in a pseudo-distributed mode and everything was working fine. But then I had to restart my computer because of some reason. And now when I am trying to start Namenode and Datanode I can find only Datanode running. Could anyone tell me the possible reason of this problem? Or am I doing something wrong?

I tried both `bin/start-all.sh` and `bin/start-dfs.sh`.


Namenode not getting started

I am new to hadoop distributed file system, I have done complete installation of hadoop single node on my machine.but after that when i am going to upload data to hdfs it give an error message `Permission Denied`.

Message from terminal with command:

    hduser@ubuntu:/usr/local/hadoop$ hadoop fs -put /usr/local/input-data/ /input
    put: /usr/local/input-data (Permission denied)

    hduser@ubuntu:/usr/local/hadoop$ 


After using sudo and adding hduser to sudouser:

    hduser@ubuntu:/usr/local/hadoop$ sudo bin/hadoop fs -put /usr/local/input-data/ /inwe
    put: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode=&quot;&quot;:hduser:supergroup:rwxr-xr-x

    hduser@ubuntu:/usr/local/hadoop$ 

Permission denied at hdfs

I am trying to load a CSV file into a Hive table like so:

    CREATE TABLE mytable
    (
    num1 INT,
    text1 STRING,
    num2 INT,
    text2 STRING
    )
    ROW FORMAT DELIMITED FIELDS TERMINATED BY &quot;,&quot;;

    LOAD DATA LOCAL INPATH &#39;/data.csv&#39;
    OVERWRITE INTO TABLE mytable;    
&lt;br /&gt;
The csv is delimited by an comma (,) and looks like this:

    1, &quot;some text, with comma in it&quot;, 123, &quot;more text&quot;

This will return corrupt data since there is a &#39;,&#39; in the first string.&lt;br/&gt;
Is there a way to set an text delimiter or make Hive ignore the &#39;,&#39; in strings?

I can&#39;t change the delimiter of the csv since it gets pulled from an external source.

Hive load CSV with commas in quoted fields

I am running Spark on Windows 7. When I use Hive, I see the following error
    
    The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- 

The permissions are set as the following 

    C:\tmp&gt;ls -la
    total 20
    drwxr-xr-x    1 ADMIN Administ        0 Dec 10 13:06 .
    drwxr-xr-x    1 ADMIN Administ    28672 Dec 10 09:53 ..
    drwxr-xr-x    2 ADMIN Administ        0 Dec 10 12:22 hive

I have set &quot;full control&quot; to all users from Windows-&gt;properties-&gt;security-&gt;Advanced.

But I still see the same error. Any help please?
I have checked a bunch of links, some say this is a bug on Spark 1.5. Is this true?

Thanks
Aarthi

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- (on Windows)

The default data block size of HDFS/Hadoop is 64MB. The block size in the disk is generally 4KB.

What does 64MB block size mean? -&gt;Does it mean that the smallest unit of reading from disk is 64MB?

If yes, what is the advantage of doing that?-&gt; easy for continuous access of large files in HDFS?

Can we do the same by using the disk&#39;s original 4KB block size?

data block size in HDFS, why 64MB?

Characteristics of Apache Parquet are :

 - Self-describing 
 - Columnar format
 - Language-independent

In comparison to Avro, Sequence Files, RC File etc. I want an overview of the formats. I have already read : [How Impala Works with Hadoop File Formats](https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_file_formats.html) , it gives some insights on the formats but I would like to know how the access to data &amp; storage of data is done in each of these formats. How parquet has an advantage over the others?

What are the pros and cons of parquet format compared to other formats?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with `df.registerTempTable(&quot;table_name&quot;)` 

##I have tried:

    scala&gt; val df = sqlContext.load(&quot;hdfs:///csv/file/dir/file.csv&quot;)

##Error which I got:
    java.lang.RuntimeException: hdfs:///csv/file/dir/file.csv is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [49, 59, 54, 10]
    	at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:418)
    	at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:277)
    	at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:276)
    	at scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
    	at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:54)
    	at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
    	at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
    	at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:56)
    	at scala.collection.parallel.mutable.ParArray$Map.tryLeaf(ParArray.scala:650)
    	at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:165)
    	at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:514)
    	at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
    	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

What is the right command to load CSV file as DataFrame in Apache Spark?

Spark - load CSV file as DataFrame?

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through winscp .

How to copy file from HDFS to the local file system

This is kind of naive question but I am new to NoSQL paradigm and don&#39;t know much about it. So if somebody can help me clearly understand difference between the HBase and Hadoop or if give some pointers which might help me understand the difference.

Till now, I did some research and acc. to my understanding Hadoop provides framework to work with raw chunk of data(files) in HDFS and HBase is database engine above Hadoop, which basically works with structured data instead of raw data chunk. Hbase provides a logical layer over HDFS just as SQL does. Is it correct?




Difference between HBase and Hadoop/HDFS

In Map Reduce programming the reduce phase has shuffling, sorting and reduce as its sub-parts. Sorting is a costly affair.

*What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?*
  

What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?

    root# bin/hadoop fs -mkdir t
    mkdir: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user/root/t. Name node is in safe mode.

not able to create anything in hdfs

I did 

    root# bin/hadoop fs -safemode leave

But showing 

    safemode: Unknown command

what is the problem?

Solution: http://unmeshasreeveni.blogspot.com/2014/04/name-node-is-in-safe-mode-how-to-leave.html?m=1



Name node is in safe mode. Not able to leave

According to the `Hadoop - The Definitive Guide`

&gt; The logical records that FileInputFormats define do not usually fit neatly into HDFS blocks. For example, a TextInputFormat’s logical records are lines, which will cross HDFS boundaries more often than not. This has no bearing on the functioning of your program—lines are not missed or broken, for example—but it’s worth knowing about, as it does mean that data-local maps (that is, maps that are running on the same host as their input data) will perform some remote reads. The slight overhead this causes is not normally significant.

Suppose a record line is split across two blocks (b1 and b2). The mapper processing the first block (b1) will notice that the last line doesn&#39;t have a EOL separator and fetches the remaining of the line from the next block of data (b2).

How does the mapper processing the second block (b2) determine that the first record is incomplete and should process starting from the second record in the block (b2)?

How does Hadoop process records split across block boundaries?

Are they supposed to be equal?

but, why the &quot;`hadoop fs`&quot; commands show the `hdfs files` while the &quot;`hdfs dfs`&quot; commands show the local files?

here is the hadoop version information:

&gt; Hadoop 2.0.0-mr1-cdh4.2.1 Subversion
&gt; git://ubuntu-slave07.jenkins.cloudera.com/var/lib/jenkins/workspace/CDH4.2.1-Packaging-MR1/build/cdh4/mr1/2.0.0-mr1-cdh4.2.1/source
&gt; -r  Compiled by jenkins on Mon Apr 22 10:48:26 PDT 2013

what&#39;s the difference between &quot;hadoop fs&quot; shell commands and &quot;hdfs dfs&quot; shell commands?

I know `du -sh` in common Linux filesystems. But how to do that with HDFS?

The way to check a HDFS directory&#39;s size?

I need to copy a folder from local file system to HDFS. I could not find any example of moving a folder(including its all subfolders) to HDFS

`$ hadoop fs -copyFromLocal /home/ubuntu/Source-Folder-To-Copy HDFS-URI`




hadoop copy a local file system folder to HDFS

I&#39;d like to know how to find the mapping between Hive tables and the actual HDFS files (or rather, directories) that they represent. I need to access the table files directly.

Where does Hive store its files in HDFS?

Where does Hive store files in HDFS?

When I connected my Hadoop cluster to Amazon storage and downloaded files to HDFS, I found `s3://` did not work. When looking for some help on the Internet I found I can use `S3n`. When I used `S3n` it worked. I do not understand the differences between using `S3` and `S3n` with my Hadoop cluster, can someone explain?

Differences between Amazon S3 and S3n in Hadoop

How can you write to multiple outputs dependent on the key using Spark in a single Job.

Related: https://stackoverflow.com/questions/23994383/write-to-multiple-outputs-by-key-scalding-hadoop-one-mapreduce-job/

E.g.

    sc.makeRDD(Seq((1, &quot;a&quot;), (1, &quot;b&quot;), (2, &quot;c&quot;)))
    .writeAsMultiple(prefix, compressionCodecOption)

would ensure `cat prefix/1` is

    a
    b

and `cat prefix/2` would be

    c


EDIT: I&#39;ve recently added a new answer that includes full imports, pimp and compression codec, see https://stackoverflow.com/a/46118044/1586965, which may be helpful in addition to the earlier answers.


Write to multiple outputs by key Spark - one Spark job

A fast method for inspecting files on HDFS is to use [tail](http://hadoop.apache.org/docs/r0.19.0/hdfs_shell.html#tail):

    ~$ hadoop fs -tail /path/to/file

This displays the last kilobyte of data in the file, which is extremely helpful. However, the opposite command `head` does not appear to be part of the shell command collections. I find this very surprising. 

My hypothesis is that since HDFS is built for very fast streaming reads on very large files, there is some access-oriented issue that affects `head`. This makes me hesitant to do things to access the head. Does anyone have an answer?

Why is there no &#39;hadoop fs -head&#39; shell command?

Is there a hdfs command to see available free space in hdfs. We can see that through browser at master:hdfsport in browser , but for some reason I can&#39;t access this and I need some command.
I can see my disk usage through command ./bin/hadoop fs -du -h but cannot see free space available.

Thanks for answer in advance. 

HDFS free space available command

How does someone fix a HDFS that&#39;s corrupt?  I looked on the Apache/Hadoop website and it said its [`fsck` command](http://hadoop.apache.org/docs/r0.19.0/hdfs_user_guide.html#fsck), which doesn&#39;t fix it. Hopefully someone who has run into this problem before can tell me how to fix this.


&gt; Unlike a traditional fsck utility for native file systems, this command does not correct the errors it detects. Normally NameNode automatically corrects most of the recoverable failures.

When I ran `bin/hadoop fsck / -delete`, it listed the files that were corrupt or missing blocks.  How do I make it not corrupt?  This is on a practice machine so I COULD blow everything away but when we go live, I won&#39;t be able to &quot;fix&quot; it by blowing everything away so I&#39;m trying to figure it out now.

How to fix corrupt HDFS FIles

I&#39;m working with Apache Spark on a Cluster using HDFS. As far as I understand, HDFS is distributing files on data-nodes. So if a put a &quot;file.txt&quot; on the filesystem, it will be split into partitions.
Now I&#39;m calling 

    rdd = SparkContext().textFile(&quot;hdfs://.../file.txt&quot;) 

from Apache Spark. 
Has rdd now automatically the same partitions as &quot;file.txt&quot; on the filesystem? 
What happens when I call 

    rdd.repartition(x)

where x &gt; then the partitions used by hdfs? Will Spark physically rearrange the data on hdfs to work locally?

Example:
I put a 30GB Textfile on the HDFS-System, which is distributing it on 10 nodes.
Will Spark
a) use the same 10 partitons? and b) shuffle 30GB across the cluster when I call repartition(1000)?


How does Spark partition(ing) work on files in HDFS?

I want to create a file in HDFS and write data in that. I used this code:
    
    Configuration config = new Configuration();     
    FileSystem fs = FileSystem.get(config);	
    Path filenamePath = new Path(&quot;input.txt&quot;);	
	try {
        if (fs.exists(filenamePath)) {
            fs.delete(filenamePath, true);
		}
	 
        FSDataOutputStream fin = fs.create(filenamePath);
        fin.writeUTF(&quot;hello&quot;);
        fin.close();
    }

It creates the file, but it does&#39;t write anything in it. I searched a lot but 
didn&#39;t find anything. What is my problem? Do I need any permission to write in HDFS?

Thanks.	

All Hdfs Solutions on Gang of Coders

Total of 25 Hdfs Solutions