TOP 10 HADOOP BIG DATA QUESTIONS AND ANSWERS

Gartner predicts over 50 percent of new business systems will be use continuous intelligence by 2022. Leading companies such as Morgan Stanley, Standard Chartered, J-P Morgan, etc, are calling upon aspiring candidates to apply for positions like Analytics Professional, Data Management Professionals and Big Data Engineer. This brings a bright future to those skilled in Big data. Yet acing the interview where high-level questions are put forth remains a big hurdle.

Take help from these top 10 Big Data Interview Questions highly asked in job interviews.

1) How important is Big data to e-commerce?

E-commerce is one of the biggest beneficiaries of Big data processing and analytics. A lot of critical information is gathered from social media sites and search engines that ecommerce companies use to predict better and offer a more effective customer experience. Predictive analysis plays an important role in retaining customers longer on the websites and this is made smoother with Big data. It also helps to fine-tune customer interactions through better personalization. Big data has proven to reduce the cart abandonment rate through prediction and personalization.

2) Describe Operational Big Data.

Operational Big data involves real-time data that is instantly analyzed and presented without the help of advanced coding or data scientists. These include interactive systems where data is captured, processed and reported instantly without involving data scientists. NoSQL Big data systems use advanced cloud computing techniques that run complex computations involving such real-time data without having to invest in additional infrastructure. They can manage large volumes of varied data and process them quickly using easy-to-implement technology.

3) Explain the HDFS Architecture.

The HDFS architecture consists of nodes and blocks. The Namenode comprises commodity hardware that manages that operating system which is typically Linux and the namenode software. It manages the file system providing access to the file system namespace and data. It allows opening, closing and renaming of files and directories. The datanode consists of hardware, an operating system, and datanode software. The datanode performs the reading and writing of data as per the client’s request. The datanode manages the data blocks including creation, deletion, and replication based on the directions of the namenode. The data is stored in files and each file is divided into different segments and stored in different data nodes. Block is the smallest section of data that’s stored in the HDFS.

4) Explain ls, lsr, du and cat commands in HDFS.

The ls, lsr, du and cat commands are used in HDFS to access the file system.

• ls: This HDFS command is used along with the path to list out the contents of the directory specified in the path. It shows the file names, owner, permissions and modification date for each file in the directory.
• lsr: This command is used to display all files within the subdirectories in the mentioned path along with the filename, owner, permissions and modification date for each file in each subdirectory.
• du: This command takes in the path and displays the full HDFS prefixed file names along with the space occupied by each file in the given path.
• cat: This command takes in a filename and displays the content of the file on the console screen.

5) What is the difference between Hadoop and RDBMS?

Hadoop is not a database. Basically, it is a distributed system that lets you store the data in large amounts on cloud machines. An RDBMS is a distributed database system that stores the interrelated data and its dependencies. RDBMS uses relational data and stores it in rows and columns. Hadoop provides various ways to span the data across various mediums and reach out to the data. The storage is spread across multiple local servers. The Hadoop has efficient fault-tolerance, to detect and manage the defect in nodes. As Java is used in Hadoop, it is platform-independent. Hadoop has high scalability as compared to RDBMS.

6) Explain Repartition Join in MapReduce.

In Repartition Join, each map works on an input split on the table on either the Left or Right of the join. Each record is tagged with the table to identify. So the key becomes the key and value is the record. These key-value pairs are then partitioned, sorted, and combined. The reducer takes in the key-value pairs and the table tags. For every key in the Right table, all matching records in the Left tables will be fetched.

7) Explain what Flatten does in Pig.

Flatten is an operator or modifier that un-nests the tuples and bags in Pig. It is like simplifying the data of unnecessary complications, making data plain. When Flatten is used with bags, the data is simplified into tuples. When tuples are flattened, they are turned into plain data. It works like a ForEach loop where data is cross-joined with each element to relate it to the main key.

8) Does Impala use Caching?

Yes Impala uses caching to provide quicker results. In fact, Impala works better with HDFS rather than Hadoop. It caches some of the frequently accessed data so that the database or HDFS does not have to be accessed every time data is requested. You can set up the cache pool and cache limit for the same user as the ImpalaD. Once caching is enabled, for every schema object you have created with the said cache pool will be available in the cache so that it is loaded only once. Thus all frequently accessed tables and partitions can be stored in the cache for quicker access.

9) What are the attributes of AVRO Schemas?

AVRO schemas contain four attributes – type of the schema, its namespace, schema name, and the fields in it. The type of the schema determines whether it is a record, map, array or any other primitive or complex data type supported by AVRO. The namespace is where we can find the schema object. Name is the identifier of the schema and the field is an array that contains the name and datatype of the fields used in the schema.

10) Explain the Fork and Join control node in Workflow.

A Fork is used when there’s parallel processing required. In the end, the fork consolidates the results that are fed into another job. This consolidation is done by the Join. So every fork ends with a join. After the start node, the forks run parallel to each other and process the jobs that are consolidated by a join. The join passes on the data to the next node only when all nodes connected complete their tasks.

Pave the route to your dream job by preparing for your interview with the questions we discussed, along with more questions from our books HR Interview Questions You’ll Most Likely Be Asked (Third Edition) and Leadership Interview Questions You’ll Most Likely Be Asked. These provide a comprehensive list of questions for your interview regardless of your area of expertise.

Good luck with your job hunt!