Big Data + Artificial Intelligence(Series 1)

HOW BIG DATA IS SOLVING AI’S PROBLEM?

We generate an enormous amount of data daily by searching on websites, watching online videos etc. These data are needed to be handled for better performance in the future. Lots of websites, apps, self-driving cars, robots are made to implement them we need Learning Algorithms. To Train Learning Algorithms, We Need Data. Web Provides a huge amount of data, but these are unstructured data. BIG DATA extracted from the web, are used to train the algorithms. These Data are highly useful and with increasing amount of data, we need a way to utilize this amount of data in a smart way. We can see some good results and improvements in sentiment analysis and stock market prediction.

Do you know what is BIG DATA?

Today’s ultra-connected world generate massive volumes of data at ever-accelerating rates. As a result, big data analytics has become a powerful tool for businesses looking to leverage mountains of valuable data for profit and competitive advantage. The technologies we used earlier to process data are insufficient to process large datasets. The functionality that Relational Database Management System provides cannot fulfill the requirements of today’s World. It can only process Structured and non-complex data. It is very difficult to process data of Companies whose workloads are unpredictable and varies with time. It is also a slower process when it comes to processing Bigdata. To eliminate these problems Hadoop came into existence.

What is Hadoop?

Hadoop is an open source project that offers a new way to store and process big data. The software framework is written in Java for distributed storage and distributed processing of very large datasets on computer system built on commodity hardware. Large Web companies such as Google and Facebook use Hadoop to store and manage their huge data sets, Hadoop has also proven valuable for many other more traditional enterprise based on its five big advantages. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs.

There are two concepts available for Hadoop:

1.Hadoop 64mb by default

2.Hadoop 128mb by default

Hadoop is an ecosystem stored in cluster form based on master and slave Architecture are*Master-named node and *Cluster data node.

Advantages of using Hadoop: -

 1)Scalability

It is a highly scalable platform because it can store and process a very large data sets across hundreds of inexpensive servers that operate in parallel. Unlike a traditional relational database.Systems (RDBMS) that can’t scale to process large amounts of data, Hadoop enables businesses to run applications of nodes involving thousands of terabyte of data.

2)Cost-effective

Hadoop also offers a cost-effective storage solution for businesses exploding data sets. The problem with traditional relational database management systems is that it is extremely cost prohibitive to scale to such a degree in order to process such massive volumes of data. In an effort to reduce costs, many companies in the past would have had to down-sample data and classify it based on certain assumptions as to which data was the most valuable. The raw data would be deleted, as it would be too cost-prohibitive to keep. While this approach may have worked in the short term, this meant that when business priorities changed, the complete raw data set was not available, s it was too expensive to store. Hadoop, on the other hand, is designed as a scale-out architecture that can affordably store all of a company’s data for later use.

3)Flexible

 Hadoop enables businesses to easily access new data sources and tap into different types of data (both structured and unstructured) to generate value from that data. This means businesses can use Hadoop to derive valuable business insights from data sources such as social media, email conversations or clickstream data. In addition, Hadoop can be used for a wide variety of purposes, such as log processing, recommendation systems, data warehousing, market campaign analysis and fraud detection.

4)Fast

Hadoop's unique storage method is based on a distributed file system that basically 'maps' data wherever it is located on a cluster. The tools for data processing are often on the same servers where the data is located, resulting in much faster data processing. If you're dealing with large volumes of unstructured data, Hadoop is able to efficiently process terabytes of data in just minutes, and petabytes in hours.

5)Resilient to Failure

 A key advantage of using Hadoop is its fault tolerance. When data is sent to an individual node, that data is also replicated to other nodes in the cluster, which means that in the event of failure, there is another copy available for use.The Mar distribution goes beyond that by eliminating the Name Node and replacing it with a distributed No Name Node architecture that provides true high availability. Our architecture provides protection from both single and multiple failures.When it comes to handling large data sets in a safe and cost-effective manner, Hadoop has the advantage over relational database management systems, and its value for any size business will continue to increase as unstructured data continues to grow.

Do you know some challenges that occur while using Hadoop?

• MapReduce programming is not a good match for all problems. It’s good for simple information requests and problems that can be divided into independent units, but it's not efficient for iterative and interactive analytic tasks. MapReduce is file-intensive. Because the nodes don’t intercommunicate except through sorts and shuffles, iterative algorithms require multiple map-shuffle/sort-reduce phases to complete. This creates multiple files between MapReduce phases and is inefficient for advanced analytic computing.
• There’s a widely acknowledged talent gap. It can be difficult to find entry-level programmers who have sufficient Java skills to be productive with MapReduce. That's one reason distribution providers are racing to put relational (SQL) technology on top of Hadoop. It is much easier to find programmers with SQL skills than MapReduce skills. And, Hadoop administration seems part art and part science, requiring low-level knowledge of operating systems, hardware and Hadoop kernel setting.
• Data security. Another challenge centers around the fragmented data security issues, though new tools and technologies are surfacing. The Kerberos authentication protocol is a great step toward making Hadoop environments secure.
• Full-fledged data management and governance. Hadoop does not have easy-to-use, full-featured tools for data management, data cleansing, governance, and metadata. Especially lacking are tools for data quality and standardization.

Fun Fact:- “Hadoop” was the name of a yellow toy elephant owned by the son of one of its investors.

This article is written by 

Akshay Tyagi

and managed by:

For more such blogs and updates follow my facebook link:

https://www.facebook.com/societyofai/