Short Introduction To Cassandra

Cassandra is a distributed database from Apache that is highly scalable and designed to manage very large amounts of structured data. It provides high availability with no single point of failure.

Apache Cassandra is a highly scalable, high-performance distributed database designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. It is a type of NoSQL database. Let us first understand what a NoSQL database does.

Listed below are some of the notable points of Apache Cassandra:

• It is scalable, fault-tolerant, and consistent.

• Cassandra is being used by some of the biggest companies such as Facebook, Twitter, Cisco, Rackspace, ebay, Twitter, Netflix, and more.

• It is a column-oriented database.

• Cassandra implements a Dynamo-style replication model with no single point of failure, but adds a more powerful “column family” data model.

• Its distribution design is based on Amazon’s Dynamo and its data model on Google’s Bigtable.

• Created at Facebook, it differs sharply from relational database management systems.

Cassandra Commands and Simple Example

To start Cassandra afresh

sudo rm -rf /var/log/cassandra/*
sudo rm -rf /var/lib/cassandra/*

Starting the server

cd $CASSANDRA_HOME
bin/cassandra -f

Starting the CLI

bin/cassandra-cli
quit;

Describing the Environment

show cluster name;
show keyspaces;
show api version;

DML Operations

A keyspace is similar to the database, a column familiy similar to a table and columns are similar to columns in RDBMS.

create keyspace Twissandra;
use Twissandra;
create column family User with comparator = UTF8Type;
update column family User with
column_metadata =
[
{column_name: first, validation_class: UTF8Type},
{column_name: last, validation_class: UTF8Type},
{column_name: age, validation_class: UTF8Type, index_type: KEYS}
];

DDL Operations

To insert data

assume User keys as utf8;
set User['jsmith']['first'] = 'John';
set User['jsmith']['last'] = 'Smith';
set User['jsmith']['age'] = '38';

To Get Data

get User['jsmith'];

To Query Data

get User where age = '38';

Introduction to MongoDB

MongoDB is an open-source document database, and leading NoSQL database. MongoDB is written in c++

This tutorial will give you great understanding on MongoDB concepts needed to create and deploy a highly scalable and performance oriented database.

Installing Mongo

http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update
sudo apt-get install -y mongodb-org

Remove the logs and the data if something got corrupted

sudo rm -rf /var/lib/mongodb/*
sudo rm -rf /var/log/mongodb/*

Start Mongo

sudo mongod --config /etc/mongod.conf --smallfiles
Verify that the mongod process has started successfully by checking the contents of the log file at /var/log/mongodb/mongod.log.

Some references to Mongo documentation

http://www.mongodb.com/architecture
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
http://docs.mongodb.org/manual/tutorial/getting-started/

Start the mongo shell

mongo

Different Mongo commands

show dbs
use mydb
j = { name : "mongo" }
k = { x : 3 }
db.testData2.insert( j )
db.testData2.insert( k )
show collections
db.testData2.find()
db.testData2.find( { x : 3 } )
db.testData.findOne()
db.testData.find().limit(3)
db.users.insert (

{

name: "praveen",
age: 30,
gender: "M",
groups: [ "news", "computer" ]

}

)

Introduction to Neo4J

Neo4J is a graph database, a database aimed at storing and querying graphs rather than tables. Many things in modern life are 'graphy': social network data is a well-known example, where the emphasis lies more on who is connected with who and what than on the data itself, like your name and address.

I recently migrated my spreadsheet analysis tool from SQL Server to Neo4J. It was 

1) healthy for my brain to work with a new type of database, 

2) fun, and 

3) queries turned out to be a lot more intuitive and efficient.

Now Take Simple Example of Neo4j

Starting and stopping the Neo service

service neo4j-service status
service neo4j-service start
service neo4j-service stop

Neo UI

http://localhost:7474/webadmin/

Starting the shell

./bin/neo4j-shell

creting a node

CREATE (ee { name: "Emil", from: "Sweden" }) RETURN ee.name;
CREATE clause to create a node
() to indicate a node
{} to add properties to a node
ee a variable to the new node

Retrieve the node

START ee=node(*) WHERE ee.name! = "Emil" RETURN ee;
START clause to being a query
ee=node(*) to search through all the nodes

Create many nodes and relations at once

CREATE (ee { name: "Emil", from: "Sweden", klout: 99 }),
(js { name: "Johan", from: "Sweden", learn: "surfing" }),
(ir { name: "Ian", from: "England", title: "author" }),
(rvb { name: "Rik", from: "Belgium", pet: "Orval" }),
(ally { name: "Allison", from: "California", hobby: "surfing" }),
(ee)-[:KNOWS {since: 2001}]->(js),(ee)-[:KNOWS {rating: 5}]->(ir),
(js)-[:KNOWS]->(ir),(js)-[:KNOWS]->(rvb),
(ir)-[:KNOWS]->(js),(ir)-[:KNOWS]->(ally),
(rvb)-[:KNOWS]->(ally);

To find Emils friend

START ee=node(*) MATCH (ee)-[:KNOWS]->(friends)
WHERE ee.name! = "Emil" RETURN friends;
MATCH clause to describe the pattern from Known nodes to Found nodes

Pattern matching can be used to make recommendations. Johan is learning to surf, so he may want to find a new friend who already does.

START js=node(*)
MATCH (js)-[:KNOWS]->()-[:KNOWS]->(surfer)
WHERE js.name! = "Johan" AND surfer.hobby! = "surfing"
RETURN DISTINCT surfer;
() is for ignoring the nodes

Hadoop Certification: Is Hortonworks or Cloudera Better?

CCDH is one of the best options that can help you to excel in your career. It’s my personal experience that professionals with this certification definitely have an edge over others as the skills required to clear this certification are at par with industry standards. To achieve this certification, one needs to have a good knowledge of entire Hadoop ArchitectureBig Data and Hadoop training with certification online including Pig, Hive, Sqoop and Flume.

Basically the certification program will test you on four objectives including Infrastructure, Data management and Querying objectives. Going as per you background, it will be not that difficult for you to excel in Hadoop. It is very easy to learn from the base as the technology is unique to itself.

For it’s preparation, you can opt for various courses available online. You can also go through various free tutorials available online. What is important here is that you must practice everything you lean practically because that is only going to help you in landing to a proper role in Hadoop World.

Big data is omnipresent. According to a recent prediction by Indian Institute Architects (IIA), Big Data Analytics will be embedded everywhere. As predicted by IDC, the Big Data market is going to reach $48.6 Billion by 2019, increasing with a CAGR of 23.1% from what is now. Over 50% of all business analytics software considers Hadoop an apt solution to managing all the data concerns in the world.

Hadoop Training is the most widely searched term by professionals and business enterprises to gain professional insights into Hadoop Distributed File System (HDFS), MapReduce, Hadoop Clusters, HBase, Yarn, Pig, Zookeeper and other related topics. This training aims at providing extensive knowledge on each topic through several hands-on exercises, examples and quizzes.

In addition to Big Data Hadoop training course, you will undertake project work and will be allotted a 2-hour session with IBM Certified Trainer to discuss project solution. IBM Certificate will be awarded on project completion.

The project will help you implement Hadoop concepts and theories in real-time in your organization. You can increase your chances of getting recognized by you dream company, as most leading enterprises and small business groups use Hadoop for handling their data on the internet and within the organization.

Recommended Audience

• Programming Developers and System Administrators
• Project managers eager to learn new techniques of maintaining large data
• Experienced working professionals aiming to become Big Data Analysts
• Mainframe Professionals, Architects & Testing Professionals
• Graduates, undergraduates and working professionals eager to learn the latest Big Data technology

Jobs in Hadoop

Both experienced and Hadoop certified professionals are approached by top-paid Multinationals for following job positions:

• Hadoop Developers
• Hadoop Analysts
• Hadoop Administrators
• Hadoop Testers
• Data Scientists

The hiring outlook for Hadoopers is at its peak. The average entry-level salaries for all the above titles have increased by 27% since 2013, and it is 6.6 lakhs per annum currently.

This course prepares you for Cloudera Certification exams: CCAH and CCDH. Once enrolled for the course, you will receive lifetime access to videos, tutorials and course material. Big Data and Hadoop training with certification online

Reviews of Hadoop Certification

★★★★☆

Both Hortonworks and Cloudera divide their trainings and certifications in 3 main groups: Data Scientists, Developers/Analysts, and Administration. 

In whichever group you are, you'll have to know the Hadoop basics: HDFS (storage), Yarn, and MapReduce. The basics should be covered in every course or certification you are going to take. What certification you want to take depends on what job role you are in: an analyst / data scientist, developer or sysadmin.

You can succeed in the certification by reading books or attending training. Getting the certification doesn't necessarily mean you can actually work with the technology. It can be good though if you are new to the technology and you don't really know where to start.

I personally don't know much about the Cloudera certification, but I am a Trainer for Hortonworks in London. I think the certification can be good additional to work experience to prove you are up-to-date with the technology. Hadoop has a lot of moving parts, lots of different technologies, lots of small things that you can learn from sitting a course or reading books covering the subjects.

★★★★★

Hadoop Interview Questions - http://www.hub4tech.com/intervie...

Cloudera (CCB-400) Certification - http://www.hub4tech.com/cloudera...

Cloudera (DS-200) Certification - http://www.hub4tech.com/data-sci...

Cloudera (CCD-410) Certification-  http://www.hub4tech.com/cloudera...

Cloudera (CCA-505) Certification-  http://www.hub4tech.com/cloudera -certification-cca-505-exam-practice-test

Cloudera (CCA-500) Certification-  http://www.hub4tech.com/cloudera...

Happy Hadooping .. 

Analyse Twitter Data Using Hadoop, Flume and Hive

Twitter Data Analysis Using Hadoop Flume

we will try the experimental Twitter Source provided by Apache’s Flume distribution to get streaming tweets into HDFS and we will process them in Hive and create structured tables which can be further utilized for analytics via any other tool like, Tableau, QlikView or Hunk.

Flume TwitterAgent Setup

 we will setup a Twitter Agent in Apache Flume distribution (apache-flume-1.5.2-bin , which is the latest version at the time of writing this post). 

Twitter Source Overview

              Apache’s Twitter Source org.apache.flume.source.twitter.TwitterSource is highly experimental and may change between minor versions of Flume. This should be used at our own risk and curiosity to analyze real-time streaming data.

Twitter Source connects via Streaming API to the twitter fire-hose, and continuously downloads tweets. These tweets are converted into Avro format and Avro events are sent to the downstream Flume sink, HDFS sink in our use case. For Connecting with Twitter streaming data, we need consumer and access tokens and secrets of a Twitter developer account.

Creation of Twitter Developer Account

Twitter Developer Account can be created at Twitter Developers apps Page. In this page we need to provide valid twitter account page in the website field from which we need to get streaming data. If we provide valid details on this page we will get our app created as shown in below screen shots. For security reasons, I have blurred consumer access key, secret values in the below screens.

We need below four values for authenticated by Twitter.

• Consumer Key (API Key)
• Consumer Secret (API Secret)
• Access Token
• Access Token Secret

In the same screen, We can create access tokens by clicking on ‘Create my access token‘. Access tokens will be as shown in the below screen.

Creation of Agent in flume.conf

Lets create the agent named TwitterAgent under flume.conf file in FLUME_CONF_DIR directory location with below configuration properties. In this we need to specify the access tokens collected from twitter correctly in Twitter source setup.

Download the flume-sources-1.0-SNAPSHOT.jar and add it to the flume class path as shown below in the conf/flume-env.sh file

FLUME_CLASSPATH="/home/training/Installations/apache-flume-1.3.1-bin/flume-sources-1.0-SNAPSHOT.jar"

The jar contains the java classes to pull the Tweets and save them into HDFS.

The conf/flume.conf should have all the agents (flume, memory and hdfs) defined as below

Twitter Network Graph using Neo4j

Neo4j is a one of the popular Graph Databases. and CQL stands for CypherQuery Language. Neo4j is written in Java Language.

Twitter Profile Graph Database

What is a Graph Database?

A graph database is an online database management system with 

Create, Read, Update and Delete (CRUD) operations working on a graph data model.

For Analyze Twitter Graph Database using Neo4j Click Given Link Below :

http://network.graphdemos.com/login?next=http://network.graphdemos.com/%23browser

Above Link Loads Twitter API data and tweet activities well 

as a mixture of other tweets which may be popular to Graph Database enthusiasts. 

Your Twitter API Loads as per users Twitter Activity and Usage Status.

Access your personal instance of Neo4j using the URL, username,

and password above.

 We'll keep your instance around for a few days, 

but you can always come back and get a new instance.