Telecommunications Blog

Telecommunications Blog

A blog for mobile communications systems GSM , UMTS and LTE

Recomended

Breaking

Saturday, February 3, 2018

Installing Hadoop 3.0.0 Single Node Cluster on Ubuntu

2/03/2018 01:42:00 PM 1
Installing Hadoop 3.0.0 Single Node Cluster on Ubuntu

Hi Hadoop observers,

     This article is sharing information with you about the simple installation of Hadoop 3.0.0 single node. Follow the steps clearly so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).


 

Prerequisites

  • UBUNTU (Ubuntu 17.10)
  • JAVA JDK 8 (minimum) or JDK 9 or OpenJDK 8
  • SSH and Pdsh

Step 1: Installing Java

1 – Oracle JAVA SDK


$ java -version

If you don’t have Java installed on your system, use one of following link to install it first.
First, we should add Oracle repository as follow:

$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
Install JAVA 8 (JDK 8u161) on Ubuntu
$ sudo apt-get install oracle-java8-installer
$ java -version
Install JAVA9 (JDK 9) on Ubuntu
$ sudo apt-get install oracle-java9-installer
$ java -version

2 – Open SDK

$ sudo apt-get update
$ sudo apt-get install openjdk-8-jdk

Installing Java


Step 2: Configuring SSH

SSH is used for remote login. SSH is required in Hadoop to manage its nodes. It must have configured in Hadoop user environment (The user whom will using Hadoop).
$ sudo apt-get install ssh
$ sudo apt-get install pdsh
$ sudo service ssh status

 Generate Key Pairs

$ ssh-keygen -t rsa -P ""

The result of above line will be:

    Generating public/private rsa key pair.

    Enter file in which to save the key (/home/elgindy/.ssh/id_rsa):

    Created directory '/home/elgindy/.ssh'.

    Your identification has been saved in /home/elgindy/.ssh/id_rsa.

    Your public key has been saved in /home/elgindy/.ssh/id_rsa.pub.

    The key fingerprint is:

    9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2 elgindy@ubuntu

    The key's randomart image is:

    [...snipp...]


Then Configure passwordless ssh then Change the permission of file that contains the key

$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
 $ chmod 0600 ~/.ssh/authorized_keys

Finally check ssh to the localhost

$ ssh localhost





Step 3: Configure Hadoop

First download Hadoop 3.x from the following link

https://hadoop.apache.org/releases.html

or download the last version till now

http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.0.0/hadoop-3.0.0.tar.gz


After download go to the folder or path containing Hadoop-3.0.0.tar.gz then extract its content using right

click then extract here or using terminal as follow:

$ tar -xzf hadoop-3.0.0.tar.gz


We will install Hadoop in Home path so, we should move extracted folder to a new folder in Home path

$ sudo mv hadoop-3.0.0/ /usr/local/Hadoop/
$ sudo chown -R elgindy:elgindy /usr/local/Hadoop/


Step 4: Update $HOME/.bashrc

$ gedit ~/.bashrc


Edit .bashrc file is located in user’s home directory and adds following parameters:
 
 

#Elgindy HADOOP VARIABLES START

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

export HADOOP_HOME=/usr/local/Hadoop

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

#HADOOP VARIABLES END





Save bachrc then reload it using:
$ source ~/.bashrc

Step 5: Setup Hadoop Configuration

As we know, Core hadoop is made up of 5 daemons viz. NameNode(NN), SecondaryNameNode(SN), DataNode(NN),ResourceManager(RM), NodeManager(NM). We need to modify the config files in the conf folder.Below are the files required for respective daemons.

NAMENODE
core-site.xml
RESOURCE MANAGER
mapred-site.xml
SECONDARY NAMENODE

DATANODE
slaves
NODEMANAGER
 slaves & yarn-site.xml

Ports used by Hadoop Daemons


Remote Procedure Call (RPC) is a protocol that one program can use to request a service from a program located in another computer in a network without having to understand network details.
WEB which is denoted in the table is the WEB port number.

Hadoop Daemons
RPC Port
WEB – UI
NameNode
9000
9870
SecondaryNameNode

50090
DataNode
50010
50075
Resource Manager
8030
8088
Node Manager
8040
8042
 
First, we need to create a temp folder for Hadoop operations 
$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown -R elgindy:elgindy /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp
Then move to the following Path for configuring hadoop
$ cd /usr/local/Hadoop/etc/hadoop/
Then open hadoop-env.sh
$ gedit hadoop-env.sh
Set JAVA_HOME environment variable. Change the JAVA path as per install on your system.
export JAVA_HOME=/usr/lib/jvm/java-8-oracle




$ gedit core-site.xml

Then write the next lines between configuration.
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description>
</property>

$ gedit hdfs-site.xml

Then write the next lines between configuration.
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

$ gedit mapred-site.xml

Then write the next lines between configuration.
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

$ gedit yarn-site.xml

Then write the next lines between configuration.
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Step 6: Formatting the HDFS filesystem via the NameNode

The first step to running your Hadoop for the first time is formatting the Hadoop filesystem which is implemented on top of the local filesystem of your “cluster”.
First moving to Hadoop path
$ cd /usr/local/Hadoop
Then apply the following line (Do not format a running Hadoop filesystem as you will lose all the data currently in the cluster (in HDFS))
$ bin/hdfs namenode -format





Step 7: Starting your single-node cluster

Start All Hadoop Related Services.
(Starting Daemon’s For NN,DN and SNN )
$ sbin/start-dfs.sh 
(Starting Daemon’s For RM and NM )
$ sbin/start-yarn.sh
Or you can run all Hadoop by the following line
$ sbin/start-all.sh

Hint:
If the upper lines gave an error at the time of start HDFS services then use:
$ echo "ssh" | sudo tee /etc/pdsh/rcmd_default
Then run the above lines again.



To check running of Hadoop just writing
$ jps
The result will be:

18096 NodeManager
17440 DataNode
17952 ResourceManager
17649 SecondaryNameNode
18451 Jps
17294 NameNode




Hadoop NameNode started on port 9870 default. Access your server on port 9870 in web browser.





Now access port 8088 for getting the information about cluster and all applications
http://localhost:8088/




Access port 50090 for getting details about secondary namenode.
http://localhost:50090/
Access port 50075 to get details about DataNode
http://localhost:50075/



Step 8: Stopping your single-node cluster

Stop All Hadoop Related Services.
(Stopping Daemon’s For NN,DN and SNN )

$ sbin/stop-dfs.sh 
(Starting Daemon’s For RM and NM )

$ sbin/stop-yarn.sh

Or you can run all Hadoop by the following line

$ sbin/stop-all.sh
  


The next lesson will installing Hadoop 3.0.0 Mylti-Node.

Friday, January 1, 2016

Make USB Storage Device Read Only and Write protected

1/01/2016 12:11:00 PM 1
Make USB Storage Device Read Only and Write protected


Elgindy USB Protector





I have made a program that protect USB Drives and make them Read Only.
It's FREE for all.
It's very clean
I scanned it for you
It's Very clean (100%).
https://www.virustotal.com/en/file/f7651fbc0ca2091f4f9fa190148f8eedbeb3bcd314029eb307166cdd17a6169f/analysis/

 First on because i Encrypt the program from viruses and it's OK.

Video








It's Require .NET Framework 4
You can download it from
https://www.microsoft.com/en-us/download/details.aspx?id=40779

You can download the program from here

http://gsmcommunications.blogspot.com.eg/p/elgindy-usb-protector.html

Friday, May 1, 2015

How to delete Shortcut Virus on your USB or removable Drive with one click

5/01/2015 10:38:00 PM 0
How to delete Shortcut Virus on your USB or removable Drive with one click

Saturday, July 19, 2014

The public telephone networks

7/19/2014 11:54:00 AM 1
The public telephone networks



Telephone networks are known as public telephone networks.

They consist of :
  •  Trunk exchange .
  •  Local exchange .
  •  And lines between them .
The lines can be divided into trunk lines and local lines.

Simplified Trunk Cable Diagram
Simplified Trunk Cable Diagram

1- The Trunk Networks


Lines which are used for trunk traffic only are described as trunk lines . These networks can be designed as :
  •  Meshed networks
  •  Star networks
  •  Or mixed networks
  • Transmission medium
  •  Symmetric copper pair
  •  Coaxial copper cable
  •  Optical fiber
  •  Air (radio )



Trunk /network Configuration
Trunk /network Configuration




Monday, December 10, 2012

Call Setup and Call Handling in GSM

12/10/2012 04:37:00 AM 0
Call Setup and Call Handling in GSM

Call Setup


Different procedures are necessary depending on the initiating and terminating party:

  • Mobile Originating Call MOC: Call setup, which are initiated by an MS
  • Mobile Terminating Call MTC: Call setup, where an MS is the called party
  • Mobile Mobile Call MMC: Call setup between two mobile subscribers; MMC thus consists of the execution of a MOC and a MTC one after the other.
  • Mobile Internal Call MIC: a special case of MMC; both MSs are in the same MSC area, possibly even in the same cell.

 Mobile Originating Call MOC


1. Channel Request: The MS requests for the allocation of a dedicated signaling channel to perform the call setup.

2. After allocation of a signaling channel the request for MOC call setup, included the TMSI (IMSI) and the last LAI, is forwarded to the VLR

3. The VLR requests the AC via HLR for Triples (if necessary).

4. The VLR initiates Authentication, Cipher start, IMEI check (optional) and TMSI Re-allocation (optional).

5. If all this procedures have been successful, MS sends the Setup information (number of requested subscriber and detailed service description) to the MSC.

6. The MSC requests the VLR to check from the subscriber data whether the requested service an number can be handled (or if there are restrictions which do not allow further proceeding of the call setup)

7. If the VLR indicates that the call should be proceeded, the MSC commands the BSC to assign a Traffic Channel (i.e. resources for speech data transmission) to the MS

8. The BSC assigns a Traffic Channel TCH to the MS

9. The MSC sets up the connection to requested number (called party).

Remark: This MOC as well as the MTC described in the following describes only the principles of an MOC / MTC, not the detailed signaling flow.


Wednesday, April 18, 2012

VoIP protocols

4/18/2012 07:03:00 PM 2
VoIP protocols

Real-Time Transport Protocol (RTP)

RTP is used for transmitting audio and video packets between communicating computers. RTP is defined in RFC 3550 and "profiles", which specify how to packetize various audio and video codecs used in an RTP session, are defined in RFC 3551.

 RTP also addresses issues like packet order and provides mechanisms to help address delay and jitter. RTP was improved upon with the result being called Secure RTP which provides for encryption, authentication, and integrity of the audio and video packets transmitted between communicating devices.

H.323 

Developed by Study Group 16 of the ITU-T, H.323 is an umbrella standard encompassing
many subcomponent standards and annexes for transmitting multimedia (voice, video and data) across packet based networks. The two main subcomponent protocols are H.225 (call control), & H.245 (bearer control and capabilities exchange). H.225 consists of two main parts Q.931 (basic call control as used in ISDN networks) and RAS (Registration, Admission & Status).

H.323 also references other standards such as Real Time Protocol (RTP), G.xxx audio codecs, H.26x video codecs and T.120 real time data conferencing protocol.

SIP 

Developed by IETF, SIP is a mechanism to initiate, terminate & modify sessions in an IP
network. It uses a client / server architecture and the protocol is request-response based. It
enables personal mobility by tracking down users and delivering calls to an endpoint. It is a
lightweight, text-based protocol and reuses much of the construct of other internet protocols
such as HTTP and SMTP. SIP does not know about the underlying details of a session and relies on IETF protocol Session Description Protocol (SDP) to describe the session. It also interworks with other IETF protocols such as Megacop, RTP, RTSP, RSVP and SAP.


Both H.323 and SIP can be referred to as "intelligent endpoint protocols". What this means is that all of the intelligence required to locate the remote endpoint and to establish media streams between the local and remote device is an integral part of the protocol.

MGCP & Megacop/H.248 

There is another class of protocols which is complementary to H.323 and SIP referred to as "device control protocols". Those protocols are H.248 and MGCP. 

MGCP (Media Gateway Control Protocol) & Megacop/H.248 are relatively low level, master/slave protocols used between Call Agents and MGs. MGCP is not officially a standard but does exist as an informational RFC (RFC 3435). Megacop (IETF RFC 3015) & H.248 (ITU Rec.) are the same protocol developed by an IETF and ITU collaboration. It is derived from, and draws heavily from MGCP but adds several new enhancements.

Some service providers provide users with devices that implement H.248 or MGCP (or comparable protocols). In the core of the network, some device serving as the MGC provides the H.323 or SIP logic necessary to properly terminate VoIP calls around the world.

VoIP OSI Protocol Stack

Telecommunications Blog
Telecommunications Blog
I will explain QOS at next article .

Tuesday, April 10, 2012

What is VoIP ? - Introduction

4/10/2012 06:37:00 PM 4
What is VoIP ? - Introduction
What is VoIP

Around 1995 there was a revolutionary advancement in the way traditional voice traffic was
carried when voice, video and data were transmitted over a single ‘converged’ network. The
terms Voice Over IP (VoIP) and IP Telephony (IPT) were introduced to describe this method of
transport of voice signals as well as fax, DTMF tones, signalling traffic, and network management
traffic.


There are a number of protocols that may be employed in order to provide for VoIP communication services like :

  • SIP (Session Initiation Protocol) / RTP  (Real-Time Protocol)
  • IMS
  • H.323
  • H.248 / MGCP   ( comparable protocols )
  • Skype
VoIP can deliver voice, video, and data communications capabilities across a number of separate devices that work together, such as a mobile phone, a PC, and even an HD TV!.

You can enjoy VoIP sevice by using:
  1. An analog Telephone Adapter (ATA). It connected existing telephone jack, so you no need to change your old telephone hardware (system).
  2. Dedicated VoIP Phones. It is a Phones that create specially for IP/Packet Switch connection. Just plug this phone to your IP/internet network then you can enjoy VoIP service.
  3. A Softphone. It is a piece of software that installed at PC/Laptop to make you can enjoy VoIP service without dedicated hardware.
I will explain the protocols at the next article.