#Configure jbod in disk genius software#

Software as a Service (SaaS) & Financing.StarWind Virtual Tape Library Appliance (VTLA).Then adjust vm.overcommit_ratio to control for how much over allocation is done. Set vm.overcommit_memory to 1 and (this will allow kernel to spawn processes even though memory+swap is not enough, we know when child processes are spawned they use 2 times as much memory as they will actually use, so it is better to set it to 1). This is very important and you can read all about it in here.ДЄnother parameter to tune is vm.overcommit_memory. These parameters could be tuning in /etc/nf sysctl -w vm.swappiness=0

On DataNodes the partition of each of them should follow these guidelines so that the performance is optimized.ДЄpart from hard disk configuration couple of Kernel tuning will also give better performance as well as less cluster issues. DataNode hard disks have 3 types of usage: mnt/hdfs/1/datanode,/mnt/hdfs/2/datanode,/mnt/hdfs/3/datanode,/mnt/hdfs/4/datanodeДЇor DataNodes the disks should be partitioned for maximum efficiency. mnt/hdfs/1/namenode,/mnt/hdfs/2/namenode,/mnt/hdfs/3/namenode,/mnt/hdfs/4/namenode It specifies local storage on each node, used by the DataNodes and NameNode services to store HDFS data. This is where we tell Hadoop to use the directories we created earlier. etc/hadoop-0.20/conf.Mењњluster/hdfs-site.xml This config file will tell Hadoop where to find the NameNode and its default file system. etc/hadoop-0.20/conf.Mењњluster/core-site.xml To configure these mounted drives (which are separately mounted) use the following configurations. dev/sde1 /mnt/hdfs/4 ext4 inode_readahead_blks=128,commit=30,data=writeback,noatime,nodiratime,nodev,nobarrier 0 0 dev/sdd1 /mnt/hdfs/3 ext4 inode_readahead_blks=128,commit=30,data=writeback,noatime,nodiratime,nodev,nobarrier 0 0 dev/sdc1 /mnt/hdfs/2 ext4 inode_readahead_blks=128,commit=30,data=writeback,noatime,nodiratime,nodev,nobarrier 0 0 dev/sdb1 /mnt/hdfs/1 ext4 inode_readahead_blks=128,commit=30,data=writeback,noatime,nodiratime,nodev,nobarrier 0 0 To configure these you could follow these instructions. Also because of inherent replication and design for failure architecture there are few configuration which are highly desirable when mounting hard disk to DataNodes. Hadoop workload is lot more read intensive than write intensive. So donвЂ™t configure DataNode for RAID0 but use JBOD. In RAID0 if one of the disk fails then the whole DataNode is useless. in previous example you will get 800 MB/sec instead of 1 GB/sec).Д«ut more than the read speed, Reliability is the main concern. So for example if you have 10 disk and one of the disk is 80% of maximum throughput, then when you configure RAID0 your total read speed will be 80% of total (e.g. But as described in the вЂњ Disk are like SnowflakesвЂќ paper each disk performance are different. if you have 1o disk and each one had theoritical maximum read speed of 100 MB/sec (to check the actual speed use the script I wrote earlier) then the maxium read thoughput for RAID0 would be 1 GB/sec. The main idea is that if you configure RAID0 you will get higher read throughput e.g. You can read all about the RAID vs JBOD discussion here. There are few parameters that is very important while configuring DataNode disks. This is where all the data is stored read and processed. For NameNode Heap usage please read my previous post.ДataNodes are the heart of the Hadoop system. NameNode relies heavily on Memory so Hard disk is not primary important. This will give you ample speed and when you set the NameNode directories in Hadoop as separate drives then Hadoop will take copies of NameNode data into each drive. This will give you ample protection.ДЇor Namenode data directory use RAID0 configuration.

in the cluster use separate hard disks asВ bootdisk0 and bootdisk1 and configure to RAID1 (mirror).

So it is very important to configure the Hadoop disk the right way so that you get maximum workВ out of them. According to major Hadoop cluster admins 50% of the support tickets are because of disk failure. In a typical cluster 100,s if not 1,000,s of hard disk work in tandem to get the job done. But, even in this massive parallel processing system the least reliable components are the hard disks. It is build for failure with all itвЂ™s redundancy mechanisms.

Configure jbod in disk genius