Integrating Hadoop with LVM to provide elasticity to cluster
In this article we are integrating hadoop with LVM to provide custom elasticity to hadoop cluster. so that we can increase or decrease storage according to over requirement without affecting data.
for this demonstration i am creating a cluster locally with the help of virtual machine box with Red hat enterprise linux 8 as base os . For storage attaching the virtual storage of 10GB to the slave node.
Now let’s start the node and verify that is storage connected or not
As u can see the disk is connected i.e /dev/sdb
now we have to implement the LVM in slave node for this we need to create logical volume, volume group and persistent volume
As logical volume is created let format this volume and mount it to the folder so hdaoop can use it storing data . here we are using ext4 file system format .
as u can see data node is started for cluster below is configuration of data node
Now let’s see configuration of namenode i.e controller node for cluster with IP 192.168.43.30
So our cluster is ready and in running status so we can check how much storage it has
As u can see we have created cluster with storage size equal to 5 GB now we if sudden requirement increased we can increase our storage of cluster on runtime with the help of LVM . just increase size of logical volume from our volume group
here we added more 3 GB of storage to our data node so it should be donated to cluster
see it’s working without any !!
This type of application are very useful when there’s sudden requirement comes and if u want flexiblity in your storage without any downtime.
