Sunday, April 15, 2018

Enabling hive Authorization in Qubole

Once the Hive authorization is enabled in qubole we need to mange the users and permission by hive authentication,  following are the some of the commands which will be used for the same.

1. Listing the Current Roles

Set role admin;
show roles

2. Create the roles

CREATE ROLE <role_name>;
Creates a new role. Only the admin role has privilege for this.


Eg:
Set role admin;
Create role sysadmin;

3. Grant Role to users


GRANT ROLE <role_name> TO USER <user_name>
 
Eg:
Set role admin;
Grant Role sysadmin to user rahul ;


4. Revoke a role from user

REVOKE ROLE <role_name> FROM USER <user_name>;


Eg:
Set role admin;
REVOKE Role sysadmin from user rahul;


5. List  Roles attached to a user

SHOW ROLE GRANT USER <user_name>;


Eg.
Set role admin;
show role grant user `rahul`;


6. List Users under a role

SHOW PRINCIPALS <Role_name>


Eg
Set role admin;
SHOW PRINCIPALS sysadmin


7. Assign Role access to tables



Sample Permission
SELECT privilege: It provides read access to an object (table).
INSERT privilege: It provides ability for adding data to an object (table).
UPDATE privilege: It provides ability for running UPDATE queries on an object (table).
DELETE privilege: It provides ability for deleting data in an object (table).
ALL privilege: It provides all privileges. In other words, this privilege gets translated into all the above privileges.


GRANT <Permission> ON <table_name> TO ROLE <role_name>;


Eg:
Grant all on default.testtable to role sysadmin


8. View Role/user Permissions on tables

Check all users who have been granted with a specific role


SHOW GRANT USER <user_name> ON <table_name|All>;
SHOW GRANT ROLE <user_name> ON <table_name|All>;


Eg:
SHOW GRANT user analytics on all

Saturday, March 31, 2018

Parsing Value from a Json Field in Qubole.

When the data in one of the Filed in the hive env is in Json format and when we need to extract the value out of the Json we can use the following commands

get_json_object(column_name, '$.keyvalue')

The column name is : jdata and json the Column is as followes.

{
    "Foo": "ABC",
    "Bar": "20090101100000",
    "Quux": {
        "QuuxId": 1234,
        "QuuxName": "Sam"
    }
}

if we have to extract ABC : get_json_object(jdata, '$.Foo') 

Friday, February 16, 2018

Azure VPN Gateway with Cisco ASA using Routing

When we configure the Azure VPN Gateway with Cisco ASA there will be issue realted to routing type so we need to enable UsePolicyBasedTrafficSelectors in the Azure Connection to Solve the issue .



$RG1          = "****************"
$Connection16 = "****************"

$connection6  = Get-AzureRmVirtualNetworkGatewayConnection -Name $Connection16 -ResourceGroupName $RG1

$newpolicy6   = New-AzureRmIpsecPolicy -IkeEncryption AES256 -IkeIntegrity SHA384 -DhGroup DHGroup24 -IpsecEncryption AES256 -IpsecIntegrity SHA1 -PfsGroup PFS24 -SALifeTimeSeconds 28800 -SADataSizeKilobytes 4608000


Set-AzureRmVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $connection6 -IpsecPolicies $newpolicy6



Set-AzureRmVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $connection6 -IpsecPolicies $newpolicy6 -UsePolicyBasedTrafficSelectors $True



PS Azure:\> $connection6.UsePolicyBasedTrafficSelectors

True

Azure:\



PS Azure:\> $connection6.IpsecPolicies

Docker Management using Portainer

mkdir -p /mnt/docker
yum install -y rsync


* * * * * rsync -avzh /mnt/docker/ root@dm01:/mnt/docker/
* * * * * rsync -avzh /mnt/docker/ root@dm02:/mnt/docker/
* * * * * rsync -avzh /mnt/docker/ root@dm03:/mnt/docker/
* * * * * rsync -avzh /mnt/docker/ root@dm04:/mnt/docker/
~                                                          


Install Portainer with a persistent container
Code:
mkdir -p /mnt/docker/portainer/data


docker pull portainer/portainer
docker service create \
    --name portainer \
    --publish 9090:9000 \
    --constraint 'node.role == manager' \
    --mount type=bind,src=/mnt/shared/portainer,dst=/data \
    --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
    portainer/portainer \
   -H unix:///var/run/docker.sock

[root@dm01 ~]#



Sunday, December 3, 2017

Qubole load CSV with spark




val df = sqlContext.read.format("com.databricks.spark.csv")
                    .option("delimiter", "|")
                    .option("header", "true")
                    .option("inferSchema", "true")
                    .load("s3://*****.CSV")


df.registerTempTable("temp-table")


sqlContext.sql("""
create table database.table as
select * from temp-table
""")

Tuesday, November 28, 2017

Increases swap in azure linux machine

To create a swap file in the directory that's defined by the ResourceDisk.MountPoint parameter, you can update the /etc/waagent.conf file by setting the following three parameters:

ResourceDisk.Format=y
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=xx

Note The xx placeholder represents the desired number of megabytes (MB) for the swap file.
Restart the WALinuxAgent service by running one of the following commands, depending on the system in question:

Ubuntu: service walinuxagent restart
Red Hat/Centos: service waagent restart

Run one of the following commands to show the new swap apace that's being used after the restart:

dmesg | grep swap
swapon -s
cat /proc/swaps
file /mnt/resource/swapfile
free| grep -i swap

If the swap file isn't created, you can restart the virtual machine by using one of the following commands:

shutdown -r now
init 6

Wednesday, November 22, 2017

Docker Clustering with Swarm in Centos7

Installing Docker
mkdir /install-files ; cd /install-files

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-1.13.1-1.el7.centos.x86_64.rpm

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm

#package for docker-engine-selinux
yum install -y policycoreutils-python
rpm -i docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm

#package for docker-engine
yum install -y libtool-ltdl libseccomp
rpm -i docker-engine-1.13.1-1.el7.centos.x86_64.rpm

#remove rpm packages
rm docker-engine-* -f

#enable systemd service
systemctl enable docker

#start docker
systemctl start docker

firewall-cmd --permanent --add-source=192.168.56.0/24
firewall-cmd --permanent --add-port=2377/tcp
firewall-cmd --permanent --add-port=7946/tcp
firewall-cmd --permanent --add-port=7946/udp
firewall-cmd --permanent --add-port=4789/udp
firewall-cmd --reload

Firewalld
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-using-firewalld-on-centos-7

firewall-cmd --get-active-zones
firewall-cmd --list-all
firewall-cmd --zone=public --add-port=2377/tcp --permanent


systemctl enable docker;
systemctl restart docker

Docker Cluster Env

docker swarm init --advertise-addr=192.168.56.105
Swarm initialized: current node (b4b79zi3t1mq1572r0iubxdhc) is now a manager.


To add a worker to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-1wcz7xfyvhewvj3dd4wcbhufw4lub3b1vgpuoybh90myzookbf-4ksxoxrilifb2tmvuligp9krs \
    192.168.56.101:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.


To join as a Swarm manager

docker swarm join-token manager
  docker swarm join \
    --token SWMTKN-1-10cqx6yryq5kyfe128m2xhyxzplsc90lzksqggmscv1nfipsbb-bfdbvfhuw9sg8mx2i1a4rkvlv \
    192.168.56.101:2377