Sunday, December 3, 2017

Qubole load CSV with spark




val df = sqlContext.read.format("com.databricks.spark.csv")
                    .option("delimiter", "|")
                    .option("header", "true")
                    .option("inferSchema", "true")
                    .load("s3://*****.CSV")


df.registerTempTable("temp-table")


sqlContext.sql("""
create table database.table as
select * from temp-table
""")

Tuesday, November 28, 2017

Increases swap in azure linux machine

To create a swap file in the directory that's defined by the ResourceDisk.MountPoint parameter, you can update the /etc/waagent.conf file by setting the following three parameters:

ResourceDisk.Format=y
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=xx

Note The xx placeholder represents the desired number of megabytes (MB) for the swap file.
Restart the WALinuxAgent service by running one of the following commands, depending on the system in question:

Ubuntu: service walinuxagent restart
Red Hat/Centos: service waagent restart

Run one of the following commands to show the new swap apace that's being used after the restart:

dmesg | grep swap
swapon -s
cat /proc/swaps
file /mnt/resource/swapfile
free| grep -i swap

If the swap file isn't created, you can restart the virtual machine by using one of the following commands:

shutdown -r now
init 6

Wednesday, November 22, 2017

Docker Clustering with Swarm in Centos7

Installing Docker
mkdir /install-files ; cd /install-files

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-1.13.1-1.el7.centos.x86_64.rpm

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm

#package for docker-engine-selinux
yum install -y policycoreutils-python
rpm -i docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm

#package for docker-engine
yum install -y libtool-ltdl libseccomp
rpm -i docker-engine-1.13.1-1.el7.centos.x86_64.rpm

#remove rpm packages
rm docker-engine-* -f

#enable systemd service
systemctl enable docker

#start docker
systemctl start docker

firewall-cmd --permanent --add-source=192.168.56.0/24
firewall-cmd --permanent --add-port=2377/tcp
firewall-cmd --permanent --add-port=7946/tcp
firewall-cmd --permanent --add-port=7946/udp
firewall-cmd --permanent --add-port=4789/udp
firewall-cmd --reload

Firewalld
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-using-firewalld-on-centos-7

firewall-cmd --get-active-zones
firewall-cmd --list-all
firewall-cmd --zone=public --add-port=2377/tcp --permanent


systemctl enable docker;
systemctl restart docker

Docker Cluster Env

docker swarm init --advertise-addr=192.168.56.105
Swarm initialized: current node (b4b79zi3t1mq1572r0iubxdhc) is now a manager.


To add a worker to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-1wcz7xfyvhewvj3dd4wcbhufw4lub3b1vgpuoybh90myzookbf-4ksxoxrilifb2tmvuligp9krs \
    192.168.56.101:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.


To join as a Swarm manager

docker swarm join-token manager
  docker swarm join \
    --token SWMTKN-1-10cqx6yryq5kyfe128m2xhyxzplsc90lzksqggmscv1nfipsbb-bfdbvfhuw9sg8mx2i1a4rkvlv \
    192.168.56.101:2377

Sunday, November 5, 2017

Creating CSR with multiple Domains



Create an Conf File with following details : openssl.conf
===================
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
countryName = Country Name (2 letter code)
countryName_default = GB
stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Lancashire
localityName = Locality Name (eg, city)
localityName_default = Altham
organizationalUnitName = Organizational Unit Name (eg, section)
commonName = Common Name (eg, YOUR name)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 40

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = abc.com
DNS.2 = xyz.abc.com
DNS.3 = *.abc.com
DNS.4 = *.xyz.abc.com
===================

Create the Private 
  openssl genrsa -out acb.com.key 2048


Create the CSR with Help of conf file created. 

 openssl req -new -sha256 -key abc.com.key -out abc.com.csr -config openssl.conf 


Tuesday, October 24, 2017

docker: 'stack' is not a docker command.

While deploying the docker services using stack deploy command. We got following error.

docker stack deploy -c docker-compose.yml appslab
docker: 'stack' is not a docker command.
See 'docker --help'.

Resolution
Upgrade docker to 1.13

In Centos 7 we used the following to get the docker upgraded. Now the docket-latest package in centos7 is upgraded to 1.13

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-1.13.1-1.el7.centos.x86_64.rpm

wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm

#package for docker-engine-selinux
yum install -y policycoreutils-python
rpm -i docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm

#package for docker-engine
yum install -y libtool-ltdl libseccomp
rpm -i docker-engine-1.13.1-1.el7.centos.x86_64.rpm

#remove rpm packages
rm docker-engine-* -f

#enable systemd service
systemctl enable docker

#start docker
systemctl start docker

Friday, September 22, 2017

Fedora 26 + Virtualbox 5.1 + kenel 4.12

Error: After Upgrading the VMs will not starting

/tmp/vbox.0/r0drv/linux/memuserkernel-r0drv-linux.o: warning: objtool: .fixup: unexpected end of section
  if [ "-pg" = "-pg" ]; then if [ /tmp/vbox.0/r0drv/linux/memuserkernel-r0drv-linux.o != "scripts/mod/empty.o" ]; then ./scripts/recordmcount  "/tmp/vbox.0/r0drv/linux/memuserkernel-r0drv-linux.o"; fi; fi;
make[1]: *** [Makefile:1519: _module_/tmp/vbox.0] Error 2
make: *** [Makefile:304: vboxdrv] Error 2


Change to the Virtualbox source directory
cd /usr/share/virtualbox/src

get this patch and place it in /usr/share/virtualbox/src
sudo wget http://paste.siduction.org/20170629003423

apply patch
sudo patch -Np0 < 20170629003423

run vboxconfig by executing
/sbin/vboxconfig

have fun!

Friday, September 8, 2017

Minio Running as Service



In this case we  are just running minio server as a service in a docker swarm so that the services are up. For this we need a shared Storage so that when the service is running in different hosts the data’s are synced.


echo "AKIAIOSFODNN7EXAMPLE" | docker secret create access_key -
echo "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" | docker secret create secret_key -

docker service create --name="minio-service" --publish 9000:9000   --constraint 'node.role == manager' --mount type=bind,src=/mnt/minio/,dst=/data --secret="access_key" --secret="secret_key" minio/minio server /data