Pages

Saturday, March 31, 2018

Parsing Value from a Json Field in Qubole.

Description of how to extract a value from a JSON field in Hive using the get_json_object function. When the data in one of the fields in the Hive environment is in JSON format, and we need to extract a value out of the JSON, we can use the get_json_object function. For example, if we have a column named jdata containing the following JSON:


get_json_object(column_name, '$.keyvalue')

The column name is : jdata and json the Column is as followes.

{
    "Foo": "ABC",
    "Bar": "20090101100000",
    "Quux": {
        "QuuxId": 1234,
        "QuuxName": "Sam"
    }
}

if we have to extract ABC : get_json_object(jdata, '$.Foo') 

Friday, February 16, 2018

Azure VPN Gateway with Cisco ASA using Routing

The Azure VPN Gateway and Cisco ASA can encounter routing-type issues when configured together. To resolve this, the UsePolicyBasedTrafficSelectors must be enabled in the Azure Connection. The provided code is a PowerShell script that retrieves the specified Azure virtual network gateway connection and creates a new IPsec policy with specific parameters. The script then sets the IPsec policies for the connection to the new policy and enables UsePolicyBasedTrafficSelectors to solve the routing issue.

$RG1 = "****************"

This line declares a variable $RG1 and sets its value to a string of asterisks. This is likely just a placeholder for the actual resource group name.

$Connection16 = "****************"

Similar to the first line, this line declares a variable $Connection16 and sets its value to a string of asterisks. This is likely just a placeholder for the actual connection name.

$connection6 = Get-AzureRmVirtualNetworkGatewayConnection -Name $Connection16 -ResourceGroupName $RG1

This line retrieves the virtual network gateway connection object for a connection with the specified name ($Connection16) in the specified resource group ($RG1). The connection object is assigned to the variable $connection6.

$newpolicy6 = New-AzureRmIpsecPolicy -IkeEncryption AES256 -IkeIntegrity SHA384 -DhGroup DHGroup24 -IpsecEncryption AES256 -IpsecIntegrity SHA1 -PfsGroup PFS24 -SALifeTimeSeconds 28800 -SADataSizeKilobytes 4608000

This line creates a new IPsec policy object ($newpolicy6) with the specified settings for IKE encryption, integrity, DH group, IPsec encryption, integrity, Perfect Forward Secrecy (PFS) group, Security Association (SA) lifetime, and SA data size.

Set-AzureRmVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $connection6 -IpsecPolicies $newpolicy6

This line updates the virtual network gateway connection object ($connection6) with the new IPsec policy ($newpolicy6) created in the previous step.

Set-AzureRmVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $connection6 -IpsecPolicies $newpolicy6 -UsePolicyBasedTrafficSelectors $True

This line updates the virtual network gateway connection object ($connection6) again, this time enabling policy-based traffic selectors by setting the -UsePolicyBasedTrafficSelectors parameter to $True. This is necessary to resolve routing issues that can occur when configuring the Azure VPN Gateway with a Cisco ASA.


PS Azure:\> $connection6.UsePolicyBasedTrafficSelectors

True

Azure:\

PS Azure:\> $connection6.IpsecPolicies


Docker Management using Portainer

Portainer is a lightweight management UI that allows easy management of Docker environments, including creating, deploying, and managing containers, services, and stacks. It is particularly useful for those who are new to Docker or those who prefer a visual interface over command-line management.

To install Portainer with a persistent container, you can follow these steps:
  • Pull the Portainer image: docker pull portainer/portainer
  • Create a directory for Portainer data: mkdir -p /mnt/docker/portainer/data
  • Create a Docker service for Portainer with the following command:  
docker service create \ --name portainer \ --publish 9090:9000 \ --constraint 'node.role == manager' \ --mount type=bind,src=/mnt/docker/portainer/data,dst=/data \ --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \ portainer/portainer \ -H unix:///var/run/docker.sock

 

the above command will create a new Docker service named "portainer" with a published port of 9090, mounted volume for persistent data, and a constraint for the node role of "manager".

  • Access the Portainer UI by visiting the IP address or hostname of the Docker swarm manager node on port 9090 in a web browser.
  • Create a new user account and start managing your Docker environment using the Portainer UI.

Access the Portainer UI by visiting the IP address or hostname of the Docker swarm manager node on port 9090 in a web browser.



Sunday, December 3, 2017

Qubole load CSV with spark

This is a code snippet using Spark on Qubole to load a CSV file into a DataFrame, register it as a temporary table, and create a permanent table from the data in the temporary table.

The first line of code reads the CSV file from an S3 location into a DataFrame. The options set for the format, delimiter, header, and inferSchema specify how the CSV file should be read and parsed.

val df = sqlContext.read.format("com.databricks.spark.csv")
                    .option("delimiter", "|")
                    .option("header", "true")
                    .option("inferSchema", "true")
                    .load("s3://*****.CSV")

The second line of code registers the DataFrame as a temporary table, which can be used for querying.

df.registerTempTable("temp-table")

The third line of code creates a permanent table in a specified database by executing an SQL query on the temporary table. The query selects all the columns and rows from the temporary table and creates a new table with the same data in the specified database.

sqlContext.sql("""
create table database.table as
select * from temp-table
""")

Tuesday, November 28, 2017

Increases swap in azure linux machine

In Azure to create a swap file in the directory that's defined by the ResourceDisk.MountPoint parameter, you can update the /etc/waagent.conf file by setting the following three parameters:

ResourceDisk.Format=y
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=xx


Note The xx placeholder represents the desired number of megabytes (MB) for the swap file.
Restart the WALinuxAgent service by running one of the following commands, depending on the system in question:

Ubuntu: service walinuxagent restart
Red Hat/Centos: service waagent restart


Run one of the following commands to show the new swap apace that's being used after the restart:

dmesg | grep swap
swapon -s
cat /proc/swaps
file /mnt/resource/swapfile
free| grep -i swap


If the swap file isn't created, you can restart the virtual machine by using one of the following commands:

shutdown -r now
init 6

Wednesday, November 22, 2017

Docker Clustering with Swarm in Centos7

Docker Clustering with Swarm in Centos7 is a process of creating a cluster of Docker hosts using the Docker Swarm feature in the CentOS 7 operating system. The Swarm feature is a native clustering and orchestration tool within Docker that enables users to create and manage a cluster of Docker hosts. This process involves setting up a Docker Swarm manager and one or more Docker Swarm nodes, configuring the network and storage for the cluster, and deploying and scaling Docker services across the cluster. The benefits of clustering Docker hosts with Swarm in CentOS 7 include increased scalability, high availability, and load balancing of Docker services, as well as simplified management and deployment of containerized applications.

Installing Docker

mkdir /install-files ; cd /install-files
wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-1.13.1-1.el7.centos.x86_64.rpm
wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm


Package for docker-engine-selinux
yum install -y policycoreutils-python
rpm -i docker-engine-selinux-1.13.1-1.el7.centos.noarch.rpm
Package for docker-engine
yum install -y libtool-ltdl libseccomp
rpm -i docker-engine-1.13.1-1.el7.centos.x86_64.rpm
Remove rpm packages
rm docker-engine-* -f
Enable systemd service
systemctl enable docker
Start docker

systemctl start docker

Firewalld Enabling Firewall Rules

firewall-cmd --get-active-zones
firewall-cmd --list-all
firewall-cmd --zone=public --add-port=2377/tcp --permanent
firewall-cmd --permanent --add-source=192.168.56.0/24
firewall-cmd --permanent --add-port=2377/tcp
firewall-cmd --permanent --add-port=7946/tcp
firewall-cmd --permanent --add-port=7946/udp
firewall-cmd --permanent --add-port=4789/udp
firewall-cmd --reload
Enable and Restart systemd service
systemctl enable docker;
systemctl restart docker
Docker Cluster Env

docker swarm init --advertise-addr=192.168.56.105

Swarm initialized: current node (b4b79zi3t1mq1572r0iubxdhc) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-1wcz7xfyvhewvj3dd4wcbhufw4lub3b1vgpuoybh90myzookbf-4ksxoxrilifb2tmvuligp9krs \
    192.168.56.101:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

To join as a Swarm manager

docker swarm join-token manager

  docker swarm join \
    --token SWMTKN-1-10cqx6yryq5kyfe128m2xhyxzplsc90lzksqggmscv1nfipsbb-bfdbvfhuw9sg8mx2i1a4rkvlv \
    192.168.56.101:2377


Sunday, November 5, 2017

Creating CSR with multiple Domains With Openssl

Creating a CSR (Certificate Signing Request) with multiple domains using OpenSSL involves generating a private key and a CSR file, which includes the details of the domain(s) to be included in the certificate. The process involves the following steps:

Generate a private key using the openssl command with the following syntax:

openssl genrsa -out domain.key 2048

This generates a private key file named "domain.key" with 2048 bits of encryption.

Create a configuration file (e.g. domain.conf) that contains the details of the domains to be included in the certificate. This file should contain the following details:

[req]
default_bits       = 2048
default_keyfile    = domain.key
distinguished_name = req_distinguished_name
req_extensions     = req_ext

[req_distinguished_name]
countryName             = Country Name (2 letter code)
stateOrProvinceName     = State or Province Name (full name)
localityName            = Locality Name (eg, city)
organizationName        = Organization Name (eg, company)
commonName              = Common Name (e.g. server FQDN or YOUR name)
emailAddress            = Email Address

[req_ext]
subjectAltName          = @alt_names

[alt_names]
DNS.1                  = example.com
DNS.2                  = www.example.com
DNS.3                  = subdomain.example.com


In the example above, "example.com", "www.example.com", and "subdomain.example.com" are included as the alternate domain names.

Generate a CSR file using the openssl command with the following syntax:

openssl req -new -sha256 -key domain.key -out domain.csr -config domain.conf

This generates a CSR file named "domain.csr" that contains the details of the private key and the alternate domain names specified in the configuration file.

Submit the CSR file to a Certificate Authority (CA) to obtain a signed SSL certificate that can be installed on the server.

Overall, this process allows for the creation of a CSR file with multiple domain names that can be used to obtain a signed SSL certificate to secure those domains.