Pages

Showing posts with label Aws. Show all posts
Showing posts with label Aws. Show all posts

Sunday, August 13, 2017

Qubole : Load Multiple tables to Qubole Hive table from a Data Store

API call to Load Multiple tables from a Qubole Data Store to Hive table. 


[rahul@local qubole]$ cat /databasescript 
#!/bin/bash

#Qubole API Key
AUTH="***********"
#Database Name
DB_NAME="***********"
#Host Name
DB_HOST="***********"
#User Name
DB_USER="***********
#Password 
DB_PASS='***********'

echo $DB_PASS


## request table import from tap;
function tableImport() {

request_body=$(cat <<EOF
{
   "command_type":"DbImportCommand",
   "mode":"1",
   "hive_serde":"orc",
   "hive_table":"<HIVE TABLE NAME>.$1",
   "dbtap_id":"$2",
   "db_table":"$1",
   "db_parallelism":"1",
   "use_customer_cluster":"1",
   "customer_cluster_label":"Qubole_Data_Import",
   "tags":[" Data"]
}
EOF
)

echo $request_body
   curl -X POST \
-H "X-AUTH-TOKEN: $AUTH" \
-H "Content-Type:application/json" \
-d "$request_body" https://api.qubole.com/api/v1.2/commands/
}

##register database with tap
request_body=$(cat <<EOF
{
  "db_name":"$DB_NAME",
  "db_host":"$DB_HOST",
  "db_user":"$DB_USER",
  "db_passwd":"$DB_PASS",
  "db_type":"sqlserver",
  "db_location":"on-premise",
  "gateway_ip": "***********",
  "gateway_port": "***********",
  "gateway_username": "***********",
  "gateway_private_key": "***********"}

EOF
)

echo $KEY
ID=$(curl -s -X POST \
-H "X-AUTH-TOKEN: $AUTH" \
-H "Content-Type:application/json" \
-d "$request_body" https://api.qubole.com/api/v1.2/db_taps/ | jq .id)

#get the tables and call import
curl -s -H "X-AUTH-TOKEN: $AUTH" \
     -H "Content-Type:application/json" \
     https://api.qubole.com/api/v1.2/db_taps/$ID/tables | jq -r .[] | while read x; do  tableImport $x $ID; done

# can't delete the tap at the end unless we continuously poll for no active jobs;
STATUS="null"

while [ "$STATUS" = "null" ]
do
STATUS=$(curl  -s -X DELETE \
 -H "X-AUTH-TOKEN: $AUTH" \
 -H "Content-Type:application/json" \
 https://api.qubole.com/api/v1.2/db_taps/$ID | jq .status)
echo -n "."
sleep 5
done

Friday, December 23, 2016

Aws Volume tagging Script.

Following Script will give the aws cli command to tag the volumes with tags associated to its instance.

Prerequisite
1. All instance are tagged properly
2. Aws Cli is installed and Configured properly
3. Configure the Aws Cli output to Json
4. Create the File Full_Json_Instances.json with output of describe-instances

===================
#/bin/python
import json

def fun_Instance_Volume_Tagging(Instance_ID, Instance):
    for Volumes in Instance["Instances"][0]["BlockDeviceMappings"]:
        VOL_ID = Volumes["Ebs"]["VolumeId"]
        for Tags in Instance["Instances"][0]["Tags"]:
            TAG_KEY = Tags["Key"]
            TAG_VALUE = Tags["Value"]
            print "aws ec2 create-tags --resources " + VOL_ID + " --tags Key=" + TAG_KEY + ",Value=" + TAG_VALUE +""

with open("Full_Json_Instances.json") as json_file:
    json_data = json.load(json_file)
    for Instances in json_data["Reservations"]:
        Instance_ID = Instances["Instances"][0]["InstanceId"];
        fun_Instance_Volume_Tagging(Instance_ID, Instances)
====================

Monday, August 15, 2016

Aws Flowlogs for Traffic Monitoring

VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data is stored using Amazon CloudWatch Logs. After you've created a flow log, you can view and retrieve its data in Amazon CloudWatch Logs.

Flow logs can help you with a number of tasks; for example, to troubleshoot why specific traffic is not reaching an instance, which in turn can help you diagnose overly restrictive security group rules. You can also use flow logs as a security tool to monitor the traffic that is reaching your instance.

To create Flow log for your subnet we followed these steps:
1. Creating Log group in your CloudWatch:
  - We created new log group in your CloudWatch to log your entries.
  - Please remember your can use same log group for multiple flow log you create.
  - To create log group: AWS Management console -> CloudWatch -> Logs --> Create new log group

2. Create Flow Log for VPC
  - Open the Amazon EC2 console -> Service VPC
  - In the navigation pane, choose VPC then select your VPC
  - From VPC Action select Create a Flow Log
Please refer this link to get more information on this: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#create-flow-log

Note that VPC Flow Log Records[1] is a space-separated string that has the format outline in our document, where 14 fields are available, and field #11 & #12 has recorded the time in Unix seconds)

According to the Filter & Pattern Syntax[2], we can filter the log events matching our conditions for space-delimited logs.

Example filter (as we dont care the 1st 10 fields in this case, so we use ... )

[..., start, end, action, status]

Say if we need to capture the vpc flow log between Sat, 06 Aug 2016 04:35:56 GMT and Sun, 07 Aug 2016 04:35:56 GMT

using epoch time converter(http://www.epochconverter.com/ for example), we get the Unix time in second being 1470458156 & 1470544556

so the filter we will be using become

[..., start>1470458156, end<1470544556, action, status]

So you can follow link[3], To search all log entries after a given start time using the Amazon CloudWatch console

Goto AWS CloudWatch Console-> Logs -> Select the vpc flowlog log group -> above "Log Streams List", click "Search Event"

and use the [..., start>1470458156, end<1470544556, action, status] in the filter field, then press Enter.

You can modify the filter accordingly for more conditions.


Resource Links: [1] AWS - VPC - VPC Flow Logs - Flow Log Records https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#flow-log-records [2] AWS - CloudWatch - Searching and Filtering Log Data - Filter and Pattern Syntax - Using Metric Filters to Extract Values from Space-Delimited Log Events https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html#d0e26783 [3] AWS - CloudWatch - To search all log entries after a given start time using the Amazon CloudWatch console https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/SearchDataFilterPattern.html

Monday, July 25, 2016

Import OVA to Amazon Aws

VM Import/Export enables you to easily import virtual machine images from your existing environment to Amazon EC2 instances and export them back to your on-premises environment. This offering allows you to leverage your existing investments in the virtual machines that you have built to meet your IT security, configuration management, and compliance requirements by bringing those virtual machines into Amazon EC2 as ready-to-use instances. You can also export imported instances back to your on-premises virtualization infrastructure, allowing you to deploy workloads across your IT infrastructure.


Step 1.  : installing the Aws CLI

Step 2. We can get the Access Key ID and Secret Key from Aws IAM service under the specific User.
aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: us-west-2
Default output format [None]: ENTER

Step 3
Now create two files: trust-policy.json & role-policy.json, in the second file you’ll need to replace “$bucketname” with your bucket name.

trust-policy.json:
===============
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"",
         "Effect":"Allow",
         "Principal":{
            "Service":"vmie.amazonaws.com"
         },
         "Action":"sts:AssumeRole",
         "Condition":{
            "StringEquals":{
               "sts:ExternalId":"vmimport"
            }
         }
      }
   ]
}
===============

role-policy.json:
=================
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":[
            "arn:aws:s3:::$bucketname"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObject"
         ],
         "Resource":[
            "arn:aws:s3:::$bucketname/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "ec2:ModifySnapshotAttribute",
            "ec2:CopySnapshot",
            "ec2:RegisterImage",
            "ec2:Describe*"
         ],
         "Resource":"*"
      }
   ]
}
===================

Now, use the aws cli tools to apply the policies:
$ aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json
$ aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json

Step 4 : Check VM prerequisite before exporting as OVA
========================
In regard of VM, before exporting it from Vsphere and importing to AWS cloud, please make sure that all prerequisites for Import have been fullfiled.
Compare your VM with this checklist:
-all unnecessary services are disabled,
-no unnecessary applications are placed in Windows Startup,
-there are no pending reboots (reboot flag set by Windows Update or by any other software),
-VM volumes are defragmented and the size of each disk is resized to necessary (bigger disk=longer conversion time),
-you use single network interface setup to use DHCP (this should be done prior to import),
-no ISO is attached to this VM,
-make sure that Microsoft .NET Framework 3.5 Service Pack 1 or later are installed (required to support Ec2Config),
-your VM's root volume use MBR partition table,
-your anti-virus and anti-spyware software and firewalls are disabled,
-only one partition is bootable,
-rdp access is enabled,
-the administrator account and all other user accounts use secure passwords. All accounts must have passwords or the importation might fail.
-Uninstall the VMware Tools from your VMware VM,
-the language of your OS is EN-US,
-these hotfixes are installed (according to OS version):
Install Latest Ec2Config
https://aws.amazon.com/developertools/5562082477397515
=================

Step 5 : Uploading the OVA to S3 and Creating the AMI
You can upload your VMs in OVA format to your Amazon S3 bucket using the upload tool of your choice. After you upload your VM to Amazon S3, you can use the AWS CLI to import your OVA image. These tools accept either a URL (public Amazon S3 file, a signed GET URL for private Amazon S3 files) or the Amazon S3 bucket and path to the disk file.

Use aws ec2 import-image to create a new import image task.
The syntax of the command is as follows:

$ aws ec2 import-image --description "Windows 2008 OVA" --disk-containers file://containers.json
The file containers.json is a JSON document that contains information about the image. The S3Key is the name of the image file you want to upload to the S3Bucket.

[{
    "Description": "First CLI task",
    "Format": "ova",
    "UserBucket": {
        "S3Bucket": "my-import-bucket",
        "S3Key": "my-windows-2008-vm.ova"
    }
}]

Step 6 : Checking the Status


Use the “aws ec2 describe-import-image-tasks” command to return the status of the task. The syntax of the command is as follows:

Regarding the License licensing, within the  api-call "aws ec2 import-image" we can define a "--license-type" value.
Based on this option your VM will use your license (BYOL) or will activate itself in AWS KMS[4]. Option should be set to "AWS" or "BYOL".

Wednesday, February 17, 2016

AWS IAM policy for limiting the users access to a group of instance with a particular Tag Name.

AWS IAM policy for limiting the users access to a group of instance with a particular Tag Name.


The TAGNAME and VALUE will be
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:RebootInstances"
            ],
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/TAGNAME": "VALUE"
                }
            },
            "Resource": "arn:aws:ec2:eu-east-0:123654456123:instance/*"
        }
    ]
}

Mount S3 Bucket on CentOS/RHEL and Ubuntu using S3FS

S3FS is FUSE (File System in User Space) based solution to mount an Amazon S3 buckets, We can use system commands with this drive just like as another Hard Disk in system. On s3fs mounted files systems we can simply use cp, mv and ls the basic Unix commands similar to run on locally attached disks.
This article will help you to install S3FS and Fuse by compiling from source, and also help you to mount S3 bucket on your CentOS/RHEL and Ubuntu systems.
Step 1: Remove Existing Packages
First check if you have any existing s3fs or fuse package installed on your system. If installed it already remove it to avoid any file conflicts.
CentOS/RHEL Users:
# yum remove fuse fuse-s3fs
Ubuntu Users:
$ sudo apt-get remove fuse
Step 2: Install Required Packages
After removing above packages. First we will install all dependencies for fuse and s3cmd. Install the required packages to system use following command.
CentOS/RHEL Users:
# yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap
Ubuntu Users:
$ sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support
Step 3: Download and Compile Latest Fuse
Download and compile latest version of fuse source code. For this article we are using fuse version 2.9.3. Following set of command will compile fuse and add fuse module in kernel.
# cd /usr/src/
# wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.9.3/fuse-2.9.3.tar.gz
# tar xzf fuse-2.9.3.tar.gz
# cd fuse-2.9.3
# ./configure –prefix=/usr/local
# make && make install
# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
# ldconfig
# modprobe fuse
Step 4: Download and Compile Latest S3FS
Download and compile latest version of s3fs source code. For this article we are using s3fs version 1.74. After downloading extract the archive and compile source code in system.
# cd /usr/src/
# wget https://s3fs.googlecode.com/files/s3fs-1.74.tar.gz
# tar xzf s3fs-1.74.tar.gz
# cd s3fs-1.74
# ./configure –prefix=/usr/local
# make && make install

Step 5: Setup Access Key
Also In order to configure s3fs we would required Access Key and Secret Key of your S3 Amazon account. Get these security keys from Here.
# echo AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY > ~/.passwd-s3fs
# chmod 600 ~/.passwd-s3fs
Note: Change AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.
Step 6: Mount S3 Bucket
Finally mount your s3 bucket using following set of commands. For this example, we are using s3 bucket name as mydbbackup and mount point as /s3mnt.
# mkdir /tmp/cache
# mkdir /s3mnt
# chmod 777 /tmp/cache /s3mnt
# s3fs -o use_cache=/tmp/cache mydbbackup /s3mnt

Install s3cmd in Linux to Manage Amazon s3 Bucket

s3cmd is a command line utility used for creating s3 buckets, uploading, retrieving and managing data to Amazon s3 storage. This article will help you to how to use install s3cmd on CentOS, RHEL and Ubuntu Systems and manage s3 buckets via command line in easy steps
How to Install s3cmd Package
s3cmd is available in default rpm repositories for CentOS,RHEL and Ubuntu systems, You can install it using simply executing following commands on your system.
On CentOS/RHEL:
# yum install s3cmd

If the above command is not working we need to create the following repo or enable epel repo
# vim /etc/yum.repos.d/s3tools.repo
[s3tools]
name=Tools for managing Amazon S3 - Simple Storage Service (RHEL_6)
type=rpm-md
baseurl=http://s3tools.org/repo/RHEL_6/
gpgcheck=1
gpgkey=http://s3tools.org/repo/RHEL_6/repodata/repomd.xml.key
enabled=1
On Ubuntu/Debian:
$ sudo apt-get install s3cmd
On SUSE Linux Enterprise Server 11:
# zypper addrepo http://s3tools.org/repo/SLES_11/s3tools.repo
# zypper install s3cmd
Configure s3cmd Environment
In order to configure s3cmd we would require Access Key and Secret Key of your S3 Amazon account. Get these security keys from aws securityCredentials page. If will prompt to login to your amazon account.
After getting key files, use below command to configure s3cmd.
# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: xxxxxxxxxx
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes

New settings:
  Access Key: xxxxxxxxxxxxxxxxxxxxxx
  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  Encryption password: xxxxxxxxxx
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
Uses of s3cmd Command Line
Once configuration is successfully completed. Now find below command details to how to manage s3 buckets using commands.
1. Creating New Bucket
To create a new bucket in Amazon s3 use below command. It will create bucket named apps in S3 account.
# s3cmd mb s3://apps

Bucket 's3://apps/' created
2. Uploading file in Bucket
Below command will upload file file.txt to s3 bucket using s3cmd command.
# s3cmd put file.txt s3://apps/

file.txt -> s3://appsfile.txt  [1 of 1]
 190216 of 190216   100% in    0s  1668.35 kB/s  done
3. Uploading Directory in Bucket
If we need to upload entire directory use -r to upload it recursively like below.
# s3cmd put -r backup s3://apps/

backup/file1.txt -> s3://apps/backup/file1.txt  [1 of 2]
 9984 of 9984   100% in    0s    18.78 kB/s  done
backup/file2.txt -> s3://apps/backup/file2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done
Make sure you are not adding trailing slash in upload directory named backup (eg: backup/), else it will upload only content of backup directory only.
# s3cmd put -r backup/ s3://apps/

backup/file1.txt -> s3://apps/file1.txt  [1 of 2]
 9984 of 9984   100% in    0s    21.78 kB/s  done
backup/file2.txt -> s3://apps/file2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done
4. List Data of S3 Bucket
List the objects of s3 bucket using ls switch with s3cmd.
# s3cmd ls s3://apps/

                       DIR   s3://apps/backup/
2013-09-03 10:58    190216   s3://apps/file.txt
5. Download Files from Bucket
Some times if we need to download files from s3 bucket, Use following commands to download it.
# s3cmd get s3://apps/file.txt

s3://apps/file.txt -> ./file.txt  [1 of 1]
 4 of 4   100% in    0s    10.84 B/s  done
6. Remove Data of S3 Bucket
To remove files are folder from s3 bucket use following commands.
 Removing file from s3 bucket 
# s3cmd del s3://apps/file.txt

File s3://apps/file.txt deleted

 Removing directory from s3 bucket 
# s3cmd del s3://apps/backup
File s3://apps/backup deleted
7. Remove S3 Bucket
If we don’t need s3 bucket any more, we can simply delete it using following command. Before removing bucket make sure its empty.
# s3cmd rb s3://apps
ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty
Above command failed because of s3 bucket was not empty
To remove bucket first remove all objects inside bucket and then use command again.
# s3cmd rb s3://apps
Bucket 's3://apps/' removed
8. List All S3 Bucket
Use following command to list all s3 buckets in your aws account.
# s3cmd ls

Wednesday, December 16, 2015

AWS EBS SnapShot Scipt for Linux


The script itself is fairly straightforward, but took several dry-runs in order to fine tune it so that it would reliably create the snapshots, but more importantly would clear out old snapshots older than 7 days.

You can see the for loop for deleting older snapshots. This is done by parsing snapshot dates, converting the dates to a pure numeric value and comparing said numeric value to a “7 days ago” date variable.

Take a look at the script below, hopefully it will be useful to you! There could be more error checking, but that should be fairly easy to do.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#!/bin/sh
# EBS Snapshot volume script
# Written by Star Dot Hosting
# www.stardothosting.com
 
# Constants
ec2_bin="/opt/aws/bin"
my_cert="/opt/aws/cert.txt"
my_key="/opt/aws/key.txt"
instance_id=`wget -q -O- http://169.254.169.254/latest/meta-data/instance-id`
 
# Dates
datecheck_7d=`date +%Y-%m-%d --date '7 days ago'`
datecheck_s_7d=`date --date="$datecheck_7d" +%s`
 
# Get all volume info and copy to temp file
$ec2_bin/ec2-describe-volumes -C $my_cert -K $my_key  --filter "attachment.instance-id=$instance_id" > /tmp/volume_info.txt 2>&1
 
 
# Get all snapshot info
$ec2_bin/ec2-describe-snapshots -C $my_cert -K $my_key | grep "$instance_id" > /tmp/snap_info.txt 2>&1
 
# Loop to remove any snapshots older than 7 days
for obj0 in $(cat /tmp/snap_info.txt)
do
 
        snapshot_name=`cat /tmp/snap_info.txt | grep "$obj0" | awk '{print $2}'`
        datecheck_old=`cat /tmp/snap_info.txt | grep "$snapshot_name" | awk '{print $5}' | awk -F "T" '{printf "%sn", $1}'`
        datecheck_s_old=`date "--date=$datecheck_old" +%s`
 
#       echo "snapshot name: $snapshot_name"
#       echo "datecheck 7d : $datecheck_7d"
#       echo "datecheck 7d s : $datecheck_s_7d"
#       echo "datecheck old : $datecheck_old"
#       echo "datecheck old s: $datecheck_s_old"
 
        if (( $datecheck_s_old <= $datecheck_s_7d ));
        then
                echo "deleting snapshot $snapshot_name ..."
                $ec2_bin/ec2-delete-snapshot -C $my_cert -K $my_key $snapshot_name
        else
                echo "not deleting snapshot $snapshot_name ..."
 
        fi
 
done
 
 
# Create snapshot
for volume in $(cat /tmp/volume_info.txt | grep "VOLUME" | awk '{print $2}')
do
        description="`hostname`_backup-`date +%Y-%m-%d`"
        echo "Creating Snapshot for the volume: $volume with description: $description"
        $ec2_bin/ec2-create-snapshot -C $my_cert -K $my_key -d $description $volume
done