Tuesday, October 27, 2015
Monday, October 12, 2015
AWS S3 : Limiting Access from Certain IP's
We need to our nodes to connect to S3 from a single IP address, an elastic IP attached to our gateway node.
So - based on this design, we needed a way to only allow access to a set of buckets from this single IP address. For example purposes we are using the IP of 72.309.38.2.
IF you were to follow the initial example laid out by the AWS documentation - you’ll end up with a policy that probably looks similar to this.
{
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-wicked-awesome-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "72.309.38.2/32"
}
}
}
]
}
What you’re going to find, after banging your head on the table a few times, is that this policy does not work. There does not appear to be an implied deny rule with S3 buckets (similar to how IAM access policies are setup). I did my testing with the s3-curl command - When you have that tool setup - you can make a query like the following:
./s3curl.pl --head --id=AKIAYOURACCESSKEY --key=YourSecretKey -- https://s3.amazonaws.com/my-wicked-awesome-bucket/even-more-awesome.file
Using the policy above - returned this:
HTTP/1.1 200 OK
x-amz-id-2: BLPqeibX8nZGnSDNi9zRhb+6a8fDiOW6Ij1OXhadWknJKCX9WAb7x1xNETvdXAEv
x-amz-request-id: F6AF36D912E6003B
Date: Fri, 27 Apr 2012 00:25:52 GMT
Last-Modified: Tue, 17 Apr 2012 17:50:11 GMT
ETag: "8553074962ba71d8b2b600b971ba80a8"
Accept-Ranges: bytes
Content-Type: application/pdf
Content-Length: 1451065
Server: AmazonS3
After troubleshooting this with the AWS support team - they let me know of the following:
By default accounts are restricted from accessing S3 unless they have been given access via policy. However, S3 is designed by default to allow any IP address access. So to block IP's you would have to specify denies explicitly in the policy instead of allows.
Once I learned this - the policy was easy to adjust. I flipped around the policy from allowing access from only my IP address to denying access from everywhere that was NOT my IP address.
My new policy looked like this:
{
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPDeny",
"Effect": "Deny",
"Principal": {
"AWS": "*"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-wicked-awesome-bucket/*",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": "72.309.38.2/32"
}
}
}
]
}
And when I ran my s3-curl command again - I now received the following response.
HTTP/1.1 403 Forbidden
x-amz-request-id: 3B90483F655AA692
x-amz-id-2: 0XEO2miEzZEwbU2N5zPwzLtX4J7BJx/LLXpvSLPpQ7e2wuocx49katsD+ZcqwYA7
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 27 Apr 2012 00:27:14 GMT
Server: AmazonS3
So - based on this design, we needed a way to only allow access to a set of buckets from this single IP address. For example purposes we are using the IP of 72.309.38.2.
IF you were to follow the initial example laid out by the AWS documentation - you’ll end up with a policy that probably looks similar to this.
{
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-wicked-awesome-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "72.309.38.2/32"
}
}
}
]
}
What you’re going to find, after banging your head on the table a few times, is that this policy does not work. There does not appear to be an implied deny rule with S3 buckets (similar to how IAM access policies are setup). I did my testing with the s3-curl command - When you have that tool setup - you can make a query like the following:
./s3curl.pl --head --id=AKIAYOURACCESSKEY --key=YourSecretKey -- https://s3.amazonaws.com/my-wicked-awesome-bucket/even-more-awesome.file
Using the policy above - returned this:
HTTP/1.1 200 OK
x-amz-id-2: BLPqeibX8nZGnSDNi9zRhb+6a8fDiOW6Ij1OXhadWknJKCX9WAb7x1xNETvdXAEv
x-amz-request-id: F6AF36D912E6003B
Date: Fri, 27 Apr 2012 00:25:52 GMT
Last-Modified: Tue, 17 Apr 2012 17:50:11 GMT
ETag: "8553074962ba71d8b2b600b971ba80a8"
Accept-Ranges: bytes
Content-Type: application/pdf
Content-Length: 1451065
Server: AmazonS3
After troubleshooting this with the AWS support team - they let me know of the following:
By default accounts are restricted from accessing S3 unless they have been given access via policy. However, S3 is designed by default to allow any IP address access. So to block IP's you would have to specify denies explicitly in the policy instead of allows.
Once I learned this - the policy was easy to adjust. I flipped around the policy from allowing access from only my IP address to denying access from everywhere that was NOT my IP address.
My new policy looked like this:
{
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPDeny",
"Effect": "Deny",
"Principal": {
"AWS": "*"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-wicked-awesome-bucket/*",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": "72.309.38.2/32"
}
}
}
]
}
And when I ran my s3-curl command again - I now received the following response.
HTTP/1.1 403 Forbidden
x-amz-request-id: 3B90483F655AA692
x-amz-id-2: 0XEO2miEzZEwbU2N5zPwzLtX4J7BJx/LLXpvSLPpQ7e2wuocx49katsD+ZcqwYA7
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 27 Apr 2012 00:27:14 GMT
Server: AmazonS3
Saturday, August 15, 2015
Configure MySql and ODBC Connector In Windows 2012 Server
Installing MySql and its ODBC Connector In the Windows 2012 Server
Download The Microsoft Visual C++ 2010 Redistributable
http://www.microsoft.com/en-gb/download/details.aspx?id=14632
Download and install the ODBC Connector
http://cdn.mysql.com/Downloads/Connector-ODBC/5.3/mysql-connector-odbc-5.3.4-winx64.msi
Downoading the MySQL
http://cdn.mysql.com/Downloads/MySQL-5.6/mysql-5.6.26-winx64.zip
c:\mysql>cd bin
c:\mysql\bin>mysqld --install
Service successfully installed.
Go to Services and start the Mysql Services.
c:\mysql\bin>mysql -u root -p
>>No password is needed for first time
Set the root Password
=====================
mysql>
mysql> UPDATE mysql.user SET password=PASSWORD('redhat') WHERE user='root';
Query OK, 3 rows affected (0.00 sec)
Rows matched: 3 Changed: 3 Warnings: 0
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> select user,host from mysql.user;
+------+-----------+
| user | host |
+------+-----------+
| root | 127.0.0.1 |
| root | ::1 |
| | localhost |
| root | localhost |
+------+-----------+
4 rows in set (0.00 sec)
mysql> delete from mysql.user where user='';
Query OK, 1 row affected (0.00 sec)
mysql> select user,host from mysql.user;
+------+-----------+
| user | host |
+------+-----------+
| root | 127.0.0.1 |
| root | ::1 |
| root | localhost |
+------+-----------+
3 rows in set (0.00 sec)
mysql> create database fileserver;
Query OK, 1 row affected (0.01 sec)
mysql>GRANT ALL PRIVILEGES ON fileserver.* TO 'fileserver'@'localhost' IDENTIFIED BY 'fileserver';
Query OK, 0 rows affected (0.02 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)
mysql>
Download The Microsoft Visual C++ 2010 Redistributable
http://www.microsoft.com/en-gb/download/details.aspx?id=14632
Download and install the ODBC Connector
http://cdn.mysql.com/Downloads/Connector-ODBC/5.3/mysql-connector-odbc-5.3.4-winx64.msi
Downoading the MySQL
http://cdn.mysql.com/Downloads/MySQL-5.6/mysql-5.6.26-winx64.zip
c:\mysql>cd bin
c:\mysql\bin>mysqld --install
Service successfully installed.
Go to Services and start the Mysql Services.
c:\mysql\bin>mysql -u root -p
>>No password is needed for first time
Set the root Password
=====================
mysql>
mysql> UPDATE mysql.user SET password=PASSWORD('redhat') WHERE user='root';
Query OK, 3 rows affected (0.00 sec)
Rows matched: 3 Changed: 3 Warnings: 0
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> select user,host from mysql.user;
+------+-----------+
| user | host |
+------+-----------+
| root | 127.0.0.1 |
| root | ::1 |
| | localhost |
| root | localhost |
+------+-----------+
4 rows in set (0.00 sec)
mysql> delete from mysql.user where user='';
Query OK, 1 row affected (0.00 sec)
mysql> select user,host from mysql.user;
+------+-----------+
| user | host |
+------+-----------+
| root | 127.0.0.1 |
| root | ::1 |
| root | localhost |
+------+-----------+
3 rows in set (0.00 sec)
mysql> create database fileserver;
Query OK, 1 row affected (0.01 sec)
mysql>GRANT ALL PRIVILEGES ON fileserver.* TO 'fileserver'@'localhost' IDENTIFIED BY 'fileserver';
Query OK, 0 rows affected (0.02 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)
mysql>
Wednesday, August 12, 2015
Mysql Cluster Using Mysql NDB
Mysql Cluster using NDB(Network DataBase) provides a self healing mysql Cluster which provides a good performance. Mainly the Mysql Cluster Contains 3 Components ie using Management , SQL and Data parts.
Here we will be configuring two Management and two Data/SQL (together in One server) for the HA. Once the configuration is completed we will have two end points to connect to te database so we need to keep an Load balancer in front of the SQL end points.
OS used is RHEL7
Selinux Enabed
Firewall Disabled
Management Server## Perform the Following steps in both the Management Server's.
Install Needed Packages
=================
yum install glibc.i686 ncurses-libs.i686 libstdc++.i686 libgcc.i686 -yMake Directories and Download the Cluster Files
====================================
mkdir /usr/src/mysql-mgm
cd /usr/src/mysql-mgm
wget http://cdn.mysql.com/Downloads/MySQL-Cluster-7.4/mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
tar zxvf mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
cd mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686
cp bin/ndb_mgm* /usr/bin/
chmod 755 /usr/bin/ndb_mgm*
mkdir /var/lib/mysql-cluster
vi /var/lib/mysql-cluster/config.ini
==========================================
[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=80M
IndexMemory=18M
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
DataDir=/var/lib/mysql-cluster
[TCP DEFAULT]
# Section for the cluster management node
[NDB_MGMD]
NodeId=1
# IP address of the first management node (this system)
HostName=192.168.70.130
[NDB_MGMD]
NodeId=2
#IP address of the second management node
HostName=192.168.70.131
# Section for the storage nodes
[NDBD]
# IP address of the first storage node
HostName=192.168.70.132
DataDir= /var/lib/mysql-cluster
[NDBD]
# IP address of the second storage node
HostName=192.168.70.133
DataDir=/var/lib/mysql-cluster
# one [MYSQLD] per storage node
[MYSQLD]
[MYSQLD]
==========================================
chown mysql. /var/lib/mysql-cluster -R
To start the Management Service
========================
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --configdir=/var/lib/mysql-cluster/
Data And SQL Server#Perform this on both of the Server's
==============================================
Install the needed Packages
====================
yum install libaio.i686 libaio-devel.i686 -y
yum install perl -y
yum -y install perl-Data-Dumper
Download the packages
cd /usr/local/
wget http://cdn.mysql.com/Downloads/MySQL-Cluster-7.4/mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
tar zxvf mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
mv /root/mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz mysql
chown mysql. mysql -R
cd mysql
Initializing the database
scripts/mysql_install_db --user=mysql --datadir=/usr/local/mysql/data
cp support-files/mysql.server /etc/init.d/
chmod 755 /etc/init.d/mysql.server
cd /usr/local/mysql/bin
mv * /usr/bin
cd ../
vi /etc/my.cnf
============
[mysqld]
ndbcluster
# IP address of the cluster management node
ndb-connectstring=192.168.70.130,192.168.70.131
[mysql_cluster]
# IP address of the cluster management node
ndb-connectstring=192.168.70.130,192.168.70.131
============
mkdir /var/lib/mysql-cluster
cd /var/lib/mysql-cluster
ndbd --initial
/etc/init.d/mysql.server start
After this, secure the MySQL installation by running the appropriate script:
/usr/local/mysql/bin/mysql_secure_installation
Testing
In the Management Node check the command ndb_mgm and check the status
Thursday, July 9, 2015
Delete a nat rule in iptables
First of all I list all the rules including line numbers like this;
iptables -L -t nat –line-numbers
I then look at the output that will be similar to the below
In this example lets say I want to delete rule number 2 in the PREROUTING chain, I would enter the following;
iptables -t nat -D PREROUTING 2
In English the above line means remove line number 2 from the PREOUTING chain, I would then run the first command again to check my iptables file, then save the iptables file and restart the iptables service.
iptables -L -t nat –line-numbers
service iptables save
service iptables restart
All the above is carried out running on Centos, you may have to edit slightly for your particular distribution.
Friday, June 12, 2015
Getting Client IP Behind the Aws ELB (Http/Http Mode)
We need to add the Following Logformat to get the clients IP.
We use the X-Forwarded-For entry in the apache configuration to get it done.
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined_new
#....
#...
#
# START_HOST example.com
ServerName example.com
DocumentRoot "/var/www/example.com/html"
Options Includes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
CustomLog /var/www/logs/example.com/access_log combined_new
ErrorLog /var/www/logs/example.com/error_log
# END_HOST example.com
We use the X-Forwarded-For entry in the apache configuration to get it done.
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined_new
#....
#...
#
# START_HOST example.com
ServerName example.com
DocumentRoot "/var/www/example.com/html"
Options Includes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
CustomLog /var/www/logs/example.com/access_log combined_new
ErrorLog /var/www/logs/example.com/error_log
# END_HOST example.com
Friday, June 5, 2015
Jira
===
JIRA is a commercial software product that can be licensed for running on-premises or available as a hosted application. Pricing depends on the maximum number of users.
Installing Java
yum install java-1.7.0*
Installing Database
yum install -y mariadb-server
mysql -u root -p
CREATE DATABASE jiradb CHARACTER SET utf8 COLLATE utf8_bin;
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER,INDEX on jiradb.* TO 'jira'@'localhost' IDENTIFIED BY 'jira_xuZEKE4N';
flush privileges;
SHOW GRANTS FOR 'jira'@'localhost';
exit;
Install Jira:
Download atlassian-jira-6.2.2-x64.bin (32/64 bit) from https://www.atlassian.com/software/jira/download. And install as below-
wget https://downloads.atlassian.com/software/jira/downloads/atlassian-jira-6.4.5-x64.bin
./atlassian-jira-6.4.5-x64.bin
===================================================================
[root@adancsvso002 opt]# sh atlassian-jira-6.4.5-x64.bin
Unpacking JRE ...
Starting Installer ...
May 26, 2015 6:28:39 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
This will install JIRA 6.4.5 on your computer.
OK [o, Enter], Cancel [c]
Choose the appropriate installation or upgrade option.
Please choose one of the following:
Express Install (use default settings) [1], Custom Install (recommended for advanced users) [2, Enter], Upgrade an existing JIRA installation [3]
Where should JIRA 6.4.5 be installed?
[/opt/atlassian/jira]
Default location for JIRA data
[/var/atlassian/application-data/jira]
Configure which ports JIRA will use.
JIRA requires two TCP ports that are not being used by any other
applications on this machine. The HTTP port is where you will access JIRA
through your browser. The Control port is used to Startup and Shutdown JIRA.
Use default ports (HTTP: 8080, Control: 8005) - Recommended [1, Enter], Set custom value for HTTP and Control ports [2]
JIRA can be run in the background.
You may choose to run JIRA as a service, which means it will start
automatically whenever the computer restarts.
Install JIRA as Service?
Yes [y, Enter], No [n]
Extracting files ...
Please wait a few moments while JIRA starts up.
Launching JIRA ...
Installation of JIRA 6.4.5 is complete
Your installation of JIRA 6.4.5 is now ready and can be accessed via your
browser.
JIRA 6.4.5 can be accessed at http://localhost:8080
Finishing installation ...
[root@adancsvso002 opt]#
===================================================================
firewall-cmd --zone=public --add-port=8080/tcp --permanent
firewall-cmd --zone=public --add-port=8005/tcp --permanent
firewall-cmd --reload
wget http://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-5.1.35.tar.gz
tar zxvf mysql-connector-java-5.1.35.tar.gz
cp -rp mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar /opt/atlassian/jira/lib/
systemctl restart mariadb
systemctl status mariadb
service jira start
http://xxx.xxx.xxx.xxx:8080/
===
JIRA is a commercial software product that can be licensed for running on-premises or available as a hosted application. Pricing depends on the maximum number of users.
Installing Java
yum install java-1.7.0*
Installing Database
yum install -y mariadb-server
mysql -u root -p
CREATE DATABASE jiradb CHARACTER SET utf8 COLLATE utf8_bin;
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER,INDEX on jiradb.* TO 'jira'@'localhost' IDENTIFIED BY 'jira_xuZEKE4N';
flush privileges;
SHOW GRANTS FOR 'jira'@'localhost';
exit;
Install Jira:
Download atlassian-jira-6.2.2-x64.bin (32/64 bit) from https://www.atlassian.com/software/jira/download. And install as below-
wget https://downloads.atlassian.com/software/jira/downloads/atlassian-jira-6.4.5-x64.bin
./atlassian-jira-6.4.5-x64.bin
===================================================================
[root@adancsvso002 opt]# sh atlassian-jira-6.4.5-x64.bin
Unpacking JRE ...
Starting Installer ...
May 26, 2015 6:28:39 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
This will install JIRA 6.4.5 on your computer.
OK [o, Enter], Cancel [c]
Choose the appropriate installation or upgrade option.
Please choose one of the following:
Express Install (use default settings) [1], Custom Install (recommended for advanced users) [2, Enter], Upgrade an existing JIRA installation [3]
Where should JIRA 6.4.5 be installed?
[/opt/atlassian/jira]
Default location for JIRA data
[/var/atlassian/application-data/jira]
Configure which ports JIRA will use.
JIRA requires two TCP ports that are not being used by any other
applications on this machine. The HTTP port is where you will access JIRA
through your browser. The Control port is used to Startup and Shutdown JIRA.
Use default ports (HTTP: 8080, Control: 8005) - Recommended [1, Enter], Set custom value for HTTP and Control ports [2]
JIRA can be run in the background.
You may choose to run JIRA as a service, which means it will start
automatically whenever the computer restarts.
Install JIRA as Service?
Yes [y, Enter], No [n]
Extracting files ...
Please wait a few moments while JIRA starts up.
Launching JIRA ...
Installation of JIRA 6.4.5 is complete
Your installation of JIRA 6.4.5 is now ready and can be accessed via your
browser.
JIRA 6.4.5 can be accessed at http://localhost:8080
Finishing installation ...
[root@adancsvso002 opt]#
===================================================================
firewall-cmd --zone=public --add-port=8080/tcp --permanent
firewall-cmd --zone=public --add-port=8005/tcp --permanent
firewall-cmd --reload
wget http://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-5.1.35.tar.gz
tar zxvf mysql-connector-java-5.1.35.tar.gz
cp -rp mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar /opt/atlassian/jira/lib/
systemctl restart mariadb
systemctl status mariadb
service jira start
http://xxx.xxx.xxx.xxx:8080/
Subscribe to:
Posts (Atom)