Tuesday, October 27, 2015
Monday, October 12, 2015
AWS S3 : Limiting Access from Certain IP's
So - based on this design, we needed a way to only allow access to a set of buckets from this single IP address. For example purposes we are using the IP of 72.309.38.2.
IF you were to follow the initial example laid out by the AWS documentation - you’ll end up with a policy that probably looks similar to this.
{
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-wicked-awesome-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "72.309.38.2/32"
}
}
}
]
}
What you’re going to find, after banging your head on the table a few times, is that this policy does not work. There does not appear to be an implied deny rule with S3 buckets (similar to how IAM access policies are setup). I did my testing with the s3-curl command - When you have that tool setup - you can make a query like the following:
./s3curl.pl --head --id=AKIAYOURACCESSKEY --key=YourSecretKey -- https://s3.amazonaws.com/my-wicked-awesome-bucket/even-more-awesome.file
Using the policy above - returned this:
HTTP/1.1 200 OK
x-amz-id-2: BLPqeibX8nZGnSDNi9zRhb+6a8fDiOW6Ij1OXhadWknJKCX9WAb7x1xNETvdXAEv
x-amz-request-id: F6AF36D912E6003B
Date: Fri, 27 Apr 2012 00:25:52 GMT
Last-Modified: Tue, 17 Apr 2012 17:50:11 GMT
ETag: "8553074962ba71d8b2b600b971ba80a8"
Accept-Ranges: bytes
Content-Type: application/pdf
Content-Length: 1451065
Server: AmazonS3
After troubleshooting this with the AWS support team - they let me know of the following:
By default accounts are restricted from accessing S3 unless they have been given access via policy. However, S3 is designed by default to allow any IP address access. So to block IP's you would have to specify denies explicitly in the policy instead of allows.
Once I learned this - the policy was easy to adjust. I flipped around the policy from allowing access from only my IP address to denying access from everywhere that was NOT my IP address.
My new policy looked like this:
{
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPDeny",
"Effect": "Deny",
"Principal": {
"AWS": "*"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-wicked-awesome-bucket/*",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": "72.309.38.2/32"
}
}
}
]
}
And when I ran my s3-curl command again - I now received the following response.
HTTP/1.1 403 Forbidden
x-amz-request-id: 3B90483F655AA692
x-amz-id-2: 0XEO2miEzZEwbU2N5zPwzLtX4J7BJx/LLXpvSLPpQ7e2wuocx49katsD+ZcqwYA7
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 27 Apr 2012 00:27:14 GMT
Server: AmazonS3
Saturday, August 15, 2015
Configure MySql and ODBC Connector In Windows 2012 Server
Download The Microsoft Visual C++ 2010 Redistributable
http://www.microsoft.com/en-gb/download/details.aspx?id=14632
Download and install the ODBC Connector
http://cdn.mysql.com/Downloads/Connector-ODBC/5.3/mysql-connector-odbc-5.3.4-winx64.msi
Downoading the MySQL
http://cdn.mysql.com/Downloads/MySQL-5.6/mysql-5.6.26-winx64.zip
c:\mysql>cd bin
c:\mysql\bin>mysqld --install
Service successfully installed.
Go to Services and start the Mysql Services.
c:\mysql\bin>mysql -u root -p
>>No password is needed for first time
Set the root Password
=====================
mysql>
mysql> UPDATE mysql.user SET password=PASSWORD('redhat') WHERE user='root';
Query OK, 3 rows affected (0.00 sec)
Rows matched: 3 Changed: 3 Warnings: 0
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> select user,host from mysql.user;
+------+-----------+
| user | host |
+------+-----------+
| root | 127.0.0.1 |
| root | ::1 |
| | localhost |
| root | localhost |
+------+-----------+
4 rows in set (0.00 sec)
mysql> delete from mysql.user where user='';
Query OK, 1 row affected (0.00 sec)
mysql> select user,host from mysql.user;
+------+-----------+
| user | host |
+------+-----------+
| root | 127.0.0.1 |
| root | ::1 |
| root | localhost |
+------+-----------+
3 rows in set (0.00 sec)
mysql> create database fileserver;
Query OK, 1 row affected (0.01 sec)
mysql>GRANT ALL PRIVILEGES ON fileserver.* TO 'fileserver'@'localhost' IDENTIFIED BY 'fileserver';
Query OK, 0 rows affected (0.02 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)
mysql>
Wednesday, August 12, 2015
Mysql Cluster Using Mysql NDB
OS used is RHEL7
Selinux Enabed
Firewall Disabled
Management Server## Perform the Following steps in both the Management Server's.
Make Directories and Download the Cluster Files
====================================
mkdir /usr/src/mysql-mgm
cd /usr/src/mysql-mgm
wget http://cdn.mysql.com/Downloads/MySQL-Cluster-7.4/mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
tar zxvf mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
cd mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686
cp bin/ndb_mgm* /usr/bin/
chmod 755 /usr/bin/ndb_mgm*
mkdir /var/lib/mysql-cluster
vi /var/lib/mysql-cluster/config.ini
==========================================
[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=80M
IndexMemory=18M
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
DataDir=/var/lib/mysql-cluster
[TCP DEFAULT]
# Section for the cluster management node
[NDB_MGMD]
NodeId=1
# IP address of the first management node (this system)
HostName=192.168.70.130
[NDB_MGMD]
NodeId=2
#IP address of the second management node
HostName=192.168.70.131
# Section for the storage nodes
[NDBD]
# IP address of the first storage node
HostName=192.168.70.132
DataDir= /var/lib/mysql-cluster
[NDBD]
# IP address of the second storage node
HostName=192.168.70.133
DataDir=/var/lib/mysql-cluster
# one [MYSQLD] per storage node
[MYSQLD]
[MYSQLD]
==========================================
chown mysql. /var/lib/mysql-cluster -R
To start the Management Service
========================
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --configdir=/var/lib/mysql-cluster/
Data And SQL Server#Perform this on both of the Server's
==============================================
Install the needed Packages
====================
yum install libaio.i686 libaio-devel.i686 -y
yum install perl -y
yum -y install perl-Data-Dumper
Download the packages
cd /usr/local/
wget http://cdn.mysql.com/Downloads/MySQL-Cluster-7.4/mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
tar zxvf mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz
mv /root/mysql-cluster-gpl-7.4.7-linux-glibc2.5-i686.tar.gz mysql
chown mysql. mysql -R
cd mysql
Initializing the database
scripts/mysql_install_db --user=mysql --datadir=/usr/local/mysql/data
cp support-files/mysql.server /etc/init.d/
chmod 755 /etc/init.d/mysql.server
cd /usr/local/mysql/bin
mv * /usr/bin
cd ../
vi /etc/my.cnf
============
[mysqld]
ndbcluster
# IP address of the cluster management node
ndb-connectstring=192.168.70.130,192.168.70.131
[mysql_cluster]
# IP address of the cluster management node
ndb-connectstring=192.168.70.130,192.168.70.131
============
mkdir /var/lib/mysql-cluster
cd /var/lib/mysql-cluster
ndbd --initial
/etc/init.d/mysql.server start
After this, secure the MySQL installation by running the appropriate script:
/usr/local/mysql/bin/mysql_secure_installation
Testing
In the Management Node check the command ndb_mgm and check the status
Thursday, July 9, 2015
Delete a nat rule in iptables
First of all I list all the rules including line numbers like this;
iptables -L -t nat –line-numbers
I then look at the output that will be similar to the below
In this example lets say I want to delete rule number 2 in the PREROUTING chain, I would enter the following;
iptables -t nat -D PREROUTING 2
In English the above line means remove line number 2 from the PREOUTING chain, I would then run the first command again to check my iptables file, then save the iptables file and restart the iptables service.
iptables -L -t nat –line-numbers
service iptables save
service iptables restart
All the above is carried out running on Centos, you may have to edit slightly for your particular distribution.
Wednesday, July 8, 2015
Linux tune the VM subsystem.
Tuning the memory subsystem in Linux is a powerful but delicate task. The right settings can boost your system’s performance, but incorrect changes may cause instability or slowdowns. Always adjust one parameter at a time and monitor your system before making further changes.
Exploring /proc/sys/vm
The /proc/sys/vm directory contains files that represent kernel parameters for the virtual memory subsystem. You can read and write to these files to tune system behavior.
To view the files, use:
cd /proc/sys/vm
ls -l
Sample output:
-rw-r--r-- 1 root root 0 Oct 16 04:21 block_dump
-rw-r--r-- 1 root root 0 Oct 16 04:21 dirty_background_ratio
-rw-r--r-- 1 root root 0 Oct 16 04:21 dirty_expire_centisecs
-rw-r--r-- 1 root root 0 Oct 16 04:21 dirty_ratio
-rw-r--r-- 1 root root 0 Oct 16 04:21 dirty_writeback_centisecs
-rw-r--r-- 1 root root 0 Oct 16 04:21 drop_caches
-rw-r--r-- 1 root root 0 Oct 16 04:21 swappiness
-rw-r--r-- 1 root root 0 Oct 16 04:21 vfs_cache_pressure
...
Key Parameters and Their Effects
dirty_background_ratio
Purpose: Sets the percentage of system memory filled with “dirty” pages (pages to be written to disk) before the background writeback daemon (pdflush) starts writing them out.
Check current value:
sysctl vm.dirty_background_ratio
Default example:
vm.dirty_background_ratio = 10
Tuning:
Increasing this value (for example, to 20) means less frequent flushes, which may benefit systems with fast disks but can cause larger flushes at once.
sysctl -w vm.dirty_background_ratio=20
swappiness
Purpose: Controls how aggressively the kernel swaps memory pages to disk.
Check current value:
sysctl vm.swappiness
Default example:
vm.swappiness = 60
Tuning:
Lower values reduce swapping (good for desktops), higher values increase swapping (can benefit workloads with long-sleeping processes).
sysctl -w vm.swappiness=100
dirty_ratio
Purpose: Sets the percentage of system memory that can be filled with dirty pages before processes generating writes must themselves start writing data to disk.
Check current value:
sysctl vm.dirty_ratio
Default example:
vm.dirty_ratio = 40
Tuning:
Lowering this value (for example, to 25) causes data to be written to disk more frequently, reducing the risk of large data loss but possibly impacting performance.
sysctl -w vm.dirty_ratio=25
Best Practices for VM Tuning
Change one setting at a time.
Monitor system performance after each change using tools like vmstat, top, or free.
If performance improves, keep the new setting. If not, revert to the previous value.
Document your changes for future reference and troubleshooting.
CloudStack Installation and Configuration
Managment Server
service iptables stop
chkconfig iptables off
echo " HOSTNAME=controller.example.com" > /etc/hostname
cat /etc/hostname
echo "142.0.42.46 controller.example.com controller " >> /etc/hosts
echo " HOSTNAME=controller.example.com" >> /etc/sysconfig/network
cat /etc/hosts
hostname controller.example.com
hostname
ping -c 3 controller
yum install -y yum-plugin-priorities gedit curl wget nc
yum -y install policycoreutils setroubleshoot
sed -i "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/sysconfig/selinux
sed -i "s/SELINUX=disabled/SELINUX=permissive/g" /etc/sysconfig/selinux
setenforce 0
# vi /etc/yum.repos.d/cloudstack.repo
[cloudstack]
name=cloudstack
baseurl=http://cloudstack.apt-get.eu/rhel/4.2/
enabled=1
gpgcheck=0
yum -y install ntp
service ntpd start
chkconfig ntpd on
yum -y install mysql mysql-server MySQL-python
service mysqld start
chkconfig mysqld on
mysql_install_db
mysql_secure_installation
Downloading vhd-util
wget http://download.cloud.com.s3.amazonaws.com/tools/vhd-util
If the Management Server is RHEL or CentOS, copy vhd-util to /usr/lib64/cloud/common/scripts/vm/hypervisor/xenserver.
If the Management Server is Ubuntu, copy vhd-util to /usr/lib/cloud/common/scripts/vm/hypervisor/xenserver/vhd-util.
yum -y install cloud-client
cloudstack-setup-databases cloud:cloud@localhost --deploy-as=root:mysql-password -i 142.0.42.46
cloudstack-setup-management
Mostly by default the the Dashboard password will be admin and password.
yum -y install nfs-utils
mkdir -p /export/primary
mkdir -p /export/secondary
vi /etc/exports
/export *(rw,async,no_root_squash,no_subtree_check)
# vi /etc/sysconfig/nfs
LOCKD_TCPPORT=32803
LOCKD_UDPPORT=32769
MOUNTD_PORT=892
RQUOTAD_PORT=875
STATD_PORT=662
STATD_OUTGOING_PORT=2020
service rpcbind start
service nfs start
chkconfig nfs on
chkconfig rpcbind on
mkdir -p /mnt/primary
mkdir -p /mnt/secondary
mount -t nfs 142.0.42.46:/export/primary /mnt/primary
mount -t nfs 142.0.42.46:/export/secondary /mnt/secondary
Create a System virtual machine template, you also can get it from official site.
Usage: cloud-install-sys-tmplt: -m <secondary storage mount point> -f <system vm template file> [-h <hypervisor name: kvm|vmware|xenserver> ] [ -s <mgmt server secret key, if you specified any when running cloudstack-setup-database, default is password>][-u <Url to system vm template>] [-F <clean up system templates of specified hypervisor>] [-e <Template suffix, e.g vhd, ova, qcow2>] [-o <Database server hostname or ip, e.g localhost>] [-r <Database user name, e.g root>] [-d <Database password. Fllowed by nothing if the password is empty>]
or
cloud-install-sys-tmplt: -m <secondary storage mount point> -u <http url for system vm template> [-h <hypervisor name: kvm|vmware|xenserver> ] [ -s <mgmt server secret key>]
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /mnt/secondary -u http://d21ifhcun6b1t2.cloudfront.net/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2 -h kvm -s -F
On The Compute Node
service iptables stop
chkconfig iptables off
echo " HOSTNAME=compute.example.com" > /etc/hostname
cat /etc/hostname
echo "142.0.42.46 compute.example.com controller " >> /etc/hosts
echo " HOSTNAME=compute.example.com" >> /etc/sysconfig/network
cat /etc/hosts
hostname compute.example.com
hostname
ping -c 3 controller
vi /etc/yum.repos.d/cloudstack.repo
[cloudstack]
name=cloudstack
baseurl=http://cloudstack.apt-get.eu/rhel/4.2/
enabled=1
gpgcheck=0
yum -y install ntp
yum -y install cloudstack-agent
yum -y install qemu-kvm
Modify the libvirt configuration files, remove the following comments, change the value of auth_tcp to “none”
vi /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"
mdns_adv = 0