Pages

Thursday, November 7, 2013

LVM and RAID

When it comes to managing storage on your computer, especially in professional settings, traditional methods can sometimes be limiting. This is where technologies like Logical Volume Management (LVM) and Redundant Array of Inexpensive Disks (RAID) come in handy. They offer flexible and powerful ways to handle your disk space.


LOGICAL VOLUME MANAGEMENT (LVM)

LVM allows you to manage your disk space in a more flexible way than traditional disk partitions. Instead of being stuck with fixed-size partitions on a single disk, LVM lets you create "logical" partitions that can span across multiple physical hard drives and be easily resized.


HOW LVM WORKS

LVM builds a layer of abstraction between your operating system and the physical storage.

  • Physical Volumes (PVs): Your physical disks or partitions are first converted into PVs. Think of these as the basic building blocks.

  • Volume Groups (VGs): PVs are then combined to form VGs. A VG can include PVs from different physical disks, creating a large pool of storage.

  • Logical Volumes (LVs): From these VGs, you create LVs, which are the "logical" partitions that your operating system sees and uses.


KEY ADVANTAGES OF LVM

LVM offers significant benefits over traditional disk partitioning:

  • Flexible Resizing: You can resize logical volumes (make them bigger or smaller) even while your system is running and applications are using them. This means less downtime for your services.

  • Data Migration: Easily move data from a failing or older physical disk to a newer one without taking your system offline.

  • Performance and Redundancy: Combine physical devices to improve performance (disk striping) or add redundancy for data protection (disk mirroring).

  • Snapshots: Create point-in-time "snapshots" of your logical volumes. This is incredibly useful for taking backups without interrupting your system's operations.


BASIC LVM COMMANDS

Here's a quick look at common LVM commands:

  • pvcreate: Initializes a disk or partition as a Physical Volume (PV).

    • Example: pvcreate /dev/hda

  • vgcreate: Creates a Volume Group (VG) from one or more PVs.

    • Example: vgcreate my_volume_group /dev/hda /dev/hdb

  • vgextend: Adds a PV to an existing VG.

    • Example: vgextend my_volume_group /dev/hdc

  • vgreduce: Removes a PV from a VG.

    • Example: vgreduce my_volume_group /dev/hdc (Note: This will also remove any LVs using space from that PV.)

  • lvcreate: Creates a Logical Volume (LV) from free space in a VG.

    • Example: lvcreate -n my_logical_volume --size 100G my_volume_group

  • lvextend: Extends the size of an LV.

    • Example: lvextend -L+20G /dev/my_volume_group/my_logical_volume

Once an LV is extended, you often need to extend the filesystem on it to utilize the new space. For example, with an ext3 filesystem, you might use ext2online.


LVM VERSIONS

There are different versions of LVM, with LVM2 being the current standard, offering more features and scalability compared to LVM1.

  • LVM1: Older, limited in size and number of volumes.

  • LVM2: Newer, supports much larger volumes (Exabytes) and more PVs/LVs, includes features like transactional metadata for faster recovery and volume mirroring.


REDUNDANT ARRAY OF INEXPENSIVE DISKS (RAID)

RAID combines multiple physical hard disks into a single logical unit. This is done by a "RAID controller," which can be hardware-based (transparent to the OS) or software-based (managed by the OS).


PURPOSE OF RAID

RAID aims to achieve one or more of the following:

  • Increased Storage Capacity: Combine smaller disks into one larger logical disk.

  • Improved Performance: Speed up read/write operations by distributing data across multiple disks.

  • Data Redundancy: Protect against data loss if one or more hard disks fail.


ADVANTAGES OF RAID

  • Cost-Effective: Often a more affordable way to achieve performance or redundancy compared to single, very expensive disks.

  • Flexibility: Easily adapt to changing storage needs by reconfiguring software RAID without major hardware changes.

  • Performance Boost: Certain RAID levels significantly improve data access speeds.

  • Data Protection: Other RAID levels provide excellent data redundancy, ensuring your data survives disk failures.


DISADVANTAGES OF RAID

  • Increased Complexity: RAID setups are more complex than using single disks, which can lead to issues with certain software or system recovery.

  • Trade-offs: Gains in performance often come at the cost of reduced redundancy, and vice versa.

  • Increased Wear: Spreading data across multiple disks can lead to more wear and tear, potentially increasing failure rates.

  • Backup Challenges: Backing up and restoring data on RAID arrays can sometimes be more difficult.


COMMON RAID LEVELS

Different RAID levels offer varying balances of performance, redundancy, and capacity.

  • RAID 0 (Striping):

    • How it works: Data is broken into chunks and written across all disks simultaneously.

    • Benefits: Excellent read/write performance.

    • Drawbacks: No redundancy. If one disk fails, all data is lost.

    • Best for: Non-critical data where speed is paramount (e.g., temporary files).

  • RAID 1 (Mirroring):

    • How it works: Data is duplicated (mirrored) across all disks. Each disk contains an identical copy.

    • Benefits: High redundancy. System continues to operate as long as one disk is functional.

    • Drawbacks: Reduced usable capacity (only the size of the smallest disk). Slower write performance due to data being written multiple times.

    • Best for: Critical data storage where data integrity is essential.

  • RAID 5:

    • How it works: Requires at least three disks. Data is striped across disks, and "parity" information (used for recovery) is distributed among them.

    • Benefits: Good balance of performance and redundancy. Can survive the loss of one disk.

    • Drawbacks: Slight CPU overhead for parity calculations.

    • Best for: Mission-critical scenarios requiring both good throughput and data integrity.

  • Linear RAID:

    • How it works: Disks are simply concatenated (joined end-to-end) to form one large volume. Data is written sequentially.

    • Benefits: Increases overall storage size beyond individual disk limits.

    • Drawbacks: No performance improvement, no redundancy. Loss of any disk means loss of the entire array.

    • Best for: Creating a single large volume from smaller, disparate disks, where neither performance nor redundancy is a primary concern.

  • Other Levels: More advanced levels like RAID 6 (dual parity) and nested levels (e.g., RAID 0+1, combining striping and mirroring) exist for highly specific and demanding environments.


Tuesday, October 22, 2013

Extending cPanel's Home Directory to a New Partition

Is your cPanel server's /home directory running out of space? This can stop you from creating new accounts or cause other issues. A common solution is to add a new hard drive and configure cPanel to use it for new user accounts. This guide explains how to do that simply.


PREPARING THE NEW PARTITION

Before cPanel can use the new space, you need to add and set up a new hard drive on your server.

Steps:

  1. Install New Hard Drive: Physically install the new hard drive into your server.

  2. Format the Drive: Format the new drive with a suitable file system (like ext4).

  3. Mount as /home2: Create a new directory, for example, /home2, and mount the new hard drive to this location. This makes the new storage accessible.


TELLING CPANEL TO USE THE NEW PARTITION

Once /home2 is ready, you have two main ways to tell cPanel to use it for new accounts. Both methods involve editing the cPanel configuration file: /etc/wwwacct.conf.


OPTION 1: DEDICATE NEW ACCOUNTS TO /home2

This option makes sure all future cPanel accounts are created directly in the new /home2 partition.

How to Configure:

  • Edit the /etc/wwwacct.conf file.

  • Find the line that says HOMEDIR.

  • Change its value to /home2.

Example:

HOMEDIR=/home2

Result:

After this change, any new cPanel accounts you create will automatically be placed in the /home2 directory. Existing accounts in /home will remain there.


OPTION 2: AUTO-SELECT BASED ON FREE SPACE

This is a more flexible option. cPanel will check both /home and /home2 and create new accounts on the partition with the most free space.

How to Configure:

  • Edit the /etc/wwwacct.conf file.

  • Find the line that says HOMEMATCH.

  • Set its value to /home*.

Example:

HOMEMATCH=/home*

Result:

With this setting, when a new account is created, cPanel will intelligently decide whether to put it in /home or /home2 based on which directory has more available disk space. This helps distribute user data more evenly across your storage.

Monday, October 14, 2013

NCftp - get multiple Folders with ftp

    The NcFTP client is a powerful tool for transferring files over FTP, especially useful for system administrators and developers. It offers advanced features like recursive directory downloads, making it a great alternative to standard FTP clients. This guide will walk you through installing NcFTP and using its recursive download capabilities.


    INSTALLING NCFTP

    NcFTP is compatible with various Unix-like operating systems, including FreeBSD, Solaris, and most Linux distributions.

    On Debian/Ubuntu Systems: You can easily install NcFTP using the apt-get package manager. Open your terminal and run the following command:

    sudo apt-get install ncftp
    

    This command will download and install the NcFTP client on your system.


    RECURSIVE DIRECTORY DOWNLOADS WITH NCFTPGET

    ncftpget is a command-line tool within the NcFTP suite designed for efficient file transfers, particularly for scripting and advanced usage. It allows you to download entire directories and their contents recursively.

    Basic Recursive Download Command: To download a remote directory and all its subdirectories and files, use the following format:

    ncftpget -R -v -u "USERNAME" ftp.server.com /local/directory /remote/directory
    

    Let's break down the options:

    • -R: This crucial option tells ncftpget to copy all subdirectories and files recursively from the remote server.

    • -v: Stands for "verbose." This displays detailed download activity and progress in your terminal, which is helpful for monitoring transfers.

    • -u "USERNAME": Specifies the username for logging into the FTP server. If you omit this, ncftpget will attempt to log in anonymously.

    • ftp.server.com: Replace this with the actual hostname or IP address of your FTP server.

    • /local/directory: This is the path on your local machine where you want to save the downloaded files.

    • /remote/directory: This is the path to the directory on the remote FTP server that you wish to copy.

    Example: If you wanted to download the /www-data directory from ftp.nixcraft.net to your local /home/vivek/backup directory, using the username ftpuser, the command would be:

    ncftpget -R -v -u "ftpuser" ftp.nixcraft.net /home/vivek/backup /www-data
    

    TROUBLESHOOTING: "TAR MODE" ERROR

    Sometimes, when performing recursive downloads, you might encounter an error message similar to this:

    tar: End of archive volume 1 reached
    tar: Sorry, unable to determine archive format.
    Could not read directory listing data: Connection reset by peer
    

    This error often indicates that ncftpget is attempting to use "TAR mode" for the recursive transfer, which might not be supported or configured correctly on the FTP server, or it's encountering issues with directory listings.

    Solution: Disable TAR Mode To resolve this, you can add the -T option to your ncftpget command. The -T option explicitly tells ncftpget not to try using TAR mode with recursive mode.

    Revised Command for Error Resolution:

    ncftpget -T -R -v -u "ftpuser" ftp.nixcraft.net /home/vivek/backup /www-data
    

    By adding -T, you ensure that ncftpget uses a different method for recursive downloads, bypassing the TAR mode issue.

Saturday, September 21, 2013

Add more space to /tmp in cPanel server.

The following could help you to increase more space in /tmp.

You need to make alternation in the file /scripts/securetmp

#vi /scripts/securetmp

Find the entry my $tmpdsksize under Global Variables as follows:
# Global Variables
my $tmpdsksize = 512000; # Must be larger than 250000

Change the value for that particular entry to desired size.

Then make sure that no processes are using /tmp using the command, lsof /tmp

Please stop the service /etc/init.d/mysql stop. Also delete the file, /usr/tmpDSK if it exists by rm -rf /usr/tmpDSK

Then

umount /tmp

Run the script

#/scripts/securetmp

Then you will asked for some confirmation steps.

“Would you like to secure /tmp at boot time?” Press y

“Would you like to secure /tmp now?” Press y

Eventually you can see the upgraded space to /tmp in server :)

 

//////////////////////////////////////////////

1.) Stop MySql service and process kill the tailwatchd process.

[root@server ~]# /etc/init.d/mysqld stop
Stopping MySQL: [ OK ]
[root@server ~]# pstree -p | grep tailwatchd
Find the tailwatchd process id and kill it
[root@server ~]# kill -9 2522
2.) Take a backup of /tmp as /tmp.bak

[root@server ~]#cp -prf /tmp /tmp.bak
3.) Create a 2GB file in the avaliable freespace

[root@server ~]# dd if=/dev/zero of=/usr/tmpDSK bs=1024k count=2048
2048Æ® records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 73.6908 seconds, 29.1 MB/s
[root@server~]# du -sch /usr/tmpDSK
2.1G /usr/tmpDSK
2.1G total
4.) Assign ext3 filesystem to the file

[root@server~]# mkfs -t ext3 /usr/tmpDSK
mke2fs 1.39 (29-Mayñ®–¦)
/usr/tmpDSK is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
262144 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
5.) Check the file system type:-

[root@server ~]# file /usr/tmpDSK
/usr/tmpDSK: Linux rev 1.0 ext3 filesystem data (large files)
Note:-

You may also use the following comands for making ext3 file system on a file:

[root@server ~]# mkfs.ext3 /usr/tmpDSK
[root@server ~]# mke2fs /usr/tmpDSK
6.) Unmount /tmp partition

[root@server ~]# umount /tmp
7.) Mount the new /tmp filesystem with noexec

[root@server ~]# mount -o loop,noexec,nosuid,rw /usr/tmpDSK /tmp
8.) Set the correct permission for /tmp

[root@server ~]# install -d –mode=1777 /tmp
[root@antg ~]# ls -ld /tmp
drwxrwxrwt 3 root root 4096 Feb 6 08:42 /tmp
( you may use the command chmod 1777 /tmp for doing the same )

[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 28G 6.4G 20G 25% /
/dev/sda8 99M 10M 84M 11% /boot
tmpfs 500M 0 500M 0% /dev/shm
/usr/tmpDSK 2.0G 68M 1.9G 4% /tmp
9.)Restore the content of old /tmp.bkp directory

[root@server ~]# cp -rpf /tmp.bak/* /tmp
10.) Restart the mysql and tailwathchd services.

[root@server ~]# /etc/init.d/mysql start
[root@server ~]# /scripts/restartsrv_tailwatchd
11.)Edit the fstab and replace /tmp entry line with :-

/usr/tmpDSK /tmp ext3 loop,noexec,nosuid,rw 0 0
12.) Mount all filesystems

[root@server~]# mount -a
Check it now:-

[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 28G 6.4G 20G 25% /
/dev/sda8 ɃM 10M 84M 11% /boot
tmpfs 500M 0 500M 0% /dev/shm
/usr/tmpDSK 2.0G 68M 1.9G 4% /tmp

really hope this little tutoral can help you:)

 

 

Wednesday, September 18, 2013

How to monitor file access on Linux with "auditd"

For critical servers and sensitive data, keeping an eye on who accesses or changes files is essential for security. The Linux Audit System, with its auditd daemon, helps you do just that. It monitors system calls and logs them, providing a detailed trail of file activity.


INSTALLING AUDITD

auditd is available in the package repositories for most Linux distributions.

  • Debian, Ubuntu, Linux Mint:

    sudo apt-get install auditd

    On these systems, auditd usually starts automatically on boot after installation.

  • Fedora, CentOS, RHEL:

    sudo yum install audit

    To ensure auditd starts automatically on boot for these distributions:

    sudo chkconfig auditd on

CONFIGURING AUDITD

You can configure auditd using the auditctl command-line utility or by editing its configuration file, /etc/audit/audit.rules. This guide focuses on editing the configuration file.

  • Edit the Configuration File:

    sudo vi /etc/audit/audit.rules
  • Example Configuration (/etc/audit/audit.rules):

    # First rule - delete all existing rules
    -D
    
    # Increase buffer size for busy systems to prevent lost events
    -b 1024
    
    # Monitor when files or directories are deleted (unlink and rmdir system calls)
    -a exit,always -S unlink -S rmdir
    
    # Monitor file open attempts by a specific Linux User ID (UID 1001)
    -a exit,always -S open -F loginuid=1001
    
    # Monitor write-access and changes to file properties (permissions) for critical files
    -w /etc/group -p wa
    -w /etc/passwd -p wa
    -w /etc/shadow -p wa
    -w /etc/sudoers -p wa
    
    # Monitor read-access to a specific sensitive directory
    -w /etc/secret_directory -p r
    
    # Lock the audit configuration to prevent unauthorized modifications until reboot
    -e 2
    
    • -D: Clears all previous rules.

    • -b 1024: Sets the buffer size. Increase this for active systems to avoid missing events.

    • -a exit,always -S <syscall>: Monitors specific system calls.

    • -F loginuid=<UID>: Filters events by the user ID logged in.

    • -w <path> -p <permissions>: Sets a watch on a file or directory.

      • w: write access

      • a: attribute change (e.g., permissions)

      • r: read access

    • -e 2: Puts auditd into immutable mode, preventing rule changes until reboot.

  • Restart Auditd: After making changes to /etc/audit/audit.rules, restart the service for them to take effect.

    sudo service auditd restart

ANALYZING AUDIT LOGS

auditd logs its findings to /var/log/audit/audit.log. The ausearch command-line tool is used to query these logs.

  • Check File Access: To see if /etc/passwd has been accessed or modified:

    sudo ausearch -f /etc/passwd

    The output will show details like the time, type of event, user, process, and system call involved. For example, you might see an entry indicating chmod was applied to /etc/passwd by root.

  • Check Directory Access: To see if /etc/secret_directory has been accessed:

    sudo ausearch -f /etc/secret_directory

    This will show events like ls commands being run within that directory by a specific UID.


IMMUTABLE MODE AND RULE MODIFICATION

If you set auditd to immutable mode (-e 2), you cannot modify the rules and restart the service without a reboot.

  • Error Message in Immutable Mode:

    Error deleting rule (Operation not permitted)
    The audit system is in immutable mode, no rules loaded
    
  • To Modify Rules in Immutable Mode:

    1. Edit /etc/audit/audit.rules.

    2. Reboot your machine. The new rules will be loaded upon restart.


LOG ROTATION

Audit logs can grow large quickly. It's recommended to enable daily log rotation.

  • Rotate Audit Logs Daily (for cronjob):

    sudo service auditd rotate

    You can add this command to a daily cron job to ensure logs are rotated regularly, preventing the /var/log/audit directory from filling up.

Wednesday, September 4, 2013

Resetting Your SSH Port Through WHM

Your SSH (Secure Shell) port is like a specific door on your server that allows secure remote access. Sometimes, you might need to change or reset it, for example, after a security configuration change or to resolve connection issues.


WHY RESET YOUR SSH PORT?

  • Security: If you've made changes to your SSH configuration (like changing the default port), resetting the service ensures those changes take effect.

  • Troubleshooting: Sometimes, a simple restart can resolve minor connection glitches.


HOW TO RESET YOUR SSH PORT VIA WHM

This process leverages WHM's built-in tools to safely restart the SSH service.

  1. Log in to WHM:

    • Open your web browser.

    • Go to http://your_server_ip:2086 (replace your_server_ip with your server's actual IP address).

    • Enter your WHM username and password.

  2. Access the SSH Reset Tool:

    • Once logged into WHM, browse directly to the following URL: http://your_server_ip:2086/scripts2/doautofixer?autofix=safesshrestart

    • Make sure to replace your_server_ip with your server's actual IP address.

  3. Confirmation:

    • After navigating to the URL, WHM will attempt to safely restart the SSH service.

    • You should see a message indicating the status of the operation, usually confirming that the SSH service has been restarted or reloaded.


IMPORTANT CONSIDERATIONS

  • Server IP: Always ensure you replace your_server_ip with the correct IP address of your server.

  • WHM Access: You need administrative access to WHM to perform this action.

  • SSH Configuration: This process primarily restarts the SSH service to apply existing configuration changes. It does not change the SSH port itself. To change the SSH port, you would typically edit the sshd_config file and then use this method to restart the service.