NCftp - get multiple Folders with ftp

The NcFTP client is a powerful tool for transferring files over FTP, especially useful for system administrators and developers. It offers advanced features like recursive directory downloads, making it a great alternative to standard FTP clients. This guide will walk you through installing NcFTP and using its recursive download capabilities.

---

INSTALLING NCFTP

NcFTP is compatible with various Unix-like operating systems, including FreeBSD, Solaris, and most Linux distributions.

On Debian/Ubuntu Systems: You can easily install NcFTP using the apt-get package manager. Open your terminal and run the following command:

sudo apt-get install ncftp

This command will download and install the NcFTP client on your system.

---

RECURSIVE DIRECTORY DOWNLOADS WITH NCFTPGET

ncftpget is a command-line tool within the NcFTP suite designed for efficient file transfers, particularly for scripting and advanced usage. It allows you to download entire directories and their contents recursively.

Basic Recursive Download Command: To download a remote directory and all its subdirectories and files, use the following format:

ncftpget -R -v -u "USERNAME" ftp.server.com /local/directory /remote/directory

Let's break down the options:

• -R: This crucial option tells ncftpget to copy all subdirectories and files recursively from the remote server.

• -v: Stands for "verbose." This displays detailed download activity and progress in your terminal, which is helpful for monitoring transfers.

• -u "USERNAME": Specifies the username for logging into the FTP server. If you omit this, ncftpget will attempt to log in anonymously.

• ftp.server.com: Replace this with the actual hostname or IP address of your FTP server.

• /local/directory: This is the path on your local machine where you want to save the downloaded files.

• /remote/directory: This is the path to the directory on the remote FTP server that you wish to copy.

Example: If you wanted to download the /www-data directory from ftp.nixcraft.net to your local /home/vivek/backup directory, using the username ftpuser, the command would be:

ncftpget -R -v -u "ftpuser" ftp.nixcraft.net /home/vivek/backup /www-data

---

TROUBLESHOOTING: "TAR MODE" ERROR

Sometimes, when performing recursive downloads, you might encounter an error message similar to this:

tar: End of archive volume 1 reached
tar: Sorry, unable to determine archive format.
Could not read directory listing data: Connection reset by peer

This error often indicates that ncftpget is attempting to use "TAR mode" for the recursive transfer, which might not be supported or configured correctly on the FTP server, or it's encountering issues with directory listings.

Solution: Disable TAR Mode To resolve this, you can add the -T option to your ncftpget command. The -T option explicitly tells ncftpget not to try using TAR mode with recursive mode.

Revised Command for Error Resolution:

ncftpget -T -R -v -u "ftpuser" ftp.nixcraft.net /home/vivek/backup /www-data

By adding -T, you ensure that ncftpget uses a different method for recursive downloads, bypassing the TAR mode issue.

HIDING BACKGROUND PROCESS OUTPUT: THE /DEV/NULL TRICK

When running a command on a Linux or Unix-like system, the program often produces output:

1. Standard Output (stdout): Normal results, messages, or requested data (File Descriptor 1).

2. Standard Error (stderr): Warnings, errors, and diagnostic messages (File Descriptor 2).

When you run a long-running process in the background, this output can clutter your console, even after you move on to other tasks.

---

THE SOLUTION: REDIRECTING TO /DEV/NULL

To hide all output and keep your terminal clean, you must redirect both stdout and stderr to a special location called /dev/null.

/dev/null is a virtual device known as the "bit bucket" or "black hole." Any data written to it is instantly discarded.

THE STANDARD REDIRECTION COMMAND

The standard, most effective command to silence all output is:

yourcommand > /dev/null 2>&1

How This Works:

1. > /dev/null: This redirects Standard Output (stdout, or FD 1) to the black hole (/dev/null).

2. 2>&1: This redirects Standard Error (stderr, or FD 2) to the same place where stdout (FD 1) is currently pointing. Since stdout is pointing at /dev/null, stderr is also sent there and silenced.

---

RUNNING IN THE BACKGROUND (THE FULL COMMAND)

To combine output hiding with background execution, you simply add the ampersand (&) at the end.

yourcommand > /dev/null 2>&1 &

This command accomplishes two things:

1. Silences All Output: (> /dev/null 2>&1)

2. Sends to Background: (&) - It immediately returns control of the terminal to you, allowing the process to run independently.

---

PERSISTENT BACKGROUND EXECUTION (NOHUP)

If you are running a command in the background and want it to continue running even after you log out or close your terminal session (a common need for server-side tasks), use nohup.

The Command for Persistent, Silent Background Task:

nohup yourcommand > /dev/null 2>&1 &

How This Works:

• nohup: Ensures the process ignores the "hang up" signal sent when a terminal is closed.

• > /dev/null 2>&1: Silences all output (otherwise, nohup would create a file called nohup.out).

• &: Runs the entire sequence in the background.

This is the ideal way to run a Linux application in the background that you expect to survive your session and generate no console noise.

Rsync command

Rsync (Remote Sync) is a powerful utility for copying and synchronizing files and directories, particularly across networked locations. Unlike simple copy commands, Rsync is designed for speed and efficiency by only transferring the parts of files that have actually changed.

This makes it the go-to tool for backups, file migration, and maintaining synchronized mirrors between servers.

---

THE CORE CONCEPT: DIFFERENTIAL COPYING

The key to Rsync's efficiency is its "delta transfer" algorithm.

• Checks for Differences: Rsync first compares the files in the source location to the files in the destination location. It does this quickly by comparing file sizes and modification times.

• Transfers Only the Delta: Instead of copying entire files that have been modified, Rsync calculates the differences (the "delta") within the files. It only transfers those small blocks of changed data, dramatically reducing the amount of network bandwidth and time required.

KEY BENEFIT: RESUMABLE TRANSFERS

One of Rsync's most valuable features is its ability to handle interruptions gracefully:

• If a large transfer is interrupted (e.g., due to a lost network connection, power failure, or timeout), you can simply re-run the exact same rsync command.

• Rsync will pick up where it left off, checking files and continuing the sync from the point of interruption, saving you from restarting the entire copy process.

---

UNDERSTANDING THE COMMAND SYNTAX

The basic Rsync command follows a simple pattern:

rsync [OPTIONS] [SOURCE] [DESTINATION]

EXAMPLE BREAKDOWN

The provided example is a very common and practical use of Rsync:

rsync -av source/public_html/ destination/public_html/

1. The Source and Destination:

• source/public_html/: The location where the files currently reside (the directory to be copied from).

• destination/public_html/: The location where the files should be copied to (the target directory).

2. The Options (-av):

The power of Rsync lies in its flags, with -av being the most widely used combination for general synchronization.

• -a (Archive Mode): This is a shorthand for several important flags (-rlptgoD). It ensures the synchronization is comprehensive and preserves file integrity by:

  • recursing into directories.

  • linking files.

  • preserving permissions.

  • time-stamps (modification times).

  • group ownership.

  • owner (user) ownership.

  • Device files and special files.

• -v (Verbose): This simply tells Rsync to output detailed information about the files being transferred, giving you confirmation and real-time progress.

Using this command ensures that the files in the destination directory are an exact, efficient, and attribute-preserving mirror of the source directory.

Install Grsecurity on 32 bit OS is : Kernel

Ideal way to install Grsecurity on 32 bit OS is : Just follow the steps given below.

Fetch the sources:

Download kernel from kernel.org

#wget http://www.kernel.org/pub/linux/kernel/v2.6/longterm/v2.6.32/linux-2.6.32.51.tar.gz

Downlaod latest Grsecurity patch from below URL :

#wget http://grsecurity.net/stable/grsecurity-2.2.2-2.6.32.51-201201021326.patch

Extract:
tar xjf linux-2.6.32.51.tar.gz

Patch the kernel:

#cd linux-2.6.32.51

#patch -p1 < ../grsecurity-2.2.2-2.6.32.51-201201021326.patch

Now start making the kernel :

# make clean && make mrproper

Edit your kernel as per your need :

# make menuconfig

Compile your kernel and install it:

# make bzImage

# make modules

# make modules_install

Make sure it’s working ok with the help of following command :

# depmod 2.6.32.51-grsec

Installing and booting the new kernel :

# cp arch/i386/boot/bzImage /boot/vmlinuz-2.6.32.51-grsec

There is also a file called “System.map” that must be copied to the same boot directory.

# cp System.map /boot

Do not forget to make changes in /etc/grub.conf

also go to grub prompt after this and fire below command :

# grub > savedefault –-default=0 –-once

Now reboot server :

#Shutdown -r now.

Add more space to /tmp in cPanel server.

The following could help you to increase more space in /tmp.

You need to make alternation in the file /scripts/securetmp

#vi /scripts/securetmp

Find the entry my $tmpdsksize under Global Variables as follows:
# Global Variables
my $tmpdsksize = 512000; # Must be larger than 250000

Change the value for that particular entry to desired size.

Then make sure that no processes are using /tmp using the command, lsof /tmp

Please stop the service /etc/init.d/mysql stop. Also delete the file, /usr/tmpDSK if it exists by rm -rf /usr/tmpDSK

Then

umount /tmp

Run the script

#/scripts/securetmp

Then you will asked for some confirmation steps.

“Would you like to secure /tmp at boot time?” Press y

“Would you like to secure /tmp now?” Press y

Eventually you can see the upgraded space to /tmp in server :)

 

//////////////////////////////////////////////

1.) Stop MySql service and process kill the tailwatchd process.

[root@server ~]# /etc/init.d/mysqld stop
Stopping MySQL: [ OK ]
[root@server ~]# pstree -p | grep tailwatchd
Find the tailwatchd process id and kill it
[root@server ~]# kill -9 2522
2.) Take a backup of /tmp as /tmp.bak

[root@server ~]#cp -prf /tmp /tmp.bak
3.) Create a 2GB file in the avaliable freespace

[root@server ~]# dd if=/dev/zero of=/usr/tmpDSK bs=1024k count=2048
2048Ʈ records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 73.6908 seconds, 29.1 MB/s
[root@server~]# du -sch /usr/tmpDSK
2.1G /usr/tmpDSK
2.1G total
4.) Assign ext3 filesystem to the file

[root@server~]# mkfs -t ext3 /usr/tmpDSK
mke2fs 1.39 (29-May񮖦)
/usr/tmpDSK is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
262144 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
5.) Check the file system type:-

[root@server ~]# file /usr/tmpDSK
/usr/tmpDSK: Linux rev 1.0 ext3 filesystem data (large files)
Note:-

You may also use the following comands for making ext3 file system on a file:

[root@server ~]# mkfs.ext3 /usr/tmpDSK
[root@server ~]# mke2fs /usr/tmpDSK
6.) Unmount /tmp partition

[root@server ~]# umount /tmp
7.) Mount the new /tmp filesystem with noexec

[root@server ~]# mount -o loop,noexec,nosuid,rw /usr/tmpDSK /tmp
8.) Set the correct permission for /tmp

[root@server ~]# install -d –mode=1777 /tmp
[root@antg ~]# ls -ld /tmp
drwxrwxrwt 3 root root 4096 Feb 6 08:42 /tmp
( you may use the command chmod 1777 /tmp for doing the same )

[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 28G 6.4G 20G 25% /
/dev/sda8 99M 10M 84M 11% /boot
tmpfs 500M 0 500M 0% /dev/shm
/usr/tmpDSK 2.0G 68M 1.9G 4% /tmp
9.)Restore the content of old /tmp.bkp directory

[root@server ~]# cp -rpf /tmp.bak/* /tmp
10.) Restart the mysql and tailwathchd services.

[root@server ~]# /etc/init.d/mysql start
[root@server ~]# /scripts/restartsrv_tailwatchd
11.)Edit the fstab and replace /tmp entry line with :-

/usr/tmpDSK /tmp ext3 loop,noexec,nosuid,rw 0 0
12.) Mount all filesystems

[root@server~]# mount -a
Check it now:-

[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 28G 6.4G 20G 25% /
/dev/sda8 ɃM 10M 84M 11% /boot
tmpfs 500M 0 500M 0% /dev/shm
/usr/tmpDSK 2.0G 68M 1.9G 4% /tmp

really hope this little tutoral can help you:)

 

 

How to monitor file access on Linux with "auditd"

For critical servers and sensitive data, keeping an eye on who accesses or changes files is essential for security. The Linux Audit System, with its auditd daemon, helps you do just that. It monitors system calls and logs them, providing a detailed trail of file activity.

---

INSTALLING AUDITD

auditd is available in the package repositories for most Linux distributions.

• Debian, Ubuntu, Linux Mint:

  sudo apt-get install auditd

  On these systems, auditd usually starts automatically on boot after installation.

• Fedora, CentOS, RHEL:

  sudo yum install audit

  To ensure auditd starts automatically on boot for these distributions:

  sudo chkconfig auditd on

---

CONFIGURING AUDITD

You can configure auditd using the auditctl command-line utility or by editing its configuration file, /etc/audit/audit.rules. This guide focuses on editing the configuration file.

• Edit the Configuration File:

  sudo vi /etc/audit/audit.rules

• Example Configuration (/etc/audit/audit.rules):

  # First rule - delete all existing rules
  -D
  
  # Increase buffer size for busy systems to prevent lost events
  -b 1024
  
  # Monitor when files or directories are deleted (unlink and rmdir system calls)
  -a exit,always -S unlink -S rmdir
  
  # Monitor file open attempts by a specific Linux User ID (UID 1001)
  -a exit,always -S open -F loginuid=1001
  
  # Monitor write-access and changes to file properties (permissions) for critical files
  -w /etc/group -p wa
  -w /etc/passwd -p wa
  -w /etc/shadow -p wa
  -w /etc/sudoers -p wa
  
  # Monitor read-access to a specific sensitive directory
  -w /etc/secret_directory -p r
  
  # Lock the audit configuration to prevent unauthorized modifications until reboot
  -e 2
  

  • -D: Clears all previous rules.

  • -b 1024: Sets the buffer size. Increase this for active systems to avoid missing events.

  • -a exit,always -S <syscall>: Monitors specific system calls.

  • -F loginuid=<UID>: Filters events by the user ID logged in.

  • -w <path> -p <permissions>: Sets a watch on a file or directory.

    • w: write access

    • a: attribute change (e.g., permissions)

    • r: read access

  • -e 2: Puts auditd into immutable mode, preventing rule changes until reboot.

• Restart Auditd: After making changes to /etc/audit/audit.rules, restart the service for them to take effect.

  sudo service auditd restart

---

ANALYZING AUDIT LOGS

auditd logs its findings to /var/log/audit/audit.log. The ausearch command-line tool is used to query these logs.

• Check File Access: To see if /etc/passwd has been accessed or modified:

  sudo ausearch -f /etc/passwd

  The output will show details like the time, type of event, user, process, and system call involved. For example, you might see an entry indicating chmod was applied to /etc/passwd by root.

• Check Directory Access: To see if /etc/secret_directory has been accessed:

  sudo ausearch -f /etc/secret_directory

  This will show events like ls commands being run within that directory by a specific UID.

---

IMMUTABLE MODE AND RULE MODIFICATION

If you set auditd to immutable mode (-e 2), you cannot modify the rules and restart the service without a reboot.

• Error Message in Immutable Mode:

  Error deleting rule (Operation not permitted)
  The audit system is in immutable mode, no rules loaded
  

• To Modify Rules in Immutable Mode:

  1. Edit /etc/audit/audit.rules.

  2. Reboot your machine. The new rules will be loaded upon restart.

---

LOG ROTATION

Audit logs can grow large quickly. It's recommended to enable daily log rotation.

• Rotate Audit Logs Daily (for cronjob):

  sudo service auditd rotate

  You can add this command to a daily cron job to ensure logs are rotated regularly, preventing the /var/log/audit directory from filling up.

Resetting Your SSH Port Through WHM

Your SSH (Secure Shell) port is like a specific door on your server that allows secure remote access. Sometimes, you might need to change or reset it, for example, after a security configuration change or to resolve connection issues.

---

WHY RESET YOUR SSH PORT?

• Security: If you've made changes to your SSH configuration (like changing the default port), resetting the service ensures those changes take effect.

• Troubleshooting: Sometimes, a simple restart can resolve minor connection glitches.

---

HOW TO RESET YOUR SSH PORT VIA WHM

This process leverages WHM's built-in tools to safely restart the SSH service.

1. Log in to WHM:

   • Open your web browser.

   • Go to http://your_server_ip:2086 (replace your_server_ip with your server's actual IP address).

   • Enter your WHM username and password.

2. Access the SSH Reset Tool:

   • Once logged into WHM, browse directly to the following URL: http://your_server_ip:2086/scripts2/doautofixer?autofix=safesshrestart

   • Make sure to replace your_server_ip with your server's actual IP address.

3. Confirmation:

   • After navigating to the URL, WHM will attempt to safely restart the SSH service.

   • You should see a message indicating the status of the operation, usually confirming that the SSH service has been restarted or reloaded.

---

IMPORTANT CONSIDERATIONS

• Server IP: Always ensure you replace your_server_ip with the correct IP address of your server.

• WHM Access: You need administrative access to WHM to perform this action.

• SSH Configuration: This process primarily restarts the SSH service to apply existing configuration changes. It does not change the SSH port itself. To change the SSH port, you would typically edit the sshd_config file and then use this method to restart the service.