Pages

Monday, December 16, 2013

Exim cheat sheet

Exim is a powerful mail transfer agent (MTA) used on many Linux servers. When emails can't be delivered immediately, they get put into a "queue." Managing this queue is crucial for server health and ensuring mail delivery. This guide provides quick commands for common Exim queue tasks.


CLEARING THE EXIM MAIL QUEUE

Sometimes, you need to clear out stuck or unwanted emails from the queue.

  • Remove All Mails: This command directly deletes all files from the input directory of the Exim spool, effectively clearing the entire queue. rm -rf /var/spool/exim/input/*

  • Delete All Frozen Mails: Frozen emails are those that Exim has temporarily stopped trying to deliver due to issues. exim -bpr | grep frozen | awk {'print $3'} | xargs exim -Mrm Alternatively, a more concise command: exiqgrep -z -i | xargs exim -Mrm

  • Delete Frozen Mails Older Than a Day: This is useful for clearing old, stalled messages without affecting newer ones. The 86400 represents seconds (1 day). exiqgrep -zi -o 86400 | xargs exim -Mrm You can change 86400 to any number of seconds for a different time frame.

  • Clear Spam Mails: If your logs indicate messages are marked as [SPAM]. grep -R -l [SPAM] /var/spool/exim/msglog/*|cut -b26-|xargs exim -Mrm

  • Clear Frozen Mails (Based on Log Entry): grep -R -l '*** Frozen' /var/spool/exim/msglog/*|cut -b26-|xargs exim -Mrm

  • Clear Mails for Unverified Recipients: grep -R -l 'The recipient cannot be verified' /var/spool/exim/msglog/*|cut -b26-|xargs exim -Mrm

  • Remove Mails from a Specific Sender (e.g., 'root'): Replace "" with the sender's email address or username, for example, root@yourhostname. exim -bp |grep ""|awk '{print $3}'|xargs exim -Mrm

  • Remove 'nobody' Mails: These often come from scripts. Replace HOSTNAME with your server's hostname.

    • From a specific sender (nobody@HOSTNAME): exiqgrep -i -f nobody@HOSTNAME | xargs exim -Mrm

    • For a specific recipient/domain (nobody@HOSTNAME): exiqgrep -i -r nobody@HOSTNAME | xargs exim -Mrm

  • Delete Mails for a Specific Domain: Replace yourdomain.com with the actual domain. exim -bp | grep "yourdomain.com" | awk {'print $3'} | xargs exim -Mrm


DELIVERING MAILS FROM THE QUEUE

If emails are stuck but should be delivered, you can force a delivery attempt.

  • Force Deliver All Mails: This command attempts to deliver all messages in the queue. The -P 40 option attempts 40 deliveries in parallel. exim -bpru |awk '{print $3}' | xargs -n 1 -P 40 exim -v -M

  • Flush the Mail Queue (Force Another Run): This tells Exim to process the queue again. exim -qff Alternatively: /usr/sbin/exim -qff exim -qf

  • Force Deliver Mails of a Particular Domain: Replace domain.com with the target domain. exim -v -Rff domain.com

  • Force Deliver a Specific Message: Replace MSGID with the message's unique ID. exim -M MSGID To view the transaction during delivery: exim -v -M MSGID


CHECKING THE EXIM MAIL QUEUE STATUS

These commands help you monitor the queue and inspect individual messages.

  • Exim Queue Summary: Provides details like count, volume, oldest, newest message, and domain breakdown. exim -bp | exiqsumm

  • Number of Frozen Mails: exim -bpr | grep frozen | wc -l

  • Total Number of Mails in Queue: exim -bpr | grep "<" | wc -l A simpler alternative: exim -bpc

  • View Mail in Queue for a User/Sender: Replace $name with the username or email address. exim -bp|grep $name

  • Check All Mails in the Queue: This lists all messages and their IDs. exim -bp

  • View Log for a Message: Replace message ID with the actual ID. exim -Mvl message ID

  • View Message Header: Replace $MSGID with the message ID. exim -Mvh $MSGID

  • View Message Body: Replace $MSGID with the message ID. exim -Mvb $MSGID


ADVANCED EXIM TOOLS

  • Simulate SMTP Transaction: This command helps debug Exim's checks, ACLs (Access Control Lists), and filters without actually sending a mail. Replace 127.0.0.1 with the IP you want to simulate from. exim -bh 127.0.0.1

  • Most Used Mailing Script Locations: This can help identify scripts sending a lot of mail. grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

  • Check Syntactic Errors in Configuration: Use this when modifying Exim's configuration file. exim -C /config/file.new -bV

Wednesday, December 11, 2013

Understanding Mod_Security Database Connection Issues

You've encountered an error stating that the Mod_Security plugin can't connect to its database. Specifically, it's an "Access denied" error for the user 'modsec'@'localhost', even though a password was provided. This usually means the password Mod_Security is using to connect to the database is incorrect or the database user doesn't have the right permissions.


WHY THIS MATTERS

Mod_Security is a web application firewall that helps protect your website from various attacks. If it can't connect to its database, it might not be able to log security events or function correctly, leaving your website vulnerable.


HOW TO FIX IT

There are two main steps to resolve this, focusing on ensuring the 'modsec' user can properly access the 'modsec' database.

  1. Find the Correct Password Mod_Security is Using:

    The error indicates Mod_Security is trying to connect with a specific password. You need to find out what password it's actually configured to use.

    • Action: Run the following command in your server's terminal:

      grep dbpassword /etc/cron.hourly/modsecparse.pl
      
    • Explanation: This command searches a common Mod_Security configuration file (modsecparse.pl) for the line containing dbpassword. This line will reveal the password that Mod_Security is currently trying to use for its database connection. Let's say the output of this command shows the password is 'odu6lGYKAIyP'.

  2. Grant the Correct Permissions to the Database User:

    Once you know the password Mod_Security is configured with, you need to ensure the 'modsec' database user has the correct password and permissions in MySQL.

    • Action: Log into your MySQL server (as a root user or a user with sufficient privileges) and execute the following command. Replace 'odu6lGYKAIyP' with the actual password you found in the previous step.

      SQL
      GRANT ALL ON modsec.* TO 'modsec'@localhost IDENTIFIED BY 'odu6lGYKAIyP';
      FLUSH PRIVILEGES;
      
    • Explanation:

      • GRANT ALL ON modsec.*: This gives the 'modsec' user all permissions on all tables within the modsec database.

      • TO 'modsec'@localhost: Specifies that these permissions apply to the user 'modsec' when connecting from the 'localhost' (meaning from the same server).

      • IDENTIFIED BY 'odu6lGYKAIyP': Sets or updates the password for the 'modsec' user to 'odu6lGYKAIyP'. It's crucial that this password matches what Mod_Security is configured to use.

      • FLUSH PRIVILEGES;: This command reloads the grant tables in MySQL, applying the new permissions immediately.

Tuesday, December 10, 2013

Nagios- Setting up

This guide explains how to set up a client server to be monitored by Nagios. We'll cover installing the necessary Nagios plugins and the NRPE (Nagios Remote Plugin Executor) agent, which allows the Nagios server to run checks on the client.


PREREQUISITES

Before you start, make sure you have:

  • A Linux-based client server.

  • Root access or sudo privileges.

  • Basic understanding of the Linux command line.


INSTALLING NAGIOS PLUGINS

Nagios plugins are scripts that perform checks on various aspects of your server, like disk space, CPU usage, or running services.

  1. Prepare the Environment:

    • Navigate to a temporary directory: cd /usr/local/src/

    • Create a directory for Nagios files: mkdir nagios

    • Move into the new directory: cd nagios

    • Create a nagios user: useradd nagios

  2. Download and Extract Plugins:

    • Download the plugins: wget http://pkgs.fedoraproject.org/repo/pkgs/nagios-plugins/nagios-plugins-1.4.16.tar.gz/862f5e44fb5bc65ce7e5d86d654d4da0/nagios-plugins-1.4.16.tar.gz

    • Extract the downloaded file: tar -xzf nagios-plugins-1.4.16.tar.gz

    • Change into the extracted directory: cd nagios-plugins-1.4.16

  3. Compile and Install Plugins:

    • Set a required environment variable: export LDFLAGS=-ldl

    • Configure the installation: ./configure --with-nagios-user=nagios --with-nagios-group=nagios --enable-redhat-pthread-workaround --enable-ssl

    • Compile the plugins: make

    • Install them: make install

    • Go back to the parent directory: cd ..


INSTALLING NRPE (NAGIOS REMOTE PLUGIN EXECUTOR)

NRPE allows your Nagios server to execute the plugins you just installed on this client.

  1. Download and Extract NRPE:

    • Download NRPE: wget http://sourceforge.net/projects/nagios/files/nrpe-2.x/nrpe-2.13/nrpe-2.13.tar.gz/download

    • Extract the file: tar -xzf nrpe-2.13.tar.gz

    • Change into the extracted directory: cd nrpe-2.13

  2. Compile and Install NRPE:

    • Configure NRPE: ./configure

    • Compile all components: make all

    • Install the NRPE plugin for the Nagios server (even though this is a client, it's good practice): make install-plugin

    • Install the NRPE daemon (the service that runs on the client): make install-daemon

    • Install the default NRPE configuration file: make install-daemon-config

    • Install NRPE to run under xinetd (a service that manages other services): make install-xinetd


CONFIGURE XINETD AND NRPE

NRPE often runs through xinetd, which listens for incoming connections and starts the NRPE daemon when needed.

  1. Install and Restart Xinetd:

    • Install xinetd if it's not already present: yum install xinetd (or apt-get install xinetd for Debian/Ubuntu)

    • Restart xinetd to apply changes: service xinetd restart

  2. Add NRPE Service to System Services:

    • Add the NRPE service and its default port (5666) to the /etc/services file: echo "nrpe 5666/tcp # NRPE" >> /etc/services


FIREWALL CONFIGURATION

You need to open port 5666 on the client's firewall so the Nagios server can communicate with NRPE.

  • For CSF (Config Server Firewall) Users:

    • Edit the CSF configuration file: /etc/csf/csf.conf

    • Find the TCP_IN section and add 5666 to the list of allowed incoming TCP ports.

    • Restart CSF: This usually involves csf -r or service csf restart.


NRPE CONFIGURATION AND SECURITY

You must specify which Nagios servers are allowed to connect to this NRPE agent.

  • Allow Nagios Server IP:

    • Edit the xinetd configuration for NRPE: /etc/xinetd.d/nrpe

    • Edit the main NRPE configuration file: /usr/local/nagios/etc/nrpe.cfg

    • In both files, locate the allowed_hosts directive and add the IP address of your Nagios monitoring server. For example: allowed_hosts = 127.0.0.1, <YOUR_NAGIOS_SERVER_IP>


SETTING A PASSWORD FOR THE NAGIOS USER (Optional but Recommended)

It's good practice to set a password for the nagios user created earlier.

  • Set password: passwd nagios


NAGIOS SERVER CONFIGURATION EXAMPLE (for MySQL Monitoring)

This section shows an example of how you would configure a service check on your Nagios server to monitor a MySQL database on this client. This is done on the Nagios server, not the client.

define service{
    use                     local-service       ; Name of service template to use
    host_name               sample.example.com  ; The hostname of your client server
    service_description     mySQL               ; A description of the service being monitored
    is_volatile             0
    check_period            24x7
    max_check_attempts      20
    normal_check_interval   5                   ; Check every 5 minutes
    retry_check_interval    1                   ; Retry every 1 minute on failure
    notification_options    w,u,c,r             ; Notify on warning, unknown, critical, recovery
    notification_interval   960                 ; Notify every 960 minutes (16 hours)
    notification_period     24x7
    check_command           check_mysql!nagios!password@666# ; The command to run, 
                    ;including username and password
}

Note: Replace sample.example.com with your client's hostname and adjust nagios and password@666# if your MySQL user or password differs.

Wednesday, November 20, 2013

Recovering from InnoDB Corruption in MySQL

InnoDB is a robust storage engine for MySQL, but like any system, it can encounter issues such as corruption. Understanding how to address these problems is crucial for database administrators. This guide will outline steps to recover from an InnoDB corruption using the innodb_force_recovery option.

Understanding the Error

The error log indicates an assertion failure within InnoDB, suggesting data corruption. Typical causes include hardware faults, sudden shutdowns, or bugs. The error message suggests steps for recovery and forces InnoDB into a special "recovery" mode.


130306 22:02:18 mysqld_safe Number of processes running now: 0
130306 22:02:18 mysqld_safe mysqld restarted
130306 22:02:18 [Note] Plugin 'FEDERATED' is disabled.
130306 22:02:18 InnoDB: The InnoDB memory heap is disabled
130306 22:02:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130306 22:02:18 InnoDB: Compressed tables use zlib 1.2.3
130306 22:02:18 InnoDB: Using Linux native AIO
130306 22:02:18 InnoDB: Initializing buffer pool, size = 128.0M
130306 22:02:18 InnoDB: Completed initialization of buffer pool
130306 22:02:18 InnoDB: highest supported file format is Barracuda.
130306 22:02:18 InnoDB: 5.5.30 started; log sequence number 1629186928
130306 22:02:18 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130306 22:02:18 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130306 22:02:18 [Note] Server socket created on IP: '0.0.0.0'.
130306 22:02:18 [Note] Event Scheduler: Loaded 0 events
130306 22:02:18 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.30-cll' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
130306 22:02:19 InnoDB: Assertion failure in thread 47204348393792 in file trx0purge.c line 840
InnoDB: Failing assertion: purge_sys->purge_trx_no <= purge_sys->rseg->last_trx_no
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
03:02:19 UTC - mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.


Steps to Recovery

1. Stop mysqld:

First, ensure that MySQL is not running to prevent further data corruption.

service mysqld stop

2. Backup Critical Files:

Before proceeding with any recovery steps, back up the existing InnoDB data files.

mkdir /root/mysql_backup cp /var/lib/mysql/ib* /root/mysql_backup/

3. Configure MySQL for Recovery:

Edit the MySQL configuration file (/etc/my.cnf) and add the following under the [mysqld] section:

innodb_force_recovery = 4

There are different levels of innodb_force_recovery you can use, from 1 to 6. Levels 1-4 are generally considered safe, while 5 and 6 are used as a last resort.

4. Restart mysqld:

Restart the MySQL server. It will start in recovery mode due to the innodb_force_recovery setting.

service mysqld start

5. Dump All Tables:

Create a dump of all databases. This might take some time depending on the size of your databases.

mysqldump -A > /root/mysql_backup/dump.sql

6. Drop Corrupted Databases:

Identify and drop the corrupted databases. If you're unsure which ones are corrupted, you might need to drop all databases and restore them from the dump later.

mysql -u root -p -e "DROP DATABASE corrupted_db"

7. Stop mysqld:

Once again, stop the MySQL server to proceed with the next steps.

service mysqld stop

8. Remove InnoDB Data Files:

Delete the old InnoDB data files. Be cautious with this command – ensure you have backups!

rm /var/lib/mysql/ib*

9. Comment out innodb_force_recovery:

Edit /etc/my.cnf and comment out or remove the innodb_force_recovery option.

10. Restart mysqld:

Start the MySQL server. It should recreate the InnoDB data files.

service mysqld start

11. Restore Databases:

Restore the databases from the dump you created earlier.Copy code

mysql < /root/mysql_backup/dump.sql

12. Repair and Optimize:

Finally, check and repair all databases and tables.

mysqlcheck –all-databases –repair
innodb force recovery options

Mode 1 - Doesn't crash MySQL when it sees a corrupt page
Mode 2 - Doesn't run background operations
Mode 3 - Doesn't attempt to roll back transactions
Mode 4 - Doesn't calculate stats or apply stored/buffered changes
Mode 5 - Doesn't look at the undo logs during start-up
Mode 6 - Doesn't roll-forward from the redo logs (ib_logfiles) during start-up
1 (SRV_FORCE_IGNORE_CORRUPT)
Let the server run even if it detects a corrupt page. Try to make SELECT * FROM tbl_name jump
over corrupt index records and pages, which helps in dumping tables.
2 (SRV_FORCE_NO_BACKGROUND)
Prevent the main thread from running. If a crash would occur during the purge operation,
this recovery value prevents it.
3 (SRV_FORCE_NO_TRX_UNDO)
Do not run transaction rollbacks after recovery.
4 (SRV_FORCE_NO_IBUF_MERGE)
Prevent insert buffer merge operations. If they would cause a crash, do not do them.
Do not calculate table statistics.
5 (SRV_FORCE_NO_UNDO_LOG_SCAN)
Do not look at undo logs when starting the database: InnoDB treats even incomplete transactions as committed.
6 (SRV_FORCE_NO_LOG_REDO)
Do not do the log roll-forward in connection with recovery.
The database must not otherwise be used with any nonzero value of innodb_force_recovery.
As a safety measure, InnoDB prevents users from performing INSERT, UPDATE, or
DELETE operations when innodb_force_recovery is greater than 0.

**Hint : A simple query for finding all of your InnoDB tables in case you want to specifically target the corruption.
SELECT table_schema, table_name
FROM INFORMATION_SCHEMA.TABLES
WHERE engine = 'innodb';

\

Sunday, November 17, 2013

Setting Up FFMPEG and FFMPEG-PHP on CentOS 5 & 6

Setting Up FFMPEG and FFMPEG-PHP on CentOS 5 & 6

This guide provides two straightforward methods to install FFMPEG and its PHP extension, FFMPEG-PHP, on CentOS 5 and 6 servers. FFMPEG is a powerful tool for handling multimedia files, and FFMPEG-PHP allows your PHP applications to interact with FFMPEG.


METHOD 1: USING THE INSTALLER SCRIPT

This method uses a pre-made script to automate most of the installation process.

DOWNLOAD AND RUN THE SCRIPT

First, download the installer script:

wget http://9xhost.net/scripts/ffmpeg.sh

Then, run the script:

sh ffmpeg.sh

CONFIGURE PHP

After the script finishes, you'll need to enable the FFMPEG-PHP extension in your PHP configuration. The script's output will show you the path where the PHP extension was installed, typically something like /usr/local/lib/php/extensions/no-debug-non-zts-20060613/.

Open your php.ini file for editing:

nano /usr/local/lib/php.ini

Add the following line at the end of the file to enable the extension:

extension="ffmpeg.so"

RESTART APACHE

For the changes to take effect, restart your Apache web server:

/scripts/restartsrv_httpd

VERIFY INSTALLATION

You can confirm that FFMPEG-PHP is working by checking your PHP information.

From the command line:

php -i | grep ffmpeg

This should display details about the FFMPEG and FFMPEG-PHP versions.

INSTALLATION PATHS

FFMPEG itself is typically installed at:

/usr/bin/ffmpeg

EXCLUDE FROM YUM UPDATES

To prevent potential conflicts with future system updates, it's recommended to exclude FFMPEG packages from Yum updates.

Open the yum.conf file:

nano /etc/yum.conf

Add ffmpeg* to the exclude line. If an exclude line doesn't exist, create it.


METHOD 2: MANUAL INSTALLATION VIA YUM AND COMPILATION

This method involves adding a repository, installing FFMPEG via Yum, and then compiling FFMPEG-PHP.

ADD DAG REPOSITORY

Create and open a new repository file:

nano /etc/yum.repos.d/dag.repo

Add the following content to the file:

[dag]
name=DAG RPM Repository
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=1
enabled=1

Import the GPG key for the repository:

rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt

INSTALL FFMPEG

Update your Yum packages:

yum update

Now, install FFMPEG and its development libraries:

yum install ffmpeg ffmpeg-devel ffmpeg-libpostproc

PREPARING FFMPEG-PHP

Download the FFMPEG-PHP package:

wget http://downloads.sourceforge.net/ffmpeg-php/ffmpeg-php-0.6.0.tbz2

Untar the downloaded package:

tar xjf ffmpeg-php-0.6.0.tbz2

Navigate into the extracted directory:

cd ffmpeg-php-0.6.0

Apply a necessary patch for compatibility:

sed -i 's/PIX_FMT_RGBA32/PIX_FMT_RGB32/g' ffmpeg_frame.c

Prepare for compilation:

phpize

Configure the build:

./configure

Compile the extension:

make

Install the extension:

make install

The output of make install will show you the PHP extensions path, for example:

Installing shared extensions: /usr/local/lib/php/extensions/no-debug-non-zts-20060613/

CONFIGURE PHP

Edit your php.ini file:

nano /usr/local/lib/php.ini

Ensure extension_dir is set to the path identified by make install:

extension_dir = "/usr/local/lib/php/extensions/no-debug-non-zts-20060613"

Add the following line directly below extension_dir to enable the FFMPEG-PHP extension:

extension="ffmpeg.so"

RESTART APACHE

Restart your Apache web server for changes to take effect:

/scripts/restartsrv_httpd

VERIFY INSTALLATION

Confirm FFMPEG-PHP is working from the command line:

php -i | grep ffmpeg

This should display FFMPEG and FFMPEG-PHP version information.

INSTALLATION PATHS

FFMPEG itself is typically installed at:

/usr/bin/ffmpeg

EXCLUDE FROM YUM UPDATES

To prevent potential conflicts, exclude FFMPEG packages from Yum updates.

Open the yum.conf file:

nano /etc/yum.conf

Add ffmpeg* to the exclude line.


TROUBLESHOOTING: 'list_entry' UNDECLARED ERROR

If you encounter an error during compilation similar to:

error: 'list_entry' undeclared

This usually indicates a compatibility issue between the FFMPEG-PHP version and your PHP version. The provided sed command (sed -i 's/PIX_FMT_RGBA32/PIX_FMT_RGB32/g' ffmpeg_frame.c) addresses one common issue, but newer PHP versions might require more extensive patches or a newer FFMPEG-PHP version. Consider searching for specific patches for your PHP version (e.g., "ffmpeg-php PHP 5.3 patch") or looking for a more updated FFMPEG-PHP release.