Pages

Thursday, April 25, 2013

Mastering Grep: Beyond Basic Search – Tips and Tricks for Linux Users

Advanced Grep Tips and Tricks for Power Users

The grep command is indispensable for Linux users, and its real power emerges when you move beyond basic usage. Here’s a deep dive into advanced grep techniques to maximize your productivity and search precision.

Harnessing Regular Expressions for Complex Patterns

  • Basic Regular Expressions (BRE): By default, grep uses BRE, letting you match patterns with metacharacters like:
    • . (dot): Matches any single character.
    • * (asterisk): Matches the preceding character zero or more times.
    • ^ (caret): Matches the start of a line.
    • $ (dollar): Matches the end of a line.
    • [] (square brackets): Matches any single character inside the brackets, e.g., [aeiou] for vowels.
    • (negated brackets): Matches any character not inside the brackets, e.g., [^0-9] for non-digits.

    Example:
    grep "^Error.*" logfile.txt
    Finds lines starting with "Error" followed by any characters.

  • Extended Regular Expressions (ERE): Enable with -E, unlocking:
    • + (plus): One or more occurrences.
    • ? (question mark): Zero or one occurrence.
    • | (pipe): Logical OR.
    • () (parentheses): Grouping.

    Example:
    grep -E "warning|critical" logfile.txt
    Finds lines containing either "warning" or "critical".

Refining Output with Useful Options

  • -i: Case-insensitive search.
  • -v: Invert match (show lines not matching the pattern).
  • -c: Count matching lines.
  • -n: Show line numbers.
  • -l: List filenames with matches.
  • -h: Suppress filename in output.
  • -r or -R: Recursive search through directories (-R follows symlinks).
  • -w: Match whole words only.
  • -o: Print only the matching part of the line.
  • -A <num>: Show <num> lines after a match.
  • -B <num>: Show <num> lines before a match.
  • -C <num>: Show <num> lines before and after (context).

Example:
grep -ric "exception" *.log
Counts case-insensitive matches for "exception" in all .log files recursively.

Leveraging Context for Better Understanding

When analyzing logs or code, context is crucial. The -A, -B, and -C options let you view lines before and/or after matches.

Example:
grep -C 2 "error" application.log
Shows two lines before and after each match for "error".

Searching Multiple Patterns Efficiently

  • Use -e for multiple patterns in one command:
    grep -e "pattern1" -e "pattern2" file.txt
  • Use a pattern file with -f:
    grep -f patterns.txt file.txt
    Each line in patterns.txt is treated as a separate pattern.

Combining Grep with Other Powerful Tools

  • Pipes: Filter output from other commands:
    ps aux | grep "nginx"
    Lists processes containing "nginx".
  • Find and xargs: Search files by content:
    find . -type f -print0 | xargs -0 grep "secret key"
    Finds all files and searches for "secret key" in them.
  • Parallel Execution: For large-scale searches, combine with xargs -P for parallelism:
    find /logs/ -type f | xargs -P 4 grep "error"
    This uses 4 parallel processes for faster searching.

Additional Pro Tips

  • Recursive Search with File Type Filtering:
    Use --include and --exclude to limit search scope:
    grep -r --include "*.txt" "search_term" /path/to/directory
  • Fixed String Search:
    If you don’t need regex, use fgrep or grep -F for faster fixed-string searches:
    grep -F "fixed_string" filename.txt
  • Binary Files:
    For large or binary files, use:
    grep --binary-files=text "search_term" largefile.bin

Summary Table: Key Grep Options

Option Description
-iCase-insensitive search
-vInvert match
-cCount matches
-nShow line numbers
-lList matching filenames
-r / -RRecursive search (with/without symlinks)
-wMatch whole words
-oPrint only match
-A/-B/-CShow context lines
-eMultiple patterns
-fPatterns from file
-EExtended regex
--includeInclude files by pattern
--excludeExclude files by pattern

Mastering these advanced grep techniques will dramatically improve your efficiency when searching and analyzing text on Linux systems.

Sunday, April 21, 2013

Turn OFF/ON "Spam Box" in WHM

Turn OFF "Spam Box" in WHM -> Tweak Settings -> Mail.
This just allows the user to control it, but I wanted it always on and I want to forward to the ".Junk" folder anyways.
WHM -> Tweak Settings -> Mail -> Enable SpamAssassin Spam Box: OFF
WHM -> Tweak Settings -> Mail -> Enable SpamAssassin spam filter: ON
WHM -> Exim Configuration Editor -> SpamAssassin Options -> SpamAssassin: Forced Global ON: ON

http://forums.cpanel.net/f43/can-you-change-spamassassin-spam-box-folder-223951.html

Security Update: Atomic Mod Security setup guide for cPanel

As to help our VPS and Dedicated Server customers who might also be effected by this we have designed the following guide to make installing Atmoic Mod Security into cPanel with little to no fuss.

Stage 1: Run the following commands at command line:

mkdir /var/asl
mkdir /var/asl/tmp
mkdir /var/asl/data
mkdir /var/asl/data/msa
mkdir /var/asl/data/audit
mkdir /var/asl/data/suspicious
chown nobody.nobody /var/asl/data/msa
chown nobody.nobody /var/asl/data/audit
chown nobody.nobody /var/asl/data/suspicious
chmod o-rx -R /var/asl/data/*
chmod ug+rwx -R /var/asl/data/*
mkdir /var/asl/updates
mkdir /var/asl/rules/
mkdir /var/asl/rules/clamav
mkdir /etc/asl/
touch /etc/asl/whitelist
cd /usr/local/src/
wget http://updates.atomicorp.com/channels/rules/delayed/modsec-2.7-free-latest.tar.gz
tar zxvf modsec-2.7-free-latest.tar.gz
mkdir /usr/local/apache/conf/modsec_rules/
cp modsec/* /usr/local/apache/conf/modsec_rules/
These command will create the required directory’s and download the latest free version of the Atomic Mod Security rules. It will also directly install them into the location of Apache designed for cPanel and configure the permission.

Stage 2: Configure cPanel to use the Mod Security Rules

In this stage, you can do everything from WHM as long as you have Mod Security already installed as part of your EasyApache build. If you do not, you will need to rebuild apache with Mod Security.

In go to: WHM -> Plugins -> Mod Security and then click: Edit Config

In this section, delete all the current content and then paste in the following configuration:

SecRequestBodyAccess On
SecAuditLogType Concurrent
SecResponseBodyAccess On
SecResponseBodyMimeType (null) text/html text/plain text/xml
SecResponseBodyLimit 2621440
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecServerSignature Apache
SecUploadDir /var/asl/data/suspicious
SecUploadKeepFiles Off
SecAuditLogParts ABIFHZ
SecArgumentSeparator "&"
SecCookieFormat 0
SecRequestBodyInMemoryLimit 131072
SecDataDir /var/asl/data/msa
SecTmpDir /tmp
SecAuditLogStorageDir /var/asl/data/audit
SecResponseBodyLimitAction ProcessPartial

Include /usr/local/apache/conf/modsec_rules/10_asl_antimalware.conf
Include /usr/local/apache/conf/modsec_rules/10_asl_rules.conf
Include /usr/local/apache/conf/modsec_rules/20_asl_useragents.conf
Include /usr/local/apache/conf/modsec_rules/30_asl_antispam.conf
Include /usr/local/apache/conf/modsec_rules/50_asl_rootkits.conf
Include /usr/local/apache/conf/modsec_rules/60_asl_recons.conf
Include /usr/local/apache/conf/modsec_rules/99_asl_jitp.conf
Include /usr/local/apache/conf/modsec2.whitelist.conf
Save this and restart Apache.

This should now have successfully installed the Atomic mod security rules into cPanel which are a much more secure rule base and include extra protection which is important for the latest hacks.

Saturday, April 20, 2013

Fixing su: cannot set user id: Resources temporarily

#su -
su: cannot set user id: Resource temporarily unavailable
#

After searching and googling few minutes, i found a quick resolution to fixing this issue.
This is all about VPS limit.

Edit limit.conf below or change if needed.
#vi /etc/security/limit.conf
#### add/change on these line below:
* soft nproc 2047
* hard nproc 16384
* soft nofile 2048
* hard nofile 65536

After that try to relogin using a normal access user and then try to get su -
this should fixed your issue.

Friday, April 19, 2013

Conver Putty ppk key to SSH key

In many occasions I needed to convert a Putty ppk private key nack to open ssh key to be able to use it directly from a linux box command line. It is fairly simple to do but I always need to look it up so here it is for easy reference :

Install putty using yum in Fedora.
If you are using Ubuntu you need to install putty-tools as well

From the command line give

puttygen xxxxx2.ppk -O private-openssh -o key.ssh

Enter passphrase to load key:

Then log onto your system using

ssh root@84.200.82.6 -p 8057 -i key.ssh

Tuesday, April 16, 2013

Troubleshooting apache

check httpd service is running or not

• if its not starting the check the syntax of the config file
#httpd -S (display the all files and its location )
#httpd -t or apachectl configtest
if syntax is ok the config file is correct
• check the error logs of apache
/var/log/httpd/error_log
• apache also requires working dsn client support via /etc/resolv.conf and make sure dns is working fine
• check httpd.conf file there is correct entry of server name is there or not and check the port no
• check the size of the log file if it is full then it ll cause error 500 so make sure that log files are under limit and we can us e tool called logrotate in /etc/logrorate.d/httpd create configuration file for httpd log files
• It is possible that some other process may be using port 80 or 443. Use netstat command to list open port and their owners:
#nestat -tulpn
#netstat -tulpn |grep ‘:80′
If other process using port 80 / 443, you need to stop them or assign another port to Apache

What is SPF records and How to setup?

Why Is My Emails Getting As Spam?
Check above things.

First we need to check Email Headers.
1. Return Path
2. To
3. From
4. We should not use CAPS because not only being unprofessional, but it
also triggers spam filters. To land in the Spam folder consistently,
please do not USE ALL CAPS IN THE SUBJECT LINE AND THE BODY FROM TO and
Excessive punctuation (e.g. -/, ;^%#$@+ * "').

E.g. as follows.

1. Email "From : ADMIN ENROLL <noreply@exmaple.com>" getting
spam in gmail.com sometimes and if we not change it will increasing
count.

2. Email "From: noreply@exmaple.com" it
getting in mailbox, check the CAPS deference.

http://php.net/manual/en/function.mail.php
http://www.niso.org/khelp/kmlm/user_help/html/examine_mail_headers.html

Please check this too. It can be understand better. :)
http://www.infusionblog.com/email-marketing/7-ways-for-your-email-to-land-in-the-spam-folder/

Check SPF for Domain.:
http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/

What is SPF records and How to setup?
http://www.kazeli.com/help/index.cfm?pageloc=quest&questid=435&catid=337,338

To diagnose the mail. Why its going in Spam.
Check tool:
http://mxtoolbox.com/
Here you can check following points
1. Mxlookup : For your domain and ip
2. Blacklist : For your IP
3. Diagnostics : SMTP Diagnostics, check your mail server
4. Analyze Headers : Check your mail headers.
5. SPF Records : Check your Sender Policy Framework
7. DNS Lookup : Check your Domain lookup.

If you check above all things your mail will not go in Spam.!