1.Linux utilities

1. ls (List):

   • Use Case: List files and directories in the current directory.

   • Example: ls -l (Long format listing), ls -a (List hidden files), ls /path/to/directory (List files in a specific directory).

2. pwd (Print Working Directory):

   • Use Case: Display the current working directory.

   • Example: pwd.

3. cd (Change Directory):

   • Use Case: Change the current directory.

   • Example: cd /path/to/directory.

4. mkdir (Make Directory):

   • Use Case: Create a new directory.

   • Example: mkdir my_directory.

5. rmdir (Remove Directory):

   • Use Case: Remove an empty directory.

   • Example: rmdir my_directory.

6. touch:

   • Use Case: Create an empty file or update the timestamp of an existing file.

   • Example: touch myfile.txt.

7. rm (Remove):

   • Use Case: Delete files or directories.

   • Example: rm myfile.txt (Remove a file), rm -rf my_directory (-r means Remove a directory and its contents -f means forcefully).

8. cp (Copy):

   • Use Case: Copy files or directories.

   • Example: cp file1.txt file2.txt (Copy a file), cp -r directory1/ directory2/ (Copy a directory and its contents).

9. mv (Move/Rename):

   • Use Case: Move or rename files or directories.

   • Example: mv file1.txt newfile.txt (Rename a file), mv file.txt /new/location/ (Move a file).

10. cat (Concatenate):

    • Use Case: Display the content of a file.

    • Example: cat myfile.txt.

11. less/more:

    • Use Case: View the contents of a file one screen at a time.

    • Example: less myfile.txt or more myfile.txt.

12. head/tail:

    • Use Case: Display the beginning or end of a file.

    • Example: head -n 10 myfile.txt (Display the first 10 lines), tail -f logfile.txt (Display the last few lines of a log file in real-time).

13. grep (Global Regular Expression Print):

    • Use Case: Search for a pattern in text.

    • Example: grep "pattern" file.txt.

14. find:

    • Use Case: Search for files and directories in a directory hierarchy.

    • Example: find /path/to/search -name "filename".

15. chmod (Change Mode):

    • Use Case: Change file permissions.

    • Example: chmod 755 myfile.sh (Give read, write, and execute permissions to the owner, and read and execute permissions to others).

16. chown (Change Owner):

    • Use Case: Change the owner of a file or directory.

    • Example: chown user:group myfile.txt.

17. ps (Process Status):

    • Use Case: List currently running processes.

    • Example: ps aux (List all processes).

18. kill:

    • Use Case: Terminate a running process.

    • Example: kill process_id.

19. df (Disk Free):

    • Use Case: Display disk space usage.

    • Example: df -h (Display usage in human-readable format).

20. du (Disk Usage):

    • Use Case: Show the disk space used by files and directories.

    • Example: du -sh /path/to/directory.

Use Case: Log Analysis with Pipeline

Suppose you have a large log file from a web server, and you want to analyze the number of times each unique IP address has accessed your website. You can use a pipeline of Linux utilities to achieve this:

suppose you have a log file called access.log with the following contents:

192.168.1.100 - - [12/Mar/2023:15:45:23 +0000] "GET /page1.html HTTP/1.1" 200 1234
192.168.1.101 - - [12/Mar/2023:15:46:01 +0000] "GET /page2.html HTTP/1.1" 404 5678
192.168.1.102 - - [12/Mar/2023:15:46:05 +0000] "GET /page1.html HTTP/1.1" 200 9876
192.168.1.100 - - [12/Mar/2023:15:47:30 +0000] "GET /page3.html HTTP/1.1" 200 3456
192.168.1.101 - - [12/Mar/2023:15:47:45 +0000] "GET /page4.html HTTP/1.1" 200 7890
192.168.1.102 - - [12/Mar/2023:15:48:10 +0000] "GET /page3.html HTTP/1.1" 404 1234

1. cat: Display the contents of the log file.

   cat access.log

2. grep:

• Filter lines containing IP addresses.

  cat access.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"

• The grep command with the -oE option searches for IP addresses in the log file using a regular expression.The regular expression \b([0-9]{1,3}\.){3}[0-9]{1,3}\b is used to match and extract IP addresses from a text document. Let's break down what each part of the regular expression:

• \b: This is a word boundary anchor. It matches the position where a word (sequence of alphanumeric characters) starts or ends. In this context, it ensures that we match complete IP addresses and not parts of them within other words.

• ([0-9]{1,3}\.){3}: This part of the regular expression matches the first three octets of the IP address, separated by periods (.). Let's break it down further:

  • [0-9]{1,3}: This part matches one to three digits (0-9). It specifies that an octet can consist of one to three numeric digits.

  • \.: This matches a literal period character, which separates octets in an IP address.

  • (...){3}: This whole group is enclosed in parentheses and followed by {3}, which means it should match exactly three occurrences of the preceding group. This ensures that we capture the first three octets.

• [0-9]{1,3}: This part matches the fourth octet of the IP address, which is also one to three digits, just like the previous octets.

• \b: Similar to the first \b, this is another word boundary anchor. It ensures that we match complete IP addresses and not partial ones within other words.

So, when you put it all together, \b([0-9]{1,3}\.){3}[0-9]{1,3}\b is a regular expression that matches and extracts complete IP addresses from a text document. It assumes that IP addresses are written in a standard format with four octets separated by periods and ensures that it matches only complete IP addresses, not partial ones within larger words.

1. sort: Sort the IP addresses to group identical ones together.

   cat access.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort

2. uniq -c: Count the occurrences of each unique IP address.

   cat access.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort | uniq -c

   • The uniq -c command counts and displays the occurrences of each unique line.

3. sort -n: Sort the IP addresses by the count in ascending order.

   cat access.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort | uniq -c | sort -n

   output:

    1 192.168.1.100
    1 192.168.1.101
    2 192.168.1.102

• This step is optional, but it can be useful to see the IP addresses with the highest access counts at the end.