Harshit Rawat

Posted on Feb 10

Master Shell Scripting – Zero to Interview Ready!🔥

#linux #bash #beginners #programming

Introduction

Hey Everyone, Good to see you back! For us programmers, diving into the world of Bash scripting is really important. It's all about creating scripts, which are essentially mini-programs, to automate those tedious, repetitive actions we face daily. Instead of manually running commands over and over, we use scripts to make them easy.

It's about making our lives easier by automating the mundane, freeing us to focus on the more exciting aspects of development.

Over the next few minutes, we'll explore the fundamentals of Bash scripting with practical examples. By the end, you'll have a solid understanding of streamlining your workflows and boosting your productivity.

Are you excited?

Any Prerequisites?
If you are a window user, you need to setup WSL for using Linux on your system. You can explore more about it here.

Note: In this article, the terms "shell script" and "Bash script" are used interchangeably. Bash is the most common shell, and for the purposes of this discussion, they can be considered the same.

Covering the Basics

What exactly is a Bash Script ?
Think of it as creating a mini-program to automate tasks you'd normally do manually. These scripts are plain text files containing commands you would typically type into the command line. An example could be, Let's say you need to copy all 100 files from folder A to folder B, well this task is really boring and repetitive, hence you can write a bash script to copy all the files from A to B.

Why do we need it, Is it that important ?
We need Bash scripting because it saves us time and effort. Think of it as having a assistant who can handle all the boring, repetitive tasks, so you can focus on the interesting stuff. Yes it really helps in automating big tasks for Organizations.

Where do we write these scripts ?
We write these scripts inside text editors. In our case we will be using VIM editor that comes inside our Linux distribution.

This is how a shell looks like:

Bash Commands

Here, we'll understand about the most used bash commands.

It's important to know the path you are present currently, this get's confusing when we work on multiple projects. To get the current directory we use the command pwd which means 'print working directory'

pwd

Now that we know our present directory, Let's make a new folder (u also call it a directory) where we can keep our files. To do this, we use the mkdir command, which means "make directory."

mkdir <specify your folder name>

After creating you directory you can view inside it using ls command.

ls

You can also use ls -ltr command, this lists the contents in long format (-l), sorted by modification time (-t) with the oldest entries displayed first (-r).

ls -ltr

Now, For moving inside the folder/directory you write the command cd which stands for change directory, using cd you can enter inside the folder and again typing ls will show all the files inside that folder.

cd <folder_name>

In order to come out of the directory you are currently in, you can use the command cd .., this let's you out of the directory.

cd ..

In order to remove a file or complete directory we use the command rm -r

rm -r <name of file or folder>

We can rename a file using the command mv.

mv <your-file-name> <new-file-name>

Let's make a file now, To make a file we use the touch command.

touch <filename>

To print the content inside a file we make use of the cat command.

cat <filename>

The file initially is empty because we haven't added anything inside the file. Let's add some content to the file. we'll use vim editor(complete detail ahead in the article) to edit the file , save it and then use the cat command.

Vim editor

The file is now showing the content we have written.

Lastly, If we want to print something on the shell, we use the command echo.

Okay, great! We've learned the basic commands to move around in the shell. Now we can move on to more interesting stuff. It's not really hard, so don't worry!

Bash Script & VIM editor

Now, If you have never done scripting before, you might not be familiar with vim and all the memes associated with it. Well, vim is an editor that is used for writing bash scripts on terminal, there are many such others editors like neovim, nano that can help you to write scripts but we'll be using vim in this tutorial.

Note: For first timers, you may struggle using vim, don't worry, it's pretty normal.

Let's write our first bash script

"#" symbol is used to show the content is a comment not a part of program.
# Type the below command on the shell.

vim bashscriptfile.sh

Yeahh!!, you are now inside the vim editor.

Note: You won't be able to type or Insert anything on the editor right now, because it doesn't open by default in Insert mode.

To enable Insert mode you need to press the i key on you keyboard, Now you'll be able to write like on a normal editor.

You can see the editor is in Insert mode now.

Tricky part
Now that you have done writing on vim you need to save it and come out of the editor, To save you progress we first need to come out of insert mode by pressing esc key, now we type :w command and this will save our progress, To come out of vim after saving progress, we use :q command.

We can use :wq! command directly to save and exit out of the editor.

you can also use the command :x, for the same.

You can simply write the command bash or sh (we'll talk about them soon) followed by the file name to run your sample script.

Now you might have noticed , that I've changed the file extension to .sh, The .sh extension in filenames like bashscriptfile.sh indicates that the file is a shell script, which contains a sequence of commands to be run by a Unix shell.

Now, Let's write a much better shell script. In the above file we haven't used #!/bin/bash in our vim editor, this is because it was a very normal script that was just printing the command we provided but in reality we work with complex scripts that has lots of code which needs to be compiled/interpreted.

The shebang line, #!/bin/bash, at the start of a shell script tells the operating system which interpreter, in this case, the Bash shell located at /bin/bash, should be used to execute the script, ensuring it runs correctly and consistently regardless of the user's default shell.

#! : These two characters indicate that the file is a script, and what follows is the path to the interpreter.
/bin/bash : This is the path to the Bash shell interpreter.

Now, Let's write a shell script that automates the process of file and folder creation, The script will create a folder and 5 files inside the folder when executed.

vim firstscript.sh

#!/bin/bash

# this is used to ask the name of folder/directory u wanted to create.
read -p "Enter the directory name: " dir_name

# if the folder/directory already exist, show error
if [ -d "$dir_name" ]; then
  echo "Error: Directory '$dir_name' already exists."
  exit 1
fi

# Creating the directory
mkdir "$dir_name"

# if we are able to create it or not
if [ ! -d "$dir_name" ]; then
  echo "Error: Failed to create directory '$dir_name'."
  exit 1
fi

# Creating 5 files inside the directory we made.
for i in {1..5}; do
  file_name="$dir_name/file_$i.txt"
  touch "$file_name"
  if [ ! -f "$file_name" ]; then
    echo "Error: Failed to create file '$file_name'."
  fi
done

echo "Directory '$dir_name' and 5 files created successfully."

exit 0

After writing the script save it, come back to terminal and run the below command, Let's see if we really automated file creation.

bash firstscript.sh

It'll then ask you to enter a folder name, I entered "firstdirectory".

🎉🎉 Woo hoo!, you have written you first bash script, If you are enjoying this, consider following me for more such content!!

File Permissions

While executing your shell script, when you type ./ <sample.sh> command to run the file, you will see permission error, this is because you don't have permissions to execute the file.

File permissions in Shell scripting are crucial for managing access control in Unix-like systems. The chmod command is used to change these permissions, determining who can read, write, and execute files.

The chmod command uses numeric codes to set permissions, based on the 4-2-1 rule. Each digit represents the permissions for the owner, group, and others, respectively.
4: Represents read permission.
2: Represents write permission.
1: Represents execute permission.

4+2+1 = 7, It means giving complete access of the file to everyone, (not advised).

chmod 777 demo.txt

chmod +x demo.txt

In the above command, you are giving complete access to file anyone can read write and execute this file.

You can explore more about the chmod command here.

Bash vs Sh

Above we talked command commands for executing our shell scripts. We can use both sh and bash. sh focuses on prioritizing portability and adhering to POSIX standards for maximum compatibility across Unix-like systems. bash is a more feature-rich shell common on Linux and macOS, offering conveniences like command-line editing and arrays storing command history, but sacrificing some portability.

We can say that bash is sh but with some more advanced features.

Networking in Shell scripting

Networking in shell scripting involves using command-line tools within shell scripts to interact with networks, transfer data, and manage network configurations. It allows automation of network-related tasks, such as checking network connectivity, downloading files, configuring network interfaces, connecting to remote instances etc.

Shell scripting's ability to automate tasks, manage configurations, and handle file manipulation makes it invaluable for professionals in system administration, DevOps, and development.

Common Commands
curl: Transfers data to or from a server using network protocols.
Below command will output the HTML code of the Dev Community homepage to your terminal.

curl https://dev.to

ping: Checks the connectivity of a system to another host.

ping www.google.com

ssh: Connects to a remote host securely. The command is used in connecting to remote instances like Ec2.

ssh username@ip-address

Conditional Statements & Positional Arguments

They are the normal if/else statements just like in any other language. It differentiates as fact that at the end we use fi to indicate the ending.

if [[ condition1 ]]
then
  commands1
elif [[ condition2 ]]
then
  commands2
else
  commands3
fi

let's create a very simple script.

vim condition.sh

save the script properly

Positional Arguments : We have a concept of positional arguments in shell script which stores the command-line arguments passed to a shell script. They are referred to as $1, $2, $3, and so on, where $1 is the first argument, $2 is the second argument, and so on. $0 reserved for script file name.

You can type the arguments after the bash command and file name

Loops & Functions

There are primarily three types of loops in shell scripting: for, while, and until. The for loop iterates over a sequence of items, the while loop executes as long as a condition is true, and the until loop executes as long as a condition is false.

for i in {INITIAL_VALUE..TERMINATING_VALUE}
do
  # Code 
done

vim forloop.sh

Now save the file

Functions, As we know are named blocks of code that can be called and reused throughout a script.

function_name() {
  # Commands
}

vim functions.sh

This script defines a function called follow that takes one argument and prints the statement.

Let's write a script using both functions and loops.

vim sumScript.sh

Note: If you forgot how to type inside vim , remember to press i to enable insert mode.

#!/bin/bash

# Calculate the sum of numbers from 1 to n.

sum_numbers() {
  local n=$1
  local sum=0
  local i=1

  while [[ $i -le $n ]]; do
    sum=$((sum + i))
    i=$((i + 1))
  done

  echo "Sum of numbers from 1 to $n is: $sum"
}

while true
do
  read -p "Enter a positive integer or press q to exit:" input
  if [[ "$input" == "q" ]]; then
    break
  fi

  # Check if input is a number
  if ! [[ "$input" =~ ^[0-9]+$ ]]; then
    echo "Invalid"
    continue
  fi

  sum_numbers "$input"
done

echo "Script completed."

save the script

Deep dive: Some More Interesting Commands 🚀

Piping | & Redirections (>>)

Piping, represented by the | symbol, is a powerful feature in shell scripting that allows you to chain commands together. The commands after the pipe symbol are always dependent on the commands before the symbol.

echo 'Hello, Bash piping!' | wc -w

The wc -w command counts the number of words in its input, resulting in the output ‘3’, which clearly shows how it is dependent on the input.

Grep Command

The grep command is a powerful tool in Linux used for searching text or patterns within files. In order to search for a keyword, process or anything that is specific to an organization, grep command comes in handy.

grep "keyword" <filename>

Here's how I searched for the word demo in the file created above.

Now, Let me demonstrate how to find running processes on your system using the grep command. Grep is invaluable in industries for identifying keywords of interest within large log files, very much like we searched for particular process.

ps -ef | grep "postgres"

Sed Command

The sed command, also "stream editor," is a command-line utility in Unix-like operating systems used for text manipulation.
You can perform search, replace, insert, and delete operations on text within files or input streams without opening the file in a text editor.

let's run a bash script for replacing text within a file. Let's make a touch sedfile.txt and write "This is a bash scripting tutorial" in it , followed by:

vim sedsamplefile.sh

It will open the vim editor, enable insert mode.

Now let's run our scripts.

Note: I have already written the content inside the sedfile.txt using vim editor. Now we'll run the scripts.

You can see that scripts worked as expected and bash gets replaced by shell.

sed can be used to convert data formats, clean up messy data, automating repetitive text editing tasks in shell scripts, find and replace operations across multiple files, and generating code snippets.

Awk Command

The awk command in shell scripting is a powerful text-processing tool used for pattern scanning and processing. It can filter and manipulate text data, making it useful for reports, calculations, and text transformations. AWK processes files line by line, splitting each line into fields.

vim awk_script.sh

Let's add a script to show only the usernames of logged in users.

save the file

You can clearly see it prints the name of users who logged in today.

# The $2 indicates the column 2 which was assigned to username and similarly $1 was assigned to date.

awk '{print "User: " $2 " logged in on " $1}'

AWK is commonly used for log processing, data extraction, and text transformation. For example, if we have a log file with columns like date, username, and action performed like the above one, we can use AWK to extract only the relevant information like username above.

I/O Redirections : Redirections allows you to change the input and output streams of commands. By default, commands receive input from keyboard and send output to terminal screen But Redirection lets you change where the input comes from or where the output goes.

Redirecting Output to a File (>)

Earlier we have used vim editor to edit the sedfile.txt but it can be done using the terminal only, using redirection operators.

The > operator is used to send output to a file, overwriting any existing content.

echo "Hello, World!" > sample.txt

This will insert the text "Hello, world!" inside sample.txt file.

Appending Output to a File (>>)

The >> operator appends output to an existing file instead of overwriting it.

echo "This is an appended line." >> sample.txt

This will append the line below the "Hello, world!" text.

Input Redirection (< Command)

The < operator is used to take input from a file instead of the keyboard.

 cat < output.txt

Redirecting Both Output and Errors (&> and 2>)

Redirects error messages to a file.

ls random_file 2> error_log.txt

Output will be ls: cannot access 'random_file': No such file or directory

Redirecting Both Output and Error (&> Command)

Sends both standard output and standard error to a single file.

ls valid_file Invalid_file &> output.txt

Expected output:

valid_file
ls: cannot access 'Invalid_file': No such file or directory

Earlier we have used vim editor to edit the sedfile.txt but it can be done using the terminal only, using redirection operators.

I/O redirection is a fundamental Linux concept that allows users to manage data efficiently.

Error Handling and Debugging

Error handling as always is crucial for creating reliable shell scripts. It involves managing unexpected situations during program execution and ensuring that the script behaves predictably.

Basic Error Handling

# This option causes the script to exit immediately if a command fails.
set -e

Let's write a sample script.

vim sample.sh

Adding a script that can print the content of a file but here we'll be learning how set -e comes in handy during an error.

In the above script I'm trying to view a non-existing file and set -e will be executed immediately and the script breaks.

Practical usecase: Let's say, If any of the AWS CLI commands (e.g., aws ec2 run-instances, aws ec2 authorize-security-group-ingress) fails, the script will immediately exit. This prevents the script from continuing to provision resources in a potentially broken state.

Trap : The trap command in Bash is a built-in function that allows you to execute a specific action when a script receives a particular signal.

Signal: Signals are software interrupts sent to a process to notify it of an event.

Task Scheduling

I believe we all know what Scheduling our tasks means. In shell scripting, it can be done using tools like crontab, It's a utility in Unix-like operating systems that allows you to schedule jobs to run automatically at specific intervals. These scheduled jobs are known as "cron jobs."
Practical usecases include Ec2 Instance management, syncing data at scheduled time with database etc.

To dug deeper into task scheduling and Crontab you can explore here.

Background and Foreground Jobs

Foreground jobs, runs directly in the terminal and block further input until completion, and background jobs, enabled by appending "&" to a command, which free up the terminal.

These jobs are essential for efficiency, especially in software development where compiling code, running tests, and deploying applications can be time-consuming.

By running lengthy tasks like model training or database backups in the background, while other monitoring tasks are in the foreground, users can continue to work, maximizing productivity and resource utilization.

Time to understand this through Scripts.

vim sample.sh

Copy the below script

#!/bin/bash

# Foreground task
echo "Starting a foreground task..."
sleep 3 # Wait for 3 seconds
echo "Foreground task completed."

# Background task
echo "Starting a background task..."
sleep 5 & # Run the sleep command in the background
echo "Background task started!"

echo "Doing other things while background task runs..."
sleep 1
echo "Script is still running."

#The other process runs while this message is being displayed.
sleep 2
echo "All done."

This script demonstrates the use of foreground and background jobs.

You can see "Starting a foreground task..." and then have to wait 3 seconds before anything else happens. This demonstrates that foreground tasks block further input until completion.
You can see "Starting a background task..." and "Background task started!" almost immediately, before the background task's sleep 5 command is finished. This is because the & puts the sleep 5 command in the background.
After starting the background task, the script immediately moves on to print "Doing other things while background task runs..." and then pauses for another sleep 1. This demonstrates that the terminal is free to perform other tasks.
The final "All done." message appears even though the background task (sleep 5) is likely still running. The script doesn't explicitly wait for the background task to complete before exiting.

Shell scripting best Practices 🌟

Keep Scripts Simple: Aim for simplicity by ensuring each script has a single responsibility.
Use Version Control: Store your scripts in a version control system (like Git) to track changes, collaborate with others.
Check Working Directory: Verify that the script is being executed in the correct directory.
Logging and Troubleshooting: Implement logging mechanisms to capture script execution details and errors.
Documentation and Help Options: Include documentation within the script using comments, and provide help options (e.g., -h or --help) to guide users on how to use the script effectively.
Use Local Variables in Functions: When defining functions, use local variables to limit their scope and avoid unintended interactions with global variables.

Real world applications of shell scripting 🌠

Automation of Infrastructure: AWS relies on shell scripts to automate many infrastructure management tasks. For instance, shell scripts can automatically create new servers, configure resources, and manage updates, ensuring consistent and reliable operation across its vast infrastructure.
Routine Checkups: Google uses shell scripts to automate routine system administration tasks, such as user account management, log file maintenance, and monitoring system resources. This automation helps maintain system health and stability
Configuration Management: Shell scripts ensure all the settings for applications are correct and consistent. They can set up database configurations, manage how applications interact with other services, and handle log management.
Deploying Applications: Shell scripts can be used to automate application deployment by packaging applications, copying them to servers, and configuring them to run in the desired environment.

Further Learning Resources

Conclusion

Yes, It's finally over, We've explored the depth and breadth of shell scripting, We've seen how it powers critical tasks in DevOps, system administration, data processing, and more, within organizations of all sizes, including giants like AWS and Google. By automating routine tasks, managing configurations, monitoring systems, and facilitating deployments, shell scripting continues to be an indispensable tool.

Congratulations 🎉🎉 & Thank you so much if you really made it to the end.

If you have enjoyed this whole tutorial, consider sharing it with someone who really needs this and follow me for more such content.
For paid collaborations mail me at : harshit77dev@gmail.com

Feel free to reach out to me on Twitter, LinkedIn, Youtube, Github.
Thankyou Once Again!!