Managing Storage and Packages in Linux: A Complete Guide

Okay so lets get cracking on what could be the last part of this Linux Fundamentals series, and this is a exciting one to say the least !

Storage Management

1. The Importance of High Throughput and Low Latency Storage

   In computing, the efficiency of storage systems significantly influences application performance. High throughput ensures that large volumes of data are read or written swiftly, which is crucial for tasks like data analysis and multimedia processing. Low latency guarantees immediate data retrieval, vital for real-time applications such as online gaming and financial transactions. Together, high throughput and low latency contribute to a seamless and efficient computing experience.

2. Understanding Block Storage

   Block storage is a method where data is stored in fixed-sized chunks called blocks. Each block is assigned a unique address, allowing the operating system to retrieve data efficiently. For instance, when storing a 10GB file, the system divides it into smaller blocks (commonly 4KB each). Each block is then stored separately, and an index keeps track of these blocks, facilitating quick access and management. This method enhances performance and allows for flexible data management.

AWS EBS(Elastic Block Storage) is an example of Block Storage devices. Block Storage is suitable for transactional databases and more where multiple users access the data and where parallel processing is needed.

1. Exploring Block Devices

   A block device is a hardware or virtual device that allows for reading and writing data in fixed-sized blocks. These devices include hard drives, SSDs, and virtual block devices. In Linux, block devices are represented as files and are typically found under the /dev directory. For example, /dev/sda might represent the first SATA drive in the system. These device files enable the operating system and applications to interact with the hardware seamlessly.

For our demo I will be using an EC2 instance from AWS with a default 8GB storage.

We have logged into our EC2 instance:

To list the available block devices in your system use the following command :

lsblk

What I can see is a xvda block device which has 4 partitions xvda1, xvda14, xvda15 and xvda16.

As I said all the block devices can be found under /dev directory.

1. Diving into Partitions and the GPT Scheme

Partitions are subdivisions of a storage device, allowing users to segment their storage for various purposes, such as separating system files from user data. The GUID Partition Table (GPT) is a modern partitioning scheme that offers several advantages over the older Master Boot Record (MBR) system. GPT supports larger disk sizes, allows for a nearly unlimited number of partitions, and includes redundancy and error-checking features.

Our xvda block device has 4 partitions xvda1, xvda14, xvda15 and xvda16 mounted on separate locations.

But we can also use our disk or block storage as is without necessarily creating any partitions out of it.

1. Filesystems and Mounting: Making Partitions Usable

Now lets attach another block storage to our instance in the form on an AWS EBS Volume.

Above is the default 8GB Volume (xvda) that was attached to our EC2 Instance when we created it.

Created a volume of 10GB :

Now attach this volume to our EC2 instance :

As you can see a new block storage is available under xvdf of 10GB, its the EBS volume we created just now.

Now lets create partitions out of it using gdisk and lets mount them to separate locations.

```
sudo gdisk /dev/partion_or_blockdevice 
```

Choose the n flag to create a new partition out of /dev/xvdf which was our 10GB EBS Block Storage Device.

• Press n → Create a new partition.

• Partition Number → Press Enter to accept the default.

• First sector → Type 2048 (or press Enter for the default, usually aligned).

• Last sector → Type +1G (this tells gdisk to create a 1GB partition).

• Hex Code → Press Enter (default is 8300 for Linux filesystems).

• Press w → Write the changes to the disk.

• Confirm with y

Our partition of 1Gb is created with the name xvdf1.

Possible Actions for the Remaining 9GB:

1. Create More Partitions

* You can create additional partitions using the unallocated space.

* Example: You could create a second 5GB partition and a third 4GB 
  partition.

1. Expand an Existing Partition Later

* If needed, you can extend the **1GB partition** to use more space 
  later using tools like `gdisk`, `parted`, or `resize2fs` (for 
  filesystems like ext4).

1. Leave It Unallocated

* The remaining space will be available for future use but won’t be 
  accessible by the OS until partitioned and formatted.

After creating partitions, they remain raw spaces until formatted with a filesystem, which organizes data for storage and retrieval. A filesystem defines how data is stored and retrieved on a storage device. ext4 is a widely used filesystem in Linux, known for its robustness, support for large volumes, and journaling capabilities, which help in data recovery during unexpected shutdowns.

To format a partition with the ext4 filesystem and mount it:

1. Format the partition:
   

   ```
   sudo mkfs.ext4 /dev/xvdf1
   ```
   

Replace /dev/xvdf1 with your specific partition identifier.

1. Create a mount point:
   

   ```
   sudo mkdir /mnt/xvdf1Mount
   ```
   

2. Mount the partition:
   

   ```
   sudo mount /dev/sdX1 /mnt/mydata
   ```
   

As you can see we have successfully mount our partition of a mount path in the Ubuntu OS of out EC2 Instance. Now you can create, read and write data in that mount path.

1. Persisting Mounts with /etc/fstab

To ensure that partitions mount automatically at boot, you can add entries to the /etc/fstab file specifying the device, mount point, filesystem type, and options :

1. Open /etc/fstab with a text editor:
   

   ```
   sudo vi /etc/fstab
   ```
   

2. Add a line specifying the device, mount point, filesystem type, and options:
   

   ```
   /dev/xvdf1 /mnt/xvdf1Mount ext4 rw 0 0
   ```
   

3. Save and close the file.

This configuration ensures that the system mounts the specified partitions automatically during startup.

So I just rebooted my EC2 Instance but as you can see the partition still exists and is mounted to that path :

And our file also exists :

Package Management

1. Understanding Package Management

   Package management is a fundamental aspect of Linux systems,
   facilitating the installation, upgrade, configuration, and removal
   of software packages. It ensures that software dependencies are
   handled automatically, maintaining system stability and
   consistency.

   1. Package Managers in Debian and RPM Families

   * **Debian Family:**
     - **APT (Advanced Package Tool):** 

A command-line tool that simplifies the process of managing software packages on Debian-based distributions like Ubuntu. It handles package installation, updates, and removal, resolving dependencies automatically.

     - **APT-GET:** An older command-line tool for handling packages, still widely used for script-based operations.

   * **RPM Family:**

     - **YUM (Yellowdog Updater Modified):** A package manager for RPM-based distributions like CentOS and Fedora. It manages package installations, updates, and removals, resolving dependencies automatically.

     -  **DNF (Dandified YUM):** The next-generation version of YUM, offering improved performance and better dependency resolution.

1. Operations with APT and APT-GET

* **Difference Between APT and APT-GET:**

    While both tools serve similar purposes, `apt` is designed to provide a more user-friendly experience, combining functionalities of `apt-get` and `apt-cache`. It's recommended for interactive use, whereas `apt-get` is preferred for scripting purposes.

* **Common Commands:**

    * **Update Package Lists:**

        ```
        sudo apt update
        ```

    * **Upgrade Installed Packages:**

        ```
        sudo apt upgrade
        ```

    We usually run the above commands when we launch a new VM to update and upgrade its packages.

    * **Install a Package:**

        ```bash
        sudo apt install package_name
        ```

    * **Search for a Package:**

        ```bash
        apt search package_name
        ```

    * **Remove a Package:**

        ```bash
        sudo apt remove package_name
        ```

    * **Edit Sources List:**

        The package sources are listed in `/etc/apt/sources.list`. To edit:

        ```bash
        sudo nano /etc/apt/sources.list
        ```

• List Installed Packages:

      apt list --installed
  

• How Package Managers Work:

  Package managers like APT work by connecting to remote repositories specified in the `/etc/apt/sources.list` file. They download package information and updates, ensuring that software installations and upgrades are consistent and reliable.
  

Overview

I hope this blog post helped you understand the standard partitioning scheme, storage is divided into fixed partitions during disk setup. Each partition is formatted with a filesystem and mounted for use and the Package Management in the Linux Systems. If you have any more questions or need further clarification on any topic, feel free to ask!

Next up is Traditional Partitioning vs. Logical Volume Management (LVM): Why LVM is the Better Choice. I know I said this could be the last of the series but this just got more interesting than I anticipated it to be. Also this post is getting too big! :)