The other day, I ended up in an unenviable position. I was running low on disk space on my Linux laptop, I got a warning as expected. I deleted some unwanted files, and emptied the trash. That bought me some free space.
I ran BleachBit to do a system-wide clean to reclaim more. While Bleachbit was running, I was doing quite a bit in parallel. I was processing some raw images, playing a video in the background, downloading a torrent, I had a sprawling browsing session in Firefox, and writing code. Way too much!
Long story short, I had a suspicion that BleachBit did not finish the cleaning task, but did not know for sure when I had closed windows! I did not notice anything unusual. So, I shutdown my laptop.
Next time, I turned on the laptop, the splash screen for Ubuntu came up, and then it was just a black screen. ⬛
I could see and move the cursor, but it would not boot the OS completely.
My guess was that Bleachbit hoarded the free disk space and did not release it. I just needed to troubleshoot to see how I could reverse this scenario.
Luckily, I had a bootable pen drive with Manjaro on it, so I could boot into another OS quickly and look at the disk to see what state it was in. 🦸
Spoiler alert: I fixed the issue. Maybe, this might save your bacon some day, but I hope you never get into this situation!🤞
Troubleshooting
Once Manjaro was booted, I opened the file explorer to see if the hard disk was mounted.
It was.
Next, I opened a terminal to explore. The df command can report on system disk usage. Using it with the -h flag gives you a human-readable summary of your disk usage. Here is an abbreviated view of my system (recreated):
df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.5G 0 7.5G 0% /dev
tmpfs 1.6G 2.1M 1.6G 1% /run
/dev/sda5 480G 456G 0M 95% /
...
And there you have it, my harddrive - /dev/sda5 - has no available space!
How the hell do I fix that?
I deleted some files and ran the command again. There still was no available space.
I looked up how to empty the trash bin of a disk on the command-line. Since I am doing this external to the installed operating system, it is not straightforward. I couldn't get it to work.
The issue is likely to be that the space allocated to the files that have been deleted has not been released. How do I trigger that release process?
The most common root cause suggested is that the space is being held by open files, files with a process still attached. In my case, a BleachBit process was killed improperly. Maybe that is the issue.
I looked at this troubleshooting guide - Space is not released after deleting files in Linux? - to see if I could replicate what they did. The first suggestion they made is use the lsof command. This requires that you are logged into the operating system, so that was out.
The second suggestion requires flushing the proc filesystem. This did not work for me.
I was stuck.
I called it a day. After a while away from the screen, I remembered that the Use% field showed 95% of the disk was being used. What about that 5%? 🤔
The backdoor 🚪
By default, ext3 and ext4 filesystems on Linux reserve 5% of their capacity for special processes that can continue to run when diskspace is running low. We can change this allocation and reclaim that space!
We can use the tune2fs command to adjust filesystem parameters on ext3 and ext4 filesystems. We can use this command to free the reserved space. The -m flag allows you to set the reservered blocks percentage. To set it to zero, run the following command:
tune2fs -m 0 /dev/sda5
And bingo, we have a few gigabytes of disk space available!
I restart the laptop and Ubuntu boots. Great! 🌟
However, we still have gigabytes of dead space.
I tried the lsof command to see about the open files from deleted processes. We can run the following command to get that list:
lsof | grep -i deleted
I did not see anything that looked like it belonged to BleachBit. Not a clear way to free that space.
Maybe, I can just run BleachBit?
I did a data backup and gave it a go.
And it finished. I checked the disk usage, andddddd I got the reported 30.55G of free space back! Halleujah! 🙌
We are done yet though! Don't forget to restore that reserved system space. I decided to reduce the reserved space to 3% as below:
tune2fs -m 3 /dev/sda5
Now, we are fully recovered. Business as usual! 👔
Post mortem
My solution worked, but it was slightly unorthodox. After the fact, I found some more answers to fill the gaps in my knowledge.
John P Rouillard left a comment on a cross-posted version of this article:
It's possible that your system booted just fine but couldn't bring X [X Window System] up. IIRC [If I recall correctly] parts of X (like xdm) do not run as root would not be able to get the required disk space. Did you try changing to an alternate virtual console and log in as the root user?
If I understand what you were seeing, logging in on a virtual console as root should have worked. Then you could have looked for files to clean up. With all your processing you may have filled up /tmp or /var/tmp with files. You could have deleted these files freeing enough disk space to get a graphic console back so you could run beachblit again.
The route I was unable to take due to lack of knowledge was to log into the system via a console to troubleshoot. This can done through Single User Mode, or through a virtual terminal.
Single User Mode is preferable as you are likely to be able to login as the root user (or as a "superuser") to issue commands without restrictions.
Log in via single user mode
Single User Mode that is a restricted mode for maintenance purposes. It is a mode in Unix-like operating systems such as Linux operate, where a handful of services are started at system boot for basic functionality to enable a single superuser to perform certain critical tasks. The single user mode does not have any graphical interface or any networking services running.
You can read this forum post on Ask Ubuntu to understand how to log into single user mode from Linux's default boot loader, GRUB (GRand Unified Bootloader).
Log in via a virtual terminal
Linux is a multi-user system, which allows many users to work on it simultaneously via virtual terminals. A virtual terminal gives you a way to interact with the system through a text interface. You can log into a virtual terminal as the system boots. This is useful if the graphical windows system, X (X Window System), locks up or something else goes wrong, and you need a way to try to fix the system.
A virtual terminal is the same as a regular terminal, only that you cannot use the mouse. These are some of the shortcuts that you should be aware of while working on virtual terminals.
| Shortcut | Function |
|---|---|
| Home or Ctrl + a | Move the cursor to the start of the current line |
| End or Ctrl + e | Move the cursor to the end of the current line |
| Tab | Autocomplete commands |
| Ctrl + u | Erase the current line |
| Ctrl + w | Delete the word before the cursor |
| Ctrl + k | Delete the line from the cursor position to the end |
| reset | Reset the terminal |
| history | List of commands executed by the user |
| Arrow up | Scroll up in history and enter to execute |
| Arrow down | Scroll down in history and enter to execute |
| Ctrl + d | Logout from the terminal |
| Ctrl + Alt + Del | Reboot the system |
There are six virtual terminals which can be launched using the shortcut keys Ctrl+Alt+F1 through to Ctrl+Alt+F6.
To login into a virtual terminal:
- Start the computer and press Ctrl+Alt+F1. That should move you to the first virtual terminal, which is also where the boot output is redirected.
- Enter your User ID and Password.
- The virtual terminal should be ready to interact with the system.
Top comments (5)
Once you shut down the system there were no processes remaining to keep a file open. So any file that had been deleted but kept allocated by an open process would have been de-allocated when you shutdown the system. That's why lsof didn't show anything.
The 5% free space you reclaimed using tunefs should have been usable by any process running as root. I'm not clear on what you were seeing when the system wouldn't boot. Usually when you start X windows for a graphical console, you are usually sent to a virtual console (usually number 7 IIRC). Was the "moving cursor" you described a mouse cursor?
It's possible that your system booted just fine but couldn't bring X up. IIRC parts of X (like xdm) do not run as root would not be able to get the required disk space. Did you try changing to an alternate virtual console and log in as the root user?
If I understand what you were seeing, logging in on a virtual console as root should have worked. Then you could have looked for files to clean up. With all your processing you may have filled up /tmp or /var/tmp with files. You could have deleted these files freeing enough disk space to get a graphic console back so you could run beachblit again.
Hi John,
To answer your questions.
Yes, it appeared that I had a mouse cursor after the ubuntu splash screen appeared. I was tired going through it, so I just recorded a couple facts and came back to it later.
No. I did not know how. How would I do that?
I wrote it off being able to solve it as a root user. So I looked to solve the problem by booting into Manjaro on the pen drive.
What you're saying sounds plausible. Thanks for the input.
hold alt and control keys down and press f1. That should move you to the first virtual
console which is also where the boot output is redirected.
See: makeuseof.com/what-are-linux-virtu...
Also I don't remember if root is enabled for local login. If not you could boot single user mode:
askubuntu.com/questions/132965/how...
Good to know! Thanks for sharing 🙂