DEV Community

Juan Bailon
Juan Bailon

Posted on

How to Set Up Selenium as a Linux Daemon with systemd

#linux #selenium #python #systemd

Setting up and running Chrome and Selenium on the ubuntu or debian. The guide is based on ubuntu 22.04

Selenium is great for automating web tasks, but keeping a bot running 24/7 on a server can be tricky. By using systemd, you can run your Selenium bot as a background service (daemon), ensuring it runs reliably and restarts on failure. This guide will walk you through the steps to set it up, with a focus on configuring it for a Linux VPS.

Table of Contents

  1. Installing Google Chrome

  2. Setting up the virtual environment

  3. Installing necessary packages

  4. Creating the Python script

  5. Setting up the systemd service

  6. Fixing block buffering issues

  7. Accessing logs using journalctl

  8. References

Installing Google Chrome

First, update all packages.

sudo apt update
Enter fullscreen mode Exit fullscreen mode

Download the stable Google Chrome package.

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
Enter fullscreen mode Exit fullscreen mode

Install Google Chrome.

sudo apt install -y ./google-chrome-stable_current_amd64.deb
Enter fullscreen mode Exit fullscreen mode

Check the installed Google Chrome version.

google-chrome --version
Enter fullscreen mode Exit fullscreen mode

Setting up the virtual environment

These steps are not mandatory if you're solely running the Selenium bot on your machine. However, it's recommended if you're working on other projects or need isolated environments.

Lets create our virtual environment.

python3 -m venv venv
Enter fullscreen mode Exit fullscreen mode

Activate the virtual environment.

source venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

Installing necessary packages

Now, install selenium and webdriver-manager.

pip install selenium
pip install webdriver-manager
Enter fullscreen mode Exit fullscreen mode

The purpose of webdriver-manger is to simplify the management of binary drivers for different browsers. You can learn more about it in its documentation.

Creating the Python script 

## main.py

from selenium import webdriver
## ---- Use for type hint ---- ##
from selenium.webdriver.chrome.webdriver import WebDriver
## --------------------------- ##
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager


def create_chrome_web_driver_connection(headless: bool,
                                       detach:bool,
                                       use_sandbox: bool,
                                       use_dev_shm: bool,
                                       window_width: int = 1052,
                                       window_height: int = 825
                                       ) -> WebDriver:

    service = Service(ChromeDriverManager().install())
    options = Options()
    options.add_experimental_option("detach", detach)
    options.add_argument(f"--window-size={window_width},{window_height}")
    options.add_argument("--disable-extensions")
    options.add_argument("--disable-renderer-backgrounding")
    options.page_load_strategy = 'normal'

    if not use_sandbox:
        options.add_argument('--no-sandbox')
    if not use_dev_shm:
        options.add_argument('--disable-dev-shm-usage')
    if headless:
        options.add_argument("--headless=new")

    driver = webdriver.Chrome(service= service, options=options)

    return driver



if "__main__" == __name__:
    driver =  create_chrome_web_driver_connection(headless= True,
                                                 detach= False,
                                                 use_sandbox= False,
                                                 use_dev_shm= False)

    driver.get('https://python.org')
    print(driver.title)

    driver.close()
Enter fullscreen mode Exit fullscreen mode

options.add_experimental_option("detach", detach) :

  • This allows you to configure whether the Chrome browser remains open after the script has finished executing.
  • If detach is True, the browser window will stay open even after the WebDriver session ends.

options.add_argument(f"--window-size={window_width},{window_height}") :

  • This sets the window size for the browser in terms of width and height.

You can remove this line if you want to.
If you plan to run Selenium in headless mode, make sure to set the window size this way; otherwise, in my experience, the default window size might be too small.

You can check your window size with this command driver.get_window_size()

options.add_argument("--disable-extensions") :

  • Extensions can interfere with automated browser interactions, so disabling them can improve stability.

options.add_argument("--disable-renderer-backgrounding") :

  • This prevents Chrome from deprioritizing or suspending background tabs.
  • This can be useful when performing actions across multiple tabs.

options.page_load_strategy = 'normal' :

  • This sets the page load strategy to normal, meaning Selenium will wait for the page to load fully before proceeding with further commands.
  • Other options include eager (wait until the DOMContentLoaded event) and none (not waiting for the page to load), you can learn more about it here.

options.add_argument('--no-sandbox') :

  • The sandbox is a security feature of Chrome that isolates the browser's processes.
  • Disabling it (--no-sandbox) can be useful in some testing environments (e.g., in Docker containers or when executing the script as the root user) where the sandbox causes permission issues or crashes.

options.add_argument('--disable-dev-shm-usage') :

  • /dev/shm is a shared memory space often used in Linux environments. By default, Chrome tries to use it to improve performance.
  • Disabling this (--disable-dev-shm-usage) can prevent crashes in environments where the shared memory is limited.

options.add_argument("--headless=new") :

  • This enables headless mode, which runs Chrome without a GUI.
  • Headless mode is useful for running in environments without a display, such as CI/CD pipelines or remote servers.

Setting up the systemd service

Adding ENV variables (optional)

In case your selenium program needs to use environment variables there are two ways in which you can achieve this:

  1. Using a .env file with a library like python-dotenv (the more common/popular method).

  2. Using systemd’s built-in option to set up an Environment File.

For this example, we will use the second option.

First, let's create create a conf.d directory inside the /etc directory.

sudo mkdir /etc/conf.d
Enter fullscreen mode Exit fullscreen mode

Next, create a plain text file(this will be our environment file).

sudo touch /etc/conf.d/selenium-bot.env
Enter fullscreen mode Exit fullscreen mode

Now you can add your environment variables to the file we just created.

USERNAME=Mr.user_123
Enter fullscreen mode Exit fullscreen mode

Modify the Python script to use the environment variables.

import os

""" The rest of the code stays the same """

if "__main__" == __name__:
    driver =  create_chrome_web_driver_connection(headless= True,
                                                 detach= False,
                                                 use_sandbox= False,
                                                 use_dev_shm= False)

    USERNAME = os.getenv('USERNAME')
    print(f'Hellooo! {USERNAME}')

    driver.get('https://python.org')
    print(driver.title)

    driver.close()
Enter fullscreen mode Exit fullscreen mode

Service file

You need to create a service file inside this /etc/systemd/system/ directory; this is where systemd units installed by the system administrator should be placed.

sudo touch selenium-bot.service
Enter fullscreen mode Exit fullscreen mode

For this example im going to assume that you are in a VPS and you want to run the service as the root user.

[!WARNING]
The service will run with root (administrator) privileges. This gives it full access to the system, but running as root is usually avoided unless necessary for security reasons.

Add this to the service file.

[Unit]
Description=Selenium Bot Service
After=network.target

[Service]
User=root
EnvironmentFile=/etc/conf.d/selenium-bot.env
WorkingDirectory=/root/selenium_bot/ 
ExecStart=/root/selenium_bot/venv/bin/python3 main.py
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

[Unit] Section

This section defines metadata and dependencies for the service.

Description=Selenium Bot Service : Provides a short description of what the service does. In this case, it describes it as the "Selenium Bot Service." This description is used in system logs and by systemctl to identify the service.

After=network.target: This ensures that the service starts only after the network is available. The network.target is a systemd target that indicates basic networking functionality is up.

[Service] Section

This section specifies the configuration of the service itself, including how it runs, which user runs it, and what to do if it fails.

User : Specifies the user under which the service will run. Here, it’s set to root.

EnvironmentFile : Specifies a file that contains environment variables used by the service.

WorkingDirectory: Specifies the directory from which the service will run. This is the working directory for the service process. Here, the bot files are stored in /root/selenium_bot/

ExecStart : Defines the command to start the service. Here, it runs the main.py file in /root/selenium_bot/

Restart=on-failure : Configures the service to restart automatically if it exits with a failure (i.e., non-zero exit status). This is useful to ensure that the bot service remains running, even if there are occasional failures.

RestartSec=5s : Specifies the delay between restarts in case of failure. In this case, the service will wait 5 seconds before attempting to restart after a failure.

StandardOutput=journal : Redirects the standard output (stdout) of the service to the systemd journal, which can be viewed using journalctl. This is useful for logging and debugging purposes.

StandardError=journal : Redirects the standard error (stderr) output to the systemd journal. Any errors encountered by the service will be logged and can also be viewed using journalctl

[Install] Section

This section defines how and when the service should be enabled or started.

WantedBy=multi-user.target : Specifies the target under which the service should be enabled. In this case, multi-user.target is a systemd target that is reached when the system is in a non-graphical multi-user mode (common in servers). This means the service will be started when the system reaches this target, typically when the system has booted to a multi-user environment.

To learn more about all the possible settings for a systemd service check the references

Running the service

Let's check that our service file is valid; if everything is okay, nothing should be displayed.

sudo systemd-analyze verify /etc/systemd/system/selenium-bot.service
Enter fullscreen mode Exit fullscreen mode

Reload the systemd configuration, looking for new or modified units(services).

sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Start/Restart your service.

sudo systemctl restart  selenium-bot.service
Enter fullscreen mode Exit fullscreen mode

Stop your service.

sudo systemctl stop  selenium-bot.service
Enter fullscreen mode Exit fullscreen mode

Check your service status.

sudo systemctl status  selenium-bot.service
Enter fullscreen mode Exit fullscreen mode

Do this if you want your service to start automatically at boot.

sudo systemctl enable selenium-bot.service
Enter fullscreen mode Exit fullscreen mode

Fixing block buffering issues

In python when you run your script in an interactive environment( e.g., when you manually run python3 filename.py in a terminal) Python uses line buffering. This means that output, like the one from a print() statement, is shown immediately.

However, when the Python program is run in a non-interactive environment (this is our case), the output will use block buffering. This means that the program will hold onto its output in a buffer until the buffer is full or the program ends, delaying when you can see logs/output.

You can learn more about how python's output buffering works here.

Since we want to view logs and output in real-time, we can address this issue in two ways.

Using the -u flag

The python3 docs tells us this.

-u Force the stdout and stderr streams to be unbuffered. This option has no effect on the stdin stream

By using the -u flag, Python operates in a fully unbuffered mode for both stdout and stderr. This means that each byte is sent directly to the terminal (or any output stream like a log file) as soon as it is produced. No buffering takes place at all.

Every character that would typically go to stdout (like from print() statements or errors) is immediately written, byte-by-byte, without waiting for a full line or buffer to accumulate.

To use this option run your script like this:

python3 -u main.py
Enter fullscreen mode Exit fullscreen mode

If you go with this option make sure to modify the ExecStart inside the service file.

ExecStart=/root/selenium_bot/venv/bin/python3 -u main.py
Enter fullscreen mode Exit fullscreen mode

Using the Print flush argument

In Python, the print() function buffers its output by default, meaning it stores the output in memory and only writes it out when the buffer is full or the program ends. By using flush=True, you can force Python to flush the output immediately after the print() call, ensuring that the output appears right away.

print("Hello, World!", flush=True)
Enter fullscreen mode Exit fullscreen mode

Accessing logs using journalctl

To view the full log history of your systemd unit (service), use the following command.

journalctl -u selenium-article.service
Enter fullscreen mode Exit fullscreen mode

To monitor logs in real time, use the -f flag. This will show only the most recent journal entries, and continuously print new entries as they are appended to the journal.

journalctl -f -u selenium-article.service
Enter fullscreen mode Exit fullscreen mode

References

Top comments (0)

Read next

404_chronicles profile image

🚀 Flet: The Modern Python GUI Framework That's Taking Over in 2024

404_CHRONICLES -

bascodes profile image

This Week In Python

Bas Steins -

siderthecoder profile image

A Simple Tic Tac Toe Terminal Game

Sam -

alexandrecalaca profile image

How to undo a git pull command

Alexandre Calaça -