DEV Community

Cover image for Configuring Your Home Cluster Network: A Complete Guide
Cheedge Lee
Cheedge Lee

Posted on • Originally published at notes-renovation.hashnode.dev

Configuring Your Home Cluster Network: A Complete Guide

It’s a renovated note.

Nowadays many of us will enjoy the cloud cluster rather than build a self managed cluster, as it’s less management, high availability, more secure, pay-as-you-go, and all the advantages you can think of the cloud computing. However, if you accidentally own several old computers, and don’t want to sell/transfer them and don’t know how to deal with them, a home-managed cluster will be a good choice. there is a lot of fun of a self managed cluster.

Architecture

Let’s define our architecture here: 4 nodes: 1 master/worker, 3 worker

Image description

1. master node

  • eth0: connect with institute cable

    • public interface, using DHCP
    • internet download, update
    • users can access it: ssh/scp

  • eth1: connect with worker ndoes

    • private interface, using static IP

    • communicate other code:

      • ssh/scp
      • data transfer
      • parallel communicate

config network interface eth0 and eth1 

# Configure public interface (assumes DHCP from institute network)
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
TYPE=Ethernet
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
# Request static IP from our institute's DHCP server if possible
# This makes routing more reliable
DHCP_CLIENT_ID=cluster-master
EOF

# Configure private cluster network
cat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOF
TYPE=Ethernet
DEVICE=eth1
BOOTPROTO=static
IPADDR=192.168.10.1
NETMASK=255.255.255.0
ONBOOT=yes
EOF

# Apply new network configuration
systemctl restart network
Enter fullscreen mode Exit fullscreen mode

NAT 

# Enable IP forwarding persistently
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf

# Enable connection tracking timeout optimization for HPC workloads
echo "net.netfilter.nf_conntrack_tcp_timeout_established = 86400" >> /etc/sysctl.conf
echo "net.netfilter.nf_conntrack_max = 131072" >> /etc/sysctl.conf

# Apply sysctl changes
sysctl -p

# Set up NAT with higher connection limits
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.10.0/24 -j MASQUERADE
Enter fullscreen mode Exit fullscreen mode

the last command is important, as it will contribute to the return traffic. We will explain later.

setup the firewall 

# Clear existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP  # We'll explain this choice
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established and related connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH from institute network
iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT

# Allow all traffic from the cluster's private network
iptables -A INPUT -i eth1 -s 192.168.10.0/24 -j ACCEPT

# Allow forwarding from cluster to internet
iptables -A FORWARD -i eth1 -s 192.168.10.0/24 -o eth0 -j ACCEPT

# Allow HTTP/HTTPS for package downloads
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT

# Set up local package caching repository later (optional)
iptables -A INPUT -i eth1 -p tcp --dport 80 -j ACCEPT

# Save iptables rules
iptables-save > /etc/sysconfig/iptables
Enter fullscreen mode Exit fullscreen mode

I set the default FORWARD policy to DROP for security reasons:

  • It prevents unauthorized traffic from traversing the master node

  • It creates a default-deny stance, where only explicitly allowed traffic passes

  • It prevents potential lateral movement if one node is compromised

2. worker nodes

config 

# Worker node 1 (192.168.10.2)
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
TYPE=Ethernet
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.10.2
NETMASK=255.255.255.0
GATEWAY=192.168.10.1
DNS1=8.8.8.8
ONBOOT=yes
EOF

# here GATEWAY will auto generate the route rule in table

# Worker node 2 (192.168.10.3)
# Change IPADDR=192.168.10.3 on the second worker node

# Worker node 3 (192.168.10.4)
# Change IPADDR=192.168.10.4 on the third worker node

# Restart network service on each worker node
systemctl restart network
Enter fullscreen mode Exit fullscreen mode

Routing Configuration for Worker Nodes

As we use the master node eth1 (192.168.10.1) as the gateway for work nodes (GATEWAY=192.168.10.1), above setting creates a default route on each worker node that sends all traffic not destined for the local network (192.168.10.0/24) to the master node (192.168.10.1).

$ route -n
# see the result: (send all traffic (0.0.0.0/0) to gateway 192.168.10.1)
0.0.0.0         192.168.10.1      0.0.0.0         UG    0      0        0 eth0
Enter fullscreen mode Exit fullscreen mode

Manual Command

Using manual command can also achieve the same result.

route add -net 0.0.0.0 gw 192.168.10.1
Enter fullscreen mode Exit fullscreen mode

This manually adds a default route to the current routing table. It has the same immediate effect as the configuration file setting, but it's temporary and will be lost after a reboot or network service restart.

The difference is primarily in persistence and when the configuration happens. Using the network configuration file is the standard way to set up permanent routes in CentOS/RHEL systems.

Fire wall 

# Clear existing rules
iptables -F
iptables -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH from cluster nodes only
iptables -A INPUT -p tcp --dport 22 -s 192.168.10.0/24 -j ACCEPT

# HPC/MPI Communication - comprehensive approach
# Allow all TCP/UDP between cluster nodes for parallel computing
# this is optinal if you don't want do parallel computing
iptables -A INPUT -s 192.168.10.0/24 -p tcp -j ACCEPT
iptables -A INPUT -s 192.168.10.0/24 -p udp -j ACCEPT

# Save iptables rules
iptables-save > /etc/sysconfig/iptables
systemctl enable iptables
Enter fullscreen mode Exit fullscreen mode

3. Package management

3.1 Local yum Repo

  1. because the worker node will not have the access to internet, we keep them inside the private, therefore, for package installation, updating, we need to find a way to resolve these.

  2. we want to specify the packages in our yum repo

so here we build a local yum repo

# On master node
# Install required packages
yum install -y createrepo nginx

# Create repository directory
mkdir -p /var/www/html/centos-repo

# Configure Nginx
cat > /etc/nginx/conf.d/repo.conf << EOF
server {
    listen 80;
    server_name _;
    root /var/www/html;

    location / {
        autoindex on;
    }
}
EOF

# Start and enable Nginx
systemctl enable nginx
systemctl start nginx

# Download packages to repository
yum install -y yum-utils
repotrack -p /var/www/html/centos-repo <package-name> 
# Repeat for packages we need

# Create repository metadata
createrepo /var/www/html/centos-repo

# Configure worker nodes to use this repository
cat > /etc/yum.repos.d/cluster-local.repo << EOF
[cluster-local]
name=Cluster Local Repository
baseurl=http://192.168.10.1/centos-repo
enabled=1
gpgcheck=0
EOF
Enter fullscreen mode Exit fullscreen mode

3.2 Optional for package management

In order to realise

  1. worker node package update

  2. self-managed packages

we can also use scp to transfer package from master node.

# On master node, download and transfer RPM
yum install -y yum-utils
yumdownloader <package-name>
scp <package-name>.rpm 192.168.10.1:/tmp/

# On worker node
sudo rpm -ivh /tmp/<package-name>.rpm
Enter fullscreen mode Exit fullscreen mode

3.3 directly route worker node to internet

If we don’t need high security, we can also open the private cluster to public internet. Which will configure the router table and we don’t discuss here.

4. Other Installations

ssh configuration 

# On master node, generate SSH key
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""

# Copy the key to all nodes (including itself)
for i in {1..4}; do
  ssh-copy-id -i ~/.ssh/id_rsa.pub 10.10.10.$i
done

# Do the same on each worker node to allow any-to-any communication
# (Run similar commands on each worker node)
Enter fullscreen mode Exit fullscreen mode

Network Monitor & iptables Log 

# Install tools
yum install -y tcpdump nmap iftop

# Set up automatic monitoring with fail2ban to prevent brute force attacks
yum install -y fail2ban
cat > /etc/fail2ban/jail.local << EOF
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/secure
maxretry = 5
bantime = 3600
EOF

# Start and enable fail2ban
systemctl enable fail2ban
systemctl start fail2ban

# Add logging rules before the final DROP rules
iptables -A INPUT -j LOG --log-prefix "IPTables-Input-Dropped: " --log-level 4
iptables -A FORWARD -j LOG --log-prefix "IPTables-Forward-Dropped: " --log-level 4

# Save iptables rules
iptables-save > /etc/sysconfig/iptables
Enter fullscreen mode Exit fullscreen mode

Optional Parallel Computing Configuration 

# On all nodes (master and workers)
# Install OpenMPI
yum install -y openmpi openmpi-devel

# Configure environment in /etc/profile.d/
cat > /etc/profile.d/mpi.sh << EOF
export PATH=\$PATH:/usr/lib64/openmpi/bin
export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/usr/lib64/openmpi/lib
EOF

# Source the new environment
source /etc/profile.d/mpi.sh

# Test MPI communication
# Create a hostfile
cat > /home/username/hostfile << EOF
192.168.10.1 slots=128
192.168.10.2 slots=128
192.168.10.3 slots=128
192.168.10.4 slots=128
EOF

# Run a simple MPI test
mpirun -np 4 --hostfile /home/username/hostfile hostname
Enter fullscreen mode Exit fullscreen mode

Torque/PBS 

# Install Torque on master node
yum install -y torque-server torque-scheduler torque-client

# Configure server nodes file
cat > /var/torque/server_priv/nodes << EOF
192.168.10.1 np=128
192.168.10.2 np=128
192.168.10.3 np=128
192.168.10.4 np=128
EOF

# Start Torque server
systemctl enable pbs_server
systemctl start pbs_server

# Install Torque on worker nodes
for i in {2..4}; do
  ssh 192.168.10.$i "yum install -y torque-mom torque-client; systemctl enable pbs_mom; systemctl start pbs_mom"
done
Enter fullscreen mode Exit fullscreen mode

5. Traffic Flow:

Forward Chain Traffic Flow in Both Directions

When we create the forward chain, iptables -A FORWARD -i eth1 -s 192.168.10.0/24 -o eth0 -j ACCEPT, this command let the traffic come into eth1 can be forward to eth0, which means traffic from worker nodes to master nodes, and master nodes forward it to institute internet. Here comes the question, where is the backward flow?

iptables -A FORWARD -i eth1 -s 192.168.10.0/24 -o eth0 -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

Let’s see how the traffic flows first.

Outbound Traffic (Worker → Internet)

The rule above allows packets to travel from the worker nodes (coming in on eth1) to be forwarded out to the institute network (through eth0). This handles the first half of any connection, which is the outbound request.

Return Traffic (Internet → Worker)

For the return traffic, typically we will think what we need as:

iptables -A FORWARD -i eth0 -d 192.168.10.0/24 -o eth1 -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

However, if we look closely at the original configuration, here is the command:

iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

This rule is handling the return traffic, because:

  1. When a worker node initiates a connection, the outbound packet creates an entry in the connection tracking table (create a ESTABLISHED connection)

  2. Any returning packets associated with that connection are marked as ESTABLISHED

  3. The rule above allows all ESTABLISHED connections through, regardless of interface

This is more secure than explicitly allowing all traffic from eth0 to eth1, because it only permits return traffic for connections that were initiated from inside our cluster.

If this state tracking rule wasn't present, we would absolutely need to the backward traffic rule. Without either approach, connections would work one-way only, which means worker nodes could send requests, but never receive response…

Connection Flow

Let's trace a web request from a worker node:

  1. Worker (192.168.10.2) tries to access google.com
  2. Packet travels: Worker → Master's eth1
  3. Master checks FORWARD chain, matches -i eth1 -s 192.168.10.0/24 -o eth0 rule
  4. Master performs NAT, changing source IP to its own public IP
  5. Packet leaves through eth0 to institute network
  6. Google responds to master's public IP
  7. Packet arrives at master's eth0
  8. Master checks connection tracking table, sees this is a response
  9. Packet is marked as ESTABLISHED
  10. Master checks FORWARD chain, matches the ESTABLISHED rule
  11. Master performs reverse NAT, changing destination to worker's IP
  12. Packet leaves through eth1 to worker
  13. Worker receives response

More details

more details can check the post before here.

Top comments (0)

Read next

arpitstack profile image

Code to Connection: Why Empathy is Your Secret Weapon in Coding?

Arpit Gupta -

manojratna profile image

Hands-on Android Development: Building Apps with Java & Android Studio

MANOJRATNA DUGGIRALA -

asadullah_ehsan profile image

Strategy Pattern

Asadullah Ehsan -

rifat87 profile image

Python pip installation problem

Tahzib Mahmud Rifat -