Learning Objectives:
- Develop systematic approaches to diagnosing and resolving complex system issues
- Master system recovery techniques using live environments and rescue procedures
- Learn advanced debugging tools and log analysis for root cause identification
- Build expertise in hardware troubleshooting and driver conflict resolution
Installation problems are often the first troubleshooting challenges Arch Linux users encounter. Understanding these issues and their solutions builds foundational problem-solving skills.
Boot and UEFI Problems: Modern systems with UEFI firmware present unique challenges that require specific troubleshooting approaches.
UEFI Boot Failures: When your system fails to boot after installation:
# Boot from Arch Linux ISO
# Check if EFI variables are accessible
ls /sys/firmware/efi/efivars
# Mount your system partitions
mount /dev/sda2 /mnt # Root partition
mount /dev/sda1 /mnt/boot # EFI partition
# Chroot into your system
arch-chroot /mnt
# Reinstall bootloader
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
grub-mkconfig -o /boot/grub/grub.cfg
# Check EFI boot entries
efibootmgr -v
# Add boot entry if missing
efibootmgr --create --disk /dev/sda --part 1 --loader /EFI/GRUB/grubx64.efi --label "Arch Linux"
Partition and Filesystem Issues: Resolve common storage-related problems:
# Check filesystem integrity
fsck /dev/sda2
# Repair filesystem errors
fsck -y /dev/sda2
# Check partition table
fdisk -l /dev/sda
# Repair GPT partition table
gdisk /dev/sda
# Use 'r' for recovery menu, 'b' to rebuild GPT
# Mount with specific options for problematic filesystems
mount -o ro,noload /dev/sda2 /mnt # Read-only mount for ext4
Network Configuration During Installation: Resolve connectivity issues that prevent package downloads:
# Check network interface status
ip link show
# Bring interface up
ip link set wlan0 up
# Scan for wireless networks
iwctl station wlan0 scan
iwctl station wlan0 get-networks
# Connect with manual configuration
iwctl station wlan0 connect "SSID"
# Test connectivity
ping -c 3 archlinux.org
# Check DNS resolution
nslookup archlinux.org
# Manual DNS configuration if needed
echo "nameserver 8.8.8.8" > /etc/resolv.conf
Package Installation Failures: Diagnose and resolve pacstrap and package installation issues:
# Check available disk space
df -h /mnt
# Verify package signatures
pacman-key --refresh-keys
# Force package database refresh
pacman -Syy
# Install with verbose output for debugging
pacstrap -v /mnt base linux linux-firmware
# Check for corrupted packages
pacman -Qk
# Clear package cache and retry
pacman -Scc
System recovery skills are essential when your Arch Linux installation becomes unbootable or severely damaged. These techniques can save hours of reinstallation time.
Live Environment Recovery: Use the Arch Linux ISO as a rescue system:
Accessing Your Installed System:
# Boot from Arch Linux ISO
# Identify your partitions
lsblk
fdisk -l
# Mount root partition
mount /dev/sda2 /mnt
# Mount additional partitions
mount /dev/sda1 /mnt/boot # Boot partition
mount /dev/sda3 /mnt/home # Home partition (if separate)
# Mount virtual filesystems
mount -t proc /proc /mnt/proc
mount -t sysfs /sys /mnt/sys
mount -o bind /dev /mnt/dev
mount -t devpts /dev/pts /mnt/dev/pts
# Chroot into your system
chroot /mnt /bin/bash
# Update environment
source /etc/profile
export PS1="(chroot) $PS1"
Fixing Broken Package Updates: Recover from failed system updates:
# Inside chroot environment
# Check for partial upgrades
pacman -Qk
# Force database lock removal if needed
rm /var/lib/pacman/db.lck
# Downgrade problematic packages
pacman -U /var/cache/pacman/pkg/package-old-version.pkg.tar.xz
# Reinstall core packages
pacman -S --needed base linux
# Fix broken dependencies
pacman -Syu --overwrite "*"
# Rebuild package database if corrupted
pacman-db-upgrade
Bootloader Recovery: Restore bootloader functionality:
# GRUB recovery
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
grub-mkconfig -o /boot/grub/grub.cfg
# systemd-boot recovery
bootctl --path=/boot install
bootctl --path=/boot update
# Manual boot entry creation
mkdir -p /boot/loader/entries
cat > /boot/loader/entries/arch.conf << EOF
title Arch Linux
linux /vmlinuz-linux
initrd /initramfs-linux.img
options root=/dev/sda2 rw
EOF
Kernel and Module Issues: Resolve kernel-related boot problems:
# Boot with fallback initramfs
# Edit GRUB entry and change:
# initrd /initramfs-linux.img
# to:
# initrd /initramfs-linux-fallback.img
# Rebuild initramfs
mkinitcpio -p linux
# Check for missing modules
lsmod
modprobe module-name
# Add modules to initramfs
nano /etc/mkinitcpio.conf
# Add to MODULES=() array
mkinitcpio -p linux
Advanced debugging techniques help identify root causes of complex system issues that aren't immediately obvious.
System Log Analysis: Master journalctl for comprehensive log investigation:
Advanced Journalctl Usage:
# View logs with priority filtering
journalctl -p err # Error level and above
journalctl -p warning # Warning level and above
# Time-based filtering
journalctl --since "2024-01-01 00:00:00"
journalctl --since "1 hour ago"
journalctl --until "2024-01-01 23:59:59"
# Service-specific debugging
journalctl -u service-name --no-pager
journalctl -u service-name -f # Follow logs
# Boot-specific logs
journalctl -b 0 # Current boot
journalctl -b -1 # Previous boot
journalctl --list-boots # List all boots
# Kernel message filtering
journalctl -k # Kernel messages only
journalctl -k -p err # Kernel errors only
# Export logs for analysis
journalctl --since "1 day ago" > system-logs.txt
Process and System Debugging: Use advanced debugging tools for system analysis:
Process Tracing with strace:
# Install debugging tools
sudo pacman -S strace gdb ltrace
# Trace system calls for a command
strace -o trace.log command
# Trace running process
strace -p PID
# Trace with timing information
strace -T -tt command
# Trace specific system calls
strace -e trace=open,read,write command
# Trace child processes
strace -f command
Memory and Performance Analysis:
# Memory usage analysis
cat /proc/meminfo
free -h
ps aux --sort=-%mem | head
# CPU usage analysis
top -p PID
htop
iotop # I/O usage
# System call analysis
perf top
perf record -g command
perf report
# Network debugging
ss -tuln # Show listening ports
netstat -rn # Show routing table
tcpdump -i any icmp # Capture ICMP traffic
Hardware Debugging: Diagnose hardware-related issues:
# Hardware information
lshw -short
lspci -v
lsusb -v
dmidecode
# Check hardware errors
dmesg | grep -i error
journalctl -k | grep -i error
# Memory testing (requires reboot)
# Add memtest86+ to GRUB menu
sudo pacman -S memtest86+
sudo grub-mkconfig -o /boot/grub/grub.cfg
# Disk health checking
sudo pacman -S smartmontools
sudo smartctl -a /dev/sda
sudo badblocks -v /dev/sda
# Temperature monitoring
sudo pacman -S lm_sensors
sudo sensors-detect
sensors
Service and Dependency Analysis: Debug complex service interactions:
# Analyze service dependencies
systemctl list-dependencies service-name
systemctl list-dependencies --reverse service-name
# Check service startup order
systemd-analyze critical-chain
systemd-analyze critical-chain service-name
# Debug service failures
systemctl status service-name -l
journalctl -u service-name --since "1 hour ago"
# Test service configuration
systemd-analyze verify /etc/systemd/system/service-name.service
# Check for conflicting services
systemctl --failed
systemctl list-units --state=failed
Controlled Failure Testing: Create controlled system failures in a virtual machine environment. Practice recovery procedures until you can perform them confidently.
Log Analysis Skills: Set up log monitoring and practice identifying patterns in system logs. Create alerts for critical system events.
Hardware Diagnostic Routine: Develop a systematic hardware diagnostic procedure. Test it on different systems to build familiarity with various hardware configurations.
Recovery Documentation: Document your system configuration and create recovery procedures specific to your setup. Test these procedures regularly.
Debugging Tool Mastery: Practice using strace, gdb, and other debugging tools on both working and broken systems to understand their output and capabilities.
You've now developed expert-level troubleshooting and problem-solving skills that enable you to diagnose and resolve complex Arch Linux issues. These skills are invaluable not just for Arch Linux, but for any Linux system administration role.
The systematic approach to problem-solving, combined with mastery of debugging tools and recovery techniques, gives you confidence to tackle even the most challenging system issues. Your ability to recover from system failures and diagnose root causes sets you apart as an advanced Linux user.
In the final module, we'll explore advanced topics and specialization areas that build on all your accumulated knowledge to tackle specialized use cases and advanced system configurations.