Optimizing TrueNAS on Proxmox: The Power of PCIe Passthrough for Improved Storage Management

Introduction to TrueNAS and Proxmox Setup

TrueNAS, with its ZFS-based architecture, is one of the most robust and reliable storage solutions available today. Pairing it with Proxmox, a powerful virtualization platform, allows users to maximize their hardware’s potential by combining virtualization with top-tier storage management.

However, running TrueNAS in a virtual machine (VM) on Proxmox has challenges, particularly with disk management. RAID degradation and disk identification issues are common problems. This post guides you through using PCIe passthrough for SATA expansion cards, ensuring TrueNAS can manage disks directly, reducing virtualization overhead and improving system performance.

Understanding TrueNAS in a Proxmox Environment

TrueNAS typically runs on bare metal, accessing hardware directly, including disks and network interfaces. With Proxmox, you can run TrueNAS as a virtual machine, efficiently using one machine for multiple purposes.

Advantages of running TrueNAS on Proxmox include:

• Centralized management: One server handles NAS duties and other VMs for different applications.
• Flexibility: Proxmox allows easy scaling by adding VMs or containers as needed.

However, managing hardware, especially disks, through virtualization can cause performance issues, especially after reboots. This is where PCIe passthrough becomes important.

Challenges with Disk Management in a Virtualized TrueNAS Setup

In a VM, Proxmox manages hardware, and disks are passed to TrueNAS as virtual devices, leading to issues like:

1. Disk Identifier Changes: Adding or removing disks can change virtual device names, causing TrueNAS to lose track of disks.
2. RAID Degradation: Disks not recognized after reboots may degrade RAID arrays, forcing resilvering.
3. Performance Bottlenecks: Virtualization overhead can impact read/write speeds, especially under load.

Example Commands for Passing Through Individual Disks

Passing drives via Proxmox commands:

qm set 100 -scsi1 /dev/disk/by-id/ata-HGST_HUH721010ALE604_********

qm set 100 -scsi2 /dev/disk/by-id/ata-HGST_HUH721010ALE604_********

This setup was unstable, as disk identifiers changed after reboots, causing RAID degradation and resilvering.

Why PCIe Passthrough Is a Better Approach

I passed through the entire PCIe SATA controller instead of individual disks to avoid disk recognition issues. This lets TrueNAS manage the disks directly, avoiding Proxmox’s virtualization layer.

What is PCIe Passthrough?

PCIe passthrough allows a VM to access physical PCIe devices directly. In this case, TrueNAS manages the disks as if they were running on bare metal by passing through the SATA controller.

Benefits of PCIe Passthrough

• Direct Control: TrueNAS controls the SATA controller and disks, minimizing RAID degradation risks.
• Improved Stability: Device names do not change after reboots, improving system reliability.
• Better Performance: Removing virtualization overhead improves I/O performance, especially for resolving or scrubbing.

Configuring PCIe Passthrough in Proxmox for TrueNAS

1. Enable IOMMU on Proxmox

Enable IOMMU to allow PCIe devices to be passed directly to a VM.

For Intel CPUs: Add the following to /etc/default/grub:

intel_iommu=on

For AMD CPUs: Add:

amd_iommu=on

Then update grub and reboot:

update-grub
reboot

2. Verify IOMMU Groups

Run this command to verify IOMMU is enabled and hardware is correctly identified:

find /sys/kernel/iommu_groups/ -type l

3. Pass Through the PCIe Device

In the Proxmox web interface:

1. Navigate to your TrueNAS VM.
2. Under Hardware, select Add > PCI Device.
3. Select your PCIe SATA controller from the list (e.g., Marvell 88SE9215 PCIe 2.0 x1 controller).
4. Check the PCIe option and select All Functions to pass the entire controller to TrueNAS.

4. Detach Virtualized Disks

If individual disks were passed to TrueNAS using qm set, remove these from the VM configuration, and TrueNAS will access the disks via the PCIe controller.

Resilvering and Disk Management in TrueNAS

After passing through the PCIe controller, TrueNAS detected all NAS drives and started a resilvering process for RAIDZ3 pool data integrity.

How Long Does Resilvering Take?

The duration depends on:

• Disk Size: Larger disks (e.g., 10TB HGST) take longer, around 24-48 hours.
• System Load: Reducing system load during resilvering speeds up the process.
• Disk Health: Slow or degraded disks may lengthen the process.

Monitoring the Resilvering Process

Check re-silvering status with:

zpool status

Additional Considerations: Using M.2 Slots for Storage Expansion

For additional storage, consider using M.2 SATA expansion cards if PCIe slots are occupied by other devices, such as NICs.

M.2 to SATA Expansion

M.2 SATA expansion cards allow you to connect additional SATA drives through the M.2 slot, especially useful when PCIe slots are occupied.

Key Considerations for M.2 SATA Expansion:

• Choose a Reliable Chipset: Many users recommend SATA expansion cards based on the JMicron JMB585 chipset.

Recommended Products

Why Support Matters

Creating valuable free content requires resources. While we aim to provide these without cost, they incur operational expenses. Your support is crucial for sustaining this effort. Here’s how you can help:

• Use Affiliate Links: I earn from qualifying purchases as an Amazon Associate. By using our affiliate links, you help us cover operational costs at no extra cost to you.
• Engage and Share: Like, comment, and share the content to increase our reach.

Disclaimer

As an Amazon Associate, I earn from qualifying purchases through affiliate links, at no extra cost to you.

Stay Connected with Us

For exclusive updates, training tips, fitness advice, and more, follow us across all our platforms through one easy link.

👉 Stay Connected for Exclusive Martial Arts & Fitness Tips

Join our community and never miss an update!

Return to the home section.