Local FPGA System Setup

The below sections outline what you need to install to run FireSim on each machine type in a FireSim cluster. Note that the below three machine types can all map to a single machine in your setup; in this case, you should follow all the installation instructions on your single machine, without duplication (i.e., don’t re-run a step on the same machine if it is required on multiple machine types).

Warning

We highly recommend using Ubuntu 20.04 LTS as the host operating system for all machine types in an on-premises setup, as this is the OS recommended by Xilinx.

The following steps are separated into steps that require sudo and steps that do not. After initial setup with sudo, FireSim doesn’t need sudo access. In many cases with a shared machine, sudo-based setup is already completed and thus users should continue onto the Non-sudo-based setup.

`sudo` Setup

1. Install/enable FireSim scripts to new firesim Linux group

Note

These scripts are used by the FireSim manager and other FireSim tooling (i.e. FireMarshal) to avoid needing sudo access.

Machines: Manager Machine, Run Farm Machines, Build Farm Machines.

First, let’s clone a temporary version of FireSim with the scripts within it:

cd ~/     # or any scratch directory
mkdir firesim-script-installs
cd firesim-script-installs
git clone https://github.com/firesim/firesim
cd firesim
# checkout latest official firesim release
# note: this may not be the latest release if the documentation version != "stable"
git checkout main

Next, copy the required scripts to /usr/local/bin:

sudo cp deploy/sudo-scripts/* /usr/local/bin
sudo cp platforms/xilinx_alveo_u250/scripts/* /usr/local/bin

Now we can delete the temporary clone:

rm -rf ~/firesim-script-installs    # or the temp. dir. created previously

Next, lets change the permissions of the scripts and them to a new firesim Linux group.

sudo addgroup firesim
sudo chmod 755 /usr/local/bin/firesim*
sudo chgrp firesim /usr/local/bin/firesim*

Next, lets allow the firesim Linux group to run the pre-installed commands. Enter/create the following file with sudo:

sudo visudo /etc/sudoers.d/firesim

Then add the following lines:

%firesim ALL=(ALL) NOPASSWD: /usr/local/bin/firesim-*

Then change the permissions of the file:

sudo chmod 400 /etc/sudoers.d/firesim

This allows only users in the firesim group to execute the scripts.

2. Add your user to the firesim group

Machines: Manager Machine, Run Farm Machines, Build Farm Machines.

Next, add all user who want to use FireSim to the firesim group that you created. Make sure to replace YOUR_USER_NAME with the user to run simulations with:

sudo usermod -a -G firesim YOUR_USER_NAME

Finally, verify that the user can access the FireSim installed scripts by running:

sudo -l

The output should look similar to this:

User YOUR_USER_NAME may run the following commands on MACHINE_NAME:
    (ALL) NOPASSWD: /usr/local/bin/firesim-*

3. Install Vivado Lab and Cable Drivers

Machines: Run Farm Machines.

Go to the Xilinx Downloads Website and download Vivado 2023.1: Lab Edition - Linux.

Extract the downloaded .tar.gz file, then:

cd [EXTRACTED VIVADO LAB DIRECTORY]
sudo ./installLibs.sh
sudo ./xsetup --batch Install --agree XilinxEULA,3rdPartyEULA --edition "Vivado Lab Edition (Standalone)"

This will have installed Vivado Lab to /tools/Xilinx/Vivado_Lab/2023.1/.

For ease of use, add the following to the end of your ~/.bashrc:

source /tools/Xilinx/Vivado_Lab/2023.1/settings64.sh

Then, open a new terminal or source your ~/.bashrc.

Next, install the cable drivers like so:

cd /tools/Xilinx/Vivado_Lab/2023.1/data/xicom/cable_drivers/lin64/install_script/install_drivers/
sudo ./install_drivers

4. Install the Xilinx XDMA and XVSEC drivers

Machines: Run Farm Machines.

Warning

These commands will need to be re-run everytime the kernel is updated (normally whenever the machine is rebooted).

Warning

We use a non-Xilinx given XDMA/XVSEC repository since the mainline repository hasn’t updated to 6.4.0+ kernels yet.

First, run the following to clone the XDMA kernel module source:

cd ~/   # or any directory you would like to work from
git clone https://github.com/joonho3020/dma_ip_drivers
cd dma_ip_drivers
git checkout ubuntu-24-xdma
cd XDMA/linux-kernel/xdma

Note

If using the RHS Research Nitefury II board, do the following: The directory you are now in contains the XDMA kernel module. For the Nitefury to work, we will need to make one modification to the driver. Find the line containing #define XDMA_ENGINE_XFER_MAX_DESC. Change the value on this line from 0x800 to 16. Then, build and install the driver:

sudo make install

Now, test that the module can be inserted:

sudo insmod $(find /lib/modules/$(uname -r) -name "xdma.ko") poll_mode=1
lsmod | grep -i xdma

The second command above should have produced output indicating that the XDMA driver is loaded.

Next, we will do the same for the XVSEC driver, which is pulled from a separate repository due to kernel version incompatibility:

cd ~/   # or any directory you would like to work from
git clone https://github.com/joonho3020/dma_ip_drivers dma_ip_drivers_xvsec
cd dma_ip_drivers_xvsec
git checkout ubuntu-24-xvsec
cd XVSEC/linux-kernel/

make clean all
sudo make install

Now, test that the module can be inserted:

sudo modprobe xvsec
lsmod | grep -i xvsec

The second command above should have produced output indicating that the XVSEC driver is loaded.

Also, make sure you get output for the following (usually, /usr/local/sbin/xvsecctl):

which xvsecctl

5. Install sshd

Machines: Manager Machine, Run Farm Machines, and Build Farm Machines

On Ubuntu, install openssh-server like so:

sudo apt install openssh-server

7. Check Hard File Limit

Machine: Manager Machine

Check the output of the following command:

ulimit -Hn

If the result is greater than or equal to 16384, you can continue. Otherwise, run:

echo "* hard nofile 16384" | sudo tee --append /etc/security/limits.conf

Then, reboot your machine.

8. Install your FPGA

The starter tutorials will guide you through specific installation instructions for each FPGA.

Non-`sudo` Setup

1. Fix default .bashrc

Machines: Manager Machine, Run Farm Machines, Build Farm Machines.

We need various parts of the ~/.bashrc file to execute even in non-interactive mode. To do so, edit your ~/.bashrc file so that the following section is removed:

# If not running interactively, don't do anything
case $- in
     *i*) ;;
       *) return;;
esac

2. Set up SSH Keys

Machines: Manager Machine.

On the manager machine, generate a keypair that you will use to ssh from the manager machine into the manager machine (ssh localhost), run farm machines, and build farm machines:

cd ~/.ssh
ssh-keygen -t ed25519 -C "firesim.pem" -f firesim.pem
[create passphrase]

Next, add this key to the authorized_keys file on the manager machine:

cd ~/.ssh
cat firesim.pem.pub >> authorized_keys
chmod 0600 authorized_keys

You should also copy this public key into the ~/.ssh/authorized_keys files on all of your Run Farm and Build Farm Machines.

Returning to the Manager Machine, let’s set up an ssh-agent:

cd ~/.ssh
ssh-agent -s > AGENT_VARS
source AGENT_VARS
ssh-add firesim.pem

If you reboot your machine (or otherwise kill the ssh-agent), you will need to re-run the above four commands before using FireSim. If you open a new terminal (and ssh-agent is already running), you can simply run source ~/.ssh/AGENT_VARS.

Finally, confirm that you can now ssh localhost and ssh into your Run Farm and Build Farm Machines without being prompted for a passphrase.

3. Verify Run Farm Machine environment

Machines: Manager Machine and Run Farm Machines

Finally, let’s ensure that the Xilinx Vivado Lab tools are properly sourced in your shell setup (i.e. .bashrc) so that any shell on your Run Farm Machines can use the corresponding programs. The environment variables should be visible to any non-interactive shells that are spawned.

You can check this by running the following on the Manager Machine, replacing RUN_FARM_IP with localhost if your Run Farm machine and Manager machine are the same machine, or replacing it with the Run Farm machine’s IP address if they are different machines.

ssh RUN_FARM_IP printenv

Ensure that the output of the command shows that the Xilinx Vivado Lab tools are present in the printed environment variables (i.e., PATH).

If you have multiple Run Farm machines, you should repeat this process for each Run Farm machine, replacing RUN_FARM_IP with a different Run Farm Machine’s IP address.

Congratulations! We’ve now set up your machine/cluster to run simulations.

Local FPGA System Setup

sudo Setup

Non-sudo Setup

`sudo` Setup

Non-`sudo` Setup