2.3. Setting up your Manager Instance

2.3.1. Launching a “Manager Instance”

Now, we need to launch a “Manager Instance” that acts as a “head” node that we will ssh or mosh into to work from. Since we will deploy the heavy lifting to separate c5.4xlarge and f1 instances later, the Manager Instance can be a relatively cheap instance. In this guide, however, we will use a c5.4xlarge, running the AWS FPGA Developer AMI. (Be sure to subscribe to the AMI if you have not done so. See Subscribe to the AWS FPGA Developer AMI. Note that it might take a few minutes after subscribing to the AMI to be able to launch instances using it.)

Head to the EC2 Management Console. In the top right corner, ensure that the correct region is selected.

To launch a manager instance, follow these steps:

  1. From the main page of the EC2 Management Console, click Launch Instance. We use an on-demand instance here, so that your data is preserved when you stop/start the instance, and your data is not lost when pricing spikes on the spot market.

  2. When prompted to select an AMI, search in the Community AMIs tab for FPGA Developer AMI - 1.11.1-40257ab5-6688-4c95-97d1-e251a40fd1fc and select the AMI that appears (there should be only one). DO NOT USE ANY OTHER VERSION.

  3. When prompted to choose an instance type, select the instance type of your choosing. A good choice is a c5.4xlarge.

  4. On the “Configure Instance Details” page:

    1. First make sure that the firesim VPC is selected in the drop-down box next to “Network”. Any subnet within the firesim VPC is fine.

    2. Additionally, check the box for “Protect against accidental termination.” This adds a layer of protection to prevent your manager instance from being terminated by accident. You will need to disable this setting before being able to terminate the instance using usual methods.

    3. Also on this page, expand “Advanced Details” and in the resulting text box, paste the following:

      #!/bin/bash
      
      set -ex
      set -o pipefail
      
      echo "machine launch script started" > /home/centos/machine-launchstatus
      sudo chgrp centos /home/centos/machine-launchstatus
      sudo chown centos /home/centos/machine-launchstatus
      
      {
      sudo yum install -y ca-certificates
      sudo yum install -y mosh
      sudo yum groupinstall -y "Development tools"
      sudo yum install -y gmp-devel mpfr-devel libmpc-devel zlib-devel vim git java java-devel
      curl https://www.scala-sbt.org/sbt-rpm.repo | sudo tee /etc/yum.repos.d/scala-sbt-rpm.repo
      sudo yum install -y sbt texinfo gengetopt libffi-devel
      sudo yum install -y expat-devel libusb1-devel ncurses-devel cmake "perl(ExtUtils::MakeMaker)"
      # deps for poky
      sudo yum install -y python36 patch diffstat texi2html texinfo subversion chrpath git wget
      # deps for qemu
      sudo yum install -y gtk3-devel
      # deps for firesim-software (note that rsync is installed but too old)
      sudo yum install -y python36-pip python36-devel rsync
      # Install GNU make 4.x (needed to cross-compile glibc 2.28+)
      sudo yum install -y centos-release-scl
      sudo yum install -y devtoolset-8-make
      
      # install DTC
      sudo yum -y install dtc
      
      # get a proper version of git
      sudo yum -y remove git
      sudo yum -y install epel-release
      sudo yum -y install https://repo.ius.io/ius-release-el7.rpm
      sudo yum -y install git224
      
      # install verilator
      git clone http://git.veripool.org/git/verilator
      cd verilator/
      git checkout v4.034
      autoconf && ./configure && make -j4 && sudo make install
      cd ..
      
      # bash completion for manager
      sudo yum -y install bash-completion
      
      # graphviz for manager
      sudo yum -y install graphviz python-devel
      
      # used for CI
      sudo yum -y install expect
      
      # upgrade pip
      sudo pip3 install --upgrade pip==21.3.1
      # install requirements
      sudo python3 -m pip install fab-classic==1.19.1
      sudo python3 -m pip install boto3==1.20.21
      sudo python3 -m pip install colorama==0.4.3
      sudo python3 -m pip install argcomplete==1.12.3
      sudo python3 -m pip install graphviz==0.19
      # for some of our workload plotting scripts
      sudo python3 -m pip install pyparsing==3.0.6
      sudo python3 -m pip install numpy==1.19.5
      sudo python3 -m pip install kiwisolver==1.3.1
      sudo python3 -m pip install matplotlib==3.3.4
      sudo python3 -m pip install pandas==1.1.5
      sudo python3 -m pip install awscli==1.22.21
      sudo python3 -m pip install pytest==6.2.5
      sudo python3 -m pip install moto==2.2.17
      sudo python3 -m pip install sure==2.0.0
      # needed for the awstools cmdline parsing
      sudo python3 -m pip install pyyaml==5.4.1
      
      # setup argcomplete
      activate-global-python-argcomplete
      
      } 2>&1 | tee /home/centos/machine-launchstatus.log
      
      # get a regular prompt
      echo "PS1='\u@\H:\w\\$ '" >> /home/centos/.bashrc
      echo "machine launch script completed" >> /home/centos/machine-launchstatus
      

      This will pre-install all of the dependencies needed to run FireSim on your instance.

  5. On the next page (“Add Storage”), increase the size of the root EBS volume to ~300GB. The default of 65GB can quickly become too small as you accumulate large Vivado reports/outputs, large waveforms, XSim outputs, and large root filesystems for simulations. You should get rid of the small (5GB) secondary volume that is added by default.

  6. You can skip the “Add Tags” page, unless you want tags.

  7. On the “Configure Security Group” page, select the firesim security group that was automatically created for you earlier.

  8. On the review page, click the button to launch your instance.

Make sure you select the firesim key pair that we setup earlier.

2.3.1.1. Access your instance

We HIGHLY recommend using mosh instead of ssh or using ssh with a screen/tmux session running on your manager instance to ensure that long-running jobs are not killed by a bad network connection to your manager instance. On this instance, the mosh server is installed as part of the setup script we pasted before, so we need to first ssh into the instance and make sure the setup is complete.

In either case, ssh into your instance (e.g. ssh -i firesim.pem centos@YOUR_INSTANCE_IP) and wait until the ~/machine-launchstatus file contains all the following text:

centos@ip-172-30-2-140.us-west-2.compute.internal:~$ cat machine-launchstatus
machine launch script started
machine launch script completed!

Once this line appears, exit and re-ssh into the system. If you want to use mosh, mosh back into the system.

2.3.1.2. Key Setup, Part 2

Now that our manager instance is started, copy the private key that you downloaded from AWS earlier (firesim.pem) to ~/firesim.pem on your manager instance. This step is required to give the manager access to the instances it launches for you.

2.3.2. Setting up the FireSim Repo

We’re finally ready to fetch FireSim’s sources. Run:

git clone https://github.com/firesim/firesim
cd firesim
# checkout latest official firesim release
# note: this may not be the latest release if the documentation version != "stable"
git checkout 1.13.4
./build-setup.sh fast

The build-setup.sh script will validate that you are on a tagged branch, otherwise it will prompt for confirmation. This will have initialized submodules and installed the RISC-V tools and other dependencies.

Next, run:

source sourceme-f1-manager.sh

This will have initialized the AWS shell, added the RISC-V tools to your path, and started an ssh-agent that supplies ~/firesim.pem automatically when you use ssh to access other nodes. Sourcing this the first time will take some time – however each time after that should be instantaneous. Also, if your firesim.pem key requires a passphrase, you will be asked for it here and ssh-agent should cache it.

Every time you login to your manager instance to use FireSim, you should ``cd`` into your firesim directory and source this file again.

2.3.3. Completing Setup Using the Manager

The FireSim manager contains a command that will interactively guide you through the rest of the FireSim setup process. To run it, do the following:

firesim managerinit

This will first prompt you to setup AWS credentials on the instance, which allows the manager to automatically manage build/simulation nodes. See https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html#configure-cli-launch-ec2 for more about these credentials. When prompted, you should specify the same region that you chose above and set the default output format to json.

Next, it will create initial configuration files, which we will edit in later sections. Finally, it will prompt you for an email address, which is used to send email notifications upon FPGA build completion and optionally for workload completion. You can leave this blank if you do not wish to receive any notifications, but this is not recommended.

Now you’re ready to launch FireSim simulations! Hit Next to learn how to run single-node simulations.