1.3. Setting up your Manager Instance

1.3.1. Launching a “Manager Instance”

Warning

These instructions refer to fields in EC2’s new launch instance wizard. Refer to version 1.13.4 of the documentation for references to the old wizard, being wary that specifics, such as the AMI ID selection, may be out of date.

Now, we need to launch a “Manager Instance” that acts as a “head” node that we will ssh or mosh into to work from. Since we will deploy the heavy lifting to separate c5.4xlarge and f1 instances later, the Manager Instance can be a relatively cheap instance. In this guide, however, we will use a c5.4xlarge, running the AWS FPGA Developer AMI. (Be sure to subscribe to the AMI if you have not done so. See Subscribe to the AWS FPGA Developer AMI. Note that it might take a few minutes after subscribing to the AMI to be able to launch instances using it.)

Head to the EC2 Management Console. In the top right corner, ensure that the correct region is selected.

To launch a manager instance, follow these steps:

  1. From the main page of the EC2 Management Console, click Launch Instance ▼ button and click Launch Instance in the dropdown that appears. We use an on-demand instance here, so that your data is preserved when you stop/start the instance, and your data is not lost when pricing spikes on the spot market.

  2. In the Name field, give the instance a recognizable name, for example firesim-manager-1. This is purely for your own convenience and can also be left blank.

  3. In the Application and OS Images search box, search for FPGA Developer AMI - 1.11.1-40257ab5-6688-4c95-97d1-e251a40fd1fc and select the AMI that appears under the *Community AMIs* tab (there should be only one). DO NOT USE ANY OTHER VERSION. For example, do not use FPGA Developer AMI from the AWS Marketplace AMIs tab, as you will likely get an incorrect version of the AMI.

  4. In the Instance Type drop-down, select the instance type of your choosing. A good choice is a c5.4xlarge (16 cores, 32 GiB) or a z1d.2xlarge (8 cores, 64 GiB).

  5. In the Key pair (login) drop-down, select the firesim key pair we setup earlier.

  6. In the Network settings drop-down click edit and modify the following settings:

    1. Under VPC - required, select the firesim VPC. Any subnet within the firesim VPC is fine.

    2. Under Firewall (security groups), click Select existing security group and in the Common security groups dropdown that appears, select the firesim security group that was automatically created for you earlier.

  7. In the Configure storage section, increase the size of the root volume to at least 300GB. The default of 85GB can quickly become too small as you accumulate large Vivado reports/outputs, large waveforms, XSim outputs, and large root filesystems for simulations. You should remove the small (5-8GB) secondary volume that is added by default.

  8. In the Advanced details drop-down, we’ll leave most settings unchanged. The exceptions being:

    1. Under Termination protection, select Enable. This adds a layer of protection to prevent your manager instance from being terminated by accident. You will need to disable this setting before being able to terminate the instance using usual methods.

    2. Under User data, paste the following into the provided textbox:

      #!/bin/bash
      
      CONDA_INSTALL_PREFIX=/opt/conda
      CONDA_INSTALLER_VERSION=4.12.0-0
      CONDA_INSTALLER="https://github.com/conda-forge/miniforge/releases/download/${CONDA_INSTALLER_VERSION}/Miniforge3-${CONDA_INSTALLER_VERSION}-Linux-x86_64.sh"
      CONDA_CMD="conda" # some installers install mamba or micromamba
      CONDA_ENV_NAME="firesim"
      
      DRY_RUN_OPTION=""
      DRY_RUN_ECHO=()
      REINSTALL_CONDA=0
      
      usage()
      {
          echo "Usage: $0 [options]"
          echo
          echo "Options:"
          echo "[--help]                  List this help"
          echo "[--prefix <prefix>]       Install prefix for conda. Defaults to /opt/conda."
          echo "                          If <prefix>/bin/conda already exists, it will be used and install is skipped."
          echo "[--env <name>]            Name of environment to create for conda. Defaults to 'firesim'."
          echo "[--dry-run]               Pass-through to all conda commands and only print other commands."
          echo "                          NOTE: --dry-run will still install conda to --prefix"
          echo "[--reinstall-conda]       Repairs a broken base environment by reinstalling."
          echo "                          NOTE: will only reinstall conda and exit without modifying the --env"
          echo
          echo "Examples:"
          echo "  % $0"
          echo "     Install into default system-wide prefix (using sudo if needed) and add install to system-wide /etc/profile.d"
          echo "  % $0 --prefix ~/conda --env my_custom_env"
          echo "     Install into $HOME/conda and add install to ~/.bashrc"
          echo "  % $0 --prefix \${CONDA_EXE%/bin/conda} --env my_custom_env"
          echo "     Create my_custom_env in existing conda install"
          echo "     NOTES:"
          echo "       * CONDA_EXE is set in your environment when you activate a conda env"
          echo "       * my_custom_env will not be activated by default at login see /etc/profile.d/conda.sh & ~/.bashrc"
      }
      
      
      while [ $# -gt 0 ]; do
          case "$1" in
              --help)
                  usage
                  exit 1
                  ;;
              --prefix)
                  shift
                  CONDA_INSTALL_PREFIX="$1"
                  shift
                  ;;
              --env)
                  shift
                  CONDA_ENV_NAME="$1"
                  shift
                  if [[ "$CONDA_ENV_NAME" == "base" ]]; then
                      echo "::ERROR:: best practice is to install into a named environment, not base. Aborting."
                      exit 1
                  fi
                  ;;
              --dry-run)
                  shift
                  DRY_RUN_OPTION="--dry-run"
                  DRY_RUN_ECHO=(echo "Would Run:")
                  ;;
              --reinstall-conda)
                  shift
                  REINSTALL_CONDA=1
                  ;;
              *)
                  echo "Invalid Argument: $1"
                  usage
                  exit 1
                  ;;
          esac
      done
      
      if [[ $REINSTALL_CONDA -eq 1 && -n "$DRY_RUN_OPTION" ]]; then
          echo "::ERROR:: --dry-run and --reinstall-conda are mutually exclusive.  Pick one or the other."
      fi
      
      set -ex
      set -o pipefail
      
      {
      
          # uname options are not portable so do what https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#uname-is-system-specific
          # suggests and iteratively probe the system type
          if ! type uname >&/dev/null; then
              echo "::ERROR:: need 'uname' command available to determine if we support this sytem"
              exit 1
          fi
      
          if [[ "$(uname)" != "Linux" ]]; then
              echo "::ERROR:: $0 only supports 'Linux' not '$(uname)'"
              exit 1
          fi
      
          if [[ "$(uname -mo)" != "x86_64 GNU/Linux" ]]; then
              echo "::ERROR:: $0 only supports 'x86_64 GNU/Linux' not '$(uname -io)'"
              exit 1
          fi
      
          if [[ ! -r /etc/os-release ]]; then
              echo "::ERROR:: $0 depends on /etc/os-release for distro-specific setup and it doesn't exist here"
              exit 1
          fi
      
          OS_FLAVOR=$(grep '^ID=' /etc/os-release | awk -F= '{print $2}' | tr -d '"')
          OS_VERSION=$(grep '^VERSION_ID=' /etc/os-release | awk -F= '{print $2}' | tr -d '"')
      
          echo "machine launch script started" > machine-launchstatus
          chmod ugo+r machine-launchstatus
      
          # platform-specific setup
          case "$OS_FLAVOR" in
              ubuntu)
                  ;;
              centos)
                  ;;
              *)
                  echo "::ERROR:: Unknown OS flavor '$OS_FLAVOR'. Unable to do platform-specific setup."
                  exit 1
                  ;;
          esac
      
      
          # everything else is platform-agnostic and could easily be expanded to Windows and/or OSX
      
          SUDO=""
          prefix_parent=$(dirname "$CONDA_INSTALL_PREFIX")
          if [[ ! -e "$prefix_parent" ]]; then
              mkdir -p "$prefix_parent" || SUDO=sudo
          elif [[ ! -w "$prefix_parent" ]]; then
              SUDO=sudo
          fi
      
          if [[ -n "$SUDO" ]]; then
              echo "::INFO:: using 'sudo' to install conda"
              # ensure files are read-execute for everyone
              umask 022
          fi
      
          if [[ -n "$SUDO"  || "$(id -u)" == 0 ]]; then
              INSTALL_TYPE=system
          else
              INSTALL_TYPE=user
          fi
      
          # to enable use of sudo and avoid modifying 'secure_path' in /etc/sudoers, we specify the full path to conda
          CONDA_EXE="${CONDA_INSTALL_PREFIX}/bin/$CONDA_CMD"
      
          if [[ -x "$CONDA_EXE" && $REINSTALL_CONDA -eq 0 ]]; then
              echo "::INFO:: '$CONDA_EXE' already exists, skipping conda install"
          else
              wget -O install_conda.sh "$CONDA_INSTALLER"  || curl -fsSLo install_conda.sh "$CONDA_INSTALLER"
              if [[ $REINSTALL_CONDA -eq 1 ]]; then
                  conda_install_extra="-u"
                  echo "::INFO:: RE-installing conda to '$CONDA_INSTALL_PREFIX'"
              else
                  conda_install_extra=""
                  echo "::INFO:: installing conda to '$CONDA_INSTALL_PREFIX'"
              fi
              # -b for non-interactive install
              $SUDO bash ./install_conda.sh -b -p "$CONDA_INSTALL_PREFIX" $conda_install_extra
              rm ./install_conda.sh
      
              # see https://conda-forge.org/docs/user/tipsandtricks.html#multiple-channels
              # for more information on strict channel_priority
              "${DRY_RUN_ECHO[@]}" $SUDO "$CONDA_EXE" config --system --set channel_priority strict
              # By default, don't mess with people's PS1, I personally find it annoying
              "${DRY_RUN_ECHO[@]}" $SUDO "$CONDA_EXE" config --system --set changeps1 false
              # don't automatically activate the 'base' environment when intializing shells
              "${DRY_RUN_ECHO[@]}" $SUDO "$CONDA_EXE" config --system --set auto_activate_base false
              # don't automatically update conda to avoid https://github.com/conda-forge/conda-libmamba-solver-feedstock/issues/2
              "${DRY_RUN_ECHO[@]}" $SUDO "$CONDA_EXE" config --system --set auto_update_conda false
      
              # conda-build is a special case and must always be installed into the base environment
              $SUDO "$CONDA_EXE" install $DRY_RUN_OPTION -y -n base conda-build
      
              # conda-libmamba-solver is a special case and must always be installed into the base environment
              # see https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community
              $SUDO "$CONDA_EXE" install $DRY_RUN_OPTION -y -n base conda-libmamba-solver
              # Use the fast solver by default
              "${DRY_RUN_ECHO[@]}" $SUDO "$CONDA_EXE" config --system --set experimental_solver libmamba
      
              conda_init_extra_args=()
              if [[ "$INSTALL_TYPE" == system ]]; then
                  # if we're installing into a root-owned directory using sudo, or we're already root
                  # initialize conda in the system-wide rcfiles
                  conda_init_extra_args=(--no-user --system)
              fi
              # run conda-init and look at it's output to insert 'conda activate $CONDA_ENV_NAME' into the
              # block that conda-init will update if ever conda is installed to a different prefix and
              # this is rerun.
              $SUDO "${CONDA_EXE}" init $DRY_RUN_OPTION "${conda_init_extra_args[@]}" bash 2>&1 | \
                  tee >(grep '^modified' | grep -v "$CONDA_INSTALL_PREFIX" | awk '{print $NF}' | \
                  "${DRY_RUN_ECHO[@]}" $SUDO xargs -r sed -i -e "/<<< conda initialize <<</iconda activate $CONDA_ENV_NAME")
      
              if [[ $REINSTALL_CONDA -eq 1 ]]; then
                  echo "::INFO:: Done reinstalling conda. Exiting"
                  exit 0
              fi
          fi
      
          # https://conda-forge.org/feedstock-outputs/
          #   filterable list of all conda-forge packages
          # https://conda-forge.org/#contribute
          #   instructions on adding a recipe
          # https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html#package-match-specifications
          #   documentation on package_spec syntax for constraining versions
          CONDA_PACKAGE_SPECS=()
      
      
          # handy tool for introspecting package relationships and file ownership
          # see https://github.com/rvalieris/conda-tree
          CONDA_PACKAGE_SPECS=( conda-tree )
      
          # bundle FireSim driver with deps into installer shell-script
          CONDA_PACKAGE_SPECS=( constructor )
      
          # https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html#using-the-compiler-packages
          #    for notes on using the conda compilers
          # elfutils has trouble building with gcc 11 for something that looks like it needs to be fixed upstream
          # ebl_syscall_abi.c:37:64: error: argument 5 of type 'int *' declared as a pointer [-Werror=array-parameter=]
          #    37 | ebl_syscall_abi (Ebl *ebl, int *sp, int *pc, int *callno, int *args)
          #       |                                                           ~~~~~^~~~
          # In file included from ./../libasm/libasm.h:35,
          #                  from ./libeblP.h:33,
          #                  from ebl_syscall_abi.c:33:
          # ./libebl.h:254:46: note: previously declared as an array 'int[6]'
          #   254 |                             int *callno, int args[6]);
          #       |                                          ~~~~^~~~~~~
          #
          # pin to gcc=10 until we get that fixed.
      
          CONDA_PACKAGE_SPECS+=( gcc=10 gxx=10 binutils conda-gcc-specs )
      
          # if building riscv-toolchain from source, we need to use bison=3.4 until we have
          # https://github.com/riscv-collab/riscv-binutils-gdb/commit/314ec7aeeb1b2e68f0d8fb9990f2335f475a6e33
          CONDA_PACKAGE_SPECS+=( bison=3.4 )
      
          # poky deps
          CONDA_PACKAGE_SPECS+=( python=3.8 patch texinfo subversion chrpath git wget )
          # qemu deps
          CONDA_PACKAGE_SPECS+=( gtk3 glib pkg-config bison flex )
          # qemu also requires being built against kernel-headers >= 2.6.38 because of it's use of
          # MADV_NOHUGEPAGE in a call to madvise.  See:
          # https://man7.org/linux/man-pages/man2/madvise.2.html
          # https://conda-forge.org/docs/maintainer/knowledge_base.html#using-centos-7
          # obvi this would need to be made linux-specific if we supported other MacOS or Windows
          CONDA_PACKAGE_SPECS+=( "kernel-headers_linux-64>=2.6.38" )
          # firemarshal deps
          CONDA_PACKAGE_SPECS+=( rsync psutil doit=0.35.0 gitpython humanfriendly e2fsprogs ctags bison flex expat )
          # cross-compile glibc 2.28+ deps
          # current version of buildroot won't build with make 4.3 https://github.com/firesim/FireMarshal/issues/236
          CONDA_PACKAGE_SPECS+=( make!=4.3 )
          # build-libelf wants autoconf
          CONDA_PACKAGE_SPECS+=( autoconf automake libtool )
          # other misc deps
          CONDA_PACKAGE_SPECS+=(
              bash-completion \
              sbt \
              ca-certificates \
              mosh \
              gmp \
              mpfr \
              mpc \
              zlib \
              vim \
              git  \
              openjdk \
              gengetopt \
              libffi \
              expat \
              libusb1 \
              ncurses \
              cmake \
              graphviz \
              expect \
              dtc \
              verilator==4.034 \
              screen \
          )
          # clang-format for driver coding style enforcement.
          CONDA_PACKAGE_SPECS+=( clang-format clang-tools )
      
          # python packages
          # While it is possible to install using pip after creating the
          # conda environment, pip's dependency resolution can conflict with
          # conda and create broken environments.  It's best to use the conda
          # packages so that the environment is consistent
          CONDA_PACKAGE_SPECS+=( \
              boto3==1.20.21 \
              colorama==0.4.3 \
              argcomplete==1.12.3 \
              python-graphviz==0.19 \
              pyparsing==3.0.6 \
              numpy==1.19.5 \
              kiwisolver==1.3.1 \
              matplotlib-base==3.3.4 \
              pandas==1.1.5 \
              awscli==1.22.21 \
              pytest==6.2.5 \
              pytest-dependency==0.5.1 \
              pytest-mock==3.7.0 \
              moto==3.1.0 \
              pyyaml==5.4.1 \
              mypy==0.931 \
              types-pyyaml==6.0.4 \
              boto3-stubs==1.21.6 \
              botocore-stubs==1.24.7 \
              mypy-boto3-s3==1.21.0 \
              fsspec \
              s3fs \
          )
      
          if [[ "$CONDA_ENV_NAME" == "base" ]]; then
              # NOTE: arg parsing disallows installing to base but this logic is correct if we ever change
              CONDA_SUBCOMMAND=install
              CONDA_ENV_BIN="${CONDA_INSTALL_PREFIX}/bin"
          else
              CONDA_ENV_BIN="${CONDA_INSTALL_PREFIX}/envs/${CONDA_ENV_NAME}/bin"
              if [[ -d "${CONDA_INSTALL_PREFIX}/envs/${CONDA_ENV_NAME}" ]]; then
                  # 'create' clobbers the existing environment and doesn't leave a revision entry in
                  # `conda list --revisions`, so use install instead
                  CONDA_SUBCOMMAND=install
              else
                  CONDA_SUBCOMMAND=create
              fi
          fi
      
          # to enable use of sudo and avoid modifying 'secure_path' in /etc/sudoers, we specify the full path to pip
          CONDA_PIP_EXE="${CONDA_ENV_BIN}/pip"
      
          # to enable use of sudo and avoid modifying 'secure_path' in /etc/sudoers, we specify the full path to conda
          $SUDO "${CONDA_EXE}" "$CONDA_SUBCOMMAND" $DRY_RUN_OPTION -n "$CONDA_ENV_NAME" -c conda-forge -y "${CONDA_PACKAGE_SPECS[@]}"
      
      
          # Install python packages using pip that are not available from conda
          #
          # Installing things with pip is possible.  However, to get
          # the most complete solution to all dependencies, you should
          # prefer creating the environment with a single invocation of
          # conda
          PIP_PKGS=( \
              fab-classic==1.19.1 \
              mypy-boto3-ec2==1.21.9 \
              sure==2.0.0 \
              pylddwrap==1.2.1 \
          )
          if [[ -n "$PIP_PKGS[*]" ]]; then
              "${DRY_RUN_ECHO[@]}" $SUDO "${CONDA_PIP_EXE}" install "${PIP_PKGS[@]}"
          fi
      
      
          argcomplete_extra_args=()
          if [[ "$INSTALL_TYPE" == system ]]; then
              BASH_COMPLETION_COMPAT_DIR="${CONDA_ENV_BIN}/../etc/bash_completion.d"
              "${DRY_RUN_ECHO[@]}" $SUDO mkdir -p "${BASH_COMPLETION_COMPAT_DIR}"
              argcomplete_extra_args=( --dest "${BASH_COMPLETION_COMPAT_DIR}" )
      
          else
              # if we're aren't installing into a system directory, then initialize argcomplete
              # with --user so that it goes into the home directory
              argcomplete_extra_args=( --user )
          fi
          "${DRY_RUN_ECHO[@]}" $SUDO "${CONDA_ENV_BIN}/activate-global-python-argcomplete" "${argcomplete_extra_args[@]}"
      
      } 2>&1 | tee machine-launchstatus.log
      chmod ugo+r machine-launchstatus.log
      
      
      echo "machine launch script completed" >>machine-launchstatus
      

    When your instance boots, this will install a compatible set of all the dependencies needed to run FireSim on your instance using conda.

  9. Double check your configuration. The most common misconfigurations that may require repeating this process include:

    1. Not selecting the firesim vpc.

    2. Not selecting the firesim security group.

    3. Not selecting the firesim key pair.

    4. Selecting the wrong AMI.

  10. Click the orange Launch Instance button.

1.3.1.1. Access your instance

We HIGHLY recommend using mosh instead of ssh or using ssh with a screen/tmux session running on your manager instance to ensure that long-running jobs are not killed by a bad network connection to your manager instance. On this instance, the mosh server is installed as part of the setup script we pasted before, so we need to first ssh into the instance and make sure the setup is complete.

In either case, ssh into your instance (e.g. ssh -i firesim.pem centos@YOUR_INSTANCE_IP) and wait until the /machine-launchstatus file contains all the following text:

$ cat /machine-launchstatus
machine launch script started
machine launch script completed

Once this line appears, exit and re-ssh into the system. If you want to use mosh, mosh back into the system.

1.3.1.2. Key Setup, Part 2

Now that our manager instance is started, copy the private key that you downloaded from AWS earlier (firesim.pem) to ~/firesim.pem on your manager instance. This step is required to give the manager access to the instances it launches for you.

1.3.2. Setting up the FireSim Repo

We’re finally ready to fetch FireSim’s sources. Run:

git clone https://github.com/firesim/firesim
cd firesim
# checkout latest official firesim release
# note: this may not be the latest release if the documentation version != "stable"
git checkout main
./build-setup.sh fast

The build-setup.sh script will validate that you are on a tagged branch, otherwise it will prompt for confirmation. This will have initialized submodules and installed the RISC-V tools and other dependencies.

Next, run:

source sourceme-f1-manager.sh

This will have initialized the AWS shell, added the RISC-V tools to your path, and started an ssh-agent that supplies ~/firesim.pem automatically when you use ssh to access other nodes. Sourcing this the first time will take some time – however each time after that should be instantaneous. Also, if your firesim.pem key requires a passphrase, you will be asked for it here and ssh-agent should cache it.

Every time you login to your manager instance to use FireSim, you should ``cd`` into your firesim directory and source this file again.

1.3.3. Completing Setup Using the Manager

The FireSim manager contains a command that will interactively guide you through the rest of the FireSim setup process. To run it, do the following:

firesim managerinit --platform f1

This will first prompt you to setup AWS credentials on the instance, which allows the manager to automatically manage build/simulation nodes. See https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html#configure-cli-launch-ec2 for more about these credentials. When prompted, you should specify the same region that you chose above and set the default output format to json.

Next, it will prompt you for an email address, which is used to send email notifications upon FPGA build completion and optionally for workload completion. You can leave this blank if you do not wish to receive any notifications, but this is not recommended. Next, it will create initial configuration files, which we will edit in later sections.

Now you’re ready to launch FireSim simulations! Hit Next to learn how to run single-node simulations.