Running Fast Mode Simulations

In this section, we provide a step-by-step tutorial on how to partition a RocketTile out from the SoC and run fast-mode simulations using 2 FPGAs on EC2 F1. This assumes that you have completed the AWS F1 Getting Started Guide and that you are familiar with the FireSim workflow.

1. Building Partitioned Sims: Setting up FireAxe Target configs

To build bitstreams for partitioned simulations, we need specify which modules we want to partition out. We can do that in :gh-file-ref:`sim/midas/src/main/scala/configs/FireAxeTargetConfigs.scala`.

//////////////////////////////////////////////////////////////////////////////
// - F1 partition a RocketTile out
// - Connect FPGAs using PCIe peer to peer communication scheme
//
// FPGA 0 (RocketTile) ----------- FPGA 1 (SoC subsystem)
//////////////////////////////////////////////////////////////////////////////
class RocketTileF1Config
    extends Config(
      new WithPCIM ++ // Use PCIM (PCIe peer to peer) communication scheme
        // WithPartitionGlobalInfo takes in a Seq of Seq
        // The inner Seq specifies the list of modules that should be grouped together in a single partition.
        // The outer Seq specifies the list of partition groups.
        // Each partition group is mapped onto a separate FPGA.
        new WithPartitionGlobalInfo(
          Seq(
            Seq("RocketTile")
          )
        ) ++
        new BaseF1Config
    )

class RocketTileF1PCIMBase
    extends Config(
      new WithPartitionBase ++ // Base partition (SoC subsystem)
        new RocketTileF1Config
    )

class RocketTileF1PCIMPartition0
    extends Config(
      new WithPartitionIndex(0) ++ // Partition 0 containing the partitioned RocketTile
        new RocketTileF1Config
    )

The WithPartitionGlobalInfo fragment takes in an argument of type Seq[Seq[String]] where each Seq[String] indicates the group of modules to be extracted out to the same partition.

new WithPartitionGlobalInfo(Seq(
  Seq("A", "B"),
  Seq("C", "D"))

For instance, in the above example, modules “A”, “B” will be grouped and extracted out to one FPGA, “C”, “D” will be grouped and extracted out to another FPGA, and the base SoC will be placed on the third FPGA.

As in the above example, we can use WithPartitionBase and WithPartitionIndex(idx) to specify the partitions. WithPartitionIndex(0) for example, will choose Seq("A", "B") as the group of modules to partition out. How each partition group is mapped to the FPGA is specified in :gh-file-ref:`deploy/runtools/user_topology.py`.

2. Building Partitioned Sims: config_build_recipes.yaml

Now, we can specify the config_build_recipes.yaml. We need to set the TARGET_CONFIG to the FireChip configuration that we want to build, and the PLATFORM_CONFIG to the partition configs that we defined in the above step.

You can now use the firesim buildbitstream command to build bitstreams of these partition configurations.

3. Running Partitioned Simulations: user_topology.py

Once the bitstreams are built and you copied the hwdb entries into config_hwdb.yaml, we need to setup config_runtime.yaml and :gh-file-ref:`deploy/runtools/user_topology.py` to run FireAxe simulations.

    def fireaxe_rocket_fastmode_config(self) -> None:
        # DOC include start: fireaxe_fastmode_config hwdb_entries
        # hwdb_entries maps the partition index to the hwdb name
        hwdb_entries = {0: "f1_rocket_split_soc_fast", 1: "f1_rocket_split_tile_fast"}
        # DOC include end: fireaxe_fastmode_config hwdb_entries
        # DOC include start: fireaxe_fastmode_config slot_to_pidx
        # slotid_to_pidx maps the partition index to the FPGA slotid
        # For instance, `slotid_to_pidx = [2, 1, 0]` will map partition
        # index 2 to simulation slot 0, partition index 1 to simulation slot 1,
        # and partition index 0 to simulation slot 2.
        slotid_to_pidx = [0, 1]
        # DOC include end: fireaxe_fastmode_config slot_to_pidx
        # DOC include start: fireaxe_fastmode_config edges
        # The `FireAxeEdge` class is used to depict the connections between the partitions.
        edges = [FireAxeEdge(FireAxeNodeBridgePair(0, 0), FireAxeNodeBridgePair(1, 0))]
        # DOC include end: fireaxe_fastmode_config edges
        # DOC include start: fireaxe_fastmode_config mode
        # The PartitionMode enum contains the different partition modes that are available
        mode = PartitionMode.FAST_MODE
        # DOC include end: fireaxe_fastmode_config mode
        # DOC include start: fireaxe_fastmode_config summing it all up
        self.fireaxe_topology_config(hwdb_entries, edges, slotid_to_pidx, mode)
        # DOC include end: fireaxe_fastmode_config summing it all up

This is how the FireAxe topology looks like when partitioning a RocketTile out from the SoC. Lets look at each line individually.

The hwdb_entries is a dictionary mapping the partition index to the hwdb name.

        # hwdb_entries maps the partition index to the hwdb name
        hwdb_entries = {0: "f1_rocket_split_soc_fast", 1: "f1_rocket_split_tile_fast"}

The slot_to_pidx is a list mapping the partition index to the FPGA slotid. The list entries are the partition indices and the slotid is dictated by the position of the list. For instance, slotid_to_pidx = [2, 1, 0] will map partition index 2 to simulation slot 0, partition index 1 to simulation slot 1 and partition index 0 to simulation slot 2.

        # slotid_to_pidx maps the partition index to the FPGA slotid
        # For instance, `slotid_to_pidx = [2, 1, 0]` will map partition
        # index 2 to simulation slot 0, partition index 1 to simulation slot 1,
        # and partition index 0 to simulation slot 2.
        slotid_to_pidx = [0, 1]

We need to specify the SoC partition topology to run FireAxe simulations. The below figure depicts the current SoC partition topology. The pidx indicates the partition index : idx 0 corresponds to f1_rocket_split_soc_fast and idx 1 corresponds to f1_rocket_split_tile_fast. In FireAxe, each partition contains a set of bridges that are used to communicate with other partitions. The bridges attached to a partition all have a unique id starting from zero. In our example, partition 0 and partition 1 communicates through a single edge where the bridge id is both 0.

The FireAxeEdge class is used to depict the connections between the partitions. When connecting partition X to partition Y, we need to be aware of the bridge index that connects X and Y (Xbidx, Ybidx). The vertices of the edge can be described as a tuple of the partition index and bridge index: Xpair = FireAxeNodeBridgedPair(X, Xbidx) and Ypair = FireAxeNodeBridgedPair(Y, Ybidx). Then, the edge can be described as FireAxeEdge(Xpair, Ypair). So the edge in the above figure can be described as follows:

        # The `FireAxeEdge` class is used to depict the connections between the partitions.
        edges = [FireAxeEdge(FireAxeNodeBridgePair(0, 0), FireAxeNodeBridgePair(1, 0))]

Next, we need to specify the partitioning mode that we chose to build the bitstreams. In this example, we chose the FAST_MODE.

        # The PartitionMode enum contains the different partition modes that are available
        mode = PartitionMode.FAST_MODE

At this point, all we need to do is call the fireaxe_topology_config with the above parameters.

        self.fireaxe_topology_config(hwdb_entries, edges, slotid_to_pidx, mode)

Users shouldn’t need to worry about this code. All it is doing is generating the partition topology and creating a PartitionConfig instance that contains information about the topology and passing it to the FireSimServerNode.

    def fireaxe_topology_config(
        self,
        hwdb_entries: Dict[int, str],
        edges: List[FireAxeEdge],
        slotid_to_pidx: List[int],
        mode: PartitionMode,
    ) -> None:
        pidx_to_slotid = dict()
        for sid, pid in enumerate(slotid_to_pidx):
            pidx_to_slotid[pid] = sid

        # Create partition nodes
        pidx_to_partition_node: Dict[int, PartitionNode] = dict()
        for pidx, hwdb in hwdb_entries.items():
            node = PartitionNode(hwdb, pidx)
            pidx_to_partition_node[pidx] = node

        # Add edges to nodes
        for edge in edges:
            u_node = pidx_to_partition_node[edge.u.pidx]
            v_node = pidx_to_partition_node[edge.v.pidx]
            u_node.add_edge(edge.u.bidx, edge.v.bidx, v_node)
            v_node.add_edge(edge.v.bidx, edge.u.bidx, u_node)

        # Create PartitionConfigs and FireSimServerNode
        servers: Dict[int, FireSimServerNode] = dict()
        for pidx, node in pidx_to_partition_node.items():
            partition_cfg = PartitionConfig(node, pidx_to_slotid, mode)
            servers[pidx_to_slotid[node.pidx]] = FireSimServerNode(
                partition_cfg.get_hwdb(), partition_config=partition_cfg
            )

        # Sort the servers by their sim slot id
        servers = dict(sorted(servers.items()))
        self.roots = list(servers.values())

4. Running Partitioned Simulations: config_runtime.yaml

Now, we need to setup config_runtime.yaml to run FireAxe simulations. All we need to do is change the config_runtime[target_config][topology] to the topology that we defined in step 3.

target_config:
    topology: fireaxe_rocket_fastmode_config

At this point, you can run the firesim runworkload to kick off simulations.