Running NoC Partition Mode Simulations

In this section, we provide a step-by-step tutorial on how to partition a SoC with a ring NoC containing four tiles into 3 FPGAs connected as a ring. We will run this on the local Xilinx Alveo U250 FPGAs. Note that this feature is highly experimental and has not been tested on EC2 F1. This assumes that you have read the FireAxe running fast-mode simulations.

Some caveats of this mode:

• You have to disable TracerV by mixing in the WithNoTraceIO configuration fragment to the TARGET_CONFIG.

• Works only for mesh or ring based NoC topologies where each FPGA is connected to exactly 2 other FPGAs.

1. Building Partitioned Sims: Setting up FireAxe Target configs

Similarily to the fast-mode, we need to setup the FireAxe target configurations. One change to note is that we are now using the WithQSFP configuration fragment. With this information, FireAxe will generate an Aurora IP in the FPGA shell to exchange data between multiple FPGAs using the QSFP interconnects. Also, note that we added the WithFireAxeNoCPart. This tells the compiler to perform the NoC-partition pass by grouping router nodes and modules connected to those router nodes. The indices in WithPartitionGlobalInfo now indicates the router node indices.

//////////////////////////////////////////////////////////////////////////////
// - Xilinx U250 partition a ring NoC onto 3 FPGA connected as a ring
//
//        FPGA 0     ------------------ FPGA 1
// - router 0 & tile 0               - router 2 & tile 2
// - router 1 & tile 1               - router 3 & tile 3
//                \                     /
//                 \                   /
//                  \                 /
//                   \               /
//                         FPGA 2
//                  - router nodes 4 ~ 10
//                  - SoC subsystem
//////////////////////////////////////////////////////////////////////////////
class QuadTileRingNoCTopoQSFPXilinxAlveoConfig
    extends Config(
      new WithQSFP ++             // FireSim will instantiate Aurora IPs
        new WithFireAxeNoCPart ++ // Compiler will detect NoC router nodes to partition
        new WithPartitionGlobalInfo(
          Seq(
            Seq("0", "1"),
            Seq("2", "3"),
            // base group has to be put last
            (4 until 10).map(i => s"${i}"),
          )
        ) ++
        new BaseXilinxAlveoU250Config
    )

class QuadTileRingNoCU250Base
    extends Config(
      new WithPartitionBase ++
        new QuadTileRingNoCTopoQSFPXilinxAlveoConfig
    )

class QuadTileRingNoCU250Partition0
    extends Config(
      new WithPartitionIndex(0) ++
        new QuadTileRingNoCTopoQSFPXilinxAlveoConfig
    )

class QuadTileRingNoCU250Partition1
    extends Config(
      new WithPartitionIndex(1) ++
        new QuadTileRingNoCTopoQSFPXilinxAlveoConfig
    )

2. Building Partitioned Sims: config_build_recipes.yaml

We can specify the config_build_recipes.yaml at this point.

##############################################################################
# Using the NoC-partition-mode to partition the design across 3 FPGAs
# connected as a ring.
##############################################################################
xilinx_u250_quad_rocket_ring_base:
  PLATFORM: xilinx_alveo_u250
  TARGET_PROJECT: firesim
  DESIGN: FireSim
  TARGET_CONFIG: FireSimQuadRocketSbusRingNoCConfig
  PLATFORM_CONFIG: QuadTileRingNoCBase
  deploy_quintuplet: null
  platform_config_args:
    fpga_frequency: 90
    build_strategy: TIMING
  post_build_hook: null
  metasim_customruntimeconfig: null
  bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u250.yaml

xilinx_u250_quad_rocket_ring_0:
  PLATFORM: xilinx_alveo_u250
  TARGET_PROJECT: firesim
  DESIGN: FireSim
  TARGET_CONFIG: FireSimQuadRocketSbusRingNoCConfig
  PLATFORM_CONFIG: QuadTileRingNoC0
  deploy_quintuplet: null
  platform_config_args:
    fpga_frequency: 90
    build_strategy: TIMING
  post_build_hook: null
  metasim_customruntimeconfig: null
  bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u250.yaml

xilinx_u250_quad_rocket_ring_1:
  PLATFORM: xilinx_alveo_u250
  TARGET_PROJECT: firesim
  DESIGN: FireSim
  TARGET_CONFIG: FireSimQuadRocketSbusRingNoCConfig
  PLATFORM_CONFIG: QuadTileRingNoC1
  deploy_quintuplet: null
  platform_config_args:
    fpga_frequency: 90
    build_strategy: TIMING
  post_build_hook: null
  metasim_customruntimeconfig: null
  bit_builder_recipe: bit-builder-recipes/xilinx_alveo_u250.yaml

3. Running Partitioned Simulations: user_topology.py

Once again, we need to specify the FireAxe topology to run simulations.

    def fireaxe_ring_noc_config(self) -> None:
        hwdb_entries = {
            0 : "xilinx_u250_quad_rocket_ring_0",
            1 : "xilinx_u250_quad_rocket_ring_1",
            2 : "xilinx_u250_quad_rocket_ring_base"
        }
        slotid_to_pidx = [0, 1, 2]
        # DOC include start: fireaxe_ring_noc_config edges
        edges = [
            FireAxeEdge(FireAxeNodeBridgePair(0, 0), FireAxeNodeBridgePair(2, 1)),
            FireAxeEdge(FireAxeNodeBridgePair(2, 0), FireAxeNodeBridgePair(1, 1)),
            FireAxeEdge(FireAxeNodeBridgePair(1, 0), FireAxeNodeBridgePair(0, 1))
        ]
        # DOC include end: fireaxe_ring_noc_config edges
        mode = PartitionMode.NOC_MODE
        self.fireaxe_topology_config(hwdb_entries, edges, slotid_to_pidx, mode)

Lets look at how the FireAxe topology is set in this example. We can see that bridge 0 of partition N is always connected to bridge 1 of partition (N + 1) % NFPGAs and bridge 1 of partition N is always connected to bridge 0 of partition (N + NFPGAs - 1) % NFPGAs.

We can specify the above topology as below:

        edges = [
            FireAxeEdge(FireAxeNodeBridgePair(0, 0), FireAxeNodeBridgePair(2, 1)),
            FireAxeEdge(FireAxeNodeBridgePair(2, 0), FireAxeNodeBridgePair(1, 1)),
            FireAxeEdge(FireAxeNodeBridgePair(1, 0), FireAxeNodeBridgePair(0, 1))
        ]

4. Running Partitioned Simulations: config_runtime.yaml

Now we can update config_runtime.yaml to run FireAxe simulations.

target_config:
    topology: fireaxe_rocket_ring_noc_config