Supernode - Multiple Simulated SoCs Per FPGA
Supernode allows users to run multiple simulated SoCs per-FPGA in order to improve
FPGA resource utilization and reduce cost. For example, in the case of using
FireSim to simulate a datacenter scale system, supernode mode allows realistic
rack topology simulation (32 simulated nodes) using a single
instance (8 FPGAs).
Below, we outline the build and runtime configuration changes needed to utilize supernode designs. Supernode is currently only enabled for RocketChip designs with NICs. More details about supernode can be found in the FireSim ISCA 2018 Paper.
By default, supernode packs 4 identical designs into a single FPGA, and utilizes all 4 DDR channels available on each FPGA on AWS F1 instances. It currently does so by generating a wrapper top level target which encapsualtes the four simulated target nodes. The packed nodes are treated as 4 separate nodes, are assigned their own individual MAC addresses, and can perform any action a single node could: run different programs, interact with each other over the network, utilize different block device images, etc. In the networked case, 4 separate network links are presented to the switch-side.
Building Supernode Designs
Here, we outline some of the changes between supernode and regular simulations that are required to build supernode designs.
The Supernode target configuration wrapper can be found in Chipyard in
chipyard/generators/firechip/src/main/scala/TargetConfigs.scala. An example wrapper
class SupernodeFireSimRocketConfig extends Config( new WithNumNodes(4) ++ new freechips.rocketchip.subsystem.WithExtMemSize((1 << 30) * 8L) ++ // 8 GB new FireSimRocketConfig)
In this example,
SupernodeFireSimRocketConfig is the wrapper, while
FireSimRocketConfig is the target node configuration. To simulate a
different target configuration, we will generate a new supernode wrapper, with
the new target configuration. For example, to simulate 4 quad-core nodes on one
FPGA, you can use:
class SupernodeFireSimQuadRocketConfig extends Config( new WithNumNodes(4) ++ new freechips.rocketchip.subsystem.WithExtMemSize((1 << 30) * 8L) ++ // 8 GB new FireSimQuadRocketConfig)
Next, when defining the build recipe, we must remember to use the supernode
DESIGN parameter should always be set to
FireSim, while the
TARGET_CONFIG parameter should be set to
the wrapper configuration that was defined in
PLATFORM_CONFIG can be selected the same as in regular FireSim
configurations. For example:
DESIGN: FireSim TARGET_CONFIG: SupernodeFireSimQuadRocketConfig PLATFORM_CONFIG: BaseF1Config deploy_quintuplet: null
We currently provide a single pre-built AGFI for supernode of 4 quad-core
RocketChips with DDR3 memory models. You can build your own AGFI, using the supplied samples in
config_build_recipes.yaml. Importantly, in order to meet FPGA timing
contraints, Supernode target may require lower host clock frequencies.
Host clock frequencies can be configured as parts of the
(this must be done using
PLATFORM_CONFIG if not using F1) in
Running Supernode Simulations
Running FireSim in supernode mode follows the same process as in
“regular” mode. Currently, the only difference is that the main simulation
screen remains with the name
fsim0, while the three other simulation screens
can be accessed by attaching
respectively. All simulation screens will generate uart logs (
uartlog3). Notice that you must use
sudo in order to
attach to the uartpty or view the uart logs. The additional uart logs will not
be copied back to the manager instance by default (as in a “regular” FireSim
simulation). It is neccessary to specify the copying of the additional uartlogs
(uartlog1, uartlog2, uartlog3) in the workload definition.
Supernode topologies utilize a
FireSimSuperNodeServerNode class in order to
represent one of the 4 simulated target nodes which also represents a single
FPGA mapping, while using a
FireSimDummyServerNode class which represent
the other three simulated target nodes which do not represent an FPGA mapping.
In supernode mode, topologies should always add nodes in pairs of 4, as one
FireSimSuperNodeServerNode and three
Various example Supernode topologies are provided, ranging from 4 simulated target nodes to 1024 simulated target nodes.
Below are a couple of useful examples as templates for writing custom Supernode topologies.
A sample Supernode topology of 4 simulated target nodes which can fit on a
def supernode_example_4config(self): self.roots = [FireSimSwitchNode()] servers = [FireSimSuperNodeServerNode()] + [FireSimDummyServerNode() for x in range(3)] self.roots.add_downlinks(servers)
A sample Supernode topology of 32 simulated target nodes which can fit on a
def supernode_example_32config(self): self.roots = [FireSimSwitchNode()] servers = UserTopologies.supernode_flatten([[FireSimSuperNodeServerNode(), FireSimDummyServerNode(), FireSimDummyServerNode(), FireSimDummyServerNode()] for y in range(8)]) self.roots.add_downlinks(servers)
config_runtime.yaml requires selecting a supernode agfi in conjunction with a defined supernode topology.
Work in Progress!
We are currently working on restructuring supernode to support a wider-variety of use cases (including non-networked cases, and increased packing of nodes). More documentation will follow. Not all FireSim features are currently available on Supernode. As a rule-of-thumb, target-related features have a higher likelihood of being supported “out-of-the-box”, while features which involve external interfaces (such as TracerV) has a lesser likelihood of being supported “out-of-the-box”