Driver

The Driver class is the top-level class which runs chombo-discharge simulations. The primary purpose of the Driver class is to

  1. Coordinate the simulation advance, i.e., run the relevant TimeStepper that is supplied. This can also involve restarting simulations from a user-provided time step.

  2. Initiate and coordinate regrid operations.

  3. Coordinate HDF5 I/O operations.

The constructor for this class is

/*!
  @brief Full constructor. 
  @param[in] a_computationalGeometry Computational geometry. 
  @param[in] a_timeStepper           The time stepper
  @param[in] a_amrMesh               AmrMesh core class. 
  @param[in] a_cellTagger            Cell tagger which flags cell for refinement. 
*/
Driver(const RefCountedPtr<ComputationalGeometry>& a_computationalGeometry,
       const RefCountedPtr<TimeStepper>&           a_timeStepper,
       const RefCountedPtr<AmrMesh>&               a_amrMesh,
       const RefCountedPtr<CellTagger>&            a_cellTagger = RefCountedPtr<CellTagger>(nullptr));

Here, the Driver class takes as input a user-provided geometry a_computationalGeometry, a user-provided physics module a_timestepper, and the AMR infrastructure a_amrMesh. The Driver class does not require an instance of CellTagger, which is the responsible for flagging cells for refinement. If users decide to omit a cell tagger as in the above constructor, regridding functionality is completely turned off and only the initially generated grids will be used throughout the entire simulation.

Tip

Here is the Driver C++ API.

Warning

Usage of injection of the AmrMesh object is likely to change in future versions of chombo-discharge.

Simulation setup

For setting up an running simulations, only a single routine is used:

void setupAndRun();

This routine will set up and run a simulation.

New simulations

If a simulation starts from the first time step, the Driver class will perform the following steps in setupAndRun().

  1. Ask ComputationalGeometry to generate the cut-cell moments.

  2. Collect all the cut-cells and ask AmrMesh to set up an initial grid and generate the initial AMR infrastructure. This initial grid is always generated by flagging cells for refinement along solid boundaries. If no such cells exist, a uniform grid on the coarsest domain is generated. Various options are available for configuring this initial grid, see Cell refinement philosophy. Also note that it is possible to restrict the maximum level that can be generated from the geometric tags.

  3. Ask the TimeStepper to set up relevant solvers and fill them with initial data.

  4. Perform the number of initial regrids that the user asks for. After the regrid, the solvers are re-filled with initial data. I.e., initial data is always regenerated on the finest grid and is not interpolated from coarse grid.

  5. Let TimeStepper perform a post-initialization routine. Whether or not this post-initialization does anything depends on the physics module begin used.

Restarting simulations

If a simulation uses the restart functionality (i.e., does not start from the first time step), Driver will execute the following steps in setupAndRun(...).

  1. Ask ComputationalGeometry to generate the cut-cell moments.

  2. Read a checkpoint file that contains the grids and all the data that have been checkpointed by the solvers. Driver will issue an error and abort if the checkpoint file does not exist.

  3. Ask TimeStepper to perform a post-checkpoint step. This functionality has been included because not all data in every solver needs to be checkpointed. For example, an electric field solver only needs to write the electric potential to the checkpoint file because the electric field is obtained by taking the gradient.

  4. Perform the number of initial regrids that the user asks for.

Simulation advancement

The algorithm for running a simulation is conceptually simple. Driver calls TimeStepper::computeDt for computing a reasonable time step for advancing the equations, and uses Real TimeStepper::advance(Real dt) to actually perform the advance. Note that these functions are abstract functions in TimeStepper, so the actual contents of these will differ between physics modules.

Regrids, plot files, and checkpoint files are written at certain step intervals, or specified time intervals. In brief, the algorithm looks like this:

Driver::run(...){

   while(KeepRunningTheSimulation){
      if(RegridEverything){
         Driver->regrid()
      }

      tryDt    = TimeStepper->computeDt()
      actualDt = TimeStepper->advance(tryDt)

      if(WriteAPlotFile or EndOfSimulation){
         Driver->writePlotFile();
      }
      if(TimeToWriteACheckpointFile or EndOfSimulation){
         Driver->writeCheckpointFile()
      }

      KeepRunningTheSimulation = true or false
   }
}

Regridding

Regrids are called by the Driver class and proceed as follows:

  1. CellTagger generates tags for grid refinement and coarsening.

  2. TimeStepper stores data that is required during regrids. This is necessary because we often need storage containers to store the solver states on both the old and the new grids. For mesh-based data, data is usually generated by interpolating data from the old grid to the new one.

  3. AmrMesh generates the new grid boxes and associated EB information, and performs an initial load balancing.

  4. TimeStepper checks if the application should re-balance the grids.

  5. AmrMesh reinstantiates the EB and AMR grid infrastructure (e.g., interpolation operators).

  6. TimeStepper performs a regrid operation of the physics module.

  7. TimeStepper performs a post-regrid operation (e.g., filling solvers with auxiliary data).

In C++ pseudo-code, this looks something like:

Driver::regrid(){

   // Tag cells
   CellTagger->tagCellsForRefinement()

   // Store old data and free up some memory
   TimeStepper->storeOldGridData()

   // Generate the new grids
   AmrMesh->makeNewGrids()

   if(loadBalance) {
     TimeStepper->loadBalance();
   }

   // AmrMesh finalizes the EBAMR grids
   AmrMesh->regridOperators()

   // Regrid timestepper
   TimeStepper->regrid()

   // Do a post-regrid step
   TimeStepper->postRegrid()
}

Class options

Various class options are available for adjusting the behavior of the Driver class. Below, we include the current template options file for the Driver class.

Listing 1 Template input options for the Driver class. Runtime adjustable options are highlighted.
# ====================================================================================================
# Driver class options
# ====================================================================================================
Driver.verbosity                    = 2                   ## Driver verbosity.
Driver.geometry_generation          = chombo-discharge    ## Grid generation method, 'chombo-discharge' or 'chombo'
Driver.geometry_scan_level          = 0                   ## Geometry scan level for chombo-discharge geometry generator
Driver.ebis_memory_load_balance     = false               ## If using Chombo geo-gen, use memory as loads for EBIS generation
Driver.output_dt                    = -1.0                ## Output interval (values <= 0 enforces step-based output)
Driver.plot_interval                = 10                  ## Plot interval
Driver.checkpoint_interval          = 100                 ## Checkpoint interval
Driver.regrid_interval              = 10                  ## Regrid interval
Driver.write_regrid_files           = false               ## Write or do not write plot files during regrids.
Driver.write_restart_files          = false               ## Write or do not write plot files during restart or not.
Driver.initial_regrids              = 0                   ## Number of initial regrids.
Driver.do_init_load_balance         = false               ## If true, load balance the first step in a fresh simulation.
Driver.start_time                   = 0                   ## Start time (fresh simulations only).
Driver.stop_time                    = 1.0                 ## Stop time.
Driver.max_steps                    = 100                 ## Maximum number of steps.
Driver.geometry_only                = false               ## Special option that ONLY plots the geometry.
Driver.write_memory                 = false               ## Write MPI memory report.
Driver.write_loads                  = false               ## Write (accumulated) computational loads.
Driver.output_directory             = ./                  ## Output directory.
Driver.output_names                 = simulation          ## Simulation output names.
Driver.max_plot_depth               = -1                  ## Restrict maximum plot depth (-1 => finest simulation level).
Driver.max_chk_depth                = -1                  ## Restrict chechkpoint depth (-1 => finest simulation level).
Driver.num_plot_ghost               = 1                   ## Number of ghost cells to include in plots.
Driver.plt_vars                     = levelset            ## 'tags', 'mpi_rank', 'levelset', 'loads'.
Driver.restart                      = 0                   ## Restart step (less or equal to 0 implies fresh simulation).
Driver.allow_coarsening             = true                ## Allows removal of grid levels according to CellTagger.
Driver.grow_geo_tags                = 2                   ## How much to grow tags when using geometry-based refinement.
Driver.refine_angles                = 15.                 ## Refine cells if angle between elements exceed this value.
Driver.refine_electrodes            = 0                   ## Refine electrode surfaces to specified level ( < 0 will refine everything)
Driver.refine_dielectrics           = 0                   ## Refine dielectric surfaces to specified level ( < 0 will refine everything)

We point out that the output_directory directory variable is only for the HDF5 plot files, whereas other files like pout.* are put in the directory in which the executable was ran.

From the user perspective, the most commonly adjusted parameters concern the I/O operations (plot and checkpoint intervals), and geometric refinement parameters.