Experiments

The experiments interface allows the user to easily run and compare a number of different scenarios.

import passengersim as pax

pax.versions()

passengersim 0.93
passengersim.core 0.93

The experimentation starts with a base config.

cfg = pax.Config.from_yaml(pax.demo_network("3MKT/08-untrunc-em"))
cfg.simulation_controls.num_trials = 1

Then, we create an Experiments object, which will manage our set of experiments somewhat automatically. This object takes our baseline config as an argument, as well as allows us to set a directory where all experimental output will be stored.

from passengersim.experiments import Experiments

experiments = Experiments(cfg, output_dir="demo-output")

Now we can define one or more experiments, although as you’ll see quickly, the real power of this interface comes when there is more than one experiment.

Each experiment is defined by a function that accepts a Config as an argument, and returns a possibly modified Config. Within this function, you can make any modifications desired: changing simulation controls, swapping out revenue management systems for one or more carriers, or even changing the network structure itself. Each experiment function is prefixed by the experiments object as decorator, and each should have a unique function name that will be used to identify it.

@experiments
def baseline(cfg: pax.Config) -> pax.Config:
    return cfg


@experiments
def low_dmd(cfg: pax.Config) -> pax.Config:
    cfg.simulation_controls.demand_multiplier = 0.9
    return cfg


@experiments
def high_dmd(cfg):
    cfg.simulation_controls.demand_multiplier = 1.1
    return cfg

Finally, we can run all the experiments as a batch using the run command on the Experiments object.

summaries = experiments.run()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[6], line 1
----> 1 summaries = experiments.run()

File ~/Git/pax/packages/passengersim-public/src/passengersim/experiments.py:1063, in Experiments.run(self, use_existing, tag, check_versions, check_content, single_process, retain_sims, write_report, cache_results)
   1052     result = self._run_experiments_in_sequence(
   1053         use_existing=use_existing,
   1054         tag=tag,
   (...)   1060         cache_results=cache_results,
   1061     )
   1062 else:
-> 1063     result = self._run_together(
   1064         use_existing=use_existing,
   1065         tag=tag,
   1066         check_versions=check_versions,
   1067         check_content=check_content,
   1068         retain_sims=retain_sims,
   1069         write_report=write_report,
   1070         cache_results=cache_results,
   1071     )
   1073 from passengersim.driver._trial_database import finalize_result
   1075 if isinstance(result, GenericSimulationTables):

File ~/Git/pax/packages/passengersim-public/src/passengersim/experiments.py:862, in Experiments._run_together(self, use_existing, tag, check_versions, check_content, retain_sims, write_report, cache_results, summarizer)
    859 else:
    860     raise TypeError("tag must be a string or None")
--> 862 prepared_configs = self._prepare_database_configs(selected_experiments)
    863 with JobExecutor() as jobber:
    864     try:

File ~/Git/pax/packages/passengersim-public/src/passengersim/experiments.py:509, in Experiments._prepare_database_configs(self, experiments)
    507     continue
    508 config = experiment.func(self.base_config.model_copy(deep=True))
--> 509 policy = resolve_storage(config.db, orchestrated=True)
    510 validate_database_reports(config, policy)
    511 if policy.filename is not None:

File ~/Git/pax/packages/passengersim-public/src/passengersim/config/storage.py:49, in resolve_storage(db, orchestrated, borrowed)
     47     raise ValueError("Persistent storage requires db.filename or db.result_directory")
     48 if mode == "memory" and orchestrated:
---> 49     raise ValueError(
     50         "Memory storage is standalone-only; use temporary or summary_only for MultiSimulation/Experiments"
     51     )
     52 return StoragePolicy(
     53     mode,
     54     Path(filename).absolute() if filename is not None else None,
     55     Path(db.result_directory).absolute() if db.result_directory is not None else None,
     56 )

ValueError: Memory storage is standalone-only; use temporary or summary_only for MultiSimulation/Experiments

The return value from this batch run is a passengersim Contrast object, which can be used to review the results interactively in a Jupyter notebook.

summaries.fig_carrier_revenues()

summaries.fig_fare_class_mix()

When we ran the experiments, the demo-output directory was populated with outputs from each experiment, including a pickle file storing the summary results, as well as an HTML output file that includes key figures and metadata describing the results from that experiment.

from passengersim.utils.show_dir import display_directory_contents

display_directory_contents("demo-output")

We can change an existing experiment explicitly by writing a new experiment with the same tag, or implicitly by editing the Jupyter notebook and re-running the entire notebook. Here, we will just edit one experiment by overwriting it (note we do get a warning when we do this).

@experiments
def high_dmd(cfg):  # noqa: F811
    cfg.simulation_controls.demand_multiplier = 1.2
    return cfg

If we now re-run the set of experiments, PassengerSim will detect that some of the experiments are have been run already, and not re-run them in favor of simply reloading from disk. The loaded results configurations are compared against the experiment configuration, to confirm it is still the same. For the first two experiments, this is the case and the simulation is not re-run. The change we made in the last experiment is detected, and the loaded results are then discarded in favor of re-running the simulation.

summaries2 = experiments.run()

summaries2.fig_carrier_revenues()

summaries2.fig_fare_class_mix()

If we inspect the demo-output directory, we will see that the results from the old experiment are still available if needed, but they are timestamped so we can clearly identify them as older.

display_directory_contents("demo-output")

We can write out a report of the experiments, which contains a variety of standard outputs.

out_file = summaries2.write_report(
    "demo-output/meta-summary.html", base_config=cfg
)

from passengersim.utils.iframe import preview_html

preview_html(out_file)