Experiments¶
The experiments interface allows the user to easily run and compare a number of different scenarios.
Defining experiments¶
import passengersim as pax
pax.versions()
passengersim 0.94
passengersim.core 0.94
The experimentation starts with a base config.
cfg = pax.Config.from_yaml(pax.demo_network("3MKT/08-untrunc-em"))
cfg.simulation_controls.num_trials = 1
cfg.db.result_storage = "temporary"
cfg.db.filename = None
When we run experiments, we’ll do a bunch of simulations, but for this demo we don’t want to use too many compute resources, so we’ve set the number of trials to 1. In a real experiment, you would probably want to use more trials to get more reliable results.
We’re also setting the database storage to “temporary” and the filename to None, which allows
PassengerSim to use a temporary on-disk database for the simulation results instead of storing them
in memory. This is necessary because the Experiments interface orchestrates data collection from
multiple simulation runs, and transferring in-memory database content across sub-processes is not
convenient or efficient.
Then, we create an Experiments object, which will manage our set of experiments somewhat
automatically. This object takes our baseline config as an argument, as well as allows us to set a
directory where all experimental output will be stored.
from passengersim.experiments import Experiments
experiments = Experiments(cfg, output_dir="demo-output")
Now we can define one or more experiments, although as you’ll see quickly, the real power of this interface comes when there is more than one experiment.
Each experiment is defined by a function that accepts a Config as an argument, and returns a possibly modified Config. Within this function, you can make any modifications desired: changing simulation controls, swapping out revenue management systems for one or more carriers, or even changing the network structure itself. Each experiment function is prefixed by the experiments object as decorator, and each should have a unique function name that will be used to identify it.
@experiments
def baseline(cfg: pax.Config) -> pax.Config:
return cfg
@experiments
def low_dmd(cfg: pax.Config) -> pax.Config:
cfg.simulation_controls.demand_multiplier = 0.9
return cfg
@experiments
def high_dmd(cfg):
cfg.simulation_controls.demand_multiplier = 1.1
return cfg
Running experiments¶
Once all our experiments are defined, we can run all the experiments as a batch using the run
command on the Experiments object.
summaries = experiments.run()
/Users/jpn/Git/pax/.venv/lib/python3.12/site-packages/pydantic/main.py:732: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results. return cls.__pydantic_validator__.validate_python(
/Users/jpn/Git/pax/.venv/lib/python3.12/site-packages/pydantic/main.py:732: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
return cls.__pydantic_validator__.validate_python(
/Users/jpn/Git/pax/packages/passengersim-public/src/passengersim/config/manipulate.py:49: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
schema_validator.validate_python(model.__dict__)
/Users/jpn/Git/pax/.venv/lib/python3.12/site-packages/pydantic/main.py:732: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
return cls.__pydantic_validator__.validate_python(
/Users/jpn/Git/pax/packages/passengersim-public/src/passengersim/config/manipulate.py:49: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
schema_validator.validate_python(model.__dict__)
/Users/jpn/Git/pax/.venv/lib/python3.12/site-packages/pydantic/main.py:732: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
return cls.__pydantic_validator__.validate_python(
/Users/jpn/Git/pax/packages/passengersim-public/src/passengersim/config/manipulate.py:49: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
schema_validator.validate_python(model.__dict__)
╭─────────────── Summary revenue statistics ────────────────╮ │ AL1 AL2 │ │ Run Samples Mean ~95% CI Mean ~95% CI │ │ low_dmd 450 83,557 ±1,247 83.8K ±1,236 │ │ baseline 450 90,874 ±1,172 90,328 ±1,171 │ │ high_dmd 450 96,560 ±1,140 96,568 ±1,154 │ │ Post-burn-in only · Approx. 95% CI for mean │ ╰───────────────────────────────────────────────────────────╯
The return value from this batch run is a PassengerSim Contrast object, which can be used to
review the results interactively in a Jupyter notebook.
summaries.fig_carrier_revenues()
summaries.fig_fare_class_mix()
When we ran the experiments, the demo-output directory was populated with outputs from each
experiment, including a pickle file storing the summary results, as well as an HTML output file that
includes key figures and metadata describing the results from that experiment.
from passengersim.utils.show_dir import display_directory_contents
display_directory_contents("demo-output")
demo-output/
high_dmd/
high_dmd.20261103-234633.pxsim
passengersim_output.20261103-234633.html
low_dmd/
passengersim_output.20261103-234633.html
low_dmd.20261103-234633.pxsim
experiments-summary.20261103-234639.html
baseline/
baseline.20261103-234633.pxsim
passengersim_output.20261103-234633.html
We can change an existing experiment explicitly by writing a new experiment with the same tag, or implicitly by editing the Jupyter notebook and re-running the entire notebook. Here, we will just edit one experiment by overwriting it (note we do get a warning when we do this).
@experiments
def high_dmd(cfg): # noqa: F811
cfg.simulation_controls.demand_multiplier = 1.2
return cfg
/var/folders/sb/mb_3f0t16lx0wm3tr8r5g9rm0000gn/T/ipykernel_25673/1986293878.py:1: OverwriteExperimentWarning: Overwriting existing experiment tag: high_dmd
@experiments
If we now re-run the set of experiments, PassengerSim will detect that some of the experiments are have been run already, and not re-run them in favor of simply reloading from disk. The loaded results configurations are compared against the experiment configuration, to confirm it is still the same. For the first two experiments, this is the case and the simulation is not re-run. The change we made in the last experiment is detected, and the loaded results are then discarded in favor of re-running the simulation.
summaries2 = experiments.run()
Using cached results for experiment baseline
Using cached results for experiment low_dmd
/Users/jpn/Git/pax/.venv/lib/python3.12/site-packages/pydantic/main.py:732: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results. return cls.__pydantic_validator__.validate_python(
/Users/jpn/Git/pax/.venv/lib/python3.12/site-packages/pydantic/main.py:732: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results. return cls.__pydantic_validator__.validate_python(
Loaded high_dmd from demo-output/high_dmd/high_dmd.20261103-234633.pxsim, but the config has changed: {'simulation_controls': {'demand_multiplier': '1.2 != 1.1'}}
/Users/jpn/Git/pax/.venv/lib/python3.12/site-packages/pydantic/main.py:732: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
return cls.__pydantic_validator__.validate_python(
/Users/jpn/Git/pax/packages/passengersim-public/src/passengersim/config/manipulate.py:49: UserWarning: Only 450 usable samples remain after burning 50 of 500 samples across 1 trials; at least 750 usable samples are recommended to ensure meaningful results.
schema_validator.validate_python(model.__dict__)
╭────────── Summary revenue statistics · high_dmd ──────────╮ │ AL1 AL2 │ │ Run Samples Mean ~95% CI Mean ~95% CI │ │ high_dmd 450 102.7K ±1,151 101.9K ±1,167 │ │ Post-burn-in only · Approx. 95% CI for mean │ ╰───────────────────────────────────────────────────────────╯
summaries2.fig_carrier_revenues()
summaries2.fig_fare_class_mix()
If we inspect the demo-output directory, we will see that the results from the old experiment are
still available if needed, but they are timestamped so we can clearly identify them as older.
display_directory_contents("demo-output")
demo-output/
experiments-summary.20261103-234644.html
high_dmd/
passengersim_output.20261103-234639.html
high_dmd.20261103-234633.pxsim
passengersim_output.20261103-234633.html
high_dmd.20261103-234639.pxsim
low_dmd/
passengersim_output.20261103-234633.html
low_dmd.20261103-234633.pxsim
experiments-summary.20261103-234639.html
baseline/
baseline.20261103-234633.pxsim
passengersim_output.20261103-234633.html
We can write out a report of the experiments, which contains a variety of standard outputs.
out_file = summaries2.write_report(
"demo-output/meta-summary.html", base_config=cfg
)
from passengersim.utils.iframe import preview_html
preview_html(out_file)
While running experiments, there are a variety of live progress monitors available, see run diagnostics for details.