SciPy

Run-time configuration parameters

As many configuration parameters as possible are passed through a single config file, in particular all those parameters that do not significantly affect the code’s memory usage. All parameters are written to the SPARTA output file (see The SPARTA HDF5 output format).

Syntax

The SPARTA config file follows a simple key-value syntax:

# This is a SPARTA config file
<key1> <value> #This parameter has a single value
<key2> <value1>, <value2>, <value3>  #This parameter has multiple values

All content after # symbols is ignored. Lists, i.e. multiple values, are comma-separated and may contain spaces after the commas.

List of parameters

If the default for a parameter is NONE, it must be set by the user or the code will abort. The following table lists only those parameters that are general to the code or govern the treatment of tracers. The parameters for the specific Results and Analyses are listed in the respective sections.

Parameter

Type

Default

Explanation

Input (Halo Catalogs)

cat_path

string

NONE

The path to the catalog files

cat_type

int

NONE

See details below

cat_tolerance_a_snap

float

0.0001

The scale factors a of catalog and snapshot files have to agree to this precision, otherwise the code raises an error.

cat_reliable_n200m

int

20

Halo radii above this number of particles are to be considered reliable

cat_halo_jump_tol_phys

float

10.0

If halo jumps more than this many Mpc/h comoving, it is ended

cat_halo_jump_tol_box

float

0.03

If halo jumps more than this many box sizes, it is ended

catkey_id

string

“id”

Identifier of halo ID column in Rockstar catalog

catkey_desc_id

string

“desc_id”

Identifier of descendant ID column in Rockstar catalog

catkey_pid

string

“pid”

Identifier of parent ID column in Rockstar catalog

catkey_desc_pid

string

“desc_pid”

Identifier of descendant parent ID column in Rockstar catalog

catkey_num_prog

string

“num_prog”

Identifier of number of progenitors column in Rockstar catalog

catkey_mmp

string

“mmp?”

Identifier of is-most-massive-progenitor column in Rockstar catalog

catkey_phantom

string

“phantom”

Identifier of phantom (reconstructed halo) column in Rockstar catalog

catkey_x

string

“x”

Identifier of x-coordinate column in Rockstar catalog

catkey_y

string

“y”

Identifier of y-coordinate column in Rockstar catalog

catkey_z

string

“z”

Identifier of z-coordinate column in Rockstar catalog

catkey_vx

string

“vx”

Identifier of x-velocity column in Rockstar catalog

catkey_vy

string

“vy”

Identifier of y-velocity column in Rockstar catalog

catkey_vz

string

“vz”

Identifier of z-velocity column in Rockstar catalog

catkey_m200m_all

string

“M200b_all”

Identifier of M200m (including all particles) column in Rockstar catalog

catkey_rbound

string

“rvir”

Identifier of R200m (bound-only) column in Rockstar catalog

catkey_mbound

string

“mvir”

Identifier of M200m (bound-only) column in Rockstar catalog

Input (Snapshots)

snap_path

string

NONE

The path to the snapshot files or folders. See below for details

snap_sim_type

int

NONE

See details below

snap_max_read_procs

int

64

The maximum number of processes that can concurrently read snapshot files.

snap_max_waiting_messages

int

64

The maximum number of particle data chunks a process keeps in memory before stopping the reading of further particles.

Input (Simulation properties)

sim_force_res

float

NONE

The force resolution in kpc/h at z = 0

sim_force_res_comoving

bool

TRUE

Whether the force resolution is fixed in physical or comoving units. In Gadget2, the softening is comoving, but if the MaxPhys parameter has the same value, it translates into a force resolution fixed in physical coordinates.

File output

output_path

string

“output”

Path where output and temporary files are stored

output_file

string

“sparta.hdf5”

Path where output and temporary files are stored

output_compression_level

int

1

Compression level of the output HDF5 file; 0 means no compression, 9 max. comp.

output_restart_files

int

FALSE

Output restart files from which a SPARTA run can be continued

output_restart_every

int

20

Frequency (in snapshots) with which restart files are written

output_min_n200m

int

100

Minimum number of particles within R200m for a halo to be saved (at any point in its history while it is a host halo). See note below.

output_minrs_<tcr>_<res>

int

0

Minimum number of results for a halo to be saved, for tracer <tcr> (e.g. particles) and result <res> (e.g. infall).

Console output

log_level

int

1

See below

log_level_memory

int

2

See below

log_level_timing

int

0

See below

log_flush

int

0

Flush console output after every line; can be useful for debugging

Error levels

err_level_cannot_fork

int

1

Ignore/warning/error when a sub-process cannot be created (for a system call)

err_level_file_copy_failed

int

1

Ignore/warning/error when the system call copying a file may have failed

err_level_cat_snap_mismatch

int

1

Ignore/warning/error when the found catalogs and snapshots do not match up exactly

err_level_skipping_snap

int

0

Ignore/warning/error when a snapshot is skipped because there is no catalog

err_level_rockstar_filename

int

0

Ignore/warning/error when the scale fac. in a Rockstar filename is slightly mismatched

err_level_invalid_domain_box

int

1

Ignore/warning/error when the domain box has a weird shape, and the entire box is used

err_level_domain_minor

int

0

Ignore/warning/error when irregularity appeared in the domain calc. (can be frequent)

err_level_halo_req_not_found

int

2

Ignore/warning/error when a halo expected from the prev. snap is not found in catalog

err_level_desc_not_found

int

1

Ignore/warning/error when a descendant halo is not found

err_level_found_unexpected

int

1

Ignore/warning/error when a halo with progenitor(s) is found in catalog, but not in SPARTA (this can happen if the halo had to be aborted due to an unusual error)

err_level_halo_jump

int

1

Ignore/warning/error when a halo jumps unphysically (see jump tolerance parameters)

err_level_missing_halo_ptl

int

0

Ignore/warning/error when many fewer ptl are found in halo than expected from catalog

err_level_cat_radius_diff

int

0

Ignore/warning/error when R200m differs from catalog

err_level_zero_ptl

int

1

Ignore/warning/error when zero particles are found in a halo

err_level_zero_ptl_phantom

int

1

Ignore/warning/error when zero particles are found in a phantom (reconstructed) halo

err_level_radius_outside_box

int

1

Ignore/warning/error when the halo radius cannot be found within the box

err_level_radius_not_found

int

1

Ignore/warning/error when R200m could not be determined due to the halo density profile

err_level_radius_bound_diff

int

1

Ignore/warning/error when R200m_all much larger than R200m_bnd and is set to R200m_bnd

err_level_search_radius

int

1

Ignore/warning/error when search radius had to be reduced (missing particles in halo)

err_level_exc_host_ended

int

1

Ignore/warning/error when a halo is exchanged and had ended due to its host ending

err_level_exc_halo_not_found

int

1

Ignore/warning/error when a halo is not found during the exchange step

err_level_max_exchange_size

int

1

Ignore/warning/error when the maximum exchange message size is exceeded

err_level_large_halo

int

1

Ignore/warning/error when a halo’s total memory exceeds a user-defined limit

Instructions

last_snap

int

-1

Limit the number of snaps (according to catalogs); if -1, all snaps are processed

last_scale_factor

int

-1.0

Limit the number of snaps; if >0, stop at this scale factor

doall_<tcr>_<res>

int

TRUE

(De-)activate a particular result for a particular tracer, in all halos

dohalo_<tcr>_<res>

[int]

NONE

(De-)activate a particular result for a particular tracer, for a number of halo IDs; see detailed description below.

Domain & Exchange

lb_do_load_balancing

bool

TRUE

This parameter can turn load balancing off altogether; not recommended.

lb_adjust_min_snap

int

10

First snap where domain readjustment is performed

lb_slabs_adjust_factor

float

0.2

Attempt to shift domain boundaries this factor times the load imbalance

lb_slabs_adjust_max

float

0.01

Max domain boundary shift in units of box size

lb_slabs_drift_tolerance

float

0.005

Fraction of box size by which halo can be outside domain before it is exchanged

exc_max_message_size

float

500.0

Max. exchange message size in MB

Halos

halo_min_radius_mass_profile

float

0.01

The smallest radial bin for the internally stored mass profile (in units of R200m)

halo_max_radius_ratio_cat

float

10.0

If R200m_all is larger than this number times R200m_cat, set it to R200m_cat

halo_correct_phantoms

bool

TRUE

If on, recalculate the position/velocity of phantoms like a ghost (if those are on)

ghost_min_n200m

int

10

Minimum number of particles for ghosts before they are abandoned

ghost_min_host_distance

float

0.05

Minimum distance from host center in host R200m units for ghost not to be merged

ghost_max_host_center_time

float

0.5

Maximum time in dynamical times for ghost to be within ghost_min_host_distance

Other parameters

random_seed

int

0

If non-zero, use this random seed. Makes many results reproducible.

tree_points_per_leaf

int

100

Max. number of particles in leaf nodes of particle tree

potential_points_per_leaf

int

10

Number of ptls. in leaf node, for which potential is computed by direct summation

potential_err_tol

float

0.1

The error tolerance for potentials; larger nums. mean more aggressive approximation

memory_allocation_factor

float

1.5

Dynamic arrays are increased in size by this factor when current size exceeded

memory_dealloc_safety_factor

float

0.8

Dynamic arrays are decreased by memory_allocation_factor when their size falls below 1 / memory_allocation_factor * memory_dealloc_safety_factor

memory_max_halo_size

float

2000.0

Memory size of a halo (incl subhalos) above which we ignore/warn/abort as desired

Parameters specific to particle tracers

tcr_ptl_create_radius

float

2.0

Particle tracers are created when they are within this factor times R200m

tcr_ptl_delete_radius

float

3.0

Particle tracers are deleted when they are outside this factor times R200m

tcr_ptl_subtag_radius

float

2.0

Select and track particles within this factor times R200m of the subhalo

tcr_ptl_subtag_inclusive

bool

TRUE

When tagging subhalo particles, tag if any (rather than all) criteria are fulfilled

tcr_ptl_subtag_ifl_age_min

float

0.5

Tag subhalo particle if infall event older than this factor times tdyn is found

tcr_ptl_subtag_ifl_dist_min

float

2.0

Tag subhalo ptl. if infall event farther than this factor times R200m of host is found

tcr_ptl_subtag_bound_radius

float

0.5

Include particles within this factor times R200m of the subhalo in the grav. potential

tcr_ptl_subtag_bound_ratio

float

1.0

Tag particles with at least this potential-to-kinetic energy ratio

tcr_ptl_subtag_host_max_age

float

0.5

If infall event in host is more recent than this factor times tdyn, tag it

Additional comments on particular parameters

Catalog type

SPARTA can read a number of different halo catalog formats. This parameter must appear in the config file before any non-default catalog keys (catkey_<field>)! Valid catalog types are:

Type

Explanation

rockstar

Path points to a folder of .hlist files created by Rockstar and consistent-trees

Note that consistent-trees must have been run on the Rockstar catalogs as SPARTA relies on host-sub and other information (see Halos & Subhalos).

Snapshot path

The snap_path parameter must probably contain keywords to indicate the snapshot index and file chunk, as well as the exact formatting of those numbers. In particular, the keyword <snap:format> translates into the snapshot number, formatted as given in the string format (when applied in the print() function in C). For example, for snapshot 72 and chunk 11, the snap path snapdir_<snap:%03d>/snapshot_<snap:%04d>.<chunk:%d>.dat will translate to snapdir_072/snapshot_0072.11.dat. If you make a mistake and no snapshots are found, the code will likely abort with the message “Found zero snapshots. Please check the snapshot path and filename format.” By setting log_level to at least 3, you can output the filenames the code attempts to find, which will probably help in fixing the problem.

Snapshot type

This parameter determines the file type of the snapshot types to be read, and obviously depends on the simulation code that was used to run the simulation.

Type

Explanation

lgadget

Gadget2 binary format

gadget3

Gadget3 format in the HDF5 version

arepo

Arepo format, very similar to Gadget3

Minimum output mass

The output_min_n200m parameter sets the minimum mass a halo has to attain at any point in its lifetime for it to be written to the output file. However, when a halo becomes a subhalo, we set its internal mass (that is, M200m_all computed by SPARTA) to the mass at accretion, M_acc, and keep it at that value until the halo becomes a host again (if that ever happens). Thus, mass gains while a halo is a subhalo are considered unphysical and are not counted into the minimum mass.

Output level parameters

The log_level parameter sets how much console output SPARTA writes in general:

Level

Explanation

0

Barebones output, only one line per snapshot

1

Default, recommended; Output is limited to [Main], i.e. global actions and statistics

2

Very detailed output from [Main], and some process-level output (e.g. current volume)

3

More process-level output, especially about sending/received halo data sets

4

Output for each processed halo, for debugging; not recommended

5

All output, only useful for serious debugging; not recommended

The log_level_memory parameter sets how much console output SPARTA writes about the memory consumption. The code tracks every single memory allocation and can give extremely detailed information about which fields consume memory.

Level

Explanation

0

Memory tracking disabled. This is NOT recommended as leaks will not be detected!

1

No memory output unless a memory leak is detected

2

Memory listing of the mean and peak usage as well as the biggest allocated variables (recommended)

3

Memory listing for each process and all variables at the point of peak usage

4

Listing for each allocation and de-allocation (NOT recommended!)

The log_level_timing parameter sets how much timing information SPARTA outputs.

Level

Explanation

0

No timing output, except the general timing line enabled by log_level = 1

1

Timing output at each snapshot, only actions that consumed significant time

2

Timing output at each snapshot, listing all actions

Error level parameters in general

All err_level_* parameters determine what happens when certain non-critical errors or inconsistencies happen during execution, for example, if halo tracks in the catalogs are inconsistent. In such cases, we can abort the halo and continue or stop. The parameteres can take on the following values:

Level

Explanation

0

Ignore this case; this is only recommended if you understand why the errors are occurring

1

Output warning, but continue

2

Throw an error, abort the SPARTA run

The default values are set depending on the severity of the issue. When reducing errors to warnings or warnings to ignore, please make sure you understand why the warning/error occurred in the first place, and that the issue really is non-critical. For example, if the majority of halos has broken histories, that indicates an issue with the merger trees and probably means that SPARTA’s output will be meaningless.

Instructions for individual halos

The dohalo_* parameter turns on or off certain results for certain tracer types. This can be useful if a result is very memory or computation-intensive, for example storing whole trajectories. In that case, you can turn the result off by setting doall_ptl_tjy  0 but turning it on for some halos, e.g. dohalo_ptl_tjy  1, 139, 3186, 128186. Note that here the first number is always 0 or 1 to indicate on/off, followed by halo IDs. Conversely, you can turn the result on for all halos and off for certain halos, for example to avoid a situation that occurs in one particular halo.

Catalog radius/mass comparisons

There are two error level parameters that can trigger a lot of warnings or errors, but that may be useful diagnostics: if err_level_missing_halo_ptl is set to warning or error, we count the exact number of particles inside R200m_all of host halos and compare it to the mass the halo finder reported. This is a tough check on particle loading. However, this setting can trigger a lot of warnings if the halo finder’s definition is not exactly an all-particle R200m, in which case it should be turned off.

Similarly, err_level_cat_radius_diff triggers warnings or errors if R200m_all from SPARTA and the halo finder do not agree quite exactly. This is a good check on the respective SO calculations, but again can lead to excessive warnings if the halo finder’s output is not exactly R200m_all. One common reason for such warnings is that SPARTA uses a truly strict SO criterion, whereas some FOF halo finders (such as ROCKSTAR) consider only particles within an FOF group. As these groups get less complete towards large radii, SPARTA may find larger radii. Thus, this warning is turned off by default.

Exchange message size

The exc_max_message_size parameter sets the maximum message size in MB for exchanging halos between processes (if SFC domain decomposition is enabled). Larger message sizes lead to fewer messages, but also larger buffers. If a single halo is larger than this size, the message will be sent anyway, but a warning/error can be output depending on the user’s settings.

Tree parameters

The tree_points_per_leaf parameter determines how many particles can be in one leaf node of the tree before that node is split again. The particle tree is used for searches, so changing this parameter does not change the outcome (all particles are always found) but it does influence how long it takes to build the tree, to search the tree, and how much memory is occupied. A smaller number means more memory, the optimum for the build and search times depends on the system.

For the potential tree, the situation is different; here, particles in a leaf node contribute to the potential collectively, so changing their maximum number influences both performance and accuracy.

Output compression

SPARTA enables the GNU gzip data compression in HDF5 files, as it tends to work with any installation of the HDF5 library (although there are fancier compression schemes). The user can choose the output_compression_level, where 0 means no compression. This is, of course, fastest. However, setting the compression level to at least 1 (minimal compression) can reduce file sizes significantly when there are large arrays of the same number, which can easily happen. For example, an array of integer zeros can be compressed to less than 1% of its actual size. Thus, it is recommended to keep the compression level at 1 or higher.

The write time for higher compression levels can increase significantly, and the file size does not usually decrease by a large factor any more. However, if file writing is a small fraction of the overall run time and file size is a concern, feel free to increase the compression level up to its maximum of 9.

Ghost parameters

With ghost halos, there is quite some freedom in when we abandon them as having merged into their host. In particular, the ghost_min_n200m parameter sets a lower limit on the mass within R200m. A ghost can, in principle, live forever after merging with its host because its particles are held together at the host center. We decide that the ghost has merged if it is within ghost_min_host_distance of the host’s R200m for at least ghost_max_host_center_time dynamical times (measured in units of the current dynamical time).

Memory allocation and de-allocation

Much (though by no means all) of the memory in SPARTA is in dynamic arrays that contain halos, tracers, and results. When the current size of such an array is exceeded, its allocation is increased by memory_allocation_factor, which is classically set to a number between 1 and 2.

Theoretically, memory_allocation_factor can be any number greater than unity. However, choosing very conservative factors (such as 1.1) will lead to a very large number of memory reallocations, slowing down the code. Numbers above the golden ratio (~1.61) are disfavored because they can be shown to lead to memory “holes” that cannot be used to reallocated the same array. For example, if memory_allocation_factor is 2, a sequence of 2, 4, 8, 16 etc will never find enough of its own freed memory to fit into it. However, in reality the validity of this argument will depend on the compiler and operating system, and many compilers still use a value of 2.

Similarly, if the array size falls below 1 / memory_allocation_factor * memory_dealloc_safety_factor, we decrease the array by memory_allocation_factor or even to n_new * memory_allocation_factor, whichever is smaller. The safety factor ensures that the array does not “oscillate”, which can happen if the thresholds for allocation and deallocation are too similar. Thus, memory_dealloc_safety_factor must be a number between 0 and 1, with a suggested value of 0.8. Assuming the default memory_allocation_factor of 1.5, the array will be reduced when it is less than memory_dealloc_factor == 1 / 1.5 * 0.8 ~ 0.53 of its elements are used.

This memory_dealloc_factor (which is displayed as part of the derived configuration parameters in the console log) sets a lower limit on the memory efficiency of all dynamic arrays in SPARTA, except for arrays that have not reached a fraction of their initial allocation. On many systems, reallocating memory is quite fast, so that even aggressive values do not increase the runtime of SPARTA too much.

To debug out-of-memory crashes, it can be useful to take a detailed look at the memory taken up by the very largest halos. For this purpose, the user can set memory_max_halo_size in MB. When a halo is found to exceed this amount of memory (including its subhalos, which must live on the same process), a very detailed accounting of its memory consumption is output unless err_level_large_halo is set to ignore. It if is set to error, we abort (not recommended).

Example config file

The following file contains those SPARTA parameters that are typically changed by the user. The file can be found at /config/sparta.cfg:

Complete config file

The following file contains all currently supported SPARTA parameters and their default values wherever applicable. The file can be found at /config/sparta_complete.cfg:

#--------------------------------------------------------------------------------------------------
# 
# Complete config file for SPARTA inclding all parameters. See the documentation for details.
#
#--------------------------------------------------------------------------------------------------

#--------------------------------------------------------------------------------------------------
# Input
#--------------------------------------------------------------------------------------------------

# Halo catalogs
cat_path                      /my/sim/dir/rockstar/hlists
cat_type                      rockstar
cat_tolerance_a_snap          0.0001
cat_reliable_n200m            20
cat_halo_jump_tol_phys        10.0
cat_halo_jump_tol_box         0.03

# For Rockstar halo catalogs: identifiers in header
catkey_id                     id
catkey_desc_id                desc_id
catkey_pid                    pid
catkey_desc_pid               desc_pid
catkey_num_prog               num_prog
catkey_mmp                    mmp?
catkey_phantom                phantom
catkey_x                      x
catkey_y                      y
catkey_z                      z
catkey_vx                     vx
catkey_vy                     vy
catkey_vz                     vz
catkey_m200m_all              M200b_all
catkey_rbound                 rvir
catkey_mbound                 mvir

# Snapshots
snap_path                     /my/sim/dir/snaps/snapdir_<snap:%04d>/snapshot_<snap:%04d>.<chunk:%d>
snap_sim_type                 lgadget
snap_max_read_procs           64
snap_max_waiting_messages     64

# Simulation
sim_force_res                 <MUST BE SET BY USER>
sim_force_res_comoving        1

#--------------------------------------------------------------------------------------------------
# Output
#--------------------------------------------------------------------------------------------------

# File output
output_path                   output
output_file                   sparta.hdf5
output_compression_level      1
output_restart_files          0
output_restart_every          20
output_min_n200m              100
# opt: output_minrs_<tcr>_<res>  0

# Console output
log_level                     1
log_level_memory              1
log_level_timing              0
log_flush                     0

# Error levels
err_level_cannot_fork         1
err_level_file_copy_failed    1
err_level_cat_snap_mismatch   1
err_level_skipping_snap       0
err_level_rockstar_filename   0
err_level_invalid_domain_box  1
err_level_domain_minor        0
err_level_halo_req_not_found  2
err_level_desc_not_found      1
err_level_found_unexpected    1
err_level_halo_jump           1
err_level_missing_halo_ptl    0
err_level_cat_radius_diff     0
err_level_zero_ptl            1
err_level_zero_ptl_phantom    1
err_level_radius_outside_box  1
err_level_radius_not_found    1
err_level_radius_bound_diff   1
err_level_search_radius       1
err_level_exc_host_ended      1
err_level_exc_halo_not_found  1
err_level_max_exchange_size   1
err_level_large_halo          1

#--------------------------------------------------------------------------------------------------
# Operations
#--------------------------------------------------------------------------------------------------

# Instructions
last_snap                     -1
last_scale_factor             -1.0
# opt: doall_<tcr>_<res>      0/1
# opt: dohalo_<tcr>_<res>     <halo_id_1>, <halo_id_2> ...

# Domain & Exchange
lb_do_load_balancing          1
lb_adjust_min_snap            10
lb_slabs_adjust_factor        0.2
lb_slabs_adjust_max           0.01
lb_slabs_drift_tolerance      0.005
exc_max_message_size          500.0

# Halos
halo_min_radius_mass_profile  0.01
halo_max_radius_ratio_cat     10.0
halo_correct_phantoms         1
ghost_min_n200m               10
ghost_min_host_distance       0.05
ghost_max_host_center_time    0.5

# Other parameters
random_seed                   0
tree_points_per_leaf          100
potential_points_per_leaf     10
potential_err_tol             0.1
memory_allocation_factor      1.5
memory_dealloc_safety_factor  0.8
memory_max_halo_size          2000.0

#--------------------------------------------------------------------------------------------------
# Tracers
#--------------------------------------------------------------------------------------------------

# Tracer particles
tcr_ptl_create_radius         2.0
tcr_ptl_delete_radius         3.0
tcr_ptl_subtag_radius         2.0
tcr_ptl_subtag_inclusive      1
tcr_ptl_subtag_ifl_age_min    0.5
tcr_ptl_subtag_ifl_dist_min   2.0
tcr_ptl_subtag_bound_radius   0.5
tcr_ptl_subtag_bound_ratio    1.0
tcr_ptl_subtag_host_max_age   0.5

#--------------------------------------------------------------------------------------------------
# Analyses
#--------------------------------------------------------------------------------------------------

# Splashback radius (rsp)
anl_rsp_redshifts             -1
anl_rsp_defs                  <MUST BE SET BY USER>
anl_rsp_min_rrm               0.0
anl_rsp_max_rrm               1.0
anl_rsp_min_smr               -2.0
anl_rsp_max_smr               0.01
anl_rsp_demand_infall_rs      0
anl_rsp_sigma_tdyn            0.2
anl_rsp_min_weight            10.0
anl_rsp_n_bootstrap           200
anl_rsp_do_correction         1

# Halo Properties (hps)
anl_hps_redshifts             -1
anl_hps_defs                  <MUST BE SET BY USER>
anl_hps_r_max_so_host         1.2
anl_hps_r_max_so_sub          2.0
anl_hps_r_unbinding_host      1.0
anl_hps_r_unbinding_sub       1.0
anl_hps_iterative_unbinding   0