Construction of Simulation Verification Metrics and Measurement Criteria in Crowd Evacuation

Description
In crowd evacuation simulation, verification is a critical step to confirm whether the model implementation correctly reflects the conceptual model. This requires the design of scientific measurement metrics to quantitatively compare the differences between simulation outputs and real-world behaviors (or theoretical expectations), thereby assessing the model's accuracy and reliability. Constructing a systematic framework of verification metrics and measurement criteria is the foundation for ensuring model credibility.

Problem-Solving Process

1. Clarify Verification Objectives

   • Verification targets the correctness of the model "implementation," for example, checking whether the code accurately implements the force formulas in a social force model or whether the path selection algorithm executes according to the preset logic.
   • Distinguish between "Verification" and "Validation": Verification focuses on "whether the model is built correctly," while Validation focuses on "whether the model is suitable for the real world." Here, the focus is on the quantitative metrics needed for verification.

2. Identify Levels of Verification Scenarios

   • Microscopic Level: Individual behavior metrics, such as speed-density relationships, whether acceleration conforms to Newton's laws, and whether obstacle avoidance trajectories are smooth.
   • Mesoscopic Level: Local group metrics, such as flow rate-density relationships, and the dynamic formation and dissipation of congestion at bottlenecks.
   • Macroscopic Level: Overall system metrics, such as total evacuation time, exit utilization rates, and the evolution of crowd distribution.

3. Select or Design Measurement Criteria

   • Direct Comparison Method: If theoretical solutions or controlled experimental data exist, error metrics can be directly defined:
     • Root Mean Square Error (RMSE): Used for continuous variables (e.g., changes in individual position over time).
     • Mean Absolute Percentage Error (MAPE): Used for metrics sensitive to relative errors (e.g., comparison of evacuation times).
   • Statistical Analysis Tests:
     • Kolmogorov-Smirnov Test: Compares the distribution of simulated and actual data (e.g., distribution of exit passage times).
     • Correlation Analysis: Calculates the Pearson correlation coefficient between simulated and reference data (e.g., correlation of spatiotemporal changes in pedestrian flow density).
   • Spatiotemporal Consistency Metrics:
     • Spatiotemporal Heatmap Differences: Convert simulation and experimental video data into grid density heatmaps and calculate the norm of frame-by-frame differences (e.g., Frobenius norm).
     • Trajectory Similarity: Use Dynamic Time Warping (DTW) distance to compare the shape of individual trajectories.

4. Construct a Verification Metric System

   • Integrate metrics from different levels and types into a multi-level verification framework:
     • Basic Physical Verification: Check for mass conservation (constant number of agents) and whether there are abnormal energy spikes (e.g., total kinetic energy in a social force model).
     • Behavioral Rule Verification: Verify decision-making logic through unit tests, for example, "When Exit A is congested, does the agent switch to Exit B according to the preset probability?"
     • Emergent Phenomenon Verification: Compare whether expected self-organizing phenomena appear in the simulation (e.g., lane formation, oscillatory flow), which can be quantified using order parameters (e.g., lane orderliness).

5. Set Verification Thresholds and Confidence Levels

   • Define acceptable error ranges based on practical application requirements, for example:
     • Micro-level trajectory error threshold (e.g., RMSE < 0.5 meters).
     • Macro-level evacuation time error threshold (e.g., MAPE < 10%).
   • Assess result stability by conducting multiple random repeated simulations and calculating confidence intervals for the metrics.

6. Example of an Automated Verification Process

   • Input: Simulation output data, reference data (experimental or theoretical values).
   • Steps:
     1. Data alignment (time synchronization, spatial coordinate matching).
     2. Calculate metrics layer by layer: Microscopic → Mesoscopic → Macroscopic.
     3. Compare with thresholds and generate a verification report (Pass/Fail).
   • Tool Example: Write Python scripts to automatically calculate RMSE, DTW distance, and plot error distribution charts.

7. Handling Special Cases

   • When Experimental Data is Unavailable: Use theoretical models (e.g., fluid dynamics analogies) to generate benchmark solutions, or verify internal consistency through sensitivity analysis (e.g., whether trends are reasonable when parameters change).
   • Impact of Randomness: Employ statistical hypothesis tests to compare whether the distribution differences between simulated and reference data are significant (e.g., t-test, Mann-Whitney U test).

Summary
Constructing verification metrics requires combining model levels and objectives, progressively checking from mathematical consistency to behavioral logic. The design of quantitative measurement criteria should balance computational cost and precision and be embedded into automated processes to improve reproducibility. Ultimately, through the output of a systematic metric system, the reliability boundaries of the model implementation are clarified, laying the foundation for subsequent validation and application.