Environments¶
SM-Bench evaluates designs with environments that are packages of their own, built on a shared evaluation contract:
| Package | Provides |
|---|---|
| sm-core | The evaluation contract, make_env, task auditing, parse_llm_json |
| sm-heat-exchanger | The heat_exchanger environment |
An environment registers itself with sm-core when it is installed, and is
then built by name:
pip install "sm-heat-exchanger @ git+https://github.com/suni-muhendis/sm-heat-exchanger.git@v0.8.0"
from sm_core import list_environments, make_env
list_environments() # ['heat_exchanger']
env = make_env("heat_exchanger", score_version="heat_exchanger_score_v4")
Factories import simulator dependencies lazily, so discovering environments costs nothing. Always install from a tag: simulator and score behaviour is versioned deliberately.
Evaluation result¶
evaluate(task_id, task_params, design_id, design_params) applies four stages
in order:
- schema validation;
- design-rule checks;
- simulation;
- benchmark scoring.
Invalid designs return an EvaluationResult with a zero score and a stage
status; they do not raise into the caller's optimisation or training loop.
Results expose:
statusanderror_message;score.normalized_totaland per-objective components;- engineering
metrics; raw_simulation_output, including warnings and fidelity notes;environment,simulator_versionandscore_version.
Configuration errors made by the experiment author, such as an unknown operating-condition key, raise instead of being attributed to the design.
Heat-exchanger tasks¶
Operating conditions belong to the task, not the design. A task may provide an
operating_conditions block and optional secondary_operating_points.
Secondary points evaluate the same geometry at additional conditions and retain
their metrics under raw_simulation_output["secondary_points"].
The design schema contains the geometric decisions available to the model.
Task-owned flow rates, temperatures, fouling assumptions, material limits, and
targets cannot be overridden by a submitted design. SM-Bench tasks name their
environment with an "environment" key; tasks without one are heat-exchanger
tasks.
Auditing a task¶
A badly calibrated task can have an unreachable target, an unavoidable warning, or a score budget dominated by merely reaching feasibility. Audit a task before spending model calls:
report = env.audit_task(task, num_samples=20_000)
print(report.summary())
print(report.is_healthy())
print(report.score_ceiling)
print(report.forced_warnings)
Sampling can demonstrate that a score is reachable; it cannot prove that a
higher score is impossible. Tasks can therefore include reference_designs as
explicit feasibility witnesses.
The audit algorithm is environment independent. Each environment supplies broad sampling, hard requirements, declared checks, and any closed-form physical limits it can derive.
Versioning¶
Simulator and score versions are independent. Every result stores both, and results are never pooled across either boundary. Score V4 is the recommended heat-exchanger score for new tasks; older scores remain available only to reproduce historical experiments.
The physics audit that motivated task auditing is kept with the environment: simulator V3 report.