SM-Bench / Engineering intelligence
Designs generated by AI.
Judged by physics.¶
A benchmark for the engineering behind the answer. Turn a machine-generated design into measurable performance with deterministic simulations and versioned scores.
Open source · Reproducible evaluation · Traceable results
About the project
Can a language model design real engineering hardware?¶
Language models write fluent answers about engineering. Whether those answers hold up physically is another question. Suni Muhendis builds the tools to answer it: deterministic physics environments that take a design as JSON and judge it, and SM-Bench, a public benchmark that records how models perform against them.
Every environment works the same way. A model receives a design task, with the operating conditions and requirements, and answers with a complete design. The environment checks its structure, rejects impossible geometry, simulates the physics, and turns the result into a score between 0 and 1: zero for a design that fails, a gated score for one that meets the requirements, and more for one that meets them efficiently. The same referee, at the same pinned version, judges every model, so results are reproducible and comparable.
The project grows one engineering domain at a time. The heat exchanger is the first environment; turbomachinery and rotordynamics are next.
The evaluation protocol
From a proposed design to an engineering result.¶
Every design follows the same four stages. The first failure stops evaluation and returns a structured status with a zero score.
Validate the schema
Check required fields, types, and the shape of the submitted JSON.
schema_errorCheck the design
Reject impossible geometry before running the physics model.
drc_errorSimulate the physics
Measure performance using the environment's versioned simulator.
simulation_errorScore the result
Convert engineering metrics into a normalized benchmark score.
score ∈ [0, 1]Evaluation is deterministic within the pinned runtime and dependency profile. Every record retains its environment, simulator version, and score version. Results from different versions are never pooled.
Available environment
Heat transfer.
Real design constraints.¶
Evaluate shell-and-tube and concentric-tube heat exchangers against thermal,
hydraulic, mechanical, and cost requirements. The environment uses ht and
fluids alongside domain-specific checks.
- Environment
- heat_exchanger
- Simulator
- V4
- Geometries
- Shell-and-tube / Concentric-tube
- Evaluation
- Versioned scores & task-owned conditions
Roadmap
More engineering domains are on the way.¶
Each new environment follows the same contract as the heat exchanger and joins SM-Bench once it is validated and released.
Heat exchangers
Shell-and-tube and concentric-tube exchangers against thermal, hydraulic, mechanical and cost requirements.
Turbomachinery
Axial compressor design: blade rows and flow path judged by a throughflow model, starting with compressors and followed by turbines.
Rotordynamics
Shafts, bearings and discs: critical speeds, stability and vibration response of rotating machinery.
3D turbomachinery flow
Full three-dimensional blade designs evaluated with computational fluid dynamics.
Aerospace and beyond
Further domains where a design can be judged by physics alone.
Build on the benchmark
Explore. Evaluate. Reproduce.¶
-
01 / RESULTS
See how models perform¶
Compare single-attempt designs, task by task. Inspect scores, requirement compliance, cost, and where attempts stop.
-
02 / ENVIRONMENTS
Evaluate your own design¶
Install a released environment, submit a JSON design, and inspect the engineering metrics behind its score.
-
03 / BENCHMARKING
Run a reproducible experiment¶
Send a task to models, preserve every run with its provenance and cost, and explore the results in the dashboard.
A few lines to your first evaluation
Quick start¶
Python 3.12 is required. Install from a release tag so simulator and scoring behavior cannot move underneath an experiment:
pip install "sm-heat-exchanger @ git+https://github.com/suni-muhendis/sm-heat-exchanger.git@v0.8.0"
The environment brings sm-core with it. Up to envs-v0.7.0 both shipped
from SM-Bench as the sunimuhendis package; those tags stay installable.
from sm_core import make_env
env = make_env("heat_exchanger", score_version="heat_exchanger_score_v4")
task = {
"task_id": "example",
"score_version": "heat_exchanger_score_v4",
"target_heat_duty": 250_000.0,
"max_dp_tube": 5_000.0,
"max_dp_shell": 5_000.0,
}
design = {
"geometry_type": "shell_and_tube",
"length": 3.0,
"inner_tube_di": 0.016,
"inner_tube_do": 0.020,
"outer_shell_di": 0.30,
"number_of_tubes": 50,
"baffle_spacing": 0.2,
}
result = env.evaluate("example", task, "design-1", design)
print(result.status, result.score.normalized_total)
Invalid model output is data, not an exception: evaluate() returns
schema_error, drc_error, or simulation_error with a score of 0.0, and a
successful simulation returns a score in [0.0, 1.0] with the engineering
metrics behind it.
Team
The people behind Suni Muhendis.¶
License and citation¶
Source code is licensed under the Apache License 2.0. Benchmark data and project-authored reports are covered by CC BY 4.0; third-party model responses are excluded from that grant. Use CITATION.cff when citing the software.
SM-Bench is part of Suni Muhendis. sm-core provides the shared evaluation contract; engineering environments are released as separate packages.