Skip to content

SM-Bench / Engineering intelligence

Designs generated by AI.
Judged by physics.

A benchmark for the engineering behind the answer. Turn a machine-generated design into measurable performance with deterministic simulations and versioned scores.

Open source · Reproducible evaluation · Traceable results

ENVIRONMENT / 01AVAILABLE
Shell-and-tube heat exchanger Conceptual cutaway showing parallel tubes inside a shell, transverse baffles, and separate hot and cold flow paths. Illustration only; not a simulated design. INOUTDESIGN GEOMETRYHX / SHELL & TUBE
Heat exchangerConceptual schematic / not to scale
Thermal dutyPressure dropGeometry
01 / ProposeOne prompt. One complete design.
02 / EvaluateEngineering checks before scoring.
03 / CompareSame task. Same versioned referee.

About the project

Can a language model design real engineering hardware?

Language models write fluent answers about engineering. Whether those answers hold up physically is another question. Suni Muhendis builds the tools to answer it: deterministic physics environments that take a design as JSON and judge it, and SM-Bench, a public benchmark that records how models perform against them.

Every environment works the same way. A model receives a design task, with the operating conditions and requirements, and answers with a complete design. The environment checks its structure, rejects impossible geometry, simulates the physics, and turns the result into a score between 0 and 1: zero for a design that fails, a gated score for one that meets the requirements, and more for one that meets them efficiently. The same referee, at the same pinned version, judges every model, so results are reproducible and comparable.

The project grows one engineering domain at a time. The heat exchanger is the first environment; turbomachinery and rotordynamics are next.

Open refereesEach environment is an Apache-2.0 package with its schema, design rules, simulator and score, released by tag.
Versioned resultsEvery record carries its environment, simulator and score version. Results from different versions are never pooled.
Our own modelsSM_heat_exchanger_v1, a fine-tuned 7B model, is benchmarked on tasks it never saw, next to frontier models.

The evaluation protocol

From a proposed design to an engineering result.

Every design follows the same four stages. The first failure stops evaluation and returns a structured status with a zero score.

01

Validate the schema

Check required fields, types, and the shape of the submitted JSON.

schema_error
02

Check the design

Reject impossible geometry before running the physics model.

drc_error
03

Simulate the physics

Measure performance using the environment's versioned simulator.

simulation_error
04

Score the result

Convert engineering metrics into a normalized benchmark score.

score ∈ [0, 1]

Evaluation is deterministic within the pinned runtime and dependency profile. Every record retains its environment, simulator version, and score version. Results from different versions are never pooled.

Available environment

Heat transfer.
Real design constraints.

Evaluate shell-and-tube and concentric-tube heat exchangers against thermal, hydraulic, mechanical, and cost requirements. The environment uses ht and fluids alongside domain-specific checks.

Explore the environment →

Environment
heat_exchanger
Simulator
V4
Geometries
Shell-and-tube / Concentric-tube
Evaluation
Versioned scores & task-owned conditions

Roadmap

More engineering domains are on the way.

Each new environment follows the same contract as the heat exchanger and joins SM-Bench once it is validated and released.

Available

Heat exchangers

Shell-and-tube and concentric-tube exchangers against thermal, hydraulic, mechanical and cost requirements.

Coming soon

Turbomachinery

Axial compressor design: blade rows and flow path judged by a throughflow model, starting with compressors and followed by turbines.

Coming soon

Rotordynamics

Shafts, bearings and discs: critical speeds, stability and vibration response of rotating machinery.

Planned

3D turbomachinery flow

Full three-dimensional blade designs evaluated with computational fluid dynamics.

Later

Aerospace and beyond

Further domains where a design can be judged by physics alone.

Build on the benchmark

Explore. Evaluate. Reproduce.

  • 01 / RESULTS

    See how models perform

    Compare single-attempt designs, task by task. Inspect scores, requirement compliance, cost, and where attempts stop.

    Open leaderboard →

  • 02 / ENVIRONMENTS

    Evaluate your own design

    Install a released environment, submit a JSON design, and inspect the engineering metrics behind its score.

    Read the environment guide →

  • 03 / BENCHMARKING

    Run a reproducible experiment

    Send a task to models, preserve every run with its provenance and cost, and explore the results in the dashboard.

    Run a benchmark →

A few lines to your first evaluation

Quick start

Python 3.12 is required. Install from a release tag so simulator and scoring behavior cannot move underneath an experiment:

pip install "sm-heat-exchanger @ git+https://github.com/suni-muhendis/sm-heat-exchanger.git@v0.8.0"

The environment brings sm-core with it. Up to envs-v0.7.0 both shipped from SM-Bench as the sunimuhendis package; those tags stay installable.

from sm_core import make_env

env = make_env("heat_exchanger", score_version="heat_exchanger_score_v4")

task = {
    "task_id": "example",
    "score_version": "heat_exchanger_score_v4",
    "target_heat_duty": 250_000.0,
    "max_dp_tube": 5_000.0,
    "max_dp_shell": 5_000.0,
}

design = {
    "geometry_type": "shell_and_tube",
    "length": 3.0,
    "inner_tube_di": 0.016,
    "inner_tube_do": 0.020,
    "outer_shell_di": 0.30,
    "number_of_tubes": 50,
    "baffle_spacing": 0.2,
}

result = env.evaluate("example", task, "design-1", design)
print(result.status, result.score.normalized_total)

Invalid model output is data, not an exception: evaluate() returns schema_error, drc_error, or simulation_error with a score of 0.0, and a successful simulation returns a score in [0.0, 1.0] with the engineering metrics behind it.

Team

The people behind Suni Muhendis.

Mehmet Fatih Reyhan

Co-founder

GitHub ↗

Ertuğrul Şahin

Co-founder

GitHub ↗

License and citation

Source code is licensed under the Apache License 2.0. Benchmark data and project-authored reports are covered by CC BY 4.0; third-party model responses are excluded from that grant. Use CITATION.cff when citing the software.

SM-Bench is part of Suni Muhendis. sm-core provides the shared evaluation contract; engineering environments are released as separate packages.