Skip to content

Instantly share code, notes, and snippets.

@pra-dan

pra-dan/compare_onnx_files.py

Created November 11, 2025 11:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save pra-dan/b8f1ca8eac281e92a0761b6853b0ba17 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save pra-dan/b8f1ca8eac281e92a0761b6853b0ba17 to your computer and use it in GitHub Desktop.
Download ZIP
Compare multiple onnx models's output similarity. Uses polygraphy version `0.49.18`
Raw
compare_onnx_files.py
from polygraphy.backend.onnxrt import OnnxrtRunner, SessionFromOnnx
from polygraphy.backend.trt import TrtRunner, EngineFromNetwork, NetworkFromOnnxPath
from polygraphy.comparator import Comparator, DataLoader
model_path_1 = "yes_dynamo.onnx"
model_path_2 = "no_dynamo.onnx"
build_onnxrt_session_1 = SessionFromOnnx(model_path_1)
build_onnxrt_session_2 = SessionFromOnnx(model_path_2)
# build_engine = EngineFromNetwork(NetworkFromOnnxPath(model_path))
runners = [
OnnxrtRunner(build_onnxrt_session_1),
# TrtRunner(build_engine),
OnnxrtRunner(build_onnxrt_session_2),
]
data_loader = DataLoader(val_range=(0, 2))
run_results = Comparator.run(runners, data_loader=data_loader)
assert bool(Comparator.compare_accuracy(run_results))
@pra-dan
Copy link
Author

pra-dan commented Nov 11, 2025

The output looks as follows

[I] onnxrt-runner-N0-11/11/25-10:58:16  | Activating and starting inference
[I] Creating ONNX-Runtime Inference Session with providers: ['CPUExecutionProvider']
2025-11-11 10:58:16.427434618 [W:onnxruntime:, graph.cc:109 MergeShapeInfo] Error merging shape info for output. 'linear_29' source:{1200,1} target:{1200}. Falling back to lenient merge.
2025-11-11 10:58:16.427885616 [W:onnxruntime:, graph.cc:109 MergeShapeInfo] Error merging shape info for output. 'linear_44' source:{1200,1} target:{1200}. Falling back to lenient merge.
2025-11-11 10:58:16.428251933 [W:onnxruntime:, graph.cc:109 MergeShapeInfo] Error merging shape info for output. 'linear_59' source:{1200,1} target:{1200}. Falling back to lenient merge.
2025-11-11 10:58:16.428612818 [W:onnxruntime:, graph.cc:109 MergeShapeInfo] Error merging shape info for output. 'linear_74' source:{1200,1} target:{1200}. Falling back to lenient merge.
2025-11-11 10:58:16.428986724 [W:onnxruntime:, graph.cc:109 MergeShapeInfo] Error merging shape info for output. 'linear_89' source:{1200,1} target:{1200}. Falling back to lenient merge.
2025-11-11 10:58:16.429352440 [W:onnxruntime:, graph.cc:109 MergeShapeInfo] Error merging shape info for output. 'linear_104' source:{1200,1} target:{1200}. Falling back to lenient merge.
[I] onnxrt-runner-N0-11/11/25-10:58:16 
    ---- Inference Input(s) ----
    {input [dtype=float32, shape=(1, 3, 1280, 1280)]}
[I] onnxrt-runner-N0-11/11/25-10:58:16 
    ---- Inference Output(s) ----
    {logits [dtype=float32, shape=(1, 300, 1)],
     boxes [dtype=float32, shape=(1, 300, 4)]}
[I] onnxrt-runner-N0-11/11/25-10:58:16  | Completed 1 iteration(s) in 727.7 ms | Average inference time: 727.7 ms.
[I] onnxrt-runner-N1-11/11/25-10:58:16  | Activating and starting inference
[I] Creating ONNX-Runtime Inference Session with providers: ['CPUExecutionProvider']
[I] onnxrt-runner-N1-11/11/25-10:58:16 
    ---- Inference Input(s) ----
    {input [dtype=float32, shape=(1, 3, 1280, 1280)]}
[I] onnxrt-runner-N1-11/11/25-10:58:16 
    ---- Inference Output(s) ----
    {logits [dtype=float32, shape=(1, 300, 1)],
     boxes [dtype=float32, shape=(1, 300, 4)]}
[I] onnxrt-runner-N1-11/11/25-10:58:16  | Completed 1 iteration(s) in 755.9 ms | Average inference time: 755.9 ms.
[I] Accuracy Comparison | onnxrt-runner-N0-11/11/25-10:58:16 vs. onnxrt-runner-N1-11/11/25-10:58:16
[I]     Comparing Output: 'logits' (dtype=float32, shape=(1, 300, 1)) with 'logits' (dtype=float32, shape=(1, 300, 1))
[I]         Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I]         onnxrt-runner-N0-11/11/25-10:58:16: logits | Stats: mean=-5.5064, std-dev=1.544, var=2.3838, median=-6.4438, min=-6.9135 at (0, 111, 0), max=-0.89488 at (0, 9, 0), avg-magnitude=5.5064, p90=-3.2779, p95=-3.1437, p99=-2.8954
[I]             ---- Histogram ----
                Bin Range        |  Num Elems | Visualization
                (-6.91 , -6.31 ) |        158 | ########################################
                (-6.31 , -5.71 ) |         10 | ##
                (-5.71 , -5.11 ) |         33 | ########
                (-5.11 , -4.51 ) |          2 | 
                (-4.51 , -3.9  ) |          3 | 
                (-3.9  , -3.3  ) |         60 | ###############
                (-3.3  , -2.7  ) |         33 | ########
                (-2.7  , -2.1  ) |          0 | 
                (-2.1  , -1.5  ) |          0 | 
                (-1.5  , -0.895) |          1 | 
[I]         onnxrt-runner-N1-11/11/25-10:58:16: logits | Stats: mean=-3.063, std-dev=0.90728, var=0.82315, median=-2.9963, min=-6.8627 at (0, 87, 0), max=-1.4198 at (0, 9, 0), avg-magnitude=3.063, p90=-2.1736, p95=-1.9756, p99=-1.5896
[I]             ---- Histogram ----
                Bin Range        |  Num Elems | Visualization
                (-6.91 , -6.31 ) |          6 | ##
                (-6.31 , -5.71 ) |          2 | 
                (-5.71 , -5.11 ) |          5 | ##
                (-5.11 , -4.51 ) |          3 | #
                (-4.51 , -3.9  ) |          2 | 
                (-3.9  , -3.3  ) |         94 | ########################################
                (-3.3  , -2.7  ) |         71 | ##############################
                (-2.7  , -2.1  ) |         91 | ######################################
                (-2.1  , -1.5  ) |         24 | ##########
                (-1.5  , -0.895) |          2 | 
[I]         Error Metrics: logits
[I]             Minimum Required Tolerance: elemwise error | [abs=5.161] OR [rel=2.9845] (requirements may be lower if both abs/rel tolerances are set)
[I]             Absolute Difference | Stats: mean=2.5993, std-dev=1.4534, var=2.1124, median=3.1043, min=0.0069666 at (0, 0, 0), max=5.161 at (0, 1, 0), avg-magnitude=2.5993, p90=4.3677, p95=4.4929, p99=4.7298
[I]                 ---- Histogram ----
                    Bin Range        |  Num Elems | Visualization
                    (0.00697, 0.522) |         36 | ######################
                    (0.522  , 1.04 ) |         33 | ####################
                    (1.04   , 1.55 ) |         25 | ###############
                    (1.55   , 2.07 ) |         19 | ############
                    (2.07   , 2.58 ) |         13 | ########
                    (2.58   , 3.1  ) |         23 | ##############
                    (3.1    , 3.61 ) |         63 | ########################################
                    (3.61   , 4.13 ) |         41 | ##########################
                    (4.13   , 4.65 ) |         41 | ##########################
                    (4.65   , 5.16 ) |          6 | ###
[I]             Relative Difference | Stats: mean=0.93397, std-dev=0.60868, var=0.3705, median=0.87517, min=0.0012719 at (0, 0, 0), max=2.9845 at (0, 1, 0), avg-magnitude=0.93397, p90=1.8045, p95=1.9245, p99=2.2035
[I]                 ---- Histogram ----
                    Bin Range        |  Num Elems | Visualization
                    (0.00127, 0.3  ) |         54 | ########################################
                    (0.3    , 0.598) |         46 | ##################################
                    (0.598  , 0.896) |         54 | ########################################
                    (0.896  , 1.19 ) |         50 | #####################################
                    (1.19   , 1.49 ) |         30 | ######################
                    (1.49   , 1.79 ) |         31 | ######################
                    (1.79   , 2.09 ) |         28 | ####################
                    (2.09   , 2.39 ) |          4 | ##
                    (2.39   , 2.69 ) |          1 | 
                    (2.69   , 2.98 ) |          2 | #
[E]         FAILED | Output: 'logits' | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[I]     Comparing Output: 'boxes' (dtype=float32, shape=(1, 300, 4)) with 'boxes' (dtype=float32, shape=(1, 300, 4))
[I]         Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I]         onnxrt-runner-N0-11/11/25-10:58:16: boxes | Stats: mean=0.31276, std-dev=0.34833, var=0.12133, median=0.15757, min=0.0031904 at (0, 161, 0), max=0.99672 at (0, 75, 0), avg-magnitude=0.31276, p90=0.9773, p95=0.99436, p99=0.99589
[I]             ---- Histogram ----
                Bin Range        |  Num Elems | Visualization
                (0.00319, 0.103) |        537 | ########################################
                (0.103  , 0.202) |        112 | ########
                (0.202  , 0.302) |         69 | #####
                (0.302  , 0.401) |         90 | ######
                (0.401  , 0.501) |         87 | ######
                (0.501  , 0.6  ) |         38 | ##
                (0.6    , 0.7  ) |         40 | ##
                (0.7    , 0.799) |         38 | ##
                (0.799  , 0.899) |         23 | #
                (0.899  , 0.998) |        166 | ############
[I]         onnxrt-runner-N1-11/11/25-10:58:16: boxes | Stats: mean=0.27635, std-dev=0.32626, var=0.10644, median=0.062345, min=0.0031928 at (0, 9, 2), max=0.99824 at (0, 9, 0), avg-magnitude=0.27635, p90=0.82677, p95=0.8816, p99=0.94035
[I]             ---- Histogram ----
                Bin Range        |  Num Elems | Visualization
                (0.00319, 0.103) |        621 | ########################################
                (0.103  , 0.202) |         68 | ####
                (0.202  , 0.302) |         59 | ###
                (0.302  , 0.401) |         61 | ###
                (0.401  , 0.501) |         65 | ####
                (0.501  , 0.6  ) |         47 | ###
                (0.6    , 0.7  ) |         53 | ###
                (0.7    , 0.799) |         81 | #####
                (0.799  , 0.899) |        108 | ######
                (0.899  , 0.998) |         37 | ##
[I]         Error Metrics: boxes
[I]             Minimum Required Tolerance: elemwise error | [abs=0.98558] OR [rel=48.486] (requirements may be lower if both abs/rel tolerances are set)
[I]             Absolute Difference | Stats: mean=0.21671, std-dev=0.24093, var=0.058048, median=0.13272, min=2.0742e-05 at (0, 44, 2), max=0.98558 at (0, 140, 0), avg-magnitude=0.21671, p90=0.59676, p95=0.74725, p99=0.88592
[I]                 ---- Histogram ----
                    Bin Range          |  Num Elems | Visualization
                    (2.07e-05, 0.0986) |        537 | ########################################
                    (0.0986  , 0.197 ) |        193 | ##############
                    (0.197   , 0.296 ) |         97 | #######
                    (0.296   , 0.394 ) |        117 | ########
                    (0.394   , 0.493 ) |         81 | ######
                    (0.493   , 0.591 ) |         54 | ####
                    (0.591   , 0.69  ) |         46 | ###
                    (0.69    , 0.788 ) |         29 | ##
                    (0.788   , 0.887 ) |         34 | ##
                    (0.887   , 0.986 ) |         12 | 
[I]             Relative Difference | Stats: mean=4.3154, std-dev=8.7106, var=75.874, median=0.83912, min=0.0020854 at (0, 251, 0), max=48.486 at (0, 196, 3), avg-magnitude=4.3154, p90=16.048, p95=27.93, p99=38.573
[I]                 ---- Histogram ----
                    Bin Range       |  Num Elems | Visualization
                    (0.00209, 4.85) |        988 | ########################################
                    (4.85   , 9.7 ) |         60 | ##
                    (9.7    , 14.5) |         23 | 
                    (14.5   , 19.4) |         20 | 
                    (19.4   , 24.2) |         32 | #
                    (24.2   , 29.1) |         25 | #
                    (29.1   , 33.9) |         26 | #
                    (33.9   , 38.8) |         14 | 
                    (38.8   , 43.6) |          5 | 
                    (43.6   , 48.5) |          7 | 
[E]         FAILED | Output: 'boxes' | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[E]     FAILED | Mismatched outputs: ['logits', 'boxes']
[E] Accuracy Summary | onnxrt-runner-N0-11/11/25-10:58:16 vs. onnxrt-runner-N1-11/11/25-10:58:16 | Passed: 0/1 iterations | Pass Rate: 0.0%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment