XPO - Updated performance benchmarks (v18.2)

Based on user feedback, we’ve updated our benchmarks against EF 6 and EF Core:

    https://github.com/DevExpress/XpoNetCoreDemos/tree/master/ORMBenchmark

What's New in the benchmark?

Run the updated benchmark tests or review our results here. Example: 

All benchmarks were executed using .NET 4.7.2, AnyCPU release builds (include warm-up), Windows 10 Enterprise x64, local Microsoft SQL Server 2016 Developer Edition, i7-7700 CPU @3.6GHz / 16GB RAM / SSD.

We also contributed to the Dapper.Tests.Performance benchmarks, so you can compare XPO with other micro and macro ORMs. See the results below:

It is an expected result that requests to macro-ORM take more time than requests to micro-ORM or direct SQL queries.

Interesting Support Center tickets

  • We fixed an issue with the "The assembly does not have persistent classes" error in the ORM Data Model Designer (T680676);
  • We described how to create a simple data layer to modify data using XPO and a stored procedure in the same transaction (T690250);
  • We updated the K18356: How to use XPO data layer caching in XAF KB article;
  • We still receive questions on Mono specifics from Xamarin developers, for instance, about using the System.Data.SqlClient and System.Drawing.Common assemblies in XPO apps (T661502 | T643780).
Find more interesting customer questions and product enhancements for your favorite products in the DevExpress Support Center (expand the "Frameworks (XAF & XPO)" group on the right) and What's New documentation (see resolved issues). 

Your feedback counts!

What do you think about the benchmarks results? Let us know in comments. We'd love to hear your feedback.


Love XPO and want to help us promote it? Add the package through Nuget.org instead of DevExpress Nuget!

9 comment(s)
Steve Sharkey

Early on I thought I'd give XPO a try, I was really surprised that there was so little performance hit compared with more traditional Direct SQL code in my own benchmarking. So I made the decision to use XPO. Since then I have found that it has made such a difference in being able to modify & maintain code. Because, on the surface, it has changed quite slowly it is possible to really know it inside out without being is such a constant churn of learning every nuance of the latest release as is the norm with technology. I am still a massive fan.

To be honest I could have easily been a winforms subscriber but went universal in my subscription because XPO was included with the web side of things (plus I have really wanted to get on with XAF - but never really got over some of the hurdles and limitations). The fact that it is now freely available is the deal of the century (and has undermined my "need" to renew).

Added to this the recent improvements to performance means code built on XPO gets a free upgrade by recompiling has great implications for our customers. Keep up the good work.

22 November, 2018
Brad Baker - MMC

Please post your actual result files.  When I run the test, XPO never come close to EF.

I created another provider and changed the name of the original, to see the difference between .Add in a loop, and .AddRange

BenchmarkDotNet=v0.11.3, OS=Windows 10.0.17134.407 (1803/April2018Update/Redstone4)

Intel Core i7-8700K CPU 3.70GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores

 [Host] : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

 Clr    : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

Job=Clr  MaxRelativeError=0.1  MinInvokeCount=1  

Runtime=Clr  InvocationCount=1  LaunchCount=1  

RunStrategy=Throughput  UnrollFactor=1  WarmupCount=1  

    Method | RowCount |     TestProvider |        Mean |     Error |    StdDev |      Median |

----------- |--------- |----------------- |------------:|----------:|----------:|------------:|

InsertMany |     2500 | DE EF Core 2.1.4 |    611.6 ms |  31.14 ms |  29.12 ms |    615.5 ms |   (This is .Add in loop)

InsertMany |     2500 |       Direct SQL |    698.2 ms |  69.07 ms | 153.06 ms |    628.3 ms |

InsertMany |     2500 |    EF Core 2.1.4 |    594.6 ms |  28.13 ms |  21.96 ms |    594.7 ms |

(This is Add To List in Loop, .AddRange to befor save.)

InsertMany |     2500 |       XPO 18.2.3 | 28,474.2 ms | 662.45 ms | 587.25 ms | 28,387.4 ms |

22 November, 2018
Brad Baker - MMC

BenchmarkDotNet=v0.11.3, OS=Windows 10.0.17134.407 (1803/April2018Update/Redstone4)

Intel Core i7-8700K CPU 3.70GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores

 [Host] : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

 Clr    : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

Job=Clr  MaxRelativeError=0.1  MinInvokeCount=1  

Runtime=Clr  InvocationCount=1  LaunchCount=1  

RunStrategy=Throughput  UnrollFactor=1  WarmupCount=1  

    Method | RowCount |     TestProvider |         Mean |      Error |     StdDev |

----------- |--------- |----------------- |-------------:|-----------:|-----------:|

InsertMany |       10 | DE EF Core 2.1.4 |     99.92 ms |   4.449 ms |   3.715 ms |

InsertMany |      100 | DE EF Core 2.1.4 |    110.15 ms |   2.653 ms |   2.215 ms |

InsertMany |      250 | DE EF Core 2.1.4 |    139.62 ms |  17.969 ms |  25.770 ms |

InsertMany |     1000 | DE EF Core 2.1.4 |    262.81 ms |  12.778 ms |   9.976 ms |

InsertMany |     2500 | DE EF Core 2.1.4 |    613.81 ms |  60.233 ms |  50.297 ms |

InsertMany |       10 |       Direct SQL |     98.79 ms |   3.625 ms |   3.027 ms |

InsertMany |      100 |       Direct SQL |    113.27 ms |   8.641 ms |   7.660 ms |

InsertMany |      250 |       Direct SQL |    124.99 ms |  11.887 ms |  12.207 ms |

InsertMany |     1000 |       Direct SQL |    292.54 ms |  29.230 ms |  44.637 ms |

InsertMany |     2500 |       Direct SQL |    649.93 ms |  64.467 ms | 117.882 ms |

InsertMany |       10 |    EF Core 2.1.4 |     97.80 ms |   2.270 ms |   1.772 ms |

InsertMany |      100 |    EF Core 2.1.4 |    111.13 ms |   3.072 ms |   2.874 ms |

InsertMany |      250 |    EF Core 2.1.4 |    128.23 ms |  23.517 ms |  24.150 ms |

InsertMany |     1000 |    EF Core 2.1.4 |    262.35 ms |   9.808 ms |   9.174 ms |

InsertMany |     2500 |    EF Core 2.1.4 |    600.01 ms |  80.961 ms |  83.141 ms |

InsertMany |       10 |       XPO 18.2.3 |    196.88 ms |   4.742 ms |   4.436 ms |

InsertMany |      100 |       XPO 18.2.3 |  1,218.71 ms |  29.284 ms |  27.393 ms |

InsertMany |      250 |       XPO 18.2.3 |  3,144.28 ms | 248.629 ms | 232.567 ms |

InsertMany |     1000 |       XPO 18.2.3 | 12,314.92 ms | 536.985 ms | 502.296 ms |

InsertMany |     2500 |       XPO 18.2.3 | 29,409.84 ms | 722.762 ms | 603.539 ms |

22 November, 2018
Uriah (DevExpress Support)

Hello Brad, 

Thank you for your feedback. The results we published were produced by the actual code without any edits. Nevertheless, test results will definitely be different in different environments. This is the reason why we provide open access to the source code. Everyone has the opportunity to run tests in their environment. 

We thoroughly analyzed your results. To be honest, we do not have any clue about the reason for such a big difference. However, we noticed that your InsertMany test result for Direct SQL (as well as for all ORMs) are much higher than ours. This indicates that the database server performance is low. 98.79 ms is too long to insert only 10 entries.  

We would love to reproduce the same result on our side to check if we can do anything about the performance loss in such environments. We would greatly appreciate it if you clarify the following points. 

1. Did you change the connection string in the app.config file? If yes, the following details will be helpful:

  - Wat is your SQL server version?

  - Where is the SQL server running (locally, in a local network, or on the Internet)?

  - What is the round-trip time (ping) to the SQL server?

2. If you did not change the connection string, the application must be using MS SQL LocalDB. Please clarify the local SQL Server version, and whether database files are located in HDD or SSD.

3. Did you have any antiviral or other software that could track and analyze the network traffic and IO operations when you were executing benchmarks?

4. Did you run benchmarks in Visual Studio or compile and run the EXE file? 

Thank you in advance for any feedback that can help us narrow down the research area.

23 November, 2018
Brad Baker - MMC

Its going against a 2008R2 DB across a VPN so it was latency.  Seems XPO doesnt like latency.  XPO still showing double time compared to EF.

Heres the test again when run from same network:

My code is here, your code doesnt actually compile.  I use Jetbrains Rider compiled to Release.

github.com/.../XPOPerf

// * Summary *

BenchmarkDotNet=v0.11.3, OS=Windows 10.0.17134.376 (1803/April2018Update/Redstone4)

Intel Core i7-4610M CPU 3.00GHz (Haswell), 1 CPU, 4 logical and 2 physical cores

 [Host] : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

 Clr    : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

Job=Clr  MaxRelativeError=0.1  MinInvokeCount=1  

Runtime=Clr  InvocationCount=1  LaunchCount=1  

RunStrategy=Throughput  UnrollFactor=1  WarmupCount=1  

    Method | RowCount |     TestProvider |          Mean |       Error |      StdDev |        Median |

----------- |--------- |----------------- |--------------:|------------:|------------:|--------------:|

InsertMany |       50 | DE EF Core 2.1.4 |      5.647 ms |   0.5602 ms |   0.9359 ms |      5.545 ms |

InsertMany |      200 | DE EF Core 2.1.4 |     17.840 ms |   1.7370 ms |   2.3188 ms |     18.041 ms |

InsertMany |     1000 | DE EF Core 2.1.4 |    174.073 ms |   6.2564 ms |   5.5461 ms |    173.383 ms |

InsertMany |     2500 | DE EF Core 2.1.4 |    363.167 ms |  35.8564 ms |  57.9014 ms |    331.047 ms |

InsertMany |     5000 | DE EF Core 2.1.4 |    789.416 ms |  74.8079 ms |  76.8222 ms |    810.013 ms |

InsertMany |    10000 | DE EF Core 2.1.4 |  1,595.862 ms | 140.9978 ms | 124.9909 ms |  1,569.111 ms |

InsertMany |    50000 | DE EF Core 2.1.4 |  7,258.381 ms | 440.8420 ms | 390.7950 ms |  7,088.249 ms |

InsertMany |       50 |       Direct SQL |      4.834 ms |   0.3965 ms |   0.3096 ms |      4.801 ms |

InsertMany |      200 |       Direct SQL |     10.891 ms |   1.0849 ms |   1.2494 ms |     10.264 ms |

InsertMany |     1000 |       Direct SQL |    140.659 ms |   5.4898 ms |   5.1352 ms |    139.461 ms |

InsertMany |     2500 |       Direct SQL |    268.768 ms |  11.0254 ms |   8.6079 ms |    270.725 ms |

InsertMany |     5000 |       Direct SQL |    862.700 ms |  86.7908 ms | 212.8992 ms |    803.886 ms |

InsertMany |    10000 |       Direct SQL |  1,300.238 ms | 101.1468 ms |  89.6640 ms |  1,321.508 ms |

InsertMany |    50000 |       Direct SQL |  5,866.863 ms | 180.5552 ms | 160.0575 ms |  5,853.468 ms |

InsertMany |       50 |    EF Core 2.1.4 |      6.106 ms |   0.6059 ms |   1.2100 ms |      5.763 ms |

InsertMany |      200 |    EF Core 2.1.4 |     14.414 ms |   1.0264 ms |   0.9099 ms |     14.487 ms |

InsertMany |     1000 |    EF Core 2.1.4 |    166.332 ms |   8.9422 ms |   8.3645 ms |    169.031 ms |

InsertMany |     2500 |    EF Core 2.1.4 |    324.917 ms |  19.8752 ms |  18.5913 ms |    316.937 ms |

InsertMany |     5000 |    EF Core 2.1.4 |    820.579 ms |  84.1034 ms | 145.0742 ms |    760.834 ms |

InsertMany |    10000 |    EF Core 2.1.4 |  1,375.334 ms |  45.0110 ms |  39.9011 ms |  1,372.024 ms |

InsertMany |    50000 |    EF Core 2.1.4 |  7,370.499 ms | 720.3312 ms | 673.7983 ms |  7,207.465 ms |

InsertMany |       50 |       XPO 18.2.3 |     18.139 ms |   1.5684 ms |   1.3904 ms |     17.794 ms |

InsertMany |      200 |       XPO 18.2.3 |     68.331 ms |   6.8001 ms |   9.5328 ms |     64.733 ms |

InsertMany |     1000 |       XPO 18.2.3 |    291.094 ms |   7.4893 ms |   6.6390 ms |    288.345 ms |

InsertMany |     2500 |       XPO 18.2.3 |    801.432 ms |  71.1621 ms |  59.4235 ms |    796.180 ms |

InsertMany |     5000 |       XPO 18.2.3 |  1,490.661 ms | 129.1948 ms | 114.5278 ms |  1,495.313 ms |

InsertMany |    10000 |       XPO 18.2.3 |  2,972.510 ms | 231.7926 ms | 216.8190 ms |  2,985.000 ms |

InsertMany |    50000 |       XPO 18.2.3 | 13,713.237 ms | 671.8771 ms | 561.0478 ms | 13,561.126 ms |

// * Legends *

 RowCount     : Value of the 'RowCount' parameter

 TestProvider : Value of the 'TestProvider' parameter

 Mean         : Arithmetic mean of all measurements

 Error        : Half of 99.9% confidence interval

 StdDev       : Standard deviation of all measurements

 Median       : Value separating the higher half of all measurements (50th percentile)

 1 ms         : 1 Millisecond (0.001 sec)

// ***** BenchmarkRunner: End *****

Run time: 00:21:45 (1305.47 sec), executed benchmarks: 28

23 November, 2018
Brad Baker - MMC

I ran the tests again using an azure managed instance.  Anything where there is latency involved the XPO numbers seems to explode and the EF ones start to become to same.

I only ran a smaller subset:

BenchmarkDotNet=v0.11.3, OS=Windows 10.0.17134.376 (1803/April2018Update/Redstone4)

Intel Core i7-4610M CPU 3.00GHz (Haswell), 1 CPU, 4 logical and 2 physical cores

 [Host] : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

 Clr    : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3221.0

Job=Clr  MaxRelativeError=0.1  MinInvokeCount=1  

Runtime=Clr  InvocationCount=1  LaunchCount=1  

RunStrategy=Throughput  UnrollFactor=1  WarmupCount=1  

    Method | RowCount |     TestProvider |        Mean |      Error |     StdDev |

----------- |--------- |----------------- |------------:|-----------:|-----------:|

InsertMany |       50 |       Direct SQL |    60.13 ms |  0.7683 ms |  0.5999 ms |

InsertMany |       50 |    EF Core 2.1.4 |    61.64 ms |  0.9006 ms |  0.7521 ms |

InsertMany |       50 | DE EF Core 2.1.4 |    62.24 ms |  1.0486 ms |  0.9295 ms |

InsertMany |       50 |       XPO 18.2.3 |   380.91 ms |  2.9043 ms |  2.7166 ms |

           |          |                  |             |            |            |

InsertMany |      200 |       Direct SQL |    65.83 ms |  1.1743 ms |  0.9806 ms |

InsertMany |      200 |    EF Core 2.1.4 |    66.92 ms |  0.7097 ms |  0.5541 ms |

InsertMany |      200 | DE EF Core 2.1.4 |    69.94 ms |  1.2526 ms |  1.0460 ms |

InsertMany |      200 |       XPO 18.2.3 | 1,277.65 ms | 22.1612 ms | 19.6453 ms |

           |          |                  |             |            |            |

InsertMany |      500 |       Direct SQL |   104.32 ms |  2.8517 ms |  2.5280 ms |

InsertMany |      500 |    EF Core 2.1.4 |   127.08 ms |  3.1451 ms |  2.9420 ms |

InsertMany |      500 | DE EF Core 2.1.4 |   135.57 ms |  1.3647 ms |  1.1396 ms |

InsertMany |      500 |       XPO 18.2.3 | 3,243.33 ms | 50.5396 ms | 44.8020 ms |

           |          |                  |             |            |            |

InsertMany |     1000 |       Direct SQL |   200.98 ms |  9.5625 ms |  8.9448 ms |

InsertMany |     1000 | DE EF Core 2.1.4 |   217.67 ms |  5.8969 ms |  5.5160 ms |

InsertMany |     1000 |    EF Core 2.1.4 |   222.81 ms |  6.8294 ms |  6.3882 ms |

InsertMany |     1000 |       XPO 18.2.3 | 6,112.20 ms | 48.0992 ms | 42.6387 ms |

25 November, 2018
Uriah (DevExpress Support)

Thank you for your assistance, Brad! Using this information, we reproduced the similar result. We found that EF Core performs bulk inserts in very large batches (several thousands records at a time). Our DirectSQL benchmark uses the same strategy. That is why it works as fast as EF Core.

This approach reduces the number of TCP packets transferred between the SQL server and client application. That is why, this strategy works fast when the network connection is slow.

XPO also uses batches, but it inserts fewer records at a time. Thus, it needs to send more TCP packets for the same amount of data. The batch size depends on various factors and is optimized to gain a good SQL server performance rather than to reduce the number of requests.

Having that the slow network connection affects the application performance in all aspects, it is worthless to optimize the bulk inserting strategy unless we find an easy and reliable way to do this without breaking existing optimizations. At the moment, we do not have an easy solution and we decided not to do anything in this regard for now.

27 November, 2018
Brad Baker - MMC

Can you please explain what you're trying to accomplish here?

This says 50 operations, but the call is Take20, and youre looping it 50x?

[Benchmark(OperationsPerInvoke = 50)]

       public void LinqTakeRecords20() {

           for(int i = 0; i < 50; i++) {

               TestProvider.LinqTakeRecords20();

           }

       }

Why are we doing a for loop? and a where to take xx records?  why not just

   var query = dataContext.Entities.AsNoTracking().Take(takeRecords).ToList();

protected override void LinqTakeRecords(int takeRecords) {

           for(int i = 0; i < RecordsCount; i += takeRecords) {

               var query = dataContext.Entities.AsNoTracking().Where(o => o.Id >= i).Take(takeRecords);

               foreach(var o in query) { }

           }

       }

Note: you should also state your testing is not a real world example and is lab testing only :)  If we could only run a SQL instance on everyone's machine with nvme storage or even better, in memory :P

28 November, 2018
Uriah (DevExpress Support)

Hello Brad,

When the Row Count value is small, LinqTakeRecordsXX benchmarks run very fast. Thus, the execution time is measured in fractions of a millisecond. Various minor factors, such as hard disk speed and operating system processes, have significant influence on test results.

We found that the LinqTakeRecordsXX benchmarks results were different from launch to launch. Such results are useless. That is why we repeat the test 50 times.

To display the actual result in a test report, the total time should be divided by 50. The [Benchmark (OperationsPerInvoke = 50)] attribute serves this purpose.

30 November, 2018

Please login or register to post comments.