XPO - A simple benchmark against EF 6 and EF Core (UPDATE)

We have received many comments and questions on our previous blog post. Thank you all for that!

In this post, I will describe our updated benchmark based on your feedback: 
    https://github.com/DevExpress/XpoNetCoreDemos/tree/master/ORMBenchmark

Let me remind that we have tested the performance of the following Object-Relational Mapping (ORM) libraries for .NET Framework 4.6.1 and higher:


What's New in the benchmark?

  1. EF benchmarks now use the AsNoTracking method in the Fetch, LinqQuery, InstantiationNative and other test cases. It has slightly, but not significantly decreased data reading time.
  2. Since XPO has no option such as the EF's AsNoTracking, we created the additional ProjectionLinq benchmark that demonstrates how much time each ORM library takes to load objects without tracking changes.
  3. For XPO benchmarks, we set the IdentityMapBehavior option to Strong. This allows us to reduce time to place loaded objects in Identity Map.
  4. Our short-time benchmarks (for example, InstantiationLinq and LinqTakeRecords* on small data set) are now more accurate because of improved the benchmark configuration. 

We also tried to use the Find method instead of First in the Fetch benchmark based on Neal's comment. In certain cases it showed lower performance, in others it did not affect the result much. As a result, we left the benchmark unchanged.

Run the updated benchmark tests or review our results here. Example: 



All benchmarks were executed using .NET 4.6.1, AnyCPU release builds (include warm-up), Windows 10 Enterprise x64, local Microsoft SQL Server 2016 Developer Edition, i7-7700 CPU @3.6GHz / 16GB RAM / SSD. Please note that DevExpress.Xpo and other referenced libraries will automatically be restored from Nuget. Edit the connection string in the App.config file, update the ORM library and target framework versions, if necessary. 

Your feedback is needed!

What do you think about this update? Did you try to extend this benchmark with your own scenarios? Let us know in comments. We'd love to hear your feedback and results.

See Also

Love XPO and want to help us promote it? Add the package through Nuget.org instead of DevExpress Nuget!

 

  Yekaterina K.
  Technical writer
  XPO team

Again, youre testing with an old version of .net core.  Use 2.1.  And use AddRange and RemoveRange like in the real world, instead of trying to inflate your performance.

``` ini

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.16299.371 (1709/FallCreatorsUpdate/Redstone3)

Intel Core i7-4610M CPU 3.00GHz (Haswell), 1 CPU, 4 logical and 2 physical cores

Frequency=2922920 Hz, Resolution=342.1236 ns, Timer=TSC

 [Host] : .NET Framework 4.7.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0  [AttachedDebugger]

MaxRelativeError=0.1  MinInvokeCount=1  Runtime=Clr  

Toolchain=InProcessToolchain  InvocationCount=1  LaunchCount=1  

RunStrategy=Throughput  UnrollFactor=1  WarmupCount=1  

```

|              Method | RowCount | TestProvider |                Mean |              Error |             StdDev |              Median |

|-------------------- |--------- |------------- |--------------------:|-------------------:|-------------------:|--------------------:|

|          DeleteMany |       50 |      EF Core |  17,449,827.5687 ns |    855,140.5748 ns |    799,898.9464 ns |  17,432,379.2600 ns |

|          DeleteMany |       50 |          XPO |  20,242,114.9426 ns |  1,944,794.1771 ns |  2,459,541.9365 ns |  20,033,836.0300 ns |

|           DeleteOne |       50 |      EF Core | 381,688,023.3264 ns |  8,936,898.1914 ns |  7,922,326.6909 ns | 380,275,785.8550 ns |

|           DeleteOne |       50 |          XPO | 418,829,349.8740 ns |  9,175,639.4914 ns |  8,582,898.0380 ns | 419,328,667.9100 ns |

|               Fetch |       50 |      EF Core | 438,341,253.4947 ns | 15,412,493.6022 ns | 14,416,854.6752 ns | 433,521,598.2600 ns |

|               Fetch |       50 |          XPO | 449,237,752.5015 ns | 15,550,402.0541 ns | 12,985,288.7491 ns | 443,748,220.9600 ns |

|          InsertMany |       50 |      EF Core |  10,371,380.2318 ns |  1,006,416.0145 ns |  1,033,515.0527 ns |  10,143,264.2700 ns |

|          InsertMany |       50 |          XPO | 148,164,472.5136 ns |  7,720,868.9895 ns |  6,844,348.5830 ns | 148,533,794.9700 ns |

|           InsertOne |       50 |      EF Core | 377,448,288.0136 ns |  9,131,772.2414 ns |  8,095,077.4434 ns | 378,676,340.7800 ns |

|           InsertOne |       50 |          XPO | 408,196,307.1173 ns |  7,473,868.3976 ns |  6,991,060.4559 ns | 407,443,361.4300 ns |

|   InstantiationLinq |       50 |      EF Core |           0.1934 ns |          0.1332 ns |          0.2566 ns |           0.0000 ns |

|   InstantiationLinq |       50 |          XPO |           0.4950 ns |          0.1861 ns |          0.2843 ns |           0.4619 ns |

| InstantiationNative |       50 |      EF Core |           0.2264 ns |          0.1204 ns |          0.2591 ns |           0.0513 ns |

| InstantiationNative |       50 |          XPO |           0.3107 ns |          0.1794 ns |          0.3046 ns |           0.3079 ns |

|           LinqQuery |       50 |      EF Core |         112.9010 ns |         96.5035 ns |        171.5348 ns |           0.0000 ns |

|           LinqQuery |       50 |          XPO |         343.8931 ns |        156.2748 ns |        229.0655 ns |         461.8700 ns |

|   LinqTakeRecords10 |       50 |      EF Core |          32.1070 ns |         15.0597 ns |         35.2015 ns |          34.2120 ns |

|   LinqTakeRecords10 |       50 |          XPO |          30.2626 ns |         16.5718 ns |         29.8824 ns |          25.0890 ns |

|  LinqTakeRecords100 |       50 |      EF Core |   6,977,855.5480 ns |    263,813.0198 ns |    220,295.7986 ns |   6,895,858.2513 ns |

|  LinqTakeRecords100 |       50 |          XPO |  10,028,753.3989 ns |    232,194.4041 ns |    193,892.8249 ns |  10,041,676.9874 ns |

|   LinqTakeRecords20 |       50 |      EF Core |           5.1922 ns |          2.8618 ns |          4.6213 ns |           2.3948 ns |

|   LinqTakeRecords20 |       50 |          XPO |           5.3307 ns |          2.9571 ns |          5.4812 ns |           6.1582 ns |

|   LinqTakeRecords50 |       50 |      EF Core |           0.7581 ns |          0.7124 ns |          1.3380 ns |           0.0000 ns |

|   LinqTakeRecords50 |       50 |          XPO |           4.4020 ns |          1.7834 ns |          2.6694 ns |           4.1055 ns |

|      ProjectionLinq |       50 |      EF Core |           0.0132 ns |          0.0505 ns |          0.0620 ns |           0.0000 ns |

|      ProjectionLinq |       50 |          XPO |           0.2843 ns |          0.1410 ns |          0.2394 ns |           0.0855 ns |

|          UpdateMany |       50 |      EF Core |  18,100,970.4200 ns |  1,586,514.2941 ns |  1,324,811.1619 ns |  17,685,105.9900 ns |

|          UpdateMany |       50 |          XPO | 157,267,641.0347 ns |  7,500,688.6892 ns |  7,016,148.1709 ns | 155,860,919.9000 ns |

|           UpdateOne |       50 |      EF Core | 384,065,757.3020 ns |  7,033,775.6255 ns |  6,579,397.4439 ns | 384,186,139.2000 ns |

|           UpdateOne |       50 |          XPO | 419,533,502.2121 ns |  9,041,276.9240 ns |  8,014,855.7096 ns | 419,769,494.2050 ns |

17 July, 2018

Brad - good point! I was just about to check EF Core results to XPO myself... cz difference shown was huge. DX - please make sure to publish results with latest version available when you do benchmarking.

Thanks!

17 July, 2018

Its not just that, its things like oh, we going to add 50 items using .Add instead of doing .AddRange.   Yeah if you call .Add you pay the change tracking penalty 50x, instead of 1x.  Not saying EF is perfect by any means, but most people arent going to out of their way to learn a new framework for marginal performance gains.  Honestly have no idea why try trying to bring XPO back to life with EF being such a huge player.  Telerik shuttered their ORM 2 years ago no?  No idea why DevExpress even keeps it around, at least Telerik use it in their CRM SiteFinity,  Whats DevExpress use it in?  Like there  not even dmo examples on their site using it lol.

17 July, 2018

Brad, is it possible to repeat and share with us your benchmarks, also with higher row counts. For example 1.000 and 5.000. It would be interesting to complete the picture. Also I think DevExpress uses XPO in the XAF framework for operations that with EF is difficult to implement.

17 July, 2018

@George: Yes, you are correct that XPO has been one of the data access options in XAF for more than a decade (see also my answer to Brad for more information). I also agree that it is interesting to see results for all row count sets, not only for 50.

@Gosha, Brad: Thanks for your suggestions! We planned to publish another benchmark with EF Core 2.1 and .NET Framework 4.7.X later. Internally, we also discussed that benchmarks for other versions may be interesting to other users as well, for various reasons. For instance, I remember that when EF 6 came, many people preferred to stay on previous versions due to breaking changes. Fortunately, the EF Core 2.1 migration looks smooth (primarily performance improvements). Anyway, what is your opinion on leaving benchmarks for the last few versions? Did I get it right that it does not make sense to keep EF Core 2.0 and even EF 6 tests?

@Brad: Thanks for testing our benchmark locally! It would also be great to see your testing results for other row counts, as I noted above. Feel free to email them to us at support@devexpress.com.

BTW, we received different results even for your 50 rows with Skylake instead of Haswell. Results also depend on CPU, and you may receive different results for different hardware. That is why we are also asking people to run this thing locally.

We tried to make this benchmark as fair as possible. For instance, we did not use magic XPO tuning tricks and tried to use EF as an average developer Joe following Microsoft tutorials like

docs.microsoft.com/.../update-related-data.

There is no mention of the AddRange/RemoveRange methods in these tutorials. You may happen to come (or not come) across them and threads like stackoverflow.com/.../entity-framework-6-dbset-addrange-vs-idbset-add-how-can-addrange-be-so-much-fa only when experiencing real problems. Anyway, we appreciate your suggestions and will be happy to update our benchmark with new test cases based on these methods. We also welcome more suggestions on what can be improved further in our tests. Hopefully, this benchmark will be helpful to non-XPO developers as well.

Besides our XAF product, we use XPO for nearly all our services: online documentation, the Support Center, our CRM, and the main website. You may not believe it, but even with the plethora of version control options like GitHub, GitLab, and HG popular today, XPO remains at the core of our long-standing version control system (DXVCS)! We still widely use it because to date no existing options could handle projects of such a capacity well! Thus, we have a great experience with using XPO in high-loaded applications. Our systems work with databases ranging in size from 30 to 100 gigabytes.

Please follow our team blog, because we are preparing a separate article on how we have been battle-testing XPO internally for almost 15 years.

18 July, 2018

Hello Dennis,

XPO is very nice ORM framework (speed, features and etc.). But - when you do comparison You should search for the best way to hit the target and then compare times + different frameworks have their own ways to do the same thing. You can do this kind of tests and then say it took me to write 100 lines of code instead of 1 linen with XPO - that will be fair.

18 July, 2018

@Brad: Regarding "Honestly have no idea why try trying to bring XPO back to life" - Many customers like us use XPO very successful in their projects and products. I appreciate this attempt very much. Some years ago I was afraid that XPO could die, since then I evaluate almost every ORM I can find. And what can I say – up to now I only found one ORM which could beat XPO (regarding my use cases), and this was not EF :-)

(of course everyone has his own measures)

19 July, 2018

Right, I don't use XAFso I would have no idea they use it.  Looking at their documentation for any of the standard DevExtreme stuff has no mention of using XPO or any of its "benefits".  And from what I read, XAF can also use EF.  But since we dont use XAF and use standard .Net technology, makes more sense to use what Microsoft recommends :)

Running the test for more than 50 takes too much time.  I was just doing it as a quick example to test my theory.  Since most of our applications only do things in small numbers.  We would never do large data loads/manipulation with EF.  Clearly we all know any ORM takes a performance hit over raw sql, but we gain the ability to not have to write SQL.

Maybe put together a simple example of a .Net Core DevExtreme XPO project.  Just looking at the XPO framework looks clunky and non lambda based.  Note:  We also dont use VisualStudio, currently using JetBrains Rider.

19 July, 2018

@Brad: Thanks for your answer. I understand your situation and it makes sense.

We indeed do not have many dedicated examples where DevExtreme and a specific ORM library (e.g. XPO) are used in one solution: www.google.com/search

We may consider creating them if we see user demand for this - thanks for your feedback in this regard.

Currently, we have more general purpose learning materials like www.devexpress.com/.../how-to-connect-to-the-database-server. They can be used by any developer who already has a basic knowledge of their favorite server side technology and ORM.

Our DevExtreme widgets know nothing about server side technologies (XPO, EF, etc.). It does not matter to them how you manipulate data on the server side, which is interchangeable. The only requirement is that it is necessary to provide data for them in the JSON format. With XPO, you are capable doing the latter in many ways. Examples:

github.com/.../how-to-implement-odata4-service-with-xpo

github.com/.../how-to-use-the-xpo-odata-v3-service-e4389

github.com/.../DataController.cs

Of course with XPO, you are also free to build a fully custom RPC without any OData, WebAPI and other standards.

We would be more than happy to review specific XPO "clunky" cases, especially with the help of your fresh eyes. As for lambdas, you may want to check our Xamarin and ASP.NET Core demos, where we try to use LINQ more. XPO should also work everywhere where .NET works:-), so I do not expect problems with Rider.

Once again, thanks for all your comments - glad that you jumped in and we had such a constructive talk.

19 July, 2018

So much hate. XPO is a tremendous product. XAF despite being long in the tooth is an amazing one stop implementation of a CRM type product. The syntax is much better than EF. The performance maybe better but who cares and the fact that it's now free and you can spin up a dataservice for free is half the battle solved.

xaf unfortunately supports LOB teams. The platform could be so much more. For now kendo UI with XPO is the framework of choice unfortunately.

19 July, 2018

Question I have is "Does XPO support SSL databases?"

I tried to connect to a Postgresql database that has SSL enabled and used

SSL=On

SSL= True

SSL = ad naseum

and never could get it

The last support ticket I saw, was nine years ago.

So what is the trick to enable SSL?

20 July, 2018

@Robert: Yes, it does, though this is not specific to XPO at all.

A connection string you can specify for an XPO data source is a standard connection string that will be parsed by a corresponding database driver (here, Npgsql), with an additional item that specifies the provider (XpoProvider=Postgres). Please refer to the PostreSQL documentation to learn how to enable the SSL protocol for it:

stackoverflow.com/.../connect-to-postgresql-with-ssl

www.npgsql.org/.../connection-string-parameters.html

You can test the database connection by creating an NpgsqlConnection instance with your connection string in code and calling the Open method. Once the connection works, use the same connection string with XPO by appending "XpoProvider=Postgres" to it.

The old ticket you were referring to is likely about the ORM Data Model Wizard: www.devexpress.com/.../postgresql-ssl. I have added a screenshot showing how to specify a custom connection string in it.

23 July, 2018

Will you also compare it to ServiceStack.OrmLite ?

1 August, 2018

This is all well and good, but what happens when you have a model in XPO with object references? The performance out-of-the-box with Devex is terrible because it will load all the associations by default - and every single damned property on the object. There is noway around it without using a XPView or similar - which is then read only.

And you can't edit properties of an object by Key, you have to load the associated object, and then edit with the object reference.

Try loading any complex object with FK in XPO into a non-readonly dataset with selected properties and compare this to Entity Framework => ProjectTo. Especially over any kind of network.

I have 10 years experience with both XPO and EF and have tried everything to try and improve XPO including wrappers sending supplemental direct SQL queries to get efficient server side aggregations. I now use exclusively EF Core 2.1 and hate going back to my XPO projects.

This is so far from a real world complex ORM example.

13 August, 2018

Please login or register to post comments.