DevExpress Dashboard - High Performance In Memory Data Processing (Coming soon in v15.2)

16 November 2015

DevExpress Dashboard v15.2 will ship with a completely redesigned client side data processing engine. This new data engine reduces memory consumption and improves the speed of client-side data shaping operations (grouping, sorting and aggregation) by loading a compressed snapshots of aggregated data into memory. As you'll read in a moment, the improvement in memory consumption/performance is significant thanks to the following techniques:

  • Data compression is used to reduce memory consumption. For instance, dictionary compression is used to replace common dimension values with integer key values.
  • The Dashboard's new data grouping algorithm uses hash tables (with integer key values obtained during data compression). The use of key values means data grouping and sort operations are executed much faster. 
  • New multi-threading algorithms are used to perform independent operations. The Dashboard's data engine uses TPL (Task Parallel Library) to perform these operations.


We've also introduced strong typing for all operations within the data engine. This removes overhead expenses related to boxing/unboxing conversions.

Comparing v15.1 to v15.2

Seeing is believing and so here are a few test results that help illustrate the performance gains you'll find in DevExpress Dashboard v15.2.

Note: These results were gathered using Microsoft's Contoso Dataset and the underlying data source was formed using the following SQL requests:

select * from [dbo].[V_CustomerOrders] where ProductCategoryName = 'TV and Video'  | Returns approximately 1.2M records.

select * from [dbo].[V_CustomerOrders] | Returns approximately 12M records.

Memory Consumption

To say that v15.2 reduces memory consumption might be an understatement.

DevExpress Dashboard Performance - 12M Records

Binding/Loading

In this test, the Dashboard's List Box was bound to the ProductSubcategory dimension. As you can see below, we have a nearly 50x improvement from v15.1 to v15.2

DevExpress Dashboard High Performance Data Loading

In the test scenario below, the Dashboard's Grid was bound to the following dimensions and measures.

  • Dimensions: ProductCategoryName, ProductSubcategory, Product
  • Measures: Amount, Quantity (summary type set to Sum).

The power of the Dashboard's data processing engine speaks for itself - a 37x improvement in calculation speed. 

DevExpress Dashboard High Performance Data Binding

In this last example, we measured load times for the Dashboard's Pivot Grid (column/row values expanded at all levels).

Row Dimensions: ProductCategoryName, ProductSubcategory, Product
Column Dimensions: Region, CalendarYear
Measures: Amount, Quantity (summary type set to Sum)

Again, the results speak for themselves - calculation speed has improved 32x.

DevExpress Dashboard High Performance Pivot Table

=========================

We'd love to hear your thoughts on DevExpress Dashboard and these performance improvements. Let us know what you think.


9 comment(s)
Gustavo Marzioni

Great News!

16 November, 2015
Jonatas Hudler

Great addition! Spectacular results!

This adds built-in in-memory/columnar-storage likewise performance.

Regarding this feature: Is it possible through an API to serialize this compressed dataset (for fast data retrieval, from a data server cache or embedded with the dashboard file for an offline analysis, for example)?

16 November, 2015
Jim Horvath

This is great news, it's encouraging to see your focus on making a good product even better.  It'll also help justify my advocating a general-purpose toolbox (DevExpress) over a dashboarding-specific product.  :)

16 November, 2015
Laurence

I wish you would enable the dashboard to be purchased separately instead of part of the universal subscription.  As a longstanding Winforms subscriber I don't want to spend an additional $1300 just to get this, so will need to look at other options.

16 November, 2015
Marcin Chojna

What about 'non-dashboard' PivotGrid and GridView from WinForms package? Is there a way to apply this changes to single components from WinForms package?

16 November, 2015
Scott G Blood

This is great news and definitely a push in the right direction.  We are still using QikView in our company despite attempts to push away from this, the in memory compression is definitely going to assist with this.

Any chance you can push something like this into the pivot grid as well as that can be quite memory intensive when reading in records from a database or even a DataTable clone with in memory compression of data would be amazing.

17 November, 2015
Roman (DevExpress)

@Jonatas Hudler

Thanks a lot for your feedback about a new feature!

Currently, it is not possible to retrieve a data file containing a compressed dataset. Further enhancements to our data processing engine will include this feature.

17 November, 2015
Maxime Savard 3

Wow

17 November, 2015
Marco Torregrossa (Global)

Great job!

Thanks to hear and consider our feedback ;)

19 November, 2015

Please login or register to post comments.