.NET Data Processing API - First Community-Sourced Scenarios Addressed

As you may recall, we prioritized solutions for the most common ETL scenarios in September (Data Processing Scenarios (ETL) - Your Feedback Matters). Thanks for all the great feedback in this regard.

The good news is that we've built our first prototype (a .NET library/API) to address the most popular usage scenarios:

  • Add ETL (Extract, Transform, Load) capabilities to .NET apps.
  • Shape data (group, sort, filter, apply analytics functions) for use within the UI on any platform.

We kindly ask you for one more favor - please test our .NET Data Processing API in your environment to help us validate our design decisions before we commit to further development efforts.

The Why: The Benefits of Our API

The main benefits of our Data Processing API are as follows:

  1. Efficient in-memory data processing for unmatched performance (we reused the core of our blazing fast Pivot Grid control - learn more) .
  2. Unified interface to connect to different data sources (relational databases, web services, Excel spreadsheets, JSON data, etc).
  3. Custom .NET logic support to intercept data processing flow.
  4. High-level analytical functions for easier data flow design.
  5. Built-in debugging capabilities to quickly fine-tune results.

How to Test Our API

This .NET data processing library/API is NOT yet part of an official DevExpress package. This notwithstanding, we made it super easy for you to test our API with the following GitHub repository: 

DevExpress / dataprocessingapi-mvp-example 

To get started quickly, please review our sample (ConsoleExample/Program.cs) that joins input data from two sources (a JSON file and an XLSX file), applies a set of transformations (aggregation, top-n, sorting) and generates the XLSX output file in the appropriate format.

Sample data processing workflow

For information on how to construct a data flow from scratch, please refer to the README.md file, download the example and explore its capabilities within Visual Studio.


Your Feedback Matters

If the Data Processing API we’ve introduced do not fully address your business needs or if you’d like to engage us further on this topic, please complete the survey below, leave a comment below or create a new support ticket via the DevExpress Support Center. We’ll be more than happy to follow-up.

Free DevExpress Products – Get Your Copy Today

The following free DevExpress product offers remain available. Should you have any questions about the free offers below, please submit a ticket via the DevExpress Support Center at your convenience. We’ll be happy to follow-up.
Hedi Guizani
Hedi Guizani
Just great, this is a very useful component, a life saver in many business solutions.  
22 December 2021
Bassam Abdelaal
Bassam Abdelaal

hello, just to understand,

1. is this something similar to sql server integration services ? but on the client where .net logic can be used and produce more flexiability ? 

2. in SSIS there is a designer with drag and drop feature to quickly create the package for ETL , will it be same in DX solution or only thru API and code ?

3. will it be part of universal subscription ?

thanks


22 December 2021
Daniel Kmínek
Daniel Kmínek

Hello Alex

This is a very useful component.

Importing data from various sources is our daily job (excel, csv, json web services..).

However, our goal is very often saving directly into XAF Persistent Object

Is this feature part of it ?

thanks



22 December 2021
James Lincoln
James Lincoln
Very impressive demo and very applicable to my work.   Ideally also support legacy .Net framework.   I'm a big fan of using a workflow engine: would love DevExpress to onboard, where this would fit in nicely as a preprocessing step
22 December 2021
Alain Bismark
Alain Bismark

Hi Team

I work a lot with similar technologies to perform data migration between ERP systems; please let me know where I can share with you several functionalities or ideas to add into the product.

Best regards 

22 December 2021
John.Molloy
John.Molloy

hi

really interesting, unfortunately  the things i would use it for not there yet i.e XML <> XPO :(


23 December 2021
Alex (DevExpress Support)
Alex (DevExpress)

@Bassam Abdelaal

 

Our product has similarities with SSIS, but it does not require a database server to operate. So yes, you can itegrate the library into an existing/new .NET app.

We prioritized the API-only solution because it was the most demanded in previous surveys. If you have had a chance to try the API, please share your feedback with us. Our future development plans depend on successful integrations of the prototype library.

 

@Daniel, John,

 

XAF/XPO integration is not currently available. We would like to learn more about your requirements to find a suitable solution. Please submit a ticket in the Support Center and tell us how you need to process and persist data in an XAF application.

 

@James,

 

For .NET Framework, reference the dll/DevExpress.DataProcessingApi.MVP.NetFramework.dll and DevExpress libraries v.21.2.4 in your project.

 

@Alain,

 

Please share your ideas with us – feel free to contact us via support@devexpress.com or post a ticket in Support Center.

23 December 2021
carlodep
carlodep
Great job, it saves me a lot of time.
I have tried it and it works well and fast.


One question, my scenario involves the union of two heterogeneous data sources, thanks to ETL I produce the same data structure.
Will there be the possibility of having a Union function to create a single DataFlow?
It's not a blocking problem, I've created two lists and added one into the other.


Best regards
23 December 2021
Alex (DevExpress Support)
Alex (DevExpress)
@carlodep,

Thank you for taking the time and effort to test the Data Processing library. I agree that the UNION function is useful in this scenario. We are yet to decide how the API will be enhanced in the future. We will take your suggestion into account.
24 December 2021
Hedi Guizani
Hedi Guizani

Great job again,

Are we going to see Pivot/Unpivot data and other transformations function,

Xpo integration (from/to) xpo collections and objects, is defenetly a must.

A grafic designer is a great addition (Devex has already all the compoment ready (diagrams, dashboards)

Dataflow as dashbords and reports datasource will be also great.

Expression fields (with dashboard like criteria operators [windows functions, aggreations...]) will be absolutly fantastic.

Will share more Ideas if your team is willing to collect more.

Happy holidays


24 December 2021
Christian Peters
Christian Peters
Searching for an ETL-Solution, I found one that also is able to visualize an execution plan using a sankey diagram.

Since you also have such a diagram ...
4 January 2022
Jascha
Jascha
Looks very promising. We wrote our own ETL framework way back and our clients use it daily so a DX one could replace what we did if it is sufficiently powerful. I presume we will be able to add our own operations?
10 January 2022
Alain Bismark
Alain Bismark

Can you please check the following code

               DataTable result = DataFlow

                .FromExcel(ExcelFileName, ExcelWorksheetName)

                .ProcessColumn("Nombre", (string v) => { return v.ToUpper(); })

                .ProcessColumn("Apellido", (string s) => { return s.ToUpper(); })

                .ToDataTable()

                .Execute()

Seems like the second call to ProcessColumn not run.

Best regards

20 January 2022
John (DevExpress Support)
John (DevExpress Support)

Hello Alain,

To resolve this issue enable the following static option:
DevExpress.DataProcessing.InMemoryDataProcessor.DataProcessingOptions.DisableReorganizeTablesGroupOptimization = true;

21 January 2022

Please login or register to post comments.