in
Forums
Blogs
Files
Devexpress.Com
ClientCenter
Support Center
DevExpress Channel

Gary's Blog

Application Scalability Using XPO

Let’s say we have an application with the following (highly simplified) object model:-

ObjectModel

As you can see we have a Customer and a Customer’s Orders which we wish to persist to the database using XPO. This will work perfectly with 10, 100, 1,000, 10,000 orders say; however, somewhere alone the orders of magnitude your database is going to start to struggle with the numbers of Customers and their Orders. You can scale up your database server of course, but there comes a time when even that will not help. When dealing with database scalability I like to use the sharding pattern. In this pattern a large database is broken up into ever smaller pieces and clients access the required piece or shard.

In the non scalable solution the following code would be used to serialize the object model:-

namespace XPOScalabilityExample {
    class Program {
        static void Main(string[] args) {

            //GS - Explicitly set the datalayer
            string conn = MSSqlConnectionProvider.GetConnectionString(
                "(local)", "Customers");

            XpoDefault.DataLayer = XpoDefault.GetDataLayer(
                conn, AutoCreateOption.DatabaseAndSchema);

            //GS - Create a Customer and an Order and persist them
            using (UnitOfWork uow = new UnitOfWork())
            {
                Customer customer = new Customer(uow);
                customer.Name = "Gary Short";
                customer.Address = "The Adress";

                Order order = new Order(uow);
                order.OrderedItem = "A widget";
                order.OrderedQuantity = 4;
                order.Price = 342.29M;

                customer.Orders.Add(order);
                order.Customer = customer;

                uow.CommitChanges();
            }
        }
    }
}

To implement sharding, in the first instance, we will partition the database geographically, placing customers in either the north or the south. The code to persist the Customer now looks like the following:-

namespace XPOScalabilityExample {
    class Program {
        static void Main(string[] args) {

            //GS - Explicitly set the datalayer
            string conn = MSSqlConnectionProvider.GetConnectionString(
                "(local)", "North");

            XpoDefault.DataLayer = XpoDefault.GetDataLayer(
                conn, AutoCreateOption.DatabaseAndSchema);

            //GS - Create a Customer and an Order and persist them
            using (UnitOfWork uow = new UnitOfWork())
            {
                Customer customer = new Customer(uow);
                customer.Name = "Gary Short";
                customer.Address = "The Adress";
                
                Order order = new Order(uow);
                order.OrderedItem = "A widget";
                order.OrderedQuantity = 4;
                order.Price = 342.29M;

                customer.Orders.Add(order);
                order.Customer = customer;

                uow.CommitChanges();
            }
        }
    }
}

For demonstration purposes I have simulated the switch between servers by switching between databases on the same server, but I’m sure you get the idea. To access Customers in the south you would replace the word “North” with the word “South” in the connection string. Now that you have the idea of sharding we will look at a more advanced example next time.

Technorati tags: , ,
Digg This
Published Jun 11 2008, 04:09 PM by Gary Short (Developer Express)
Filed under: ,
Technorati tags: XPO, Scalability

Comments

 

Alain Bismark said:

Sorry, but which is the point here?

persist in one db-server OR in another db-server, changing the connection string?

sorry if i dont understand the "wow!!" of this post.

June 11, 2008 1:35 PM
 

Nate Laff said:

Umm... I don't get it. Like Alain said, did you just change the connection string?

June 11, 2008 2:10 PM
 

Alagarsamy Rajamannar said:

Hi,

I really don't understand the pattern of sharding.  Does each database will have exact copy of the mater database.?Then what is the point?

Thanks

Raja

June 11, 2008 2:11 PM
 

Gary Short (Developer Express) said:

No each database does not have an exact copy of the master database. Each database is a shard, or part of the whole. What happened here was we had a database with N Customer records in it. That number of records put a performance strain on the database. To ease that strain the database was horizontally partitioned into two databases, one housing Customers from the north the other housing Customers from the south. For demonstration purposes assume there is an equal split of Customers between the north and the south. Now each database server has N/2 Customer records and you have eliminated the performance strain on the original database.

Each client then connects to the required database server (via a changed connection string) depending on whether they wish to access Customers from the north or from the south.

June 11, 2008 6:58 PM
 

Renaud Bompuis said:

This approach is interesting but how do you know where to find a particular customer?

What if I have a referral programme and I want to link a customer from the North to his referral from the South?

Similarly, for an Order system you would have a Product database and each customer order would reference a Product.

If you split the customer database, where do you keep the Products?

As far as I know you can't associate objects coming from different databases as they would have different sessions and XPO doesn't like mixing them. Does it mean you would have to keep the Products in both databases?

You could create a weak link using a product reference string instead of an actual Product object, but that somewhat defeats the purpose of using an ORM like XPO and you still wouldn't be able to mix data from both databases into the same session.

If only there was an easy way to fuse sessions so you could work with multiple databases as if there was only a single one and let XPO sort them out when retrieving and saving data...

It's really what I'd like to see from XPO: do more of the hard work for me, not necessarily making it more complex by having to split data access into more incompatible sessions.

June 12, 2008 9:32 AM
 

Nate Laff said:

Gary,

Something I'd like to accomplish with XPO is that each customer (of mine) would have their own database, as each one will get pretty large.

Is this bad design, and should I look into 'sharding'?

June 12, 2008 9:47 AM
 

Martin Hart said:

Gary:

I know this is off-topic, but aren't the these lines doing the same thing?

 customer.Orders.Add(order);

 order.Customer = customer;

When you call the first line, it automatically sets the Customer field of the Order.

Am I missing something?

June 13, 2008 2:38 AM
 

Gary Short said:

@Renaud How do you find a customer? Well in this simple example it is assumed that the clients accessing data work in the north and the south respectively. However, this was just an introductory post to get you used to the idea, for a more complete, and generic, answer to that question check out a post I'm making later today. :-)

With regard to products in an ordering system, you would *never* link to the actual product because when you updated the product then that would affect all historical orders linked to that product thus destroying your ordering system. Instead you would copy a snap shot of the product data into an OrderItemLine table row. A customer and all its related data should be on the same server, this will remove any server linking issues.

@Nathan, there is nothing wrong with the one database per customer architecture if that is what your particular application merits.

@Martin, yes you are right the two lines do the same thing. I guess putting both in is just part of my OCD, like I always write "private" in front of functions even though its the default if nothing is specified. :-)

June 13, 2008 4:50 AM
 

Renaud Bompuis said:

Well, I wouldn't say that you would *never* link a product in an order item to the actual product. Order fulfilment can be a complex thing and I can think of a few cases where it's actually desired to keep that reference, if only for a while.

Anyway, it's just side-stepping the bigger issue that sharding has: it's only really efficient for related data that neatly stays within the same boundaries.

It's pretty good for websites where you can store a user profile and all its data in the same database -for instance depending on geographical location- but I'm a bit sceptical about the introduced complexity for system where data is more interdependent.

In your ordering system, how do you return the list of all customers that have purchased a given item?

You would have to issue the same query in each database and merge all results, except that you wouldn't be able to simply return them in a single XPCollection as they are from different sessions.

It's of course possible to get to the data but it just breaks many of the advantages that XPO bring in the first place by adding a level of complexity to operations that would otherwise be trivial.

I think you should better define the problem domain and the type of application that would benefit from this as well as clearly mention the inherent limits of sharding.

I'm curious to see how elegantly issues of 'border-crossing' will be handled in your future posts.

Curious and eager :-)

June 13, 2008 10:24 AM
 

Gary's Blog said:

In my previous post I introduced the topic of database scalability using XPO. In that post we looked

June 19, 2008 10:17 AM
 

Ryan Britton said:

The problem that Renaud outlined extends beyond the boundaries of the logical data tier. Since XPO is not able to (at this stage) transfer an object created in one XPO Data Layer to another without creating a deep clone of the object (with all the associated hassle of discerning a strategy for managing parents, children, referential integrity, deferred deletion etc), the sharding strategy above would not allow you to incorporate cross-shard (oooo - I coined a term :)) business logic (the kind that might be necessary if one of your northern customers migrated south for the winter). In addition to this you would feel a sense of impending doom the first time your CEO asked you to "quickly bring up all the data in that new fangled XtraPivotgrid Tool" so that he could do some on the fly segmentation and analysis of XYZCorp's entire customer base on his sparkly new laptop...

June 24, 2008 5:06 AM
 

cduffy said:

"In addition to this you would feel a sense of impending doom the first time your CEO asked you to "quickly bring up all the data in that new fangled XtraPivotgrid Tool" so that he could do some on the fly segmentation and analysis of XYZCorp's entire customer base on his sparkly new laptop..."

That's easy, it's just a union of North and South.

August 28, 2008 10:57 PM

Leave a Comment

(required)  
(optional)
(required)  
Verification code: Required
   
Add
Copyright © 1998-2008 Developer Express Inc.
ALL RIGHTS RESERVED