XPO Publication: Why?

XPO Team Blog
16 October 2006

Geoff Davis commented on my announcement post: I just don’t get it! Not a problem, Geoff, and certainly not your fault – I think the topic hasn’t been discussed in depth so far and Kenneth’s newsgroup post was a bit short. So, let’s start with “more datastore aware than DB aware” (a phrase Kenneth mentioned in his post). First of all let me say that I didn’t come up with that sentence, so I can only guess – but I have a pretty good idea what was meant here.

Abstracting databases through IDataStore

As you know, XPO can work with various DB backends, while leaving your code, the code that’s written on top of it, unchanged. So how does it do this? Pretty simple: it uses two adaptation layers between the “user code” and the database specific driver code. In practice, you may actually count more layers than that – depends on what you consider “user code”, and what kind of database you use (ADO.NET can be said to provide abstraction layers of its own).

In any case, the lowest level of XPO code that has to do with database abstraction is the implementation of the “connection providers”. These classes are each specific to a database backend, and they implement an interface called IDataStore. This is the declaration of the IDataStore interface:

public interface IDataStore {
  AutoCreateOption AutoCreateOption { get; }
  ModificationResult ModifyData(params ModificationStatement[] dmlStatements);
  SelectedData SelectData(params SelectStatement[] selects);
  UpdateSchemaResult UpdateSchema(bool dontCreateIfFirstTableNotExist, params DBTable[] tables);
}

The most important thing to note about this interface is that its data structures are no longer dependent on any specific database. So the “connection providers” contain code to “translate” database specific implementation details into this non-specific interface lingo – in other words, looking from the top, I can talk the same language to every class out there that implements IDataStore, regardless of what happens inside the class. A really common thing with interfaces, but nevertheless something that’s kind of easy to miss.

Now what does it mean to be “more datastore aware than DB aware”? It means “think along the lines of what the IDataStore interface provides”. It doesn’t matter what a data store (i.e. a class that implements IDataStore) does internally. In fact it is possible to implement IDataStore and do something completely different with the data that is passed through the interface methods – this is used a lot in implementations like the DataStoreLogger, where a thin class is chained together with other IDataStore.

A final thing worth mentioning again about IDataStore is that is is structured to work well within distributed systems. I went into this in some depth in my previous posts XPO is good for distributed applications and Using .NET Remoting with XPO. Nevertheless, XPO always uses data stores as a level of abstraction, and it doesn’t matter whether it’s done locally or via a Remoting (or other) remote connection.

So why do I want to use Remoting?

So, back to that feature Kenneth was requesting. Let’s assume we are actually working on a distributed application, and so we need to enable the clients to contact the server on a different physical system. Theoretically, we could just open up the server (the DB server itself, that is) via the network and let the clients access it. This is not a good idea (Microsoft says so (right at the top, "Firewalls"), so does Oracle (page 14)). I propose to use a technology like .NET Remoting instead. Here’s why:

Security

  • Database servers are extremely powerful. Even if administrative best practices have been adhered to, a database server will still have access to some of the most important information in a company, as well as a lot of powerful functionality. At the same time, these servers implement complex communication protocols that have not originally been invented with security in mind. Regular security problems in common database systems have made it a common assessment that exposing them to the internet directly is not a good idea.
  • DB servers don’t always implement secure access protocols like SSL. Add-on solutions for this purpose alone (VPN?) can be deployed, but make additional infrastructure necessary. Remoting on the other hand can be flexibly configured to provide for secure transport. In .NET 2 this is possible out of the box, but due to the well thought through extensibility model, secure transport can be implemented separately if needed.

Performance

  • Database server protocols usually haven’t been written to work well over slow connections. They do not usually satisfy the requirements for distributed systems communication, as I outlined in this article. The IDataStore interface is structured to comply with these requirements.
  • The extensibility of .NET Remoting allows to use custom channels with arbitrary additional functionality, like compression algorithms, that can further increase performance.

Convenience

  • In many distributed scenarios, you will sooner or later want to have a “module” of your software running on the server anyway. For example, this might be necessary to implement authentication, central workflow management or, much simpler, server-based timestamps. Publishing your IDataStore interface from that server-side module is very convenient and comes with a lot of nice side-effects: it’s easy to do your own dedicated logging, tracing and tracking, which can be very helpful in problem situations.

There’s joy in repetition… there’s joy in repetition…

As long as it’s in the middle of a Prince song, I think that’s true. It’s definitely not true when it applies to everyday work – nobody likes to perform the same task more than once. (Actually I think a good programmer should react immediately in situations where he finds himself repeating something he did before, as that’s a sign of room for optimization.) So when you create a lot of applications that can run in distributed environments, you’ll end up implementing similar code again and again: the code that is particular to the task of making an XPO IDataStore implementation available using Remoting. The XPO Publication service is designed to do that work for you, by being able to publish your service for you, based on a configuration file.

Of course no two applications end up being exactly the same. With regard to Remoting, there might be different requirements  for the channels that are being used, or the published interface may not always be the same (as in the case of the data layer cache). You might not even want to use .NET Remoting to do the publishing. So it is important that the XPO Publication Service offers flexibility in all the right places, to enable you to extend it easily with the particular implementations you need. Of course you can go and hack the source code (and re-hack it when I make an update available, of course – hehe), but you should be able to implement all the important extensions without that. Tell me if that’s not true.

Conclusions

I hope I was able to shed some light on the reasons for using Remoting in the first place, and on the role the XPO Publication service can play. Feel free to ask if there’s anything else you need to know – or to say so if your opinion is different.

Tags
8 comment(s)
Anonymous
Geoff Davis
Oliver,

Ah I think I get it now, so using the XPO publication service one can simply write a server application that incorporates this XPO publishing service and the remoting configuration is done through the use of a predetermined xml file instead of handled through the server application code? This will allow multiple DB backend configuration, ports, etc... to be configured?

If so then this is wicked!!! however I do have a couple of questions that might help me to use this scenario in my application.

I need the ability to add or remove remoting configuration from the xml file on the fly from a UI, what is the best way to go about this? You see my users will be in control of what gets published; mostly they will specify the simple information like DB Type, DB Path and port and the rest will be generic but I don't want them messing with a xml file.

Is there anyway using the XPO Publishing service that the xml file can be read into some sort of collection and then configured giving the option to persist it back to the xml file? Also could we stop/start certain ports either already running or new ones added into the collection?

Does this make any sense or am I going in the wrong direction?

Geoff.
18 October, 2006
Oliver Sturm (DevExpress)
Oliver Sturm (DevExpress)
Hey Geoff
I think you got it, yes :-) The whole functionality is implemented as a library, so you can obviously use it in your own server application. The service I wrote for the project can either be used in the simple cases where you don't need any additional functionality you write yourself, or as a template to start writing your own service.

Changing the config from a file - no, there's no saving functionality so far. Might be interesting to add, you're right. When I have the time...

At the moment, the channels are bound into the publication objects, which is technically not quite correct, as somebody pointed out to me already. This was an attempt at an easier approach to the whole Remoting thing, but I think I'll shift some things around later. Anyway - you should be able to handle each publication on its own and Publish or Unpublish it as you see fit.
19 October, 2006
Anonymous
Alain
Hi Oliver.

I convert my old app that use RemObjects to do some operations that i can do now with XPO publication services.

But I want to know if is possible that using the same philosophy I can call a stored proc (or might be a SELECT * on some VIEW) on db server, and return an XPView, but using your approach of remoting communications and XPO.

in other words, i want to use the same infraestructure of your XPO publication service to do that.

regards, alain
4 March, 2007
Anonymous
XPO

There are no changes in this version -- surprising as it may be, nobody ever reported a single bug in

31 May, 2008
Anonymous
eXpress App Framework Team

This is post no. 10 in the mini series "10 exciting things to know about XAF". You can find

31 May, 2008
Anonymous
Richard B.

Hi Oliver,

I finally get what you are trying to do.  Rest assured, I will be starting with this shortly, as I'm facing this situation at the moment.  I am one step more engrossed in this than what you are, which is I need to do everything the "remoting" will allow, but I also need to be able to have Hybrid collections.  What I mean by Hybrid collections is that if an object is cached on a local SQLCE database, then the user is to use that version BEFORE the remote version.  Then, whatever is not cached in the SQLCE database needs to be pulled from the server db.

I want to take a look further at the provider, and see if there is a way where I can get the provider to do what I need it to do, or if I need to build a method to handle this on my own.

Any suggestions?

23 July, 2008
Oliver Sturm (DevExpress)
Oliver Sturm (DevExpress)

Hi Richard,

This is a complicated scenario. Not going down to the provider level has the advantage of keeping the abstraction, so that collections remain on the high level they are on now. But then your own logic will return a list of objects that come from different databases and so will belong to different sessions, which makes working with the complete set of objects complicated. And remember there are relations - so you would end up having duplicates of related objects around, one from each of the databases you access.

On the other hand, doing this in the provider will not be straight forward either, since the provider structure isn't made to go back to more than one backend. I think it can be done, but there will be a lot of challenges when it comes to modifying data.

A basic question that comes to mind: why don't you set up replication between the main and the mobile database, access only the mobile one and be done with it?

24 July, 2008
Anonymous
Richard B.

Moving this discussion to the forums quick... that way it's a bit easier to find.

30 July, 2008

Please login or register to post comments.