XPO

October 2006 - Posts

  • Publication service: a very simple example

    I was trying to stick this XML snippet in a comment on my article with the announcement of the XPO publication service, but CS is too dumb to leave the formatting of a <pre> section intact on comments. So I’m posting it here:

    <?xml version="1.0" encoding="utf-8" standalone="yes"?>
    <XPOPublication>
      <Publication Name="TestPublication1"
        SourceConnectionString="XpoProvider=MSAccess;Provider=Microsoft.Jet.OLEDB.4.0;Mode=Share Deny None;data source=test.mdb;user id=Admin;password=;"
        SourceAutoCreateOption="DatabaseAndSchema"
        Target="http://localhost:2635/something.rem" />
    </XPOPublication>
    

    So this simple piece of configuration would be all you need to publish an Access database via an HTTP Remoting channel.

  • XPO Publication: Why?

    Geoff Davis commented on my announcement post: I just don’t get it! Not a problem, Geoff, and certainly not your fault – I think the topic hasn’t been discussed in depth so far and Kenneth’s newsgroup post was a bit short. So, let’s start with “more datastore aware than DB aware” (a phrase Kenneth mentioned in his post). First of all let me say that I didn’t come up with that sentence, so I can only guess – but I have a pretty good idea what was meant here.

    Abstracting databases through IDataStore

    As you know, XPO can work with various DB backends, while leaving your code, the code that’s written on top of it, unchanged. So how does it do this? Pretty simple: it uses two adaptation layers between the “user code” and the database specific driver code. In practice, you may actually count more layers than that – depends on what you consider “user code”, and what kind of database you use (ADO.NET can be said to provide abstraction layers of its own).

    In any case, the lowest level of XPO code that has to do with database abstraction is the implementation of the “connection providers”. These classes are each specific to a database backend, and they implement an interface called IDataStore. This is the declaration of the IDataStore interface:

    public interface IDataStore {
      AutoCreateOption AutoCreateOption { get; }
      ModificationResult ModifyData(params ModificationStatement[] dmlStatements);
      SelectedData SelectData(params SelectStatement[] selects);
      UpdateSchemaResult UpdateSchema(bool dontCreateIfFirstTableNotExist, params DBTable[] tables);
    }
    

    The most important thing to note about this interface is that its data structures are no longer dependent on any specific database. So the “connection providers” contain code to “translate” database specific implementation details into this non-specific interface lingo – in other words, looking from the top, I can talk the same language to every class out there that implements IDataStore, regardless of what happens inside the class. A really common thing with interfaces, but nevertheless something that’s kind of easy to miss.

    Now what does it mean to be “more datastore aware than DB aware”? It means “think along the lines of what the IDataStore interface provides”. It doesn’t matter what a data store (i.e. a class that implements IDataStore) does internally. In fact it is possible to implement IDataStore and do something completely different with the data that is passed through the interface methods – this is used a lot in implementations like the DataStoreLogger, where a thin class is chained together with other IDataStore.

    A final thing worth mentioning again about IDataStore is that is is structured to work well within distributed systems. I went into this in some depth in my previous posts XPO is good for distributed applications and Using .NET Remoting with XPO. Nevertheless, XPO always uses data stores as a level of abstraction, and it doesn’t matter whether it’s done locally or via a Remoting (or other) remote connection.

    So why do I want to use Remoting?

    So, back to that feature Kenneth was requesting. Let’s assume we are actually working on a distributed application, and so we need to enable the clients to contact the server on a different physical system. Theoretically, we could just open up the server (the DB server itself, that is) via the network and let the clients access it. This is not a good idea (Microsoft says so (right at the top, "Firewalls"), so does Oracle (page 14)). I propose to use a technology like .NET Remoting instead. Here’s why:

    Security

    • Database servers are extremely powerful. Even if administrative best practices have been adhered to, a database server will still have access to some of the most important information in a company, as well as a lot of powerful functionality. At the same time, these servers implement complex communication protocols that have not originally been invented with security in mind. Regular security problems in common database systems have made it a common assessment that exposing them to the internet directly is not a good idea.
    • DB servers don’t always implement secure access protocols like SSL. Add-on solutions for this purpose alone (VPN?) can be deployed, but make additional infrastructure necessary. Remoting on the other hand can be flexibly configured to provide for secure transport. In .NET 2 this is possible out of the box, but due to the well thought through extensibility model, secure transport can be implemented separately if needed.

    Performance

    • Database server protocols usually haven’t been written to work well over slow connections. They do not usually satisfy the requirements for distributed systems communication, as I outlined in this article. The IDataStore interface is structured to comply with these requirements.
    • The extensibility of .NET Remoting allows to use custom channels with arbitrary additional functionality, like compression algorithms, that can further increase performance.

    Convenience

    • In many distributed scenarios, you will sooner or later want to have a “module” of your software running on the server anyway. For example, this might be necessary to implement authentication, central workflow management or, much simpler, server-based timestamps. Publishing your IDataStore interface from that server-side module is very convenient and comes with a lot of nice side-effects: it’s easy to do your own dedicated logging, tracing and tracking, which can be very helpful in problem situations.

    There’s joy in repetition… there’s joy in repetition…

    As long as it’s in the middle of a Prince song, I think that’s true. It’s definitely not true when it applies to everyday work – nobody likes to perform the same task more than once. (Actually I think a good programmer should react immediately in situations where he finds himself repeating something he did before, as that’s a sign of room for optimization.) So when you create a lot of applications that can run in distributed environments, you’ll end up implementing similar code again and again: the code that is particular to the task of making an XPO IDataStore implementation available using Remoting. The XPO Publication service is designed to do that work for you, by being able to publish your service for you, based on a configuration file.

    Of course no two applications end up being exactly the same. With regard to Remoting, there might be different requirements  for the channels that are being used, or the published interface may not always be the same (as in the case of the data layer cache). You might not even want to use .NET Remoting to do the publishing. So it is important that the XPO Publication Service offers flexibility in all the right places, to enable you to extend it easily with the particular implementations you need. Of course you can go and hack the source code (and re-hack it when I make an update available, of course – hehe), but you should be able to implement all the important extensions without that. Tell me if that’s not true.

    Conclusions

    I hope I was able to shed some light on the reasons for using Remoting in the first place, and on the role the XPO Publication service can play. Feel free to ask if there’s anything else you need to know – or to say so if your opinion is different.

  • Announcing the XPO Publication service

    Prompted by a customer’s request (thanks, Kenneth!), I have created a publication service for XPO data stores. I’m making the source code of the project available, so it may serve as a sample of a number of techniques.

    The downloads

    If you want to recompile the projects or simply have a look at the source code or the functional tests I wrote, you’ll need the source code archive: XPOPublicationService-1.0.0.0.zip

    If all you need is the binary program (you can still add your own extensions, at least you should be able to), you can get the binary archive instead: XPOPublicationService-1.0.0.0-bin.zip

    Requirements of the project

    Starting from Kenneths original request, I worked with the following requirements:

    • The resulting program should be usable as a Windows service, to publish via .NET Remoting an XPO compatible data store (IDataStore).
    • It should be configurable (without recompiling) to work with various sources for this publication.
    • It should also support further extensions, such as defining Remoting channels to use, and chaining IDataStore providers, as required by the XPO data layer caching system or the data store logger.
    • A requirement I found myself during initial planning is that it must be extensible “externally”, so that another developer can create IDataStore implementations or Remoting channels in a separate assembly and use them in a publication, without having to change the source of the XPO Publication service.

    So far I’ve left out the idea of publishing as a Web Service instead of via Remoting – I guess it might be possible to extend in that direction, if I find the time I’ll look into it.

    So what's in the archives?

    The source code archive contains a Visual Studio 2005 solution with four projects:

    • XPOPublication is the name of the functional implementation, which comes in the form of a class library, to make it easy to reuse in various contexts.
    • XPOPublication.Tests is an assembly of NUnit tests, which contains functional tests of various parts of the implementation.
    • Publish is a console application that lets you easily run publications from the command line.
    • XPOPublicationService is a Windows service wrapper for the publication functionality.

    The binary archive includes the compiled versions of Publish.exe, XPOPublicationService.exe and XPOPublication.dll, as well as the configuration files for both executables and a sample publications.xml file (for details of configuration see below).

    Configuration

    Configuration of the publications is done via an XML file. The name of the file can be passed to the console application on the command line, otherwise a file called publications.xml in the current directory is used. For the Windows service, the name of the publication configuration file is taken from the application configuration file (XPOPublicationService.exe.config), and it is pre-configured to “..\..\publications.xml” for testing purposes.

    The publication configuration file itself has roughly the following format (no, I haven’t created a schema for it… it’s very dynamic in large parts, so I didn’t really see the point in that):

    <?xml version="1.0" encoding="utf-8" standalone="yes"?>
    <XPOPublication>
      <Publication Name="MyPublication">
        <Source>
          < … source class definition … />
        </Source>
        <Target>
          < … target class definition … />
        </Target>
        <Links>
          < … link class definition 1 … />
          < … more link class definitions … />
        </Links>
      </Publication>
    
      … more publications …
    </XPOPublication>
    

    So what are all those “… class definition” things? Well, here I took a page out of XAML’s book. Pretty simple, really: the classes mentioned in these sections are instantiated directly from the XML code when the config file is read. That’s a good idea because it leaves us with the greatest flexibility – a class referenced in the config file needn’t be one that the tool developer (me!) ever heard of before.

    The ConnectionStringSource

    For example, there’s a predefined class called ConnectionStringSource, which uses a database connection string together with the XpoDefault.GetDataStore() method to create a connection provider for the publication. To use this class, the Source section might look something like this:

        <Source>
          <xpopub:ConnectionStringSource>
            <ConnectionString> … database connection string here … </ConnectionString>
            <AutoCreateOption>DatabaseAndSchema</AutoCreateOption>
          </xpopub:ConnectionStringSource>
        </Source>
    

    In addition to this code, it is necessary to let the loader know where the class actually comes from. Note the "xpopub" prefix I'm using – this corresponds to an xmlns mapping that needs to be included on the XPOPublication node, like this:

    <?xml version="1.0" encoding="utf-8" standalone="yes"?>
    <XPOPublication
      xmlns:xpopub="clr-namespace:XPOPublication">
      …
    

    Again, this is the same syntax XAML uses for namespace and assembly references. The “assembly=” part of the mapping is also supported, if needed, so this allows you to reference classes in your configuration file that live in assemblies separate from my distribution of the tool.

    As I recognize that the ConnectionStringSource will probably be the most used source, I have added attributes on the Publication element that allow to specify the two parameters of the class. So if there’s a SourceConnectionString attribute on the Publication class, the Source section (if there is one) will be ignored! The corresponding attribute for the AutoCreateOption is called SourceAutoCreateOption. To be clear, the above sample could be written like this instead:

    <?xml version="1.0" encoding="utf-8" standalone="yes"?>
    <XPOPublication>
      <Publication Name="MyPublication"
        SourceConnectionString=" … database connection string here … "
        SourceAutoCreateOption="DatabaseAndSchema">
      … no source section! …
    

    The RemotingTarget

    The “standard” (probably most used) target is called RemotingTarget. A Target section utilizing the RemotingTarget class could look like this:

        <Target>
          <xpopub:RemotingTarget>
            <Channels>
              <xpopub:HTTPTargetChannel>
                <Port>2635</Port>
              </xpopub:HTTPTargetChannel>
            </Channels>
            <ServiceName>publication.rem</ServiceName>
          </xpopub:RemotingTarget>
          <RemotableClassCreation><xpopub:CacheRootRemotableClassCreation/></RemotableClassCreation>
        </Target>
    

    As you can see, a Target definition uses the same system again to reference classes. The HTTPTargetChannel, of course, creates an HttpChannel for Remoting. There are also the TCPTargetChannel and the IPCTargetChannel, the latter with a property called “PortName” instead of “Port”.

    The last line of the Target section defines the RemotableClassCreation strategy by using the CacheRootRemotableClassCreation implementation. This results in a CacheRoot being created, and the published object should then be accessed via the ICacheToCacheCommunicationCore interface instead of the simple IDataStore (this article about caching in XPO explains this in some more detail). The standard strategy for class creation is called DefaultRemotableClassCreation and it is automatically used if the RemotableClassCreation element is simply left out from the config file.

    Finally, similar to the Source section above, there’s also a convenience attribute for the Target section. With the exception of the RemotableClassCreation element, the above configuration could more easily be written like this:

    <?xml version="1.0" encoding="utf-8" standalone="yes"?>
    <XPOPublication>
      <Publication Name="MyPublication"
         Target="http://localhost:2635/publication.rem">
      … no target section! …
    

    Also similar to above, if the Target attribute is there, a Target section will be ignored – so for advanced things like the RemotableClassCreation, or to set the EnsureSecurity property on the channels, the full syntax must be used. But the convenience system is extensible and it’s possible to register your own recognition algorithm…

    The Links

    Finally, there may be elements that need to be inserted into the chain of IDataStore implementations. These elements can be specified in the Links section of the configuration. There’s only one such element in the box, which is the FileLoggerLink. It includes a DataStoreLogger object into the chain, configured to log to a file. A Links section with a FileLoggerLink would look like this:

        <Links>
          <xpopub:FileLoggerLink>
            <FileName> … file name with path here … </FileName>
          </xpopub:FileLoggerLink>
        </Links>
    

    When there’s more than one link defined, the order of links in the chain may become important. By default there’s no guarantee about the order in which the links would become part of the chain. But each chain link class should have a property that lets you define the order of links in the chain. For the FileLoggerLink this is called ChainIndex, which I’d like to propose as a standard if you’re going to create your own link classes. If you use this index value, it’s your own responsibility to make sure that you use the index values correctly – the library simply sorts the links by this index and instantiates them in that sorted order.

    Troubleshooting

    The library is instrumented in the most important places to provide Trace (System.Diagnostics) output. A trace switch is in use called XPOPublication, which you can configure in your application config file. The default application config files for Publish.exe and XPOPublicationService.exe already contain the necessary XML code, and the level is configured to 4 (everything) for Publish.exe and 2 (Warnings and Errors) for XPOPublicationService.exe. Likewise, Publish.exe is pre-configured to log to the console, and XPOPublicationService.exe logs to the system event log by default. Change these settings as needed to find problems.

    I have purposefully handled exceptions sparingly in the library. In the console application, any exception messages are written out to the console, and in the Windows service they make it through to the event log.

    It’s beta, even if it’s called 1.0

    Let me conclude this post by saying that this software has not been tested extensively. I find this important to mention for two reasons:

    1. As always on this blog, I want your feedback. Please tell me what you think, what you don’t understand, that my code sucks, what suggestions you have.
    2. If you use the code in a real-world deployment and everything crashes around your ears, don’t look to me. I won’t feel responsible. Let me state this clearly: the code is untested in real-world scenarios and it has to be regarded as sample code only. You are yourself responsible for any problems, if you deploy it.
  • Upcoming feature: UpCasting

    If you have created persistent class hierarchies, specifically to bind to UIs, you probably know the situation: sometimes you’d like to combine base and derived classes in a single query, but it’s not easy to do or even impossible because the derived classes obviously have an extended set of properties.

    To make this clearer, imagine these classes (once again, let me point out that the properties aren't implemented in their full recommended syntax for brevity’s sake):

      public class Thing : XPObject {
        public Thing(Session session) : base(session) { } 
        
        private string name;
        public string Name {
          get { return name; }
          set { name = value; }
      }
    
      public class SpecialThing : Thing { 
        public SpecialThing(Session session) : base(session) { } 
        
        private int specialDetail;
        public int SpecialDetail {
          get { return specialDetail; }
          set { specialDetail = value; }
        }
      }
    

    Now, in the simplest case you might want to create a collection of all Things, including those of the derived type, and access all their properties. This is a really common thing when the data is being bound to a user interface. So you create a collection of the base class type:

      XPCollection<Thing> coll = new XPCollection<Thing>(session);

    The problem is that because this collection is typed for Thing, until now you were only able to access the properties of that type. So even if the collection contained elements that were really of the derived type SpecialThing, you were not able to access the property SpecialDetail, because that property was not known to the base class type. As a workaround, your only chance was to code custom property descriptors – I wrote a blog article about this way back – it’s a possibility, but cumbersome to do for ad-hoc needs.

    The good news is, with that new feature we’ll make available in XPO 6.3, you’ll no longer need to write any code for this. Instead, you’ll simply modify the DisplayableProperties property of the XPCollection instance to something like “Oid;Name;<SpecialThing>SpecialDetail”. The “Oid;Name” part is the normal thing – you get that by default. The “<SpecialThing>SpecialDetail” is what’s new here: it describes a property on a type different from the base class type.

    The feature is pretty much universal. In addition to DisplayableProperties, you can also use the same syntax in criteria, like so:

      XPCollection<Thing> coll = new XPCollection<Thing>(session, 
        new BinaryOperator("<SpecialThing>SpecialDetail", 42));
    

    Of course the XPView is also compatible with this. I’m sure it’ll be useful in some situations.

LIVE CHAT

Chat is one of the many ways you can contact members of the DevExpress Team.
We are available Monday-Friday between 7:30am and 4:30pm Pacific Time.

If you need additional product information, write to us at info@devexpress.com or call us at +1 (818) 844-3383

FOLLOW US

DevExpress engineers feature-complete Presentation Controls, IDE Productivity Tools, Business Application Frameworks, and Reporting Systems for Visual Studio, along with high-performance HTML JS Mobile Frameworks for developers targeting iOS, Android and Windows Phone. Whether using WPF, ASP.NET, WinForms, HTML5 or Windows 10, DevExpress tools help you build and deliver your best in the shortest time possible.

Copyright © 1998-2017 Developer Express Inc.
All trademarks or registered trademarks are property of their respective owners