Evaluation means, er, evaluation

13 January 2009

For some reason, this particular topic has reared its head twice in the past few days, so I thought I'd set the record straight about what we mean by the evaluation edition (or trial-run as many call it) of DXperience.

First of all, and most important, the evaluation edition comes with its own clauses in the EULA (End-User License Agreement):

If the SOFTWARE COMPONENT PRODUCT(S) you have obtained is marked as a "TRIAL" or "EVALUATION," you may install one copy of the SOFTWARE COMPONENT PRODUCT(S) for testing purposes for a period of 30 calendar days from the date of installation ("Evaluation Period"). Upon expiration of the Evaluation Period, the SOFTWARE COMPONENT PRODUCT(S) must be uninstalled and all copies destroyed.


Developer End User MAY NOT REDISTRIBUTE any SOFTWARE COMPONENT PRODUCT(s) files if using an evaluation, trial, Not for Resale, or demo version of the SOFTWARE COMPONENT PRODUCT(s).

So, the intent of our evaluation is for the potential customer to test our product with their scenarios to make sure that our product works as expected and, more importantly, can be used within their environment. 30 calendar days is well sufficient for that purpose. Of course, in order that they gain the best results from their testing, we provide support to the potential customer during that period.

Marking off the check listThe way I envisage potential customers doing an evaluation is much like an interview for a position. First of all, I would guess that there might be two or three candidate products in the running. The evaluator will have worked out a list of scenarios that they'll want to test for all candidates, a list of questions, if you like. It's very unlikely that all candidates will pass all questions adequately, so there must be some measure of success ("this question is nice to be answered correctly, but I won't lose sleep over it if no candidate can, but this one is imperative to get right").

Next up, there must be some period of uninterrupted time to set up the evaluations. Each candidate product will have a different API, and so you must spend the time to understand each sufficiently well to get over the basic learning curve. The intent here is to see whether each candidate will work in your environment. You may have, for example, a data layer that cannot be changed. Can each product work with this data layer? Similarly, you may have a particular error-notification system: are the products flexible enough to work with this subsystem? And so on.

The intent here is not to fully plug-in each product into your solution, but to gain an understanding of each candidate's strengths and weaknesses. Part of that understanding is also an evaluation of each product's support system, which is why we gladly provide such support. We want you to buy our products after all.

At the end of the evaluation period you should have your original list of questions with a set of answers against each. A check list, if you like. Using this, you should be able to work out which product would suit you best. You will certainly not know everything about each product, you will certainly not have learnt every fancy little feature and API wrinkle. But you will know enough to make a purchase decision.

So, good evaluations involve real work, sustained over a period of a couple of weeks, well within the 30 day limit. Our evaluations are not for the "play with it a couple of hours here and there, with gaps of a week or more in between" approach. I really don't see how that kind of evaluation helps you or the vendor.

Another approach I've heard of is the "I'm plugging your evaluation product completely into my application, and then I'll buy" method. This is in complete violation of our EULA on several fronts: generally this process will take longer than 30 days, the evaluation edition is being used for development and not testing, and also typically there's more than one developer working on the project. (And of course, the final purchase usually involves a single license.)

I've also heard of the "We only need to buy one or two licenses for the maintenance phase" assumption, the product being developed by a larger team using the evaluation edition. This is so in violation of our EULA, I don't know where to start.

So, we welcome potential customers to try out DXperience. We'd love for you to come to the conclusion that our product is the best choice for you, so we'll help you along the way with our support team. We believe that the best way to do the evaluation is as I described above. And if you do find that, for one reason or another, that you need more time, just email me or support. We'll certainly consider your request.

But please don't take advantage of our evaluation policy. I really don't want to get to the point where we have to put drop-dead dates in our evaluation editions.

13 comment(s)
Colin Mackay

Only in one company have I ever been given the time for a proper evaluation as you describe. In virtually all other companies I've worked for I've been allowed to evaluate products on the hour-here-couple-of-hours-there prinicple as my time is pulled around by the needs of other "more important" projects.

14 January, 2009
Scott Blood

I have the exact same experience as Colin.

You are given a task to do, you find a component that will do this or the literature says it will do it.  You tell your boss, he gives you a couple of hours if your lucky to test it as i dont know many development managers (if your lucky enough to have one) or IT Managers who see the testing of development components as a productive useage of time.

Infact i do remember one IT Manager from a company that shall remain anonymous, who shelled out nearly £3000 on a set of UI components that he had tried for about an hour and seemed to do what he wanted, it did, however, become quickly apparant that they did not do what they were suppose to and it was £3000 down the swanny.

The only way to truelly test a component set is to hit the ground running with it and build it into your application and i guess this is where your 60 day money back guarantee comes in handy.  Why violate an EULA when you can just simply do a paper transfer for a couple of weeks????

14 January, 2009
Gary Short (DevExpress)

We understand that it is not always possible to eveluate a product in the ideal way, that is why we will extend your trial period if you are actively evaluating our tools, Julian was just asking people not to take advantage of our good will. ;-)

14 January, 2009

I recognize your need to earn a living from your product however your point of view on this issue is a perfect example why American business is in so much trouble. You are attempting to enforce your vision on independent companies and assuming that every company can spend $10k to evaluate your product by assigning a developer to it for 3 weeks. This is simply bad business and cost prohibitive. In the real world a developer is allowed to evaluate new products only when they have a slow few hours.

Get with and support your customers with some other policing system that would allow evaluators a certain number of total hours but not a calendar derived clock. I can assure you that sales will go up astronomically if you partner with your customers. Your kind of management is naive and based on "managing to the bad" or stopping the bad guys instead of "managing to the good" whereby you support your loyal base. "Managing to the good" always results in word of mouth reccomendations and those are the ones that actually breed revenue.

Try it and you might just like it.

14 January, 2009

Early the evaluation period was 60 days...

14 January, 2009
David Dillon

My experience worked out just the way you described, Julian.  I had some tests to run, finshed them in 3 weeks, they worked, I bought and began developing.  Thanks for making it possible for me to do this before I was funded for my project.

14 January, 2009

I was very happy with the trial, especially the fact you can get help at a time when you probably need it most. I was mainly trying out the .net data grid which is extremely powerful but does have a learning curve. I trialled some other controls quickly and was so impressed I bought all the controls and coderush and refactor pro without even trying them out. They turn out to save me lots of programming time.

Also the license is flexible it shows you are keeping how developers work in mind. The fact you can legally install on a desktop, laptop and build machine at the same time is very handy.

14 January, 2009
Julian Bucknall (DevExpress)

Jerome: You seem to be under the misapprehension that we're somehow a large megacorporation. Nope, we're a small independent company too. Obvious piracy -- for that is what it is -- like this affects us dearly and closely. After all, we're putting our money where our mouth is by providing full support to you for those 30 days. No conditions on that at all. What I'm describing here is not the odd one or two cases, I'm afraid.

(And, man I'd like the $10K every 3 weeks job: that's $173K a year. If you're paid that much, just buy the product: it's $2K, no eval needed.)

Colin: As Gary said, several times I've extended the eval period. All it takes is a nice email to me; really, I'm quite friendly. It's rather harder to violate the policy and try to cover it up and then complain.

s_tristan: 60 days? I don't think so. We do have an unconditional 60-day money-back guarantee, so that's one way to evaluate it for double the time ;)

Cheers, Julian

14 January, 2009
Julian Bucknall (DevExpress)

Scott, et al: I suppose my viewpoint about evaluations (that I didn't bring out better) is that one of the reasons you are evaluating something is because you want to use the product to save you money and resources. Money in the original use of the product, and money in the inevitable maintenance of the product (bug fixes, enhancements, etc). Obviously the money you spend on evaluating a product, you hope to recoup as your particular application is released and matures. After all, how hard is it to write an ASP.NET grid? Them DevExpress devs do it every couple of years :).

And I hear you about the "I tried it out for an hour, looks good, let's bet the life of the company on it" scenarios. Brrr.

Cheers, Julian

14 January, 2009

I suppose I am the exception and not in your target market, but I am in a foreign company and at an exchange rate of 80:1 US Dollars are a big deal for me.

So when a product costs 2k that is a MASSIVE investment for me.

Evaluating a product for me is a more drawn out situation as I never have enough time to really delve into the features. Case in point I have been trying to evaluate AspxGrid for several months now but life and work intervenes.

Its a dilema so I just go on reputation and my gut

14 January, 2009
Luis Madaleno

"I really don't want to get to the point where we have to put drop-dead dates in our evaluation editions."

You should! After 30 days the trial should stop working.

If the user want to extend the evaluation period you could send a code to extend it. Otherwise, they should buy it.

Just my $0.02.  

As a DX customer since 1999, I would like so see DX healthy for another 10 years. I wonder how many money you loose with piracy...


15 January, 2009
Jaume Julià

Julian, you wrote:

'So, good evaluations involve real work, sustained over a period of a couple of weeks, well within the 30 day limit. Our evaluations are not for the "play with it a couple of hours here and there, with gaps of a week or more in between"'

So why you don't apply this policy to DevExpress?. Put a developer working FULL TIME on the infamous RWA XAF and be done with it. Spending a couple hours here and there is not the way to serve your customers.

16 January, 2009
Bob Marsh

You should put a hard stop in your trail version - my previous company has one full version and 15 developers using the trial version

I have no grudges in posting this

16 January, 2009

Please login or register to post comments.