Will we ever have the kahunies to publish or be damned!!

by Malcolm 19. August 2009 23:29

 

"I shared a vagrant optimism that some of us were making real progress, that we had taken an honest road, and that the best of us would inevitably make it over the top. At the same time, I shared a dark suspicion that the life we were leading was a lost cause, that we were all actors, kidding ourselves along on a senseless odyssey. It was the tension between these two poles - a restless idealism on one hand and a sense of impending doom on the other - that kept me going."

Hunter S Thompson - The rum diary

Publish or be damned is one of those scientific mantras you see sometimes, I believe it stems from needing to get your research out first before some other lab trumps you to it thus consigning you to an interesting foot note on the bibliography of a vegan honors students thesis.

Not a pleasant ending to a life's work.

In software development and testing especially it tends to be more publish and be damned. There is always a get out clause, a reason to not go live with something. In Agile there is this theory of producing potentially shippable code at the end of each sprint. What does that even mean?? I wouldn't be happy if I went to get my car from the mechanic and he told me I have potentially working brakes for example.

Now there are loads of places that will help define what "potentially shippable" is supposed to mean but I suspect that for most people it quickly translates into a BIG POTENTIALLY - and a very small shippable.

Is it shippable as soon as we fix that bug? or is it shippable as soon as we finish these five stories? Or more likely its shippable after an 8 week stabilization phase :-) so it was never potentially shippable at all.

That got me thinking, How would you change your work practices if your customer said to the team "You know what - I like this agile lark - Here are they keys to the live production environment and after every sprint review I want it in live within the hour!"

Can you imagine! A customer that put every sprints work directly into live!

I would probably react with total terror at first but then I started thinking about how the team and the process would change. I am willing to bet for a start that our velocity would half over night as various team members started thinking - "well I cant put that off, This is going live next week!". I would probably go on a one man uber crusade to automate every facet of the acceptance criteria before any of the code was written in case it came to me late and went live before I ever saw it!!

And then I thought "Why the hell am I not doing that now??"

I had a sudden image of a testers perfect world - Its a bad sprint, the developers are running late. Very little has made it through to be tested let alone marked as done and we have 5 stories in progress. Its 3 hours before sprint review and everything gets checked in all at once! - Normally that would be a nightmare scenario!!

[Scrum Master] "So what did we finish to demo?"

[Tester] "Did all the builds pass?"

[Scrum Master]  "Yes"

[Tester] "Demo the lot! Its done!"

 

WiX: Integrating into the continuous build

by John Rayner 17. November 2006 09:35

I've been working on a WiX installer recently and I was asked to integrate it into the project's continuous build process.  This turned out to be surprisingly easy.

Early on in the development, I decided to use WiX version 3.  It is nicely integrated with Visual Studio and has some great property windows for controlling compiler and linker behaviour.  So my development environment for WiX looks like this and I can create my MSI directly from the Build menu (or Ctrl-Shift-B, as I prefer):

wixproj3

The other benefit of WiX 3 is that the main project file (xxx.wixproj) is now actually an MSBuild script (this is a change to prior versions). 

Now our continuous build has CruiseControl.Net calling an MSBuild script (similar to the setup I described in a previous post). This means that once you got the latest binaries and files into the correct folders, getting the WiX project to build is as simple as:

<MSBuild Projects="$(InstallerDirectory)\Installer.wixproj" Targets="Build"/>

How cool is that?

A Windows Service to accept messages via a web service and send them over MSN Messenger

by John Rayner 1. November 2006 09:38

I was working with Russell when he blogged about sending continuous build messages over MSN Messenger.  The implementation that we ended up with at the time suffered from a few drawbacks:

  1. The network we were on had a lot of congestion.  So sometimes the sign-in process to Messenger would time out and return an error to MSBuild.  This would break the CC.Net build [:(]
  2. Every time a message was sent, the code would sign-in to Messenger.  This meant that if you were a recipient you would see the following:

This would work a lot better if there was a Windows Service which signed in to Messenger and held onto the connection.  It could then accept messages from the CC.Net build process and forward them on to the Messenger network.  And this is what I've written.  The service exposes some web services, puts the messages into an in-memory queue and then has a timer which polls this queue and pushes message out over Messenger.  It was then fairly trivial to write a custom MSBuild task which is a proxy for these web services.  You can download the whole lot from here.

Below are some details on installation and configuration, along with some discussions around the technically more interesting bits.

Installation and Configuration

Installation is pretty straightforward:

  1. Copy the entire contents (files and subdirectories) of MessengerSvc\build to where you want them
  2. Run "installutil MessengerSvc.exe" (installutil.exe is part of the .Net SDK)
  3. Open the Services control panel and configure whatever service account, startup properties, etc that you like

A word of warning - I haven't done much testing with different privileges on the service account.  I can't tell you exactly what the minimum required privileges are.  If you do install and run this service then I'd be very ineterested to hear how you got on.

Configuration settings are all contained in the MessengerSvc.exe.config file.  There are comments in there, so I'm not going to describe each setting here.

Exposing the Web Services

This turned out to be fiddly but not overly complex.  We use the ApplicationHost class in the System.Web.Hosting namespace to start it all off.  This creates a new AppDomain and loads an instance of HttpListenerWrapper into it.  This class primarily does what it says on the tin, and wraps the HttpListener class. Our service then creates a background thread which makes a blocking call to HttpListenerWrapper.ProcessRequest which uses the HttpListener class to receive an HTTP request and calls HttpRuntime.ProcessRequest to perform the heavy-lifting of processing the request.

Incoming message requests accepted via web service are recorded in a singleton instance of QueueManager.  The HttpListenerWrapper class has a method which retrieves all the values held in the instance of the QueueManager and returns them.  This method allows us to transfer the requests from the secondary AppDomain (created by ApplicationHost) into our primary AppDomain.  It is called from the background thread created by the service, once the call to HttpListenerWrapper.ProcessRequest has completed.

Sending Messages Over Messenger

Given that I started with the code described by Russell (which comes from Dr.NETjes) which already includes the DotMSN library (available here), I was really standing on the shoulders of giants.  And so the code for actually sending the message over MSN Messenger deals more with time outs, validity periods, exception conditions and retry counts than it does with networking.  All this logic is contained in the QueueManager.PollMessages method.

It's worth pointing out that I did modify the MessengerWrapper class a bit to make it (a) thread-safe; and (b) more aware of the current state of its connection.  A good example of this is in the SendMessage method, where if a login operation is underway then we attempt to wait for it to complete.  And if there isn't a login operation in progress but we aren't logged in, then we start a login operation.  Logic like this is a lot more important in a service that in a straightforward single-threaded executable.

Service Logging

I've used log4net within the service.  There isn't a config file in the download, but you can find out how to write one by going to the official log4net site.  I've tried to aim for some consistency, so you may want to consider logging Error messages from the MessengerSvc.HostService logger into the event log, or some other important place.

Using the Service From MSBuild

Creating a custom build task which makes web service calls was really quite trivial.  If you're looking for a good primer on how to create a custom task, try this one from Waaargh.Net.  There is a basic MSBuild script in the download, which looks like this:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets = "SendMessage">
   <
UsingTask TaskName="MessengerNotification.Tasks.MsnNotification" AssemblyName="MessengerNotification.Tasks" />
   <
Target Name="SendMessage">
      <
MsnNotification 
            Message="Hello, World!"
            Recipients="john.rayner@conchango.com"
            />
   </
Target>
</
Project>

Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets = "SendMessage">
   <
UsingTask TaskName="MessengerNotification.Tasks.MsnNotification" AssemblyName="MessengerNotification.Tasks" />
   <
Target Name="SendMessage">
      <
MsnNotification 
            Message="Hello, World!"
            Recipients="john.rayner@conchango.com"
            />
   </
Target>
</
Project>

 

One of the benefits of using a service is that you don't need to specify the Messenger login details in your build script.  They are located in the service's config file instead. 

Future Enhancements

  1. It occurred to me after writing this code (once I learnt a bit about WCF) that we can replace the entire web service bit with a few choice WCF calls.  If I get time to do this, then I'll make another post about it.
  2. When writing the code to push the messages out onto Messenger, it started to become clear how you could write a Messenger bot, i.e. an application that used Messenger as its UI.  Users could then connect via Messenger to send commands / queries to the service.  This is similar to something we already have at Conchango for accessing the corporate phonebook.
  3. A professional installer is a nice-to-have.  Something that would prompt the user for various details, update the config file with these values and install the service automatically.

If you do make use this service, then please drop me a line (or a comment) to let me know how it works for you.

 

Getting FxCop to break the build

by John Rayner 5. October 2006 10:39

Recently I wrote a long post on our continuous build setup and what we had to do to get it all working.  However, switching on FxCop wasn't quite as simple as I thought it was going to be.

The Problem
You might recall that we are using the FxCop task from the MSBuild Community Tasks project at http://msbuildtasks.tigris.org/ and our build script looked like this:

<FxCop 
    TargetAssemblies="@(FxcopTargets)"
    RuleLibraries="@(FxCopRuleAssemblies)" 
    AnalysisReportFileName="D:\Projects\MyProject\Build\SmokeTest\fxcop.xml"
    DependencyDirectories="$(MSBuildCommunityTasksPath);C:\Program Files\Extreme Optimization\Statistics Library for .NET\bin;D:\Projects\MyProject\Build\Tools\log4net-1.2.10\bin\net\2.0\debug;C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0"
    FailOnError="False"
    ApplyOutXsl="False"
    OutputXslFileName="C:\Program Files\Microsoft FxCop 1.35\Xml\FxCopReport.xsl"
    Verbose="False"
    includeSummaryReport="True"
    ExcludeRules="Microsoft.Design#CA2210;Microsoft.Globalization#CA1303"
   
/>

Now we set FailOnError="True" and thought that would be enough.  Turns out it isn't.  For those who interested, the reasons for this are as follows:

  • Setting FailOnError="True" ensures that the return value of the external FxCop process is returned to MSBuild as a task status (you can see this in the source code for the FxCop task)
  • The FxCop task shells out to FxCopCmd.exe
  • The return values for FxCopCmd.exe are used to indicate a catastrophic failure in the analysis process, not rule violations

In other words, this setting simply ensures that if the code analysis cannot be carried out, then the build will fail.  This is useful, but not what I wanted.

The Solution
All the violations of FxCop rules are written out to an XML file by the above script.  So we have to load that XML file and parse out any violations which we think are important.  Fortunately the MSBuild Community Tasks project include an XmlRead task, which allows you to execute some XPath against a XML document and get the results into an MSBuild property.  Our build script now includes the following:

</PropertyGroup>
    <
FxCopCriticalErrors>0</FxCopCriticalErrors>
    <
FxCopErrors>0</FxCopErrors>
    <
FxCopCriticalWarnings>0</FxCopCriticalWarnings>
</
PropertyGroup>
<XmlRead ContinueOnError="True"
    XmlFileName="D:\Projects\MyProject\Build\SmokeTest\fxcop.xml"
    XPath="string(count(//Issue[@Level='CriticalError']))">
        <
Output TaskParameter="Value" PropertyName="FxCopCriticalErrors" />
</
XmlRead>
<
XmlRead ContinueOnError="True"
    XmlFileName="D:\Projects\MyProject\Build\SmokeTest\fxcop.xml"
    XPath="string(count(//Issue[@Level='Error']))">
        <
Output TaskParameter="Value" PropertyName="FxCopErrors" />
</
XmlRead>
<
XmlRead ContinueOnError="True"
    XmlFileName="D:\Projects\MyProject\Build\SmokeTest\fxcop.xml"
    XPath="string(count(//Issue[@Level='CriticalWarning']))">
        <
Output TaskParameter="Value" PropertyName="FxCopCriticalWarnings" />
</
XmlRead>
<
Error Text="FxCop encountered $(Count) material rule violations"
    Condition="$(FxCopCriticalErrors) &gt; 0 or $(FxCopErrors) &gt; 0 or $(FxCopCriticalWarnings) &gt; 0" />

PropertyGroup>
    <
FxCopCriticalErrors>0</FxCopCriticalErrors>
    <
FxCopErrors>0</FxCopErrors>
    <
FxCopCriticalWarnings>0</FxCopCriticalWarnings>
</
PropertyGroup>
<XmlRead ContinueOnError="True"
    XmlFileName="D:\Projects\MyProject\Build\SmokeTest\fxcop.xml"
    XPath="string(count(//Issue[@Level='CriticalError']))">
        <
Output TaskParameter="Value" PropertyName="FxCopCriticalErrors" />
</
XmlRead>
<
XmlRead ContinueOnError="True"
    XmlFileName="D:\Projects\MyProject\Build\SmokeTest\fxcop.xml"
    XPath="string(count(//Issue[@Level='Error']))">
        <
Output TaskParameter="Value" PropertyName="FxCopErrors" />
</
XmlRead>
<
XmlRead ContinueOnError="True"
    XmlFileName="D:\Projects\MyProject\Build\SmokeTest\fxcop.xml"
    XPath="string(count(//Issue[@Level='CriticalWarning']))">
        <
Output TaskParameter="Value" PropertyName="FxCopCriticalWarnings" />
</
XmlRead>
<
Error Text="FxCop encountered $(Count) material rule violations"
    Condition="$(FxCopCriticalErrors) &gt; 0 or $(FxCopErrors) &gt; 0 or $(FxCopCriticalWarnings) &gt; 0" />

Not only does this work, but it also gives you a lot of control over what you consider important.  For instance, we break the build for Critical Warnings as well as Errors and Critical Errors.  You could also use this script to enforce limits on the types of violations, so you may want to break the build if there are more than 30 warnings.

You may be surprised by the ContinueOnError="True" but this is to cope with the fact the FxCop doesn't produce any XML output if there are no violations.  In this (happy) case, the default property values of 0 will be used as the XmlRead task will not set the property values.