Preparing a 1.5 subversion repo to handle 1.6 clients

When subversion 1.6 was released, one of the changes included was the a set of changes related to enhancing the functionality of the 'svn:externals' property. As part of these changes the back-slash character became an 'escape character'.

Unfortunately if you have been using subversion with windows clients/users it is quite likely some of your developers will have used the 'backslash' to provide the 'local folder' part of the external definition (e.g you have a line such as: 'foo\bar http://my_svn/folder/trunk'.) Sadly subversion 1.6 clients no longer interpret those paths in the traditional way (it has always been recommended not to use the slash in this way, but it did previously work.) This combined with the fact that if you use TortoiseSVN or similar it is likely to have forced an automatic update recently which has somewhat forced our hand with moving to 1.6.

One could change all of the svn:externals properties on all folders at the HEAD of all their branches, and this would help with all *new* revisions, but if you have a requirement to be able to reliably produce retrospective builds (as in any good change-management environment one should be able to) then this will not do (if you were to checkout a revision of the repository prior to your fixes (say an old branch of your product you've shipped to the customer) then those externals would not work as your local svn client would be much newer than when you first built the code :(

To work around this I've created the following snippet of perl that uses the rather handy SVN::DumpFilter to search through a subversion dump file and replace any '\' characters found in svn:externals declarations with a '/' character.

use SVN::Dumpfilter;

$|=1;

sub my_filter (\%;$);
my $dumpfile = shift @ARGV; # filename or '-' for STDIN
my $outfile  = shift @ARGV; # filename or '-' for STDOUT

Dumpfilter($dumpfile, $outfile, \&my_filter);
sub my_filter (\%;$)
 {
   my $href   = shift;
   my $recalc = shift || 0;
   my $header = $href->{'header'};
   my $prop   = $href->{'properties'};
   # Do something (modify, add, delete) with the current node given by the
   # hash ref $href
   # e.g.:
   if( exists $header->{'Revision-number'}) {
 print "Processing revision ".$header->{'Revision-number'}."\n"; 
   }
   if( exists $prop->{'svn:externals'} && $prop->{'svn:externals'} =~ m/\\/)
   {
 print "found external :  ". $prop->{'svn:externals'}."\n";
 $prop->{'svn:externals'} =~ s/\\/\//g;
 print "  ->  converted external :  ". $prop->{'svn:externals'}."\n";
 $recalc= 1;
   }
  
   # The node content is accessible as scalar with ${$href->{content}}
   # Can be in every possible text or binary format.

   if ($recalc)
    {
     svn_recalc_prop_header(%$href);        # call if you changed properties
    }
 }

On an 11Gb 20,000 revision dump file this can take up to 10 minutes to run to completion.

To use, either pipe a subversion dump file to it and pipe the output to a new file, or pass the filename of the dumpfile you wish to filter and the filename of the resulting file (or a crazy mix of both).

Please be careful, this script worked for me and I offer no guarantees and certainly take no responsibility if it causes damage to your repository, but it got me out of a sticky situation.

It should also be noted that as we're not adding any 'new' revisions to the repository, any existing working copies may be a little bit out of 'sync' with the real repository, so should ideally be re-fetched from clean. In our case we could delete the few folders that were affected and re-update so it wasn't too much of a big issue

1.2.3... lets get Nant to respect ANT's error codes

If you've ever struggled to get a NANT build to 'fail' when calling through to ANT via ANT's ant.bat file (which fails to return the error-code seemingly), hopefully the following snippet will help you. This is a snippet of NANT executing a target on an ANT build file.

  <target name="nant-target">
 <!-- The impressively confusing jumping through hoops of re-directing the ant.bat's standard out to a file and then reading that file back in 
       is to get around the fact that the ant.bat file always returns '0' even when the build.xml file has 'failed' *sob* -->
  <property name="ant_results_file" value="ant_result.txt"/>
  <delete file="${ant_results_file}" />
  <exec program="${environment::get-variable('ANT_HOME')}\bin\ant.bat"
   output="${ant_results_file}"
   append="true"
   commandline="-buildfile antfile.xml anttarget" />
  <loadfile file="${ant_results_file}" property="ant_result" />
  <!-- For those voyagers coming this way in the future, we're looking for either a BUILD FAILED or a BUILD SUCCEEDED, that rather crucially
        is the *LAST* one in the file -->
  <regex pattern="(?'build_status'BUILD (FAILED|SUCCEEDED))(?!.*BUILD (FAILED|SUCCEEDED)).*$" 
         input="${ant_result}" 
         options="Singleline"/>
  <delete file="${ant_results_file}" />
  <if test="${build_status == 'BUILD FAILED'}">
   <fail message="ANT target failed"/>
  </if>
 </target>

SQL Server, extracting UTF-16BE data from an image field

If like me you made a choice back in the mists of time to store UTF16-BE data in an image field in SQL Server, you may find the following functions useful.

As a bit of back-story, it seems that SQL Server stores data as UCS2 (roughly UTF16-LE) data, and the convert -> NVARCHAR approach does not recognise any form of binary data other than this :(

To that end the following code swaps the byte-order of a set of bytes stored in an 'image' field.

It will only work on SQL 2005 as I was far to lazy to sort out TEXTPTRs and TEXTREAD etc.

Hope you find it useful:

CREATE FUNCTION dbo.my_convert_image_to_text( @value AS VARBINARY(MAX) ) RETURNS nvarchar(MAX)
BEGIN
RETURN cast( @value as NVARCHAR(MAX) )
END
GO

CREATE FUNCTION dbo.swap_byte_order( @value AS VARBINARY(MAX) ) RETURNS VARBINARY(MAX)
BEGIN
DECLARE @b varbinary(MAX)
DECLARE @i bigint
DECLARE @count bigint
SET @i= 1
SET @b = CAST('' AS varbinary(1))
SET @count= datalength(@value)
WHILE (@i <= @count)
BEGIN
SET @b = @b + CAST( SUBSTRING(@value, @i+1, 1) as binary(1) )
+ CAST( SUBSTRING(@value, @i, 1) as binary(1) )
SET @i = @i + 2
END
RETURN @b
END
GO

CREATE FUNCTION dbo.image_as_string (@value AS VARBINARY(MAX) ) RETURNS nvarchar(MAX)
BEGIN
DECLARE @dataSize bigint
DECLARE @result nvarchar(MAX)

SET @dataSize= datalength(@value)
IF @dataSize >2 AND @dataSize%2 = 0
BEGIN
SET @result= dbo.my_convert_image_to_text(dbo.swap_byte_order(@value))
END
ELSE
SET @result= dbo.my_convert_image_to_text(@value)
RETURN @result
END
GO

select dbo.test_image_as_string(someImageColumn) from someTable

Utf8 Characters Encoding in Javascript Byte Arrays

As ever, just posting this snippet in case it has use for anyone else coming this way. I recently had the need to convert between a Javascript 'byte' array and a string, where the byte array contained a UTF-8 encoded string.

I started with this code but I'm unable to locate the original author to contribute my modifications back to him/her in case they want them.

The resulting code for en-coding and de-coding UTF-8 byte arrays (with or without BOMs) is as follows:

      //************************************************************************************
      // UTF-8 Encoding helpers.
      // based on the code at http://www.webtoolkit.info
      //************************************************************************************
      Utf8Utils= function() {
          function _encode(stringToEncode, insertBOM) {
              stringToEncode = stringToEncode.replace(/\r\n/g,"\n");
              var utftext = [];
              if( insertBOM == true )  {
                  utftext[0]=  0xef;
                  utftext[1]=  0xbb;
                  utftext[2]=  0xbf;
              }

              for (var n = 0; n < stringToEncode.length; n++) {

                  var c = stringToEncode.charCodeAt(n);

                  if (c < 128) {
                      utftext[utftext.length]= c;
                  }
                  else if((c > 127) && (c < 2048)) {
                      utftext[utftext.length]= (c >> 6) | 192;
                      utftext[utftext.length]= (c & 63) | 128;
                  }
                  else {
                      utftext[utftext.length]= (c >> 12) | 224;
                      utftext[utftext.length]= ((c >> 6) & 63) | 128;
                      utftext[utftext.length]= (c & 63) | 128;
                  }

              }
              return utftext;  
          };

          var obj= {
              /**
               * Encode javascript string as utf8 byte array
               */
              encode : function(stringToEncode) {
                  return _encode( stringToEncode, false);
              },
            
              /**
               * Encode javascript string as utf8 byte array, with a BOM at the start
               */
              encodeWithBOM: function(stringToEncode) {
                  return _encode(stringToEncode, true);
              },
            
              /**
               * Decode utf8 byte array to javascript string....
               */
              decode : function(dotNetBytes) {
                  var result= "";
                  var i= 0;
                  var c=c1=c2=0;
                
                  // Perform byte-order check.
                  if( dotNetBytes.length >= 3 ) {
                      if(   (dotNetBytes[0] & 0xef) == 0xef
                          && (dotNetBytes[1] & 0xbb) == 0xbb
                          && (dotNetBytes[2] & 0xbf) == 0xbf ) {
                          // Hmm byte stream has a BOM at the start, we'll skip this.
                          i= 3;
                      }
                  }
                
                  while( i < dotNetBytes.length ) {
                      c= dotNetBytes[i]&0xff;
                    
                      if( c < 128 ) {
                          result+= String.fromCharCode(c);
                          i++;
                      }
                      else if( (c > 191) && (c < 224) ) {
                          if( i+1 >= dotNetBytes.length )
                              throw "Un-expected encoding error, UTF-8 stream truncated, or incorrect";
                          c2= dotNetBytes[i+1]&0xff;
                          result+= String.fromCharCode( ((c&31)<<6) | (c2&63) );
                          i+=2;
                      }
                      else {
                          if( i+2 >= dotNetBytes.length  || i+1 >= dotNetBytes.length )
                              throw "Un-expected encoding error, UTF-8 stream truncated, or incorrect";
                          c2= dotNetBytes[i+1]&0xff;
                          c3= dotNetBytes[i+2]&0xff;
                          result+= String.fromCharCode( ((c&15)<<12) | ((c2&63)<<6) | (c3&63) );
                          i+=3;
                      }          
                  }                
                  return result;
              }
          };
          return obj;
      }();

Please note that I don't *think * it can handle stuff off the basic multi-lingual plain correctly, as that requires 4 bytes, but I don't need that just yet ;), also it doesn't understand Java's Modified-UTF 8 encoding, hopefully that won't matter <g>... incidentally character sets, gah!

A few test snippets :

      var bytes=[0xef,0xbb, 0xbf, 0x54, 0x65, 0x73, 0x74, 0x73, 0xd7, 0x90,0xc2, 0xa2];
      var decoded= Utf8Utils.decode(bytes);
      var encoded= Utf8Utils.encode(decoded);
      alert(  Utf8Utils.decode(encoded) );

      bytes=[0xef,0xbb, 0xbf, 0x54, 0x65, 0x73, 0x74, 0x73, 0xd7, 0x90,0xc2];
      try {
          decoded= Utf8Utils.decode(bytes);
          encoded= Utf8Utils.encode(decoded);
      } catch (err ) {
         alert( "Issue with en/de-coding: " + err );
      }

Cheers.

New AMD64x2 Turion Latop Pain

This is a brief note for google really, if anyone else has problems with a Packard Bell EasyNote MV51 laptop and linux. Namely the x86_64 kernels refusing to start (hanging after "io scheduler cfq registered" ) with Ubuntu Efty Edge (and in fact any 2.6.17 or 2.6.18 64 bit kernel). Meaning you have to run with 'noapic' as one of your kernel options, which drops you down to a single cpu.

The new 2.6.19 rc6 kernel fixes it all!! Hooray, I now have 2 cpu-cores again, and in all its 64 bit loveliness!

Constructing a document literal-wrapped WSDL document

caveat: This document was originally written for an intranet wiki page to aid our team to construct WSDL for ActiveBPEL that was suitable for use by XFire Web Service clients and by .Net Web Service method calls. It was my first attempt at the problem and as such is probably wrong in several ways [one such example is the stuff I'm doing with namespaces, I now know that I should have two namespaces, one for my schemas and one for my definitions (WSDL documents), I didn't know /hadn't discovered this at the time, hence the uri: approach. (sorry!) ]

Anyway, hopefully this will help someone else going down this route :)

Steps to construct a document literal wrapped WSDL document:

• Decide on your service name, and the parameters it takes .
• Parameters seem to be defined in0,in1,in2 and out etc.
• For this example we're going to define a service called sprinkleDust which takes a Pixie entity and returns a tired Pixie.
• Construct your WSDL Document's basic structure
• We need to bring in the standard namespaces/schema definitions:
  

<?xml version="1.0" encoding="UTF-8"?>
<definitions name="SprinkleDust"
  targetNamespace="http://yournamespace"
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:smf="http://yournamespace"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">
</definitions>

• Next we want to define the PortType that defines our service method:

<?xml version="1.0" encoding="UTF-8"?>
<definitions name="SprinkleDust"
  targetNamespace="http://yournamespace"
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:smf="http://yournamespace"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <portType name="SprinkleDustServiceInterface">
      <operation name="sprinkleDust">
      </operation>
  </portType>
</definitions>

• Web methods aren't that useful without inputs and outputs, for this we need to define two WSDL message elements, and tell our operation to use them, note the naming scheme used here ServiceNameRequest and ServiceNameResponse [this is a convention but I don't think it actually matters]. Also note carefully where the prefix has been used.:

<?xml version="1.0" encoding="UTF-8"?>
<definitions name="SprinkleDust"
  targetNamespace="http://yournamespace"
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:smf="http://yournamespace"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <message name="sprinkleDustRequest">
  </message>

  <message name="sprinkleDustResponse">
  </message>

  <portType name="SprinkleDustServiceInterface">
      <operation name="sprinkleDust">
          <input message="smf:sprinkleDustRequest" name="sprinkleDustRequest" />
          <output message="smf:sprinkleDustResponse" name="sprinkleDustResponse" />
      </operation>
  </portType>
</definitions>

• Now in order for the WSDL document to be truly document-literal/wrapped, we need to define our WSDL message elements carefully, they must conform to the following requirements:
• The input message has a single part
• The part is an element (not a type/reference)
• The element has the same name as the operation
• The element's complex type has no attributes
• Because this is a fairly complex set of requirements, first of all we'll flesh out the message elements, then we'll deal with the element it actually uses:
  

<?xml version="1.0" encoding="UTF-8"?>
<definitions name="SprinkleDust"
  targetNamespace="http://yournamespace"
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:smf="http://yournamespace"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:smfprocess='urn:SprinkleDustServiceProcess'>

  <message name="sprinkleDustRequest">
      <part name="parameters" element="smfprocess:sprinkleDust" />
  </message>

  <message name="sprinkleDustResponse">
      <part name="parameters"  element="smfprocess:sprinkleDustResponse"/>
  </message>

  <portType name="SprinkleDustServiceInterface">
      <operation name="sprinkleDust">
          <input message="smf:sprinkleDustRequest" name="sprinkleDustRequest" />
          <output message="smf:sprinkleDustResponse" name="sprinkleDustResponse" />
      </operation>
  </portType>
</definitions>

• Looking at the above WSDL you should be able to see we satisfy the previously defined constraints:
• The input message has a single part - in this case named parameters
• The part is an element (not a type/reference) - the element referenced is smfprocess:sprinkleDust or smfprocess:sprinkleDustResponse
• The element has the same name as the operation - smfprocess:sprinkleDust is the same name as the operation's name ( sprinkleDust )
• The element's complex type has no attributes - Can't see this just yet, but it doesn't, honest.
• Now we need to pull in the element's we're referring to that make up the document that is sent and received to/from the web service method:
  

<?xml version="1.0" encoding="UTF-8"?>
<definitions name="SprinkleDust"
  targetNamespace="http://yournamespace"
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:smf="http://yournamespace"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:smfprocess='urn:SprinkleDustServiceProcess'>
  <types>
      <xs:schema targetNamespace="urn:SprinkleDustServiceProcess">
          <xs:import schemaLocation="SAAll.xsd" namespace="http://yournamespace"/>
          <xs:element name="sprinkleDust" >
              <xs:complexType>
                  <xs:sequence>
                      <xs:element name="in0"  type="smf:Pixie" nillable="true" minOccurs="1" maxOccurs="1"/>
                  </xs:sequence>
              </xs:complexType>
          </xs:element>

          <xs:element name="sprinkleDustResponse">
              <xs:complexType>
                  <xs:sequence>
                      <xs:element name="out" type="smf:Pixie" nillable="true" minOccurs="1" maxOccurs="1"/>
                  </xs:sequence>
              </xs:complexType>
          </xs:element>            
      </xs:schema>
      </types>
  <message name="sprinkleDustRequest">
      <part name="parameters" element="smfprocess:sprinkleDust" />
  </message>

  <message name="sprinkleDustResponse">
      <part name="parameters"  element="smfprocess:sprinkleDustResponse"/>
  </message>

  <portType name="SprinkleDustServiceInterface">
      <operation name="sprinkleDust">
          <input message="smf:sprinkleDustRequest" name="sprinkleDustRequest" />
          <output message="smf:sprinkleDustResponse" name="sprinkleDustResponse" />
      </operation>
  </portType>
</definitions>

• This step was quite complex, we've added a types section, that defines our document-literal/wrapped elements, however it belongs to its own namespace. This is because in this WSDL file we're importing an external Schema that contains the definitions of our Pixie complex types and there are some special semantics around namespaces an imports + includes.

Last but not least, If we want our service to be used by other BPEL processes we need to add in a PartnerLink element, so our structure now looks like:

<?xml version="1.0" encoding="UTF-8"?>
<definitions name="SprinkleDust"
  targetNamespace="http://yournamespace"
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:smf="http://yournamespace"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:smfprocess='urn:SprinkleDustServiceProcess'
      xmlns:plnk="http://schemas.xmlsoap.org/ws/2003/05/partner-link/">
  <types>
      <xs:schema targetNamespace="urn:SprinkleDustServiceProcess">
          <xs:import schemaLocation="SAAll.xsd" namespace="http://yournamespace"/>
          <xs:element name="sprinkleDust" >
              <xs:complexType>
                  <xs:sequence>
                      <xs:element name="in0"  type="smf:Pixie" nillable="true" minOccurs="1" maxOccurs="1"/>
                  </xs:sequence>
              </xs:complexType>
          </xs:element>

          <xs:element name="sprinkleDustResponse">
              <xs:complexType>
                  <xs:sequence>
                      <xs:element name="out" type="smf:Pixie" nillable="true" minOccurs="1" maxOccurs="1"/>
                  </xs:sequence>
              </xs:complexType>
          </xs:element>            
      </xs:schema>
      </types>
  <message name="sprinkleDustRequest">
      <part name="parameters" element="smfprocess:sprinkleDust" />
  </message>

  <message name="sprinkleDustResponse">
      <part name="parameters"  element="smfprocess:sprinkleDustResponse"/>
  </message>

  <portType name="SprinkleDustServiceInterface">
      <operation name="sprinkleDust">
          <input message="smf:sprinkleDustRequest" name="sprinkleDustRequest" />
          <output message="smf:sprinkleDustResponse" name="sprinkleDustResponse" />
      </operation>
  </portType>

  <plnk:partnerLinkType name="sprinkleDustPartnerLink">
      <plnk:role name="User">
          <plnk:portType name="smf:SprinkleDustServiceInterface" />
      </plnk:role>
  </plnk:partnerLinkType>
</definitions>

svn:externals and moving servers

Having just gone through the pain of moving our subversion repository off a machine that died on us, we discovered, or rather re-discovered the pain of (ab)using svn:externals to refer internally to the repository. i.e. we had a series of 'svn:externals' properties pointing to the server that went down.

To fix this I wrote the following ruby script to iterate over a checked-out repository and replace any existing svn:externals references to the current server with a new address. This post is really meant just to hold the ruby code I wrote so I can find it in the future ! :)

directory="";
Dir.foreach( directory ) do |x|
        unless x== "." or x == ".." or x ==".svn"
            if  File.directory?( directory + "/" + x )
                # Do the externals check here.
                externals= `svn propget svn:externals "#{directory}/#{x}"`.split("\n");
                if externals.length > 0
                    puts "--------------  #{directory}/#{x} -------- "
               
                    # Swap the bala for svn.
                    externals.collect! do |line|
                        line.gsub!( /oldServer:81/, 'newServer:81' );
                    end
               
                    # Write the new svn:externals property out to a file to use with the propset...
                    File.open("externals.txt","w") do |file|
                        file.puts( externals.join("\n") );
                    end
               
                    # Apply the new svn:externals property.
                    `svn propset svn:externals "#{directory}/#{x}" --file externals.txt`
                end
                manipulateDirectory( directory+"/"+x );
            else
            end
        end
end

end

manipulateDirectory( "." ) ;