Alexander Brett

Bundling App::SFDC for fun and profit

12 August 2015

Motivation

Whilst installing perl and App::SFDC along with a (quite large) number of dependancies is fun, effective and powerful, it’s not always the best solution for Salesforce deployment tools. When you’re deploying from throwaway AWS instances or sending your tools to developers in foreign countries who want something that just works now, you may want to provide a ready-to-go bundle of code. Fortunately, programs such as PerlApp provide a pretty good way to achieve this.

I’m going to run through how to bundle App::SFDC to a standalone .exe suitable for deploying and retrieving metadata on a windows machine.

Introduction to PerlApp

The idea behind PerlApp is pretty straightforward: you point it at a script, and it calculates the module dependancies and bundles the perl interpreter along with all required modules into a .exe, which can then be run without a local perl installation - essentially, you run perlapp --exe SFDC.exe C:\perl64\site\bin\SFDC.pl .

Loading prerequisites

Of course, when you do this and run the resulting executable, there are some modules missing - it’s hard to detect all of the prerequisites, especially when they’re being dynamically loaded in. Examples of this are that WWW::SFDC loads in modules by running:

for my $module (qw'
    Apex Constants Metadata Partner Tooling
'){
    has $module,
      is => 'ro',
      lazy => 1,
      default => sub {
        my $self = shift;
        require "WWW/SFDC/$module.pm"; ## no critic
        "WWW::SFDC::$module"->new(session => $self);
      };
  }

In a similar way, when you create a screen appender for Log::Log4perl , it quietly loads in Log::Log4perl::Appender::Screen. To fix this sort of issue, we add a few more arguments to perlapp:

perlapp  --add MooX::Options::Role^
 --add App::SFDC::Role::^
 --add Log::Log4perl::Appender::Screen^
 --add WWW::SFDC::^
 --exe SFDC.exe C:\perl64\site\bin\SFDC.pl

Fixing SSL certification

Perl isn’t great at picking up a system’s SSL settings, especially installed certificates - and when the entire purpose of a script is to send HTTPS requests, it’s something that you just have to get right - lest you get errors like 500 C:\Users\ALEXAN~1\AppData\Local\Temp\pdk-alexanderbrett/Mozilla/CA/cacert.pem on disk corrupt at /<C:\Dev\App-SFDC\SFDC.exe>WWW/SFDC.pm line 66..

One successful workaround I’ve found to this sort of error, which works whenever curl is installed, is to use curl’s Certificate Authority file instead of perl’s. You can find this by running curl -v https://login.salesforce.com >nul and looking for the lines like:

* successfully set certificate verify locations:
*   CAfile: C:\Program Files (x86)\Git\bin\curl-ca-bundle.crt

Then, you set HTTPS_CA_FILE=C:\Program Files (x86)\Git\bin\curl-ca-bundle.crt and your HTTPS connections start working again. This amount of manual faffing around is more than most developers or AWS images want to do, and fortunately PerlApp has our back again - we can bind in arbitrary files, and specifiy arbitrary environment variables. Let’s add more arguments to perlapp:

...
--bind certs/cafile.crt[file="C:\Program Files (x86)\Git\bin\curl-ca-bundle.crt",text,mode=666]^
--env HTTPS_CA_FILE=certs/cafile.crt^
...

Binding Retrieve plugins

Since we’re going to be wanting to use App::SFDC::Command::Retrieve, we need to make sure the plugins and manifests mentioned are, in fact, included. By default they are installed to the perl share/ location, and PerlApp won’t see them! This is how to bind in the default values:

...
--bind manifests/base.xml[file=C:\perl64\site\lib\auto\Share\dist\App-SFDC-Metadata\manifests\base.xml,text,mode=666]^
--bind manifests/all.xml[file=C:\perl64\site\lib\auto\Share\dist\App-SFDC-Metadata\manifests\all.xml,text,mode=666]^
--bind plugins/retrieve.plugins.pm[file=C:\perl64\site\lib\auto\Share\dist\App-SFDC-Metadata\plugins\retrieve.plugins.pm,text,mode=777]^
...

We should also ensure any dependencies from retrieve.plugins.pm are loaded:

...
--scan C:\perl64\site\lib\auto\Share\dist\App-SFDC-Metadata\plugins\retrieve.plugins.pm
...

This is the point at which you may want to override these values! If you have specific requirements for your manifests, for the folders you want to retrieve, or anything like that, create your own versions of those files and bundle those in instead.

Why doesn’t it work yet?

This was all great up to v0.13, but now this approach stopped working from v0.14 onwards. At that point I moved from a monolithic everything-in-one package approach to a dynamically loading plugin-oriented architecture, which allows anybody to create a command by naming their package App::SFDC::Command::Foo. The code that makes that happen is:

find
   {
       wanted => sub {push @commands, $1 if m'App/SFDC/Command/(\w*)\.pm'},
       no_chdir => 1
   },
   grep {-e} map {$_.'/App/SFDC'} @INC;

…and this approach is completely broken by PerlApp - when running this, you get the error invalid top directory at /<C:\Dev\App-SFDC\SFDC.exe>File/Find.pm line 472., because PerlApp doesn’t create any recognisable directory structure for bundled modules - it provides an overloaded version of require which gets the required module from somewhere non-obvious.

After trying a few different things, it seems that the simplest way to achieve a nicely-bundled .exe is going to be to write a new script which avoids the pitfalls of detecting commands at runtime. We can, in fact, write a small perl program which writes the script for us (compare the output of this to SFDC.pl - it’s the same idea, but static):

#!perl
use strict;
use warnings;
use 5.12.0;
use App::SFDC;

my $commandArrayDefinition = 'my @commands = ("'
    . (join '","', @App::SFDC::commands) . '");';

say <<'HEAD';
package SFDC;
use strict;
use warnings;
HEAD

say "use App::SFDC::Command::$_;" for @App::SFDC::commands;

say 'my @commands = ("'
        . (join '","', @App::SFDC::commands)
        . '");';

say <<'BODY';

my $usage = join "\n\n",
    "SFDC: Tools for interacting with Salesforce.com",
    "Available commands:",
    (join "\n", map {"\t$_"} @commands),
    "For more detail, run: SFDC <command> --help";

my $command = shift;
exit 1 unless do {
    if ($command) {
        if (my ($correct_command) = grep {/^$command$/i} @commands) {
            "App::SFDC::Command::$correct_command"->new_with_options->execute();
        } else {
            print $usage;
            0;
        }
    } else {
        print $usage;
    }
}
BODY

__END__

Tying it all together

Using perl -x in a batch file, we can combine the perl script-writing script and the call to PerlApp into one easy-to-digest package, by using some syntax like:

perl -x %0 > static_SFDC.pl

perlapp ^
 ...
 --info CompanyName=Sophos;LegalCopyright="This software is Copyright (c) 2015 by Sophos Limited https://www.sophos.com/. This is free software, licensed under the MIT (X11) License"^
 --norunlib --force --exe SFDC.exe static_SFDC.pl

goto :endofperl

#!perl
use strict;

...

__END__

:endofperl

For a full version, I’ve created a gist to play with.

Tags: SFDC Perl

Creating attachments with WWW::SFDC

27 April 2015

At my company, we have a problem with attachments; namely, one of the things we have to do most releases is update some mail merge templates which are stored as attachments against Salesforce records. Historically, this has been a great cause for error; at some point the release manager ends up with a sea of .docx files in some directory or other, having to remember which ones she’s already uploaded, which ones need to be uploaded against which records, and so on.

Checking these files into source control helped; now, instead of a soup of randomly named files kicking around, at least she knows she has the latest version, and has an easy way to ascertain which ones have changed. It’s still a tedious and error-prone manual process though, and one slip-up means that a distributor recieves a million dollars worth of junk on their license schedule.

Fortunately, the correct automation solution is at hand: WWW::SFDC to the rescue. Let’s imagine we’ve been sensible and we’ve stored a load of .docx and .xslx files inside source control already, and with even more foresight, we have named each file the same as the Name on the Doc_Template__c record against which we need to store them. What we need to do is create Attachment records for each changed document against each relevent record.

Let’s automate it in 10 easy steps.

  1. Get the changed files:

     git diff --name-only --diff-filter=AM --relative attachments/Doc_Template__c/
     # Or, if you'd rather just have everything (for instance, populating a new sandbox)
     git ls-files attachments/Doc_Template__c/
    
  2. Basic perl script setup, including making sure creds are set up:

     use strict;
     use warnings;
     use Getopt::Long;
     use WWW::SFDC::Partner;
     use File::Slurp 'read_file';
    
     my %creds = (
       url => 'https://login.salesforce.com'
     );
    
     GetOptions
       'p|password:s'    => \$creds{password},
       'url:s'           => \$creds{url},
       'u|username:s'    => \$creds{username};
          
     WWW::SFDC::Partner->instance(creds=>\%creds);
    
  3. Read in the changed files, checking that they exist and trimming whitespace:

     my @filenames = grep {chomp and -e } <>;
    
     exit unless scalar @filenames; # if there aren't any, we have no work to do.
    
  4. Parse the filenames and read the files. We’ll store the file contents as base64-encoded data which is what the Salesforce.com partner API expects up to provide in the body field. Fortunately, SOAP::Lite makes this a breeze, via the SOAP::Data->type() method.

     my @documents = map {
       /([^\/]*)\.(docx|xlsx)$/
         ? +{                             # +{} forces this to be a hashref, rather than an arbitrary code block
             name => $1,
             extension => $2,
             body => SOAP::Data->name(
                     body => read_file $_ # read_file is exported by File::Slurp and does what it says on the tin
                 )->type('base64')        # prepackaged SOAP::Lite magic.
           }
         : ();                            # return empty list. 
     } @filenames;
    
  5. We’re need the IDs of the Doc_Template__c records to store these against, so we’ll start by constructing a where clause…

     my $whereClause = join " OR ", map { "Name='$$_{name}'" } @documents; # that was easy
    
  6. …and we’ll go and execute that query.

     my @parentRecords = WWW::SFDC::Partner->instance()->query(
       "SELECT Id, Name FROM Doc_Template__c WHERE $whereClause"
     );
    
  7. We’re going to need to look up IDs from the name, so we’ll create a hash:

     my %parentIds = map { $_->{Name} => $_->{Id} } @parentRecords;
    
  8. The interesting bit. From each document, create an Attachment suitable for passing into a create call.

     my @recordsToCreate = map {
         $parentIds{$_->{name}} # check that the record exists;
         ? +{
             type => 'Attachment',
             ParentId => $parentIds{$_->{name}},
             name => "$$_{name}.$$_{extension}",
             body => $_->{body},
         }
         : ()
     } @documents;
    
  9. Wrap it all up with a create call

     WWW::SFDC::Partner->instance()->create(@recordsToCreate);
    
  10. Put it all together (tada!):

    git diff --name-only --diff-filter=AM --relative attachments/Doc_Template__c/^
     | perl automagic_attachments.pl -u myUsername -p passwordPlusToken
    

Now, this may feel trivial. However, having repeatable, automatic, guaranteed-error-free deployments of templates every month saves up hours of effort on release day, and hours of tracking down bugs later on.

Tags: SFDC Perl

A New Pattern for Apex Triggers

24 April 2015

This is going to be a pretty long post. If you want the meaty bit, scroll down to ‘Solving the problem by using object-orientation properly’

A brief history of trigger patterns

1: The anti-pattern

I feel like an organisation or developer will go through different stages as they learn better ways of dealing with Apex Triggers. A rookie developer in a simple instance with only 501 training might write triggers that look like this:

trigger QuoteTaxUpdate on Quote (before insert, before update) {
  for (Quote q : Trigger.new) {
    if(Trigger.isInsert || Trigger.oldMap.get(q.Id).Value__c != q.Value__c) {
      q.Tax__c = TaxCalculator.calculateTaxes(q);
      // ... more logic goes here
    }
  }
}
trigger QuoteRelatedOpportunityUpdate on Quote (after insert, after update) {
  for (Quote q : Trigger.new) {
    // Do Stuff
  }
}

There are several major problems with this approach:

  • Having multiple triggers on the same object means that it’s hard to isolate the code file causing problematic behaviour
  • It can be hard to debug complex interactions between triggered processes
  • Once you hit a couple of hundred objects with a handful of processes on each, you have a huge number of source code files to track

2: One trigger per object

With this in mind, the tendancy is to combine triggers so that each object has precisely one trigger class:

trigger QuoteTrigger on Quote (before insert, before update, after insert, after update) {
  if (Trigger.isBefore) {
    if (Trigger.isInsert) {
      //Do some things
    } else if (Trigger.isUpdate) {
      //Do other things
    }
  } else if (Trigger.isAfter) {
    if (Trigger.isInsert) {
      // More stuff
    } else if (Trigger.isUpdate) {
      // You're getting the hang of this...
    }
  }
}

Having now refactored your code into some beautiful, well-organised triggers, you notice that the stuff you’re doing in the Before Update and Before Insert is close enough that you want a method to parameterise - but you can’t have methods in triggers. Additionally, this makes it very difficult to unit test a trigger. All this leads to:

3: The Trigger Handler

You move your logic into its own class, defin

public class QuoteTriggerHandler {
  public static void onBeforeUpdate () {
    sharedMethod();
  }
  
  public static void onAfterUpdate () {
    sharedMethod();
  }
  
  // ...
  
  public sharedMethod(){
    if (Trigger.isInsert) {
      // logic goes here
    }
  }
}

This is great, because you can have some unit tests, some shared methods, and so on and so forth. However, you often find that you’re looping through the records once to choose which ones to update the taxes on, again to work out the new owner, a third time for some dynamic sharing rules, and before long you run through the ‘total lines of code executed in one transaction’ governor limit (or your customers complain about your Quotes taking agonising seconds to save).

4: The Advanced Trigger Handler

How do we fix this? We need some architecture. Let’s set it up so that we only iterate over the records once to ascertain which ones are appropriate to our needs, then each process will need to iterate over fewer records. Once you’ve got more than a couple of jobs, this really pays off. Additionally, it’s very easy to create shared and testable logic like this, because you require that each process has its own method which you can unit test to your heart’s content.

public class QuoteTriggerHandler {
	private List<Quote> QuotesForTaxUpdate = new List<Quote>();
	private List<Quote> QuotesForOpportunityOwnerUpdate = new List<Quote>();
	// more lists

	private Boolean DoTaxUpdate = false;
	private Boolean DoOpportunityOwnerUpdate;
	// more toggles

	public void onBeforeUpdate () {
		DoTaxUpdate = true;
		chooseRecords();
		doLogic();
	}

	public void onAfterInsert () {
		DoOpportunityUpdate = true;
		chooseRecords();
		doLogic();
	}

	public void chooseRecords () {
		for (Quote q : Trigger.new) {
			if (DoTaxUpdate && /*conditions*/) {
				QuotesForTaxUpdate.add(q);
			}
			if (DoOpportunityUpdate && /*conditions*/) {
				QuotesForOpportunityUpdate.add(q);
			}
		}
	}

	public void doLogic () {
		if (DoTaxUpdate)
			TaxUpdateLogic(QuotesForTaxUpdate);
		if (DoOpportunityUpdate) 
			OpportunityUpdateLogic(QuotesForOpportunityUpdate);
	}
}

So why don’t I like this? There are several remaining problems.

  • Your trigger handler will probably be somewhere between quite long and REALLY REALLY long. If I’ve identified a bug in a particular operation, I don’t want to sift through a thousand lines of code to find the method implementing that operation, and try to work out whether the bug is there, or in the if clause somewhere inside chooseRecords() (which is probably a hundred lines long all by itself).
  • This pattern is quite complicated, and it’s going to take a good while for your new hires to be confident enough to dive into said several-hundred-lines long chunk of code and make changes, out of sheer fear of breaking things.
  • You’re going to have lots of code which looks similar in the trigger handler for every object, which smells.

Solving the problem by using object orientation properly

A rough outline

If you take another look at the example above, you’ll see that we’ve basically created a contract for each TriggerOperation (I’m starting to outline the names I’ll use in the solution). This is:

  • An OperationName, which tells people what you’re trying to do - for instance, UpdateTaxes or UpdateOpportunityOwners.
  • An InitialLoopOperation, which will normally select an appropriate subset of records for your operation
  • Some OperationLogic, which is executed at the end.

With this is mind, it seems like a TriggerOperation would be a very good candidate for creating a generic class, which could then be extended and overridden, but which crucially would define a standard interface. This means that when I know roughly what’s gone wrong, I only have one small class to check through for code errors, and when a new developer want to write a new process, he knows where to start without reading reams of documentation and still breaking the build for everyone else.

The second part of this puzzle is going to have to be a TriggerOperationDispatcher, which consumes the TriggerOperation implementations and calls their operations as applicable. This class can then be shared between every SObject, reducing duplication and meaning that if you want to add functionality, you can do it once and benefit everywhere (I’ll go into this idea more further on).

Sample Implementation (MKI)

With the introductions out of the way, let’s give an outline of the classes. It’s going to get very code-heavy hereafter:

// This SHOULD have a type parameter to allow derivation for
// a particular object, but that was deprecated in API v26 :(
public virtual TriggerOperation {
	public abstract void initialLoopOperation (SObject obj);

	public void execute() {
		this.logic();
	}

	// protected means that execute() can access the
	// overridden version in the implementation class
	protected abstract void logic();
}
public TriggerOperationDispatcher {

	private List<TriggerOperation> operations = new List<TriggerOperation>();

	private Boolean isUpdate;
	private Boolean isBefore;
	private Map<Id,SObject> oldMap;
	//etc.
	
	public TriggerOperationDispatcher () {
		this(
			Trigger.newMap, Trigger.oldMap, Trigger.isUpdate,
			Trigger.isInsert, Trigger.isDelete, Trigger.isUndelete,
			//etc
		);
	}
	
	// We're going to want to carefully unit test this class, and
	// this constructor is an easy way to simulate trigger conditions
	@testVisible 
	private TriggerOperationDispatcher (Map<Id,SObject> newMap, /*etc*/) {
		this.newMap = newMap;
		//and so on...
	}

	public TriggerOperationDispatcher addOperation (TriggerOperation op) {
		operations.add(op);
		return this; //chainable
	}

	public void dispatch () {
		List<SObject> relevantObjects = (isInsert || isUpdate || isUndelete)
			? newList
			: oldList;
		
		for (SObject obj : relevantObjects)
			for (TriggerOperation op : operations)
				operation.initialLoopOperation(obj);
		
		for (TriggerOperation op : operations)
			operation.execute();
	}
}

Let’s look at how we’d refactor the QuoteTrigger to take account of this:

trigger QuoteTrigger on Quote (before insert, /*blah blah*/) {
	TriggerOperationDispatcher dispatcher = new TriggerOperationDispatcher();
	if (Trigger.isBefore) {
		if (Trigger.isInsert)
			dispatcher.addOperation(new QuoteTaxOperation(Trigger.oldMap))
				.addOperation(new SomeOtherOperation());
		if (Trigger.isUpdate)
			// The rest of this is predictable
	} else if () {
	} //etc.
}

and to make our QuoteTaxOperation:

public class QuoteTaxOperation : TriggerOperation {
	private List<Quote> QuotesForTaxUpdate;
	private Map<Id,Quote> oldMap;

	public QuoteTaxOperation (Map<Id,Quote> oldMap) {
		this.oldMap = oldMap;
	}
	
	public void override initialLoopOperation (SObject obj) {
		Quote q = (Quote)obj;
		if (oldMap == null || q.Value__c != oldMap.get(q.Id).Value__c)
			QuotesForTaxUpdate.add(q);
	}

	protected void logic () {
		for (Quote q : QuotesForTaxUpdate) {
			calculateTax(q);
		}
	}
}

Turning stuff on and off

Let’s say we’ve just updated a sandbox and we’re going to go through every email field and add something to the end, so that our customers don’t get spammed whilst we test. We’ve got an awesome apex script to accomplish this, but if all our trigger run, it’ll take forever to perform the update! We need to turn off the operations. Luckily, most of the framework for this is in place.

What we need is:

  • A catalogue of operations
public enum TriggerOperationName {
	QUOTE_UPDATE_TAX,
	QUOTE_UPDATE_OPPORTUNITY_OWNER,
	...
}
  • Each operation to be named
public virtual class TriggerOperation {
	public TriggerOperationName name;
}
public class QuoteTaxOperation {
	public QuoteTaxOperation (...) {
		this.name = TriggerOperationName.QUOTE_UPDATE_TAX
		...
	}
	
	...
}
  • A static variable to store disabled operations
public class TriggerOperationManager {
	private static Set<TriggerOperationName> disabledOperations = new List<TriggerOperationName>();
	
	private static Boolean allOperationsDisabled = false;
	
	public static void disableOperation (TriggerOperationName op) {
		disabledOperations.add(op);

	public static void disableAllOperations () {
		allOperationsDisabled = true;
	}
	
	public static boolean isOperationDisabled (TriggerOperation op) {
		return (allOperationsDisabled || disabledOperations.contains(op.name));
	}
}
  • A check for whether to use that operation
//in TriggerOperationDispatcher
	public TriggerOperationDispatcher addOperation (TriggerOperation op) {
		if (!TriggerOperationManager.isOperationDisabled(op)) operations.add(op);
		return this; //chainable
	}

… well, that was pretty easy. We can now exercise fine-grained control over whether to turn things on or off. Want to do that bulk job from anonymous apex? Chuck TriggerOperationManager.disableAllOperations() at the top of your script! Of course, you could also create a custom setting so that operations were disabled on certain profiles (such as your data migration user?) or similar.

This demonstrates the power of using object-oriented patterns well. By using inheritance and by having a single shared dispatcher class, we can achieve powerful org-wide changes with only a couple of dozen lines of code.

Logging

It’s obviously good to know what you’ve done in a transaction. What if we modified these classes to help us do that, even when we’ve burnt waaaaaaaay past the 2MB logging limit?

You could create a new object called Operation_Log__c, and a custom setting called Operation_Logging_Settings__c (to temporarily turn on logging per-profile).

Then, create a class called TriggerOperationLogger with a private list of TriggerOperationName and methods markAsExecuted and logExecution. At the end of TriggerOperationDispatcher.dispatch(), add a call to logExecution, and at the end of TriggerOperation.execute() add a call to markAsExecuted.

Again, you could achieve powerful and flexible logging of trigger operations across the entire org with only a handful of changes.

Summary

Most of the apex code I’ve seen fails to leverage even simple object-oriented concepts such as inheritance, but making use them can lead to elegant and powerful designs which save effort and increase maintainability. Give it a try!

Have I got something wrong? Propose an edit to this page on GitHub.

Tags: SFDC Apex

Running unit tests using WWW::SFDC

02 February 2015

This post is going to start basic and get gradually more complicated - I suggest that you stop once you’ve satisfied your own requirements! The basic premise is that, in order to keep a consistantly high-quality and deployable set of metadata, we keep a Salesforce.com org up to date with a branch, and nightly (or more often) we run every unit test against it. This makes sure that, even though developers are keeping abreast of the effects their changes have on other parts of the code base and so on, there is one extra highly visible and reportable checksum on quality.

Fortunately, the tooling API makes our job quite easy. If you have simple needs, the following 3 statements should be sufficient for running your tests:

my $parentJobId = WWW::SFDC::Tooling->instance()->runTestsAsynchronous(
  map {
    $_->{Id}
  } WWW::SFDC::Tooling->instance()->query(
    "Select Id FROM ApexClass WHERE NamespacePrefix = ''"
  )
);

sleep 60 while WWW::SFDC::Tooling->instance()->query(
  "SELECT Id, Status, ApexClassId FROM ApexTestQueueItem"
  . " WHERE ParentJobId = '$parentJobId' AND ("
  . " Status='Queued' OR Status='Processing'"
  . " OR Status='Preparing' OR Status='Holding')"
);

my @results = WWW::SFDC::Tooling->instance()->query(
  "SELECT Id, Outcome, MethodName, Message, StackTrace, ApexClass.Name "
  ."FROM ApexTestResult WHERE AsyncApexJobId = '$parentJobId'"
);

However, this is far from a perfect implementation. The first thing to notice is that we’re just passing every Apex class into runTestsAsynchronous, which is probably a highly inefficient way to do things (according to my measurements, it adds about 5% to the total time for the tests). It would be more elegant and quicker if we filtered the results of the first query to find tests classes, and, fortunately, we can do this using the SymbolTable field on ApexClass - a class needs to be enqueued if it, or any of its methods, have the TEST modifier. This can be achieve thus:

sub isTestModified {
  my $thing = shift;
  defined $thing->{modifiers} and (
    $thing->{modifiers} eq 'TEST'
    or (
      ref $thing->{modifiers} eq 'ARRAY'
      and grep {$_ eq 'TEST'} @{$thing->{modifiers}}
     )
   );
}

sub filterTests {
  defined $_->{SymbolTable}->{methods} and (
    (
      ref $_->{SymbolTable}->{methods} eq 'ARRAY'
      and grep {isTestModified($_)} @{$_->{SymbolTable}->{methods}}
    ) or (
      ref $_->{SymbolTable}->{methods} eq 'HASH'
      and isTestModified($_->{SymbolTable}->{methods})
    )
  )
}

my $parentJobId = WWW::SFDC::Tooling->instance()->runTestsAsynchronous(
  map { $_->{Id} } grep { filterTests } WWW::SFDC::Tooling->query(
    "Select Id, SymbolTable FROM ApexClass WHERE NamespacePrefix = ''"
  )
);

Now, you’re pretty happy with how this is working, but sometimes this query seemingly-randomly times out. The reason this happens is that if you have deployed one or more Apex classes since the last compilation, requesting the SymbolTables triggers a behind-the-scenes recompilation of your entire code base, which will take longer that the 120s timeout once you get to a certain size. My solution to this was just a brute-force retry mechanism, which can also mitigate any brief networking issues on the client (it suck if you’re halfway through a several-hour test run and you get no results because the was a momentary VPN lapse…), and I achieved it by replacing all the calls to WWW::SFDC::Tooling->instance()->query() with retryQuery():

sub retryQuery {
  my $query = shift;
  my @result;
  for (0..5) {
    eval { @result = WWW::SFDC::Tooling->instance()->query($query); };
    next if $@;
    return (scalar @result == 1 and not defined $result[0] ? undef : @result);
  }
  die "There was an error retrieving the information from Salesforce\nQuery: $query\nError: $@";
}

Bear in mind that I’ve recently modified WWW::SFDC to automatically re-authenticate in the event of a session timeout, which is extremely useful for long test runs!

At this point you feel ready to put your code on a continuous integration server and let it rip. When I did this I ran into a wierd problem where the perl process used up all of the available RAM and started thrashing swap space, taking 7 minutes to even start off the tests; it turns out that the SymbolTable for every class, all at the same time, is quite a mouthful to deserialise. The obvious solution is to query in batches, but the tooling API, as far as I can tell, does not support paged queries in the same way as the Partner API does. Queue more grep {} map {} grep {} chaining:

my $parentJobId = WWW::SFDC::Tooling->instance()->runTestsAsynchronous(
  map { $_->{Id} } grep { filterTests } map {
    retryQuery(
	"SELECT Id, SymbolTable FROM ApexClass WHERE NamespacePrefix = ''"
	. " LIMIT 200 OFFSET $_"
    )
  } grep {
    $_%200 == 0
  } 0..(scalar retryQuery(
    "Select Id FROM ApexClass WHERE NamespacePrefix = ''"
    ) - 1)
);

Now we have a robust and efficient way to run all of our unit tests on SFDC. The last thing I wanted to do was to have all of these test results aggregated and reported on with Atlassian Bamboo. Bamboo comes with a build-in JUnit parser, and JUnit has a fairly simple syntax, so for me the path of least resistance was to be a terrible-person and roll my own XML formatter:

sub jUnitFormat {
  my $result = shift;
  my $className = $$result{ApexClass}{Name};
  my $methodName = $$result{MethodName} eq "<compile>"
     ? "CompileFailed"
     : $$result{MethodName};
  return "<testcase name='$methodName' classname='$className' "
    . "status='$$result{Outcome}'>"
    . (defined $$result{Message}
      ? "<failure><![CDATA[$$result{StackTrace}\n$$result{Message}]]></failure>"
      : "")
    . "</testcase>";
}

{
  open my $fh, ">", $output;
  print $fh join "\n",
    '<?xml version="1.0" encoding="UTF-8"?>',
    '<testsuite name="SFDC Unit Tests">',
    (map {jUnitFormat($_)} @results),
    '</testsuite>';
  close $fh;
}

…easy peasy.

I’ve uploaded the final version as a runnable perl script as a gist - I very much encourage you to give it a try, and maybe even help me come up with more improvements.

Tags: SFDC Perl

Why Perl for Salesforce Tools?

23 January 2015

This explanation is going to take the form of a long story.

When I started writing developer tools for Salesforce.com, I got hold of the ant salesforce deployment library and went wild with it, with the ant zip task, loads of exec calls to git, and some extreme contortions to allow me to switch manifest files depending on whether we wanted to retrieve everything or just everything a developer normally needs. I quickly hit some situations where ant just wasn’t expressive enough for my needs:

  • Deploying to multiple environments and adding a -v for validation flag prompted me to write this post about mixing in some javascript with my ant.
  • Our QA org has some different outbound message endpoints from production. Since the GNU coreutils are installed alongside git, using perl for this was a no-brainer:
<exec executable="perl">
	<arg value="-i.bak"/>
	<arg value="-pe"/>
	<arg value="s/&lt;endpointUrl&gt;https:\\\/\\\/.*\\.sophos\\.com\\\/&lt;endpointUrl&gt;https:\\\/\\\/${endpoint}\/"/>
	<arg value="src/workflows/*.workflow"/>
</exec>

Once I’d got over these, I found that I needed to convert a list of files into a package.xml file. This turned out to be far too complicated for inline scripting or a perl one-liner, so I wrote a separate javascript file called list2xml.js and called it like:

<script src="lib/list2xml.js"
            input="${fileList}"
            outputProperty="${xmlBody}" />

However, the javascript I wrote for this turned out to have terrible performance. To process a file you need to:

  • strip off any src/ prefix
  • split up the file path
  • remove the file extension
  • get the metadata type (for a folder called classes/ the metadata type is ApexClass, for instance) which is effectively a hash lookup
  • store each component name against its metadata type
  • dedupe the resulting arrays
  • write out the xml string

This is quite a lot of string manipulation and array munging, and javascript is not great at this, especially running in Nashorn/JDK1.8. I had quite a few developers sitting me about how long all this was taking, so eventually I realised I needed to reimplement it in some faster language. Given the performance hit was in string processing, I came to the obvious conclusion: no language has faster string processing than Perl, and all my users already had it installed! The reimplemented list2xml eventually became manifest.pm.

After coming across a few of each of these situations, my codebase was a mess of xml with intermingled javascript, perl system calls, and perl and javascript files. Not only was it inconsistent, it was also entirely untested and essentially untestable, since ant is not designed for heavy lifting and as such has no testing framework, and nor does javascript (well, it probably does, but not a well-known and easy-to-use one). My users were back at my desk demanding fewer bugs.

My boss and I sat down to examine the options: we wanted ease of development and maintenance, high performance and a robust unit testing framework as a minimum. We narrowed the selection down to C# and Perl because of the skills available, and then did a head-to-head on a string processing algorithm; not only did Perl come out conclusively faster, it also had the advantage of not requiring a compilation stage, being usable for our contractors on their macs, and already having critical pain areas implemented in it.

I also found that juggling several different APIs at once was a pleasure using Perl, as was unit testing; I’m feeling pretty happy about the interface I’ve written to the metadata API, and I’m going to demonstrate some really powerful uses for the other APIs over the next couple of weeks.

I was pretty apprehensive about Perl before I started to use it in earnest; most of the reactions I get to it are along the lines of “It’s a write-only language”, “It isn’t maintainable”, “It’s out-of-date” and so on; but I’ve found that it’s quite easy. and in fact quite fun, to write highly performant, highly legible, well-documented and unit tested code.

Tags: Perl SFDC

WWW::SFDC - Metadata, Manifests and Zip Files

21 January 2015

In my last post I provided a brief introduction to WWW::SFDC, a Perl wrapper around Salesforce’s API interfaces. Today I thought I’d look slightly more in-depth at WWW::SFDC::Metadata, WWW::SFDC::Manifest, and WWW::SFDC::Zip, and how they interact with each other.

The motivation for including the Manifest and Zip modules in addition to the core API interaction modules was that even after a mechanism is provided for interfacing with the metadata API, the fact remains that deploy and retrieve accept and return .zip files as base64-encoded strings, and deploy requires a package.xml file which is populated according to a stringent set of rules, and so on: writing the logic around these has consumed a large proportion of my time as a tooling developer, and will be the same for every use case, so it makes sense to tie these in to the API interface.

The idea behind the Manifest object is to make it extremely easy to juggle lists of files, package.xml manifest files, and the inputs and outputs of the metadata API calls. For instance, the simplest way to generate empty manifest files is now:

WWW::SFDC::Manifest->new()->writeToFile('src/package.xml');

If we want to generate a custom package based on a list of modified files can look something like this:

my @filesToAdd = `git diff --name-only release master`;
WWW::SFDC::Manifest->new()->addList(@filesToAdd);

The last major problem that this package solves is the generation of a list of files, given a package, bearing in mind that certain metadata types require -meta.xml files, and others are in folders. To build a valid .zip file for deployment, you need to know exactly which files to include, and you can do this thus:

WWW::SFDC::Zip::makezip(
  'src/',
  WWW::SFDC::Manifest->new()->readFromFile('src/package.xml')->getFileList(),
  'package.xml';
);

This object then plays extremely well with the metadata API functions. If you want to retrieve every file in your org, you’d normally need to write out a package.xml including every document and email template you cared about. With the listMetadata call, you can just list the folders you care about, and you can chain this call with the appropriate manifest methods, to get extremely powerful one-liners such as:

WWW::SFDC::Zip::unzip(
  'src/',
  WWW::SFDC::Metadata->instance()->retrieveMetadata(
    WWW::SFDC::Manifest->new()->readFromFile('manifests/basic.xml')->add(
      WWW::SFDC::Metadata->instance()->listMetadata(
        {type => 'Document', folder => 'Developer_Documents'},
        {type => 'Document', folder => 'Documents'},
        {type => 'Document', folder => 'Invoices'},
        {type => 'Document', folder => 'Lead_Images'},
        {type => 'Document', folder => 'Logos'},
        {type => 'Document', folder => 'Tab_Images'},
        {type => 'EmailTemplate', folder => 'Asset'},
        {type => 'EmailTemplate', folder => 'Contact_User'},
        {type => 'EmailTemplate', folder => 'Error_Reporting'},
        {type => 'EmailTemplate', folder => 'Marketing_Templates'},
        {type => 'EmailTemplate', folder => 'Support_Templates'},
        {type => 'Report', folder => 'Merge_Reports'},
        {type => 'Report', folder => 'Finance_Reports'},
      )
    )->manifest()
  )
);

And once you’ve done this, you can deploy them all again like this:

WWW::SFDC::Metadata->instance()->deployMetadata(
  WWW::SFDC::Zip::makezip(
    'src/',
    WWW::SFDC::Manifest->new()->readFromFile('src/package.xml')->getFileList(),
    'package.xml';
  ),
  {
    singlePackage => 'true',
    $IS_VALIDATE ? checkOnly => 'true' : (),
    rollbackOnError => 'true',
  }
);

Of course, if you dynamically regenerate your package.xml, you probably won’t check it into source control, and you don’t want to take it as the truth when working out what to deploy. I actually construct my zip file like this:

WWW::SFDC::Zip::makezip(
  'src/',
  WWW::SFDC::Manifest->new()->addList(`git ls-files src/`)->getFileList(),
  'package.xml';
);

One final element of note is that WWW::SFDC::Zip::unzip accepts an optional third parameter: a function reference, applied to each file retrieved before being written to disk. I use this to achieve profile compression (see my recent post on that topic) like this:

sub _compressProfile {
  my $content = shift;
  my @lines = split /^/, $content;
  for (@lines) {
    s/\r//g;			            # remove all CR characters
    s/\t/    /g;		          # replace all tabs with 4 spaces
    if (/^\s/) {		          # ignore the the xml root node
      s/\n//;                 # remove newlines
      s/^    (?=<(?!\/))/\n/;	# insert newlines where appropriate
      s/^(    )+//;		        # trim remaining whitespace
    }
  }
  return join "", @lines;
}

sub _retrieveTimeMetadataChanges {
  my ($path, $content) = @_;
  $content = _compressProfile $content if $path =~ /\.profile|\.permissionset/;
  return $content
}

MAIN: {
  WWW::SFDC::Zip::unzip(
    'src/',
    WWW::SFDC::Metadata->instance()->retrieveMetadata($manifest->manifest()),
    &_retrieveTimeMetadataChanges
  );
}

I think that covers the main uses for those modules, and to those like me who have been grappling for months with the quirks of the ant deployment tool, the benefits of using a real programming language to achieve these tasks with minimum fuss are really obvious.

Tags: Perl SFDC

Introducing WWW::SFDC - a perl module for Salesforce.com interactions

19 January 2015

I’ve been working with the ‘Force.com migration tool’ - an ant wrapper around the Metadata API - for almost a year now, and I’ve been getting increasingly more frustrated with attempting to coerce ant (a tool for compiling java code) to be a deployment, retrieval, and git integration controller. It just isn’t meant for that purpose!

This means that when I have a list of EmailTemplate folders and want to retrieve all the templates within them, I have to spend 200 lines of xml to:

  • use some inline javascript to get the login credentials from a file depending which environment is specified
  • call a cSharp executable to ascertain whether the login credentials are valid, otherwise the user will get frozen out of Salesforce.com because of too many failed login attempts
  • call listMetadata once for each folder, which outputs the results to a file
  • call a perl script to parse those files
  • call another perl script to write the manifest file
  • call retrieve (finally!)

This is a terrible way to operate for any organisation moderately invested in getting a proper handle on SFDC metadata, and is also really difficult to test in any meaningful manner.

The issues I was having with ant inspired me to write a new perl module, WWW::SFDC, for interacting with Salesforce.com APIs. The aim is to provide Do-What-I-Mean wrappers around all of the SFDC APIs, such that they all work together and achieving moderately complicated interactions is as quick and easy as possible. The module is only about 40% complete, if you consider ‘complete’ to mean ‘has a wrapper around every function in the Partner, Metadata and Tooling API and is unit tested’, but it’s got enough to achieve some seriously useful stuff. For instance, the above-mentioned retrieve task can now be accomplished like this:

WWW::SFDC::Metadata->instance(
    password  => $password,
    username  => $username,
    url       => $url
);

WWW::SFDC::Zip::unzip(
	"src",
	WWW::SFDC::Metadata->instance()->retrieveMetadata(
		WWW::SFDC::Manifest->new()->add(
			WWW::SFDC::Metadata->instance()->listMetadata(
				{type => 'Document', folder => 'Apps'},
				{type => 'Document', folder => 'Developer_Documents'},
				{type => 'Document', folder => 'Documents'},
				{type => 'EmailTemplate', folder => 'Sales_Templates'},
				{type => 'Report', folder => 'Merge_Reports'},
			)
		)
	)
);

…which I think is pretty neat. There are some basic examples in SFDC.pm, and I intend to make some blog posts demonstrating things that I already use this module for. I’m not planning to release in on CPAN until it’s more complete, but it is installable from git.

Note that it also currently has a dependency on Logging::Trivial: I’m not sure what I’m going to do with this, since it’s difficult to differentiate Logging::Trivial amongst all the other logging modules, but it is my favourite! I may eventually rename Logging::Trivial, or change the logging engine to Logging::Minimal.

Tags: SFDC Perl

How to handle whitespace with Salesforce.com and git

15 January 2015

A common problem when working in a git repository in a cross-platform environment is end-of-line handling, as testified by the number of stackoverflow questions on the topic! I found that the most useful guide to getting whitespace right in a repository was github’s, but that there were some additional concerns when working with Salesforce.com.

Firstly, it’s important to bear in mind that SFDC provides all of its text files with unix-style (LF) line endings, and I think that the path of least resistance is to stick with what they provide! However, if you’re a windows shop, your developers are probably using tools which introduce windows-style line endings (CRLF) into the files which they touch. The problem with letting this go unchecked is that you are liable to end up with a huge number of merge conflicts which are extremely frustrating, and eventually you put --ignore-space-change or --ignore-whitespace on every git command.

The first recommendation of github’s guide is to set core.autocrlf=true and call it a day. However, you must not do this! The reason why not is that when you retrieve from SFDC, your static resources are saved as src/staticresources/foo.resource, and git does not by default recognise that these are binary files. This means if you just set up autocrlf, git will mangle these files by deleting bytes which it thinks are CR characters and are in fact useful information, and then SFDC will stop being able to read the files.

So the correct solution is to set up a .gitattributes file in the root of your git repository which looks a lot like this:

# ensure all salesforce code is normalised to LF upon commit      
*.cls text=auto eol=lf                                            
*.xml text=auto eol=lf                                            
*.profile text=auto eol=lf                                        
*.permissionset text=auto eol=lf                                  
*.layout text=auto eol=lf                                         
*.queue text=auto eol=lf                                          
*.app text=auto eol=lf                                            
*.component text=auto eol=lf                                      
*.email text=auto eol=lf                                          
*.page text=auto eol=lf                                           
*.object text=auto eol=lf                                         
*.report text=auto eol=lf                                         
*.site text=auto eol=lf                                           
*.tab text=auto eol=lf                                            
*.trigger text=auto eol=lf                                        
*.weblink text=auto eol=lf                                        
*.workflow text=auto eol=lf                                       
*.reportType text=auto eol=lf                                     
*.homePageLayout text=auto eol=lf                                 
*.homePageComponent text=auto eol=lf                              
*.labels text=auto eol=lf                                         
*.group text=auto eol=lf                                          
*.quickAction text=auto eol=lf                                    
*.js text=auto eol=lf                                             
*.py text=auto eol=lf                                             
*.pl text=auto eol=lf                                             
*.csv text=auto eol=lf                                            

… which is to say, every metadata type which you know is going to be text gets an entry, but those types which might be binary get no entry (or you can add *.staticresource binary etc). This probably isn’t quite comprehensive depending on your setup, because inside documents/*/ you can end up with arbitrary file endings - however, normally the files you have there have ‘normal’ filenames, such as downArrow.png or footer.html which git has a chance of being able to recognise as binary or not.

Once you’ve set up your .gitattributes properly, if you’re starting off a new repository you’re good to go, but if you’re having to apply these changes to a repository which you already have developers working on, you need to be quite careful about rolling them out. I’m going to write a post on that topic soon.

Tags: SFDC Git

How and why to compress your Salesforce.com profiles

15 January 2015

Why compressing your profiles is a good idea

When handling Salesforce.com metadata, especially attempting to store it in source control, it doesn’t take long to notice the following:

  • Profiles are big. In fact, they contain 3-10 lines for every Apex Class, Visualforce Page, Object, Field, App, and so on, and so forth. Before long you’ve got thousands of lines, which means…
  • It’s difficult to commit changes to a profile, because you’ve got to scroll down to line 10243 to check that that’s the change you meant.
  • It takes ages to diff your profiles because they take up many megabytes on disk.
  • Profiles are vulnerable to merge errors because git’s standard diff algorithm doesn’t respect xml structure, and good luck finding an algorithm which does which can handle such huge files.

I work with a large salesforce installation with about 110 profiles and 30 permissionsets, each of which is some 25,000 lines long, and they take up 120mb on disk. These are real problems for my organisation, and I had to come up with a solution. I realised that there’s no reason to have exactly what you retrieve from Salesforce.com stored on disk. You can apply retrieve-time transformations to your code so long as:

  • Whatever you store is still deployable.
  • The tools used to retrieve your metadata are uniform across your organisation.

I write developer tools for my colleagues, so I am in a position to guarantee the latter. As for the former, all you have to do is remove a lot of line breaks. The idea is to transform this:

    <applicationVisibilities>
        <application>Order_Management</application>
        <default>false</default>
        <visible>true</visible>
    </applicationVisibilities>
    <applicationVisibilities>
        <application>SendGrid</application>
        <default>false</default>
        <visible>true</visible>
    </applicationVisibilities>
    <applicationVisibilities>
        <application>Territory_Management</application>
        <default>false</default>
        <visible>true</visible>
    </applicationVisibilities>
    <applicationVisibilities>
        <application>standard__AppLauncher</application>
        <default>false</default>
        <visible>true</visible>
    </applicationVisibilities>
    <applicationVisibilities>
        <application>standard__Chatter</application>
        <default>false</default>
        <visible>true</visible>
    </applicationVisibilities>
    <applicationVisibilities>
        <application>standard__Community</application>
        <default>false</default>
        <visible>true</visible>
    </applicationVisibilities>
    <applicationVisibilities>
        <application>standard__Content</application>
        <default>false</default>
        <visible>true</visible>
    </applicationVisibilities>

into this:

<applicationVisibilities><application>Order_Management</application><default>false</default><visible>true</visible></applicationVisibilities>
<applicationVisibilities><application>SendGrid</application><default>false</default><visible>true</visible></applicationVisibilities>
<applicationVisibilities><application>Territory_Management</application><default>false</default><visible>true</visible></applicationVisibilities>
<applicationVisibilities><application>standard__AppLauncher</application><default>false</default><visible>true</visible></applicationVisibilities>
<applicationVisibilities><application>standard__Chatter</application><default>false</default><visible>true</visible></applicationVisibilities>
<applicationVisibilities><application>standard__Community</application><default>false</default><visible>true</visible></applicationVisibilities>
<applicationVisibilities><application>standard__Content</application><default>false</default><visible>true</visible></applicationVisibilities>

The key idea is that each metadata component, whether an application, a custom field, a visualforce page or anything else, gets precisely one line in the resulting document, which means:

  • Any addition, deletion or modification of a component changes exactly one line
  • The addition or removal of lines is guaranteed to result in well-formed XML which is deployable.
  • Merges are much, much easier to perform.
  • Since git diff works line-by-line and we’re reducing the file from 25,000 to 2,500 lines, we gain a huge increase in efficiency when working with git.
  • We get back about 500kb of disk space per file.

For really tiny Salesforce instances, this might be overkill, but you can see that once you get big enough, this makes a real impact.

##How to do this compression

I produced an extremely simple Perl script to carry out this compression. Why Perl?

  • Unmatched string processing ability
  • Perl 5.8.8 comes bundled with a git installation on windows

Save this file as profileCompress.pl:

BEGIN { $\ = undef; }
s/\r//g;                  # remove all CR characters
s/\t/    /g;              # replace all tabs with 4 spaces
if (/^\s/) {              # ignore the the xml root node
  s/\n//;                 # remove newlines
  s/^    (?=<(?!\/))/\n/; # insert newlines where appropriate
  s/^(    )+//;		      # trim remaining whitespace
}

Then every time you do a retrieve, invoke it with perl -i.bak -p profileCompress.pl src/profiles/*.profile src/permissionsets/*.permissionset. The obvious disclaimer about backing up your code first because it might get mangled and I can’t take any responsibility for that applies!

I handle this by adding

<exec executable = "perl">
	<arg value = "-pi.bak"/>
	<arg value = "${lib.dir}/script/profileCompress.pl"/>
	<arg value = "${src.dir}/profiles/*.profile"/>
	<arg value = "${src.dir}/permissionsets/*.permissionset"/>
</exec>

to my ant script right after I retrieve, once I’ve stored all of my stuff inside the folders stored in those variables.

Tags: SFDC Git

The singlePackage option in Salesforce.com metadata deployments

09 December 2014

I’ve been working on a metadata client application for Salesforce.com to completely replace ant, because once you reach a certain level of complexity, ant really doesn’t cut it anymore! However, I found that when deploying a large .zip file (130MB of data compressed to 19MB using deflate with compression level 9) my deployments were taking far, far longer than the equivalent ant deployment of the same files.

In fact, the deployment would hang in the ‘waiting’ status for over an hour before starting to make progress. I finally found the solution by essentially iterating over every combination of deployment options in my SOAP call, and thought I’d write a quick post in case anybody else needs to fix the issue.

The problem occurred when deploying a package using this file structure, without the singlepackage option set.

zip root
|  unpackaged
|  |  classes
|  |  | files
|  |  | ...
|  |  triggers
|  |  | files
|  |  | ...
|  |  ...
|  |  package.xml

I found that by setting singlePackage=true and re-arranging the zip file as follows, I was able to completely eliminate the time spent hung in ‘waiting’.

zip root
|  classes
|  | files
|  | ...
|  triggers
|  | files
|  | ...
|  ...
|  package.xml

I hope somebody else stumbling across this issue finds this and it helps!

Tags: SFDC

Running all tests during a Salesforce.com deployment

21 August 2014

Salesforce.com deployments to a production environment require all unit tests to be run at the time of deployment, and roll back if any of them encounter an error. This is a desirable behaviour for a continuous integration environment, but although the deployment tools provide a runAllTests = true option for sandbox deployments, it has the disadvantage that it runs tests in all namespaces - and depending how many managed packages you have, that could be a very lengthy process indeed.

Following on from the last post though, it turns out that there is a way to achieve exactly this behaviour by gratuitous use of javascript within the ant deploy script. The idea is this: the SFDC deploy task can have nested runTest elements, but natively in ant we can’t dynamically generate them. However, if we can get a list of test classes, we can use javascript to add them to a dynamically generated deploy task.

This approach does have limitations: you can only use it when you are deploying every test class on your org, because this won’t run tests which aren’t in your deployment, but it will calculate code coverage based on which tests are run.

I’ll start where we left off by defining simple build.xml, build.properties, and deploy.js files:

###build.xml

<project name="ant javascript example">
  <property file="build.properties"/>
  
  <target name="deploy">
    <deployscript/>
  </target>
  
  <scriptdef name="deployscript"
    language="javascript"
    src="deploy.js"/>
</project>

###deploy.js

var env  = project.getProperty("e");
username = project.getProperty(env + ".username");
password = project.getProperty(env + ".password");
url      = project.getProperty(env + ".url");

if (!(username && password && url)) {
  fail = project.createTask("fail");
    fail.setMessage("Either you didn't specify an environment, or "
      + "the specified environment didn't have the correct properties "
      + "defined in build.properties.");
  fail.execute();
}

var task = java.lang.Class.forName("com.salesforce.ant.DeployTask").newInstance();
task.setTaskName("SF deploy");
task.setPassword(password);
task.setUsername(username);
task.setServerURL(url);
task.setDeployRoot("myPkg");
task.setProject(project);
task.setMaxPoll(project.getProperty("sf.maxpoll"));

if (project.getProperty("v")) task.setCheckOnly(true);
if (project.getProperty("t")) task.setRunAllTests(true);

task.execute();

###build.properties

dev.username  = user@example.com.dev
dev.password  = passwordToken
dev.serverurl = https://test.salesforce.com

qa.username   = user@example.com.qa
qa.password   = passwordToken
qa.serverurl  = https://test.salesforce.com

prod.username = user@example.com
prod.password = passwordToken
prod.serverurl= https://login.salesforce.com

sf.maxPoll   = 200

Now, we need a way to find all of the test classes. Since I have grep installed on my computer and it’s a great cross-platform solution, I’ll use a quick-and-dirty grep through the files. This assumes you’ve got all of your classes on your hard drive in the src/classes directory, which will be the case if you use the eclipse Force.com IDE or similar, and/or if you use source control. All tests have to be denoted with @isTest or testMethod, so I’m going to do a search on those terms. This would catch any file with testMethod in a comment, for instance, so it’s not perfect, but it’s pretty good.

Once we’ve got that output, we’ll store it in a variable to use in our javascript later on.

###build.xml

<project name="ant javascript example">
  <property file="build.properties"/>
  
  <target name="deploy">
    <exec executable = "grep"
      outputproperty = "testClasses">
      <arg value = "-lEr"/>
      <arg value = "@is[tT]est|test[mM]ethod"/>
      <arg value = "src/classes/"/>
    </exec>
    <deployscript/>
  </target>
  
  <scriptdef name="deployscript"
    language="javascript"
    src="deploy.js"/>
</project>

If we now dive back into the ant-salesforce.jar source code and look in com/salesforce/ant/DeployTast.class, there’s a class called CodeNameElement, and a method called addRunTest. These are what we need to use to run the tests. Just like the DeployTask itself, we can instatiate CodeNameElement with java.lang.Class.forName().newInstance(), and add that instance as a child:

if (project.getProperty("t")) {
  var tests = (project.getProperty(testClasses)|| '').split("\n"));
  
  tests = tests
    .map   (function(e,i,a) {
      return (m = /(\w+)\.cls/.exec(e)) ? m[0] : e; // get just the test names
    }).filter (function(e,i,a) {
      return e  && a.indexOf(e) == i // uniqueness filter 
    });
  
  for each (test in tests) {
    var codeNameElement = java.lang.Class.forName(
        "com.salesforce.ant.DeployTask$CodeNameElement"
      ).newInstance();
    codeNameElement.addText(test);
    task.addRunTest(codeNameElement);
  }
};

This is really ready for use at this point, but for my own satisfaction we can make this even more javascript-oriented. If you crack open lib/ant.jar in your ant installation, you’ll see ExecTask.class halfway down. This is the underlying class for <exec/>, and the only thing different about this class is that in order to append a child ‘’ element, you need to use ‘createArg().setValue()’. Putting this into action, we can finish bringing all the logic into javascript - this javascript works with the build.xml defined at the top of this post.

###deploy.js

function getTestClasses(){
	var propertyName = "testClassesList";
	var exec = project.createTask("exec");
	exec.setExecutable("grep");
	exec.setOutputproperty(propertyName);
	
	exec.createArg().setValue("-lEr");
	exec.createArg().setValue("@is[tT]est|test[mM]ethod");
	exec.createArg().setValue("src/classes/");
	exec.execute();
	
	var result = project.getProperty(propertyName);
	return (typeof result == "string" ? result.split("\n") : [])
		.map   (function(e,i,a){ return (m = /(\w+)\.cls/.exec(e)) ? m[0] : e;})
		.filter(function(e,i,a){ return a.indexOf(e) == i });
}

(function main(){
  var env  = project.getProperty("e");
  username = project.getProperty(env + ".username");
  password = project.getProperty(env + ".password");
  url      = project.getProperty(env + ".url");
  
  if (!(username && password && url)) {
    fail = project.createTask("fail");
    fail.setMessage("Either you didn't specify an environment, or "
      + "the specified environment didn't have the correct properties "
      + "defined in build.properties.");
    fail.execute();
  }
  
  var task = java.lang.Class.forName("com.salesforce.ant.DeployTask")
    .newInstance();
  task.setTaskName("SF deploy");
  task.setPassword(password);
  task.setUsername(username);
  task.setServerURL(url);
  task.setDeployRoot("myPkg");
  task.setProject(project);
  task.setMaxPoll(project.getProperty("sf.maxpoll"));
  
  if (project.getProperty("v")) task.setCheckOnly(true);
  
  if (project.getProperty("t")) {
    for each (test in getTestClasses()) {
      var codeNameElement = java.lang.Class.forName(
          "com.salesforce.ant.DeployTask$CodeNameElement"
        ).newInstance();
      codeNameElement.addText(test);
      task.addRunTest(codeNameElement);
    }
  };
  
  task.execute();
}());

Tags: SFDC

Writing Salesforce ant deployment tasks using Javascript

19 August 2014

The Force.com deployment tool is a .jar file defining some extra tasks such as sf:deploy and sf:retrieve. Examining the example build.xml file we see several calls of the form:

<sf:retrieve 
    username="${sf.username}"
    password="${sf.password}"
    serverurl="${sf.serverurl}"
    maxPoll="${sf.maxPoll}"
    retrieveTarget="retrieveOutput"
    packageNames="MyPkg"/>
    
<sf:bulkRetrieve
    username="${sf.username}"
    password="${sf.password}"
    serverurl="${sf.serverurl}"
    maxPoll="${sf.maxPoll}"
    metadataType="${sf.metadataType}"
    retrieveTarget="retrieveUnpackaged"/>

In a mature development team with several different sandboxes, you may end up wanting to retrieve and deploy from any of several different sandboxes, with or without the bulk API, with or without running all tests1. If you tried to use ant to write logic to decide between them, you’d end up with a great many targets, each with only a slight permutation to the others. What we want to do is make these decisions using Javascript, which will make the code shorter, easier-to-read, and more flexible. We’ll use JDK1.8, and specifically the Nashorn Javascript engine.

Let’s start with the simplest possible build.xml file:

<project name="ant javascript example">
    <property file="build.properties"/>
    <target name="deploy">
        <sf:deploy 
            username="${sf.username}"
            password="${sf.password}"
            serverurl="${sf.serverurl}"
            maxPoll="${sf.maxPoll}"
            deployRoot="mypkg"
            rollbackOnError="true"/>
    </target>
</project>

and a similarly trivial build.properties file:

sf.username  = user@example.com
sf.password  = passwordToken
sf.serverurl = https://login.salesforce.com
sf.maxPoll   = 200

The first thing I want to do is add an optional ‘validate’ flag to my deployment such that when it is set, we only perform a dry-run. The idea is that when you call ant deploy -Dv=1 the deploy task gets the checkonly="true" attribute added. One way to do this natively is:

<project name="ant javascript example">
    <property file="build.properties"/>
    <target name="deploy"
            depends="dryrun,wetrun" >
    </target>
    
    <target name="dryrun"
            if="v">
        <sf:deploy 
            username="${sf.username}"
            password="${sf.password}"
            serverurl="${sf.serverurl}"
            maxPoll="${sf.maxPoll}"
            deployRoot="mypkg"
            checkonly="true"/>
    </target>
    
    <target name="wetrun"
            unless="v">
        <sf:deploy 
            username="${sf.username}"
            password="${sf.password}"
            serverurl="${sf.serverurl}"
            maxPoll="${sf.maxPoll}"
            deployRoot="mypkg"/>
    </target>
</project>

This code is over twice as long, and it’s much harder to follow the flow of what’s happening, and if I subsequently need to add another flag such as -Dt=1 to add runAllTests = "true", I’ll double my code again2. Luckily, ant has our back.

Let’s rip open ant-salesforce.jar and see what’s inside:

  • com
    • salesforce
      • ant
        • BulkRetrieveTask
        • CompileAndTest
        • Configuration
        • ConnectionFactory
        • DeployTask
        • DescribeMetadataTask
        • ListMetadataTask
        • RetrieveTask
        • SFDCAntTask
        • SFDCMDAPIAntTask
        • SFDCMDAPIAntTaskRunner
        • ZipUtil

DeployTask looks quite promising, and we can easily ascertain that DeployTask extends SFDCMDAPIAntTask, which extends SFDCAntTask, which extends Task, and glancing through those classes to find appropriate getters and setters, we can write:

<project name="ant javascript example">
    <property file="build.properties"/>
    
    <target name="deploy">
        <deployscript/>
    </target>
    
    <scriptdef name="deployscript" language="javascript">
      var task = java.lang.Class.forName("com.salesforce.ant.DeployTask")
        .newInstance();
      task.setTaskName("SF deploy");
      task.setPassword(project.getProperty("sf.password"));
      task.setUsername(project.getProperty("sf.username"));
      task.setServerURL(project.getProperty("sf.serverurl"));
      task.setDeployRoot("myPkg");
      task.setProject(project);
      task.setMaxPoll(project.getProperty("sf.maxpoll"));
      
      if (project.getProperty("v")) task.setCheckOnly(true);
      if (project.getProperty("t")) task.setRunAllTests(true);
      
      task.execute();
    </scriptdef>
</project>

Now it’s clear how we can improve and extend this functionality: next, let’s imagine that we might want to deploy to any of arbitrarily many environments (for instance, you’re a sysadmin with dev, QA, and production instances). We want to be able to call ant deploy -De=<env>. Let’s rewrite our build.properties file…

dev.username  = user@example.com.dev
dev.password  = passwordToken
dev.serverurl = https://test.salesforce.com

qa.username   = user@example.com.qa
qa.password   = passwordToken
qa.serverurl  = https://test.salesforce.com

prod.username = user@example.com
prod.password = passwordToken
prod.serverurl= https://login.salesforce.com

sf.maxPoll   = 200

…and adapt our deploy task to use this new format:

<project name="ant javascript example">
    <property file="build.properties"/>
    
    <target name="deploy">
        <deployscript/>
    </target>
    
    <scriptdef name="deployscript" language="javascript">
      <![CDATA[
        var env  = project.getProperty("e");
        username = project.getProperty(env + ".username");
        password = project.getProperty(env + ".password");
        url      = project.getProperty(env + ".url");
        
        if (!(username && password && url)) {
            fail = project.createTask("fail");
            fail.setMessage("Either you didn't specify an environment, or the specified "
              + "environment didn't have the correct properties defined in "
              + "build.properties.local.");
            fail.execute();
        }
        
        var task = java.lang.Class.forName("com.salesforce.ant.DeployTask")
          .newInstance();
        task.setTaskName("SF deploy");
        task.setPassword(password);
        task.setUsername(username);
        task.setServerURL(url);
        task.setDeployRoot("myPkg");
        task.setProject(project);
        task.setMaxPoll(project.getProperty("sf.maxpoll"));
        
        if (project.getProperty("v")) task.setCheckOnly(true);
        if (project.getProperty("t")) task.setRunAllTests(true);
        
        task.execute();
      ]]>
    </scriptdef>
</project>

There are lots of other options to be explored here, but I think this should provide a solid starting point for making powerful deployment scripts that help get the job done.

  1. By default the run all tests flag is quite broken when you have managed packages, in that it runs tests from all namespaces. I’ll write up a solution to that issue in a later post. 

  2. There is a more efficient and complicated way to do this by setting extra properties using <condition/>, but javascript still beats it. 

Tags: SFDC