Alexander Brett

Creating attachments with WWW::SFDC

27 April 2015

At my company, we have a problem with attachments; namely, one of the things we have to do most releases is update some mail merge templates which are stored as attachments against Salesforce records. Historically, this has been a great cause for error; at some point the release manager ends up with a sea of .docx files in some directory or other, having to remember which ones she’s already uploaded, which ones need to be uploaded against which records, and so on.

Checking these files into source control helped; now, instead of a soup of randomly named files kicking around, at least she knows she has the latest version, and has an easy way to ascertain which ones have changed. It’s still a tedious and error-prone manual process though, and one slip-up means that a distributor recieves a million dollars worth of junk on their license schedule.

Fortunately, the correct automation solution is at hand: WWW::SFDC to the rescue. Let’s imagine we’ve been sensible and we’ve stored a load of .docx and .xslx files inside source control already, and with even more foresight, we have named each file the same as the Name on the Doc_Template__c record against which we need to store them. What we need to do is create Attachment records for each changed document against each relevent record.

Let’s automate it in 10 easy steps.

  1. Get the changed files:

     git diff --name-only --diff-filter=AM --relative attachments/Doc_Template__c/
     # Or, if you'd rather just have everything (for instance, populating a new sandbox)
     git ls-files attachments/Doc_Template__c/
    
  2. Basic perl script setup, including making sure creds are set up:

     use strict;
     use warnings;
     use Getopt::Long;
     use WWW::SFDC::Partner;
     use File::Slurp 'read_file';
    
     my %creds = (
       url => 'https://login.salesforce.com'
     );
    
     GetOptions
       'p|password:s'    => \$creds{password},
       'url:s'           => \$creds{url},
       'u|username:s'    => \$creds{username};
          
     WWW::SFDC::Partner->instance(creds=>\%creds);
    
  3. Read in the changed files, checking that they exist and trimming whitespace:

     my @filenames = grep {chomp and -e } <>;
    
     exit unless scalar @filenames; # if there aren't any, we have no work to do.
    
  4. Parse the filenames and read the files. We’ll store the file contents as base64-encoded data which is what the Salesforce.com partner API expects up to provide in the body field. Fortunately, SOAP::Lite makes this a breeze, via the SOAP::Data->type() method.

     my @documents = map {
       /([^\/]*)\.(docx|xlsx)$/
         ? +{                             # +{} forces this to be a hashref, rather than an arbitrary code block
             name => $1,
             extension => $2,
             body => SOAP::Data->name(
                     body => read_file $_ # read_file is exported by File::Slurp and does what it says on the tin
                 )->type('base64')        # prepackaged SOAP::Lite magic.
           }
         : ();                            # return empty list. 
     } @filenames;
    
  5. We’re need the IDs of the Doc_Template__c records to store these against, so we’ll start by constructing a where clause…

     my $whereClause = join " OR ", map { "Name='$$_{name}'" } @documents; # that was easy
    
  6. …and we’ll go and execute that query.

     my @parentRecords = WWW::SFDC::Partner->instance()->query(
       "SELECT Id, Name FROM Doc_Template__c WHERE $whereClause"
     );
    
  7. We’re going to need to look up IDs from the name, so we’ll create a hash:

     my %parentIds = map { $_->{Name} => $_->{Id} } @parentRecords;
    
  8. The interesting bit. From each document, create an Attachment suitable for passing into a create call.

     my @recordsToCreate = map {
         $parentIds{$_->{name}} # check that the record exists;
         ? +{
             type => 'Attachment',
             ParentId => $parentIds{$_->{name}},
             name => "$$_{name}.$$_{extension}",
             body => $_->{body},
         }
         : ()
     } @documents;
    
  9. Wrap it all up with a create call

     WWW::SFDC::Partner->instance()->create(@recordsToCreate);
    
  10. Put it all together (tada!):

    git diff --name-only --diff-filter=AM --relative attachments/Doc_Template__c/^
     | perl automagic_attachments.pl -u myUsername -p passwordPlusToken
    

Now, this may feel trivial. However, having repeatable, automatic, guaranteed-error-free deployments of templates every month saves up hours of effort on release day, and hours of tracking down bugs later on.

Tags: SFDC Perl

A New Pattern for Apex Triggers

24 April 2015

This is going to be a pretty long post. If you want the meaty bit, scroll down to ‘Solving the problem by using object-orientation properly’

A brief history of trigger patterns

1: The anti-pattern

I feel like an organisation or developer will go through different stages as they learn better ways of dealing with Apex Triggers. A rookie developer in a simple instance with only 501 training might write triggers that look like this:

trigger QuoteTaxUpdate on Quote (before insert, before update) {
  for (Quote q : Trigger.new) {
    if(Trigger.isInsert || Trigger.oldMap.get(q.Id).Value__c != q.Value__c) {
      q.Tax__c = TaxCalculator.calculateTaxes(q);
      // ... more logic goes here
    }
  }
}
trigger QuoteRelatedOpportunityUpdate on Quote (after insert, after update) {
  for (Quote q : Trigger.new) {
    // Do Stuff
  }
}

There are several major problems with this approach:

  • Having multiple triggers on the same object means that it’s hard to isolate the code file causing problematic behaviour
  • It can be hard to debug complex interactions between triggered processes
  • Once you hit a couple of hundred objects with a handful of processes on each, you have a huge number of source code files to track

2: One trigger per object

With this in mind, the tendancy is to combine triggers so that each object has precisely one trigger class:

trigger QuoteTrigger on Quote (before insert, before update, after insert, after update) {
  if (Trigger.isBefore) {
    if (Trigger.isInsert) {
      //Do some things
    } else if (Trigger.isUpdate) {
      //Do other things
    }
  } else if (Trigger.isAfter) {
    if (Trigger.isInsert) {
      // More stuff
    } else if (Trigger.isUpdate) {
      // You're getting the hang of this...
    }
  }
}

Having now refactored your code into some beautiful, well-organised triggers, you notice that the stuff you’re doing in the Before Update and Before Insert is close enough that you want a method to parameterise - but you can’t have methods in triggers. Additionally, this makes it very difficult to unit test a trigger. All this leads to:

3: The Trigger Handler

You move your logic into its own class, defin

public class QuoteTriggerHandler {
  public static void onBeforeUpdate () {
    sharedMethod();
  }
  
  public static void onAfterUpdate () {
    sharedMethod();
  }
  
  // ...
  
  public sharedMethod(){
    if (Trigger.isInsert) {
      // logic goes here
    }
  }
}

This is great, because you can have some unit tests, some shared methods, and so on and so forth. However, you often find that you’re looping through the records once to choose which ones to update the taxes on, again to work out the new owner, a third time for some dynamic sharing rules, and before long you run through the ‘total lines of code executed in one transaction’ governor limit (or your customers complain about your Quotes taking agonising seconds to save).

4: The Advanced Trigger Handler

How do we fix this? We need some architecture. Let’s set it up so that we only iterate over the records once to ascertain which ones are appropriate to our needs, then each process will need to iterate over fewer records. Once you’ve got more than a couple of jobs, this really pays off. Additionally, it’s very easy to create shared and testable logic like this, because you require that each process has its own method which you can unit test to your heart’s content.

public class QuoteTriggerHandler {
	private List<Quote> QuotesForTaxUpdate = new List<Quote>();
	private List<Quote> QuotesForOpportunityOwnerUpdate = new List<Quote>();
	// more lists

	private Boolean DoTaxUpdate = false;
	private Boolean DoOpportunityOwnerUpdate;
	// more toggles

	public void onBeforeUpdate () {
		DoTaxUpdate = true;
		chooseRecords();
		doLogic();
	}

	public void onAfterInsert () {
		DoOpportunityUpdate = true;
		chooseRecords();
		doLogic();
	}

	public void chooseRecords () {
		for (Quote q : Trigger.new) {
			if (DoTaxUpdate && /*conditions*/) {
				QuotesForTaxUpdate.add(q);
			}
			if (DoOpportunityUpdate && /*conditions*/) {
				QuotesForOpportunityUpdate.add(q);
			}
		}
	}

	public void doLogic () {
		if (DoTaxUpdate)
			TaxUpdateLogic(QuotesForTaxUpdate);
		if (DoOpportunityUpdate) 
			OpportunityUpdateLogic(QuotesForOpportunityUpdate);
	}
}

So why don’t I like this? There are several remaining problems.

  • Your trigger handler will probably be somewhere between quite long and REALLY REALLY long. If I’ve identified a bug in a particular operation, I don’t want to sift through a thousand lines of code to find the method implementing that operation, and try to work out whether the bug is there, or in the if clause somewhere inside chooseRecords() (which is probably a hundred lines long all by itself).
  • This pattern is quite complicated, and it’s going to take a good while for your new hires to be confident enough to dive into said several-hundred-lines long chunk of code and make changes, out of sheer fear of breaking things.
  • You’re going to have lots of code which looks similar in the trigger handler for every object, which smells.

Solving the problem by using object orientation properly

A rough outline

If you take another look at the example above, you’ll see that we’ve basically created a contract for each TriggerOperation (I’m starting to outline the names I’ll use in the solution). This is:

  • An OperationName, which tells people what you’re trying to do - for instance, UpdateTaxes or UpdateOpportunityOwners.
  • An InitialLoopOperation, which will normally select an appropriate subset of records for your operation
  • Some OperationLogic, which is executed at the end.

With this is mind, it seems like a TriggerOperation would be a very good candidate for creating a generic class, which could then be extended and overridden, but which crucially would define a standard interface. This means that when I know roughly what’s gone wrong, I only have one small class to check through for code errors, and when a new developer want to write a new process, he knows where to start without reading reams of documentation and still breaking the build for everyone else.

The second part of this puzzle is going to have to be a TriggerOperationDispatcher, which consumes the TriggerOperation implementations and calls their operations as applicable. This class can then be shared between every SObject, reducing duplication and meaning that if you want to add functionality, you can do it once and benefit everywhere (I’ll go into this idea more further on).

Sample Implementation (MKI)

With the introductions out of the way, let’s give an outline of the classes. It’s going to get very code-heavy hereafter:

// This SHOULD have a type parameter to allow derivation for
// a particular object, but that was deprecated in API v26 :(
public virtual TriggerOperation {
	public abstract void initialLoopOperation (SObject obj);

	public void execute() {
		this.logic();
	}

	// protected means that execute() can access the
	// overridden version in the implementation class
	protected abstract void logic();
}
public TriggerOperationDispatcher {

	private List<TriggerOperation> operations = new List<TriggerOperation>();

	private Boolean isUpdate;
	private Boolean isBefore;
	private Map<Id,SObject> oldMap;
	//etc.
	
	public TriggerOperationDispatcher () {
		this(
			Trigger.newMap, Trigger.oldMap, Trigger.isUpdate,
			Trigger.isInsert, Trigger.isDelete, Trigger.isUndelete,
			//etc
		);
	}
	
	// We're going to want to carefully unit test this class, and
	// this constructor is an easy way to simulate trigger conditions
	@testVisible 
	private TriggerOperationDispatcher (Map<Id,SObject> newMap, /*etc*/) {
		this.newMap = newMap;
		//and so on...
	}

	public TriggerOperationDispatcher addOperation (TriggerOperation op) {
		operations.add(op);
		return this; //chainable
	}

	public void dispatch () {
		List<SObject> relevantObjects = (isInsert || isUpdate || isUndelete)
			? newList
			: oldList;
		
		for (SObject obj : relevantObjects)
			for (TriggerOperation op : operations)
				operation.initialLoopOperation(obj);
		
		for (TriggerOperation op : operations)
			operation.execute();
	}
}

Let’s look at how we’d refactor the QuoteTrigger to take account of this:

trigger QuoteTrigger on Quote (before insert, /*blah blah*/) {
	TriggerOperationDispatcher dispatcher = new TriggerOperationDispatcher();
	if (Trigger.isBefore) {
		if (Trigger.isInsert)
			dispatcher.addOperation(new QuoteTaxOperation(Trigger.oldMap))
				.addOperation(new SomeOtherOperation());
		if (Trigger.isUpdate)
			// The rest of this is predictable
	} else if () {
	} //etc.
}

and to make our QuoteTaxOperation:

public class QuoteTaxOperation : TriggerOperation {
	private List<Quote> QuotesForTaxUpdate;
	private Map<Id,Quote> oldMap;

	public QuoteTaxOperation (Map<Id,Quote> oldMap) {
		this.oldMap = oldMap;
	}
	
	public void override initialLoopOperation (SObject obj) {
		Quote q = (Quote)obj;
		if (oldMap == null || q.Value__c != oldMap.get(q.Id).Value__c)
			QuotesForTaxUpdate.add(q);
	}

	protected void logic () {
		for (Quote q : QuotesForTaxUpdate) {
			calculateTax(q);
		}
	}
}

Turning stuff on and off

Let’s say we’ve just updated a sandbox and we’re going to go through every email field and add something to the end, so that our customers don’t get spammed whilst we test. We’ve got an awesome apex script to accomplish this, but if all our trigger run, it’ll take forever to perform the update! We need to turn off the operations. Luckily, most of the framework for this is in place.

What we need is:

  • A catalogue of operations
public enum TriggerOperationName {
	QUOTE_UPDATE_TAX,
	QUOTE_UPDATE_OPPORTUNITY_OWNER,
	...
}
  • Each operation to be named
public virtual class TriggerOperation {
	public TriggerOperationName name;
}
public class QuoteTaxOperation {
	public QuoteTaxOperation (...) {
		this.name = TriggerOperationName.QUOTE_UPDATE_TAX
		...
	}
	
	...
}
  • A static variable to store disabled operations
public class TriggerOperationManager {
	private static Set<TriggerOperationName> disabledOperations = new List<TriggerOperationName>();
	
	private static Boolean allOperationsDisabled = false;
	
	public static void disableOperation (TriggerOperationName op) {
		disabledOperations.add(op);

	public static void disableAllOperations () {
		allOperationsDisabled = true;
	}
	
	public static boolean isOperationDisabled (TriggerOperation op) {
		return (allOperationsDisabled || disabledOperations.contains(op.name));
	}
}
  • A check for whether to use that operation
//in TriggerOperationDispatcher
	public TriggerOperationDispatcher addOperation (TriggerOperation op) {
		if (!TriggerOperationManager.isOperationDisabled(op)) operations.add(op);
		return this; //chainable
	}

… well, that was pretty easy. We can now exercise fine-grained control over whether to turn things on or off. Want to do that bulk job from anonymous apex? Chuck TriggerOperationManager.disableAllOperations() at the top of your script! Of course, you could also create a custom setting so that operations were disabled on certain profiles (such as your data migration user?) or similar.

This demonstrates the power of using object-oriented patterns well. By using inheritance and by having a single shared dispatcher class, we can achieve powerful org-wide changes with only a couple of dozen lines of code.

Logging

It’s obviously good to know what you’ve done in a transaction. What if we modified these classes to help us do that, even when we’ve burnt waaaaaaaay past the 2MB logging limit?

You could create a new object called Operation_Log__c, and a custom setting called Operation_Logging_Settings__c (to temporarily turn on logging per-profile).

Then, create a class called TriggerOperationLogger with a private list of TriggerOperationName and methods markAsExecuted and logExecution. At the end of TriggerOperationDispatcher.dispatch(), add a call to logExecution, and at the end of TriggerOperation.execute() add a call to markAsExecuted.

Again, you could achieve powerful and flexible logging of trigger operations across the entire org with only a handful of changes.

Summary

Most of the apex code I’ve seen fails to leverage even simple object-oriented concepts such as inheritance, but making use them can lead to elegant and powerful designs which save effort and increase maintainability. Give it a try!

Have I got something wrong? Propose an edit to this page on GitHub.

Tags: SFDC Apex

Running unit tests using WWW::SFDC

02 February 2015

This post is going to start basic and get gradually more complicated - I suggest that you stop once you’ve satisfied your own requirements! The basic premise is that, in order to keep a consistantly high-quality and deployable set of metadata, we keep a Salesforce.com org up to date with a branch, and nightly (or more often) we run every unit test against it. This makes sure that, even though developers are keeping abreast of the effects their changes have on other parts of the code base and so on, there is one extra highly visible and reportable checksum on quality.

Fortunately, the tooling API makes our job quite easy. If you have simple needs, the following 3 statements should be sufficient for running your tests:

my $parentJobId = WWW::SFDC::Tooling->instance()->runTestsAsynchronous(
  map {
    $_->{Id}
  } WWW::SFDC::Tooling->instance()->query(
    "Select Id FROM ApexClass WHERE NamespacePrefix = ''"
  )
);

sleep 60 while WWW::SFDC::Tooling->instance()->query(
  "SELECT Id, Status, ApexClassId FROM ApexTestQueueItem"
  . " WHERE ParentJobId = '$parentJobId' AND ("
  . " Status='Queued' OR Status='Processing'"
  . " OR Status='Preparing' OR Status='Holding')"
);

my @results = WWW::SFDC::Tooling->instance()->query(
  "SELECT Id, Outcome, MethodName, Message, StackTrace, ApexClass.Name "
  ."FROM ApexTestResult WHERE AsyncApexJobId = '$parentJobId'"
);

However, this is far from a perfect implementation. The first thing to notice is that we’re just passing every Apex class into runTestsAsynchronous, which is probably a highly inefficient way to do things (according to my measurements, it adds about 5% to the total time for the tests). It would be more elegant and quicker if we filtered the results of the first query to find tests classes, and, fortunately, we can do this using the SymbolTable field on ApexClass - a class needs to be enqueued if it, or any of its methods, have the TEST modifier. This can be achieve thus:

sub isTestModified {
  my $thing = shift;
  defined $thing->{modifiers} and (
    $thing->{modifiers} eq 'TEST'
    or (
      ref $thing->{modifiers} eq 'ARRAY'
      and grep {$_ eq 'TEST'} @{$thing->{modifiers}}
     )
   );
}

sub filterTests {
  defined $_->{SymbolTable}->{methods} and (
    (
      ref $_->{SymbolTable}->{methods} eq 'ARRAY'
      and grep {isTestModified($_)} @{$_->{SymbolTable}->{methods}}
    ) or (
      ref $_->{SymbolTable}->{methods} eq 'HASH'
      and isTestModified($_->{SymbolTable}->{methods})
    )
  )
}

my $parentJobId = WWW::SFDC::Tooling->instance()->runTestsAsynchronous(
  map { $_->{Id} } grep { filterTests } WWW::SFDC::Tooling->query(
    "Select Id, SymbolTable FROM ApexClass WHERE NamespacePrefix = ''"
  )
);

Now, you’re pretty happy with how this is working, but sometimes this query seemingly-randomly times out. The reason this happens is that if you have deployed one or more Apex classes since the last compilation, requesting the SymbolTables triggers a behind-the-scenes recompilation of your entire code base, which will take longer that the 120s timeout once you get to a certain size. My solution to this was just a brute-force retry mechanism, which can also mitigate any brief networking issues on the client (it suck if you’re halfway through a several-hour test run and you get no results because the was a momentary VPN lapse…), and I achieved it by replacing all the calls to WWW::SFDC::Tooling->instance()->query() with retryQuery():

sub retryQuery {
  my $query = shift;
  my @result;
  for (0..5) {
    eval { @result = WWW::SFDC::Tooling->instance()->query($query); };
    next if $@;
    return (scalar @result == 1 and not defined $result[0] ? undef : @result);
  }
  die "There was an error retrieving the information from Salesforce\nQuery: $query\nError: $@";
}

Bear in mind that I’ve recently modified WWW::SFDC to automatically re-authenticate in the event of a session timeout, which is extremely useful for long test runs!

At this point you feel ready to put your code on a continuous integration server and let it rip. When I did this I ran into a wierd problem where the perl process used up all of the available RAM and started thrashing swap space, taking 7 minutes to even start off the tests; it turns out that the SymbolTable for every class, all at the same time, is quite a mouthful to deserialise. The obvious solution is to query in batches, but the tooling API, as far as I can tell, does not support paged queries in the same way as the Partner API does. Queue more grep {} map {} grep {} chaining:

my $parentJobId = WWW::SFDC::Tooling->instance()->runTestsAsynchronous(
  map { $_->{Id} } grep { filterTests } map {
    retryQuery(
	"SELECT Id, SymbolTable FROM ApexClass WHERE NamespacePrefix = ''"
	. " LIMIT 200 OFFSET $_"
    )
  } grep {
    $_%200 == 0
  } 0..(scalar retryQuery(
    "Select Id FROM ApexClass WHERE NamespacePrefix = ''"
    ) - 1)
);

Now we have a robust and efficient way to run all of our unit tests on SFDC. The last thing I wanted to do was to have all of these test results aggregated and reported on with Atlassian Bamboo. Bamboo comes with a build-in JUnit parser, and JUnit has a fairly simple syntax, so for me the path of least resistance was to be a terrible-person and roll my own XML formatter:

sub jUnitFormat {
  my $result = shift;
  my $className = $$result{ApexClass}{Name};
  my $methodName = $$result{MethodName} eq "<compile>"
     ? "CompileFailed"
     : $$result{MethodName};
  return "<testcase name='$methodName' classname='$className' "
    . "status='$$result{Outcome}'>"
    . (defined $$result{Message}
      ? "<failure><![CDATA[$$result{StackTrace}\n$$result{Message}]]></failure>"
      : "")
    . "</testcase>";
}

{
  open my $fh, ">", $output;
  print $fh join "\n",
    '<?xml version="1.0" encoding="UTF-8"?>',
    '<testsuite name="SFDC Unit Tests">',
    (map {jUnitFormat($_)} @results),
    '</testsuite>';
  close $fh;
}

…easy peasy.

I’ve uploaded the final version as a runnable perl script as a gist - I very much encourage you to give it a try, and maybe even help me come up with more improvements.

Tags: SFDC Perl

Why Perl for Salesforce Tools?

23 January 2015

This explanation is going to take the form of a long story.

When I started writing developer tools for Salesforce.com, I got hold of the ant salesforce deployment library and went wild with it, with the ant zip task, loads of exec calls to git, and some extreme contortions to allow me to switch manifest files depending on whether we wanted to retrieve everything or just everything a developer normally needs. I quickly hit some situations where ant just wasn’t expressive enough for my needs:

  • Deploying to multiple environments and adding a -v for validation flag prompted me to write this post about mixing in some javascript with my ant.
  • Our QA org has some different outbound message endpoints from production. Since the GNU coreutils are installed alongside git, using perl for this was a no-brainer:
<exec executable="perl">
	<arg value="-i.bak"/>
	<arg value="-pe"/>
	<arg value="s/&lt;endpointUrl&gt;https:\\\/\\\/.*\\.sophos\\.com\\\/&lt;endpointUrl&gt;https:\\\/\\\/${endpoint}\/"/>
	<arg value="src/workflows/*.workflow"/>
</exec>

Once I’d got over these, I found that I needed to convert a list of files into a package.xml file. This turned out to be far too complicated for inline scripting or a perl one-liner, so I wrote a separate javascript file called list2xml.js and called it like:

<script src="lib/list2xml.js"
            input="${fileList}"
            outputProperty="${xmlBody}" />

However, the javascript I wrote for this turned out to have terrible performance. To process a file you need to:

  • strip off any src/ prefix
  • split up the file path
  • remove the file extension
  • get the metadata type (for a folder called classes/ the metadata type is ApexClass, for instance) which is effectively a hash lookup
  • store each component name against its metadata type
  • dedupe the resulting arrays
  • write out the xml string

This is quite a lot of string manipulation and array munging, and javascript is not great at this, especially running in Nashorn/JDK1.8. I had quite a few developers sitting me about how long all this was taking, so eventually I realised I needed to reimplement it in some faster language. Given the performance hit was in string processing, I came to the obvious conclusion: no language has faster string processing than Perl, and all my users already had it installed! The reimplemented list2xml eventually became manifest.pm.

After coming across a few of each of these situations, my codebase was a mess of xml with intermingled javascript, perl system calls, and perl and javascript files. Not only was it inconsistent, it was also entirely untested and essentially untestable, since ant is not designed for heavy lifting and as such has no testing framework, and nor does javascript (well, it probably does, but not a well-known and easy-to-use one). My users were back at my desk demanding fewer bugs.

My boss and I sat down to examine the options: we wanted ease of development and maintenance, high performance and a robust unit testing framework as a minimum. We narrowed the selection down to C# and Perl because of the skills available, and then did a head-to-head on a string processing algorithm; not only did Perl come out conclusively faster, it also had the advantage of not requiring a compilation stage, being usable for our contractors on their macs, and already having critical pain areas implemented in it.

I also found that juggling several different APIs at once was a pleasure using Perl, as was unit testing; I’m feeling pretty happy about the interface I’ve written to the metadata API, and I’m going to demonstrate some really powerful uses for the other APIs over the next couple of weeks.

I was pretty apprehensive about Perl before I started to use it in earnest; most of the reactions I get to it are along the lines of “It’s a write-only language”, “It isn’t maintainable”, “It’s out-of-date” and so on; but I’ve found that it’s quite easy. and in fact quite fun, to write highly performant, highly legible, well-documented and unit tested code.

Tags: Perl SFDC

WWW::SFDC - Metadata, Manifests and Zip Files

21 January 2015

In my last post I provided a brief introduction to WWW::SFDC, a Perl wrapper around Salesforce’s API interfaces. Today I thought I’d look slightly more in-depth at WWW::SFDC::Metadata, WWW::SFDC::Manifest, and WWW::SFDC::Zip, and how they interact with each other.

The motivation for including the Manifest and Zip modules in addition to the core API interaction modules was that even after a mechanism is provided for interfacing with the metadata API, the fact remains that deploy and retrieve accept and return .zip files as base64-encoded strings, and deploy requires a package.xml file which is populated according to a stringent set of rules, and so on: writing the logic around these has consumed a large proportion of my time as a tooling developer, and will be the same for every use case, so it makes sense to tie these in to the API interface.

The idea behind the Manifest object is to make it extremely easy to juggle lists of files, package.xml manifest files, and the inputs and outputs of the metadata API calls. For instance, the simplest way to generate empty manifest files is now:

WWW::SFDC::Manifest->new()->writeToFile('src/package.xml');

If we want to generate a custom package based on a list of modified files can look something like this:

my @filesToAdd = `git diff --name-only release master`;
WWW::SFDC::Manifest->new()->addList(@filesToAdd);

The last major problem that this package solves is the generation of a list of files, given a package, bearing in mind that certain metadata types require -meta.xml files, and others are in folders. To build a valid .zip file for deployment, you need to know exactly which files to include, and you can do this thus:

WWW::SFDC::Zip::makezip(
  'src/',
  WWW::SFDC::Manifest->new()->readFromFile('src/package.xml')->getFileList(),
  'package.xml';
);

This object then plays extremely well with the metadata API functions. If you want to retrieve every file in your org, you’d normally need to write out a package.xml including every document and email template you cared about. With the listMetadata call, you can just list the folders you care about, and you can chain this call with the appropriate manifest methods, to get extremely powerful one-liners such as:

WWW::SFDC::Zip::unzip(
  'src/',
  WWW::SFDC::Metadata->instance()->retrieveMetadata(
    WWW::SFDC::Manifest->new()->readFromFile('manifests/basic.xml')->add(
      WWW::SFDC::Metadata->instance()->listMetadata(
        {type => 'Document', folder => 'Developer_Documents'},
        {type => 'Document', folder => 'Documents'},
        {type => 'Document', folder => 'Invoices'},
        {type => 'Document', folder => 'Lead_Images'},
        {type => 'Document', folder => 'Logos'},
        {type => 'Document', folder => 'Tab_Images'},
        {type => 'EmailTemplate', folder => 'Asset'},
        {type => 'EmailTemplate', folder => 'Contact_User'},
        {type => 'EmailTemplate', folder => 'Error_Reporting'},
        {type => 'EmailTemplate', folder => 'Marketing_Templates'},
        {type => 'EmailTemplate', folder => 'Support_Templates'},
        {type => 'Report', folder => 'Merge_Reports'},
        {type => 'Report', folder => 'Finance_Reports'},
      )
    )->manifest()
  )
);

And once you’ve done this, you can deploy them all again like this:

WWW::SFDC::Metadata->instance()->deployMetadata(
  WWW::SFDC::Zip::makezip(
    'src/',
    WWW::SFDC::Manifest->new()->readFromFile('src/package.xml')->getFileList(),
    'package.xml';
  ),
  {
    singlePackage => 'true',
    $IS_VALIDATE ? checkOnly => 'true' : (),
    rollbackOnError => 'true',
  }
);

Of course, if you dynamically regenerate your package.xml, you probably won’t check it into source control, and you don’t want to take it as the truth when working out what to deploy. I actually construct my zip file like this:

WWW::SFDC::Zip::makezip(
  'src/',
  WWW::SFDC::Manifest->new()->addList(`git ls-files src/`)->getFileList(),
  'package.xml';
);

One final element of note is that WWW::SFDC::Zip::unzip accepts an optional third parameter: a function reference, applied to each file retrieved before being written to disk. I use this to achieve profile compression (see my recent post on that topic) like this:

sub _compressProfile {
  my $content = shift;
  my @lines = split /^/, $content;
  for (@lines) {
    s/\r//g;			            # remove all CR characters
    s/\t/    /g;		          # replace all tabs with 4 spaces
    if (/^\s/) {		          # ignore the the xml root node
      s/\n//;                 # remove newlines
      s/^    (?=<(?!\/))/\n/;	# insert newlines where appropriate
      s/^(    )+//;		        # trim remaining whitespace
    }
  }
  return join "", @lines;
}

sub _retrieveTimeMetadataChanges {
  my ($path, $content) = @_;
  $content = _compressProfile $content if $path =~ /\.profile|\.permissionset/;
  return $content
}

MAIN: {
  WWW::SFDC::Zip::unzip(
    'src/',
    WWW::SFDC::Metadata->instance()->retrieveMetadata($manifest->manifest()),
    &_retrieveTimeMetadataChanges
  );
}

I think that covers the main uses for those modules, and to those like me who have been grappling for months with the quirks of the ant deployment tool, the benefits of using a real programming language to achieve these tasks with minimum fuss are really obvious.

Tags: Perl SFDC

Introducing WWW::SFDC - a perl module for Salesforce.com interactions

19 January 2015

I’ve been working with the ‘Force.com migration tool’ - an ant wrapper around the Metadata API - for almost a year now, and I’ve been getting increasingly more frustrated with attempting to coerce ant (a tool for compiling java code) to be a deployment, retrieval, and git integration controller. It just isn’t meant for that purpose!

This means that when I have a list of EmailTemplate folders and want to retrieve all the templates within them, I have to spend 200 lines of xml to:

  • use some inline javascript to get the login credentials from a file depending which environment is specified
  • call a cSharp executable to ascertain whether the login credentials are valid, otherwise the user will get frozen out of Salesforce.com because of too many failed login attempts
  • call listMetadata once for each folder, which outputs the results to a file
  • call a perl script to parse those files
  • call another perl script to write the manifest file
  • call retrieve (finally!)

This is a terrible way to operate for any organisation moderately invested in getting a proper handle on SFDC metadata, and is also really difficult to test in any meaningful manner.

The issues I was having with ant inspired me to write a new perl module, WWW::SFDC, for interacting with Salesforce.com APIs. The aim is to provide Do-What-I-Mean wrappers around all of the SFDC APIs, such that they all work together and achieving moderately complicated interactions is as quick and easy as possible. The module is only about 40% complete, if you consider ‘complete’ to mean ‘has a wrapper around every function in the Partner, Metadata and Tooling API and is unit tested’, but it’s got enough to achieve some seriously useful stuff. For instance, the above-mentioned retrieve task can now be accomplished like this:

WWW::SFDC::Metadata->instance(
    password  => $password,
    username  => $username,
    url       => $url
);

WWW::SFDC::Zip::unzip(
	"src",
	WWW::SFDC::Metadata->instance()->retrieveMetadata(
		WWW::SFDC::Manifest->new()->add(
			WWW::SFDC::Metadata->instance()->listMetadata(
				{type => 'Document', folder => 'Apps'},
				{type => 'Document', folder => 'Developer_Documents'},
				{type => 'Document', folder => 'Documents'},
				{type => 'EmailTemplate', folder => 'Sales_Templates'},
				{type => 'Report', folder => 'Merge_Reports'},
			)
		)
	)
);

…which I think is pretty neat. There are some basic examples in SFDC.pm, and I intend to make some blog posts demonstrating things that I already use this module for. I’m not planning to release in on CPAN until it’s more complete, but it is installable from git.

Note that it also currently has a dependency on Logging::Trivial: I’m not sure what I’m going to do with this, since it’s difficult to differentiate Logging::Trivial amongst all the other logging modules, but it is my favourite! I may eventually rename Logging::Trivial, or change the logging engine to Logging::Minimal.

Tags: SFDC Perl