Asynchronous processing in Force.com

This (2 year old) Salesforce document Asynchronous Processing in Force.com offers good insight into how asynchronous Apex is processed including concepts like “extended delay” and diagrams like this one:

PeekSet

It also offers best practice advice. Well worth a read if you are using @future or Batchable or the Bulk API.

Apex Jobs error reporting – “Status Detail” needs to be bigger

Force.com provides the Batchable mechanism to allow the processing of large volumes of data to be broken down into batches and spread out over time. This helps Salesforce level demand on its servers. The Apex Jobs list view in Setup provides some basic monitoring and error reporting.

But in a world of nested exceptions where the reported exception may have a “caused by” exception attached (and perhaps multiple levels of this), I can affirm that the 255 character “Status Detail” field (AsyncApexJob.ExtendedStatus) is too short. Here is the error report from a batchable I ran today:

First error: Update failed. First exception on row 0 with id a0AE0000009W2N9MAK; first error: CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY, cve.BenefitClaimedAfterTrigger: execution of AfterUpdate

caused by: System.DmlException: Update failed. First exception…

Pretty hard to figure out the root cause based on the text “…”.

Note that implementing your own pattern of error reporting inside your Batchable won’t always help as some exceptions (such as governor limit ones) can’t be caught. So please vote for Apex Jobs listing “Status Detail” improvements for the benefit of (probably far) future generations of Force.com developers.

Avoiding “too many concurrent batch jobs” in an InstallHandler

A managed package application I work on has a data model that makes it necessary to calculate many values via triggers rather than via SObject formulas in response to specific data changes. The application has been developed incrementally meaning that the fields and logic were introduced in various versions. So InstallHandler code is needed to populate those fields when an upgrade is done.

The original approach to this was to make a series of calls to Database.executeBatch with a Database.Batchable passed in to each one which worked fine until there were 6 of these calls. Then when upgrading an early version (or running the install handler’s tests) this governor limit came in to play:

System.LimitException: Attempted to schedule too many concurrent batch jobs in this org (limit is 5).

So how to get the many benefits of using batch Apex while limiting the number of batch jobs created? The approach shown below makes use of batch chaining where in the finish method of one batchable kicks off the next batchable. To allow the existing batchables to be re-used without change, they are wrapped in a ChainedJob class. This class holds the reference to the next ChainedJob, and delegates to the batchable it wraps; it also holds the scope size value used to execute the batchable.

The install handler can then be coded like this where zero or more ChainedJobs are created and then executed in a chain:

public class XyzInstallHandler implements InstallHandler {

    private Version previousVersion;
    
    public void onInstall(InstallContext context) {
        
        previousVersion = context.previousVersion();
        
        List<ChainedJob> jobs = new List<ChainedJob>();
        if (isInstalledVersionBefore(new Version(2, 12))) {
            jobs.add(new ChainedJob(new ContactBatchable(), 200, 'contact fields'));
        }
        if (isInstalledVersionBefore(new Version(2, 19))) {
            jobs.add(new ChainedJob(new AccountBatchable(), 1000, 'account fields'));
        }
        if (isInstalledVersionBefore(new Version(2, 44))) {
            jobs.add(new ChainedJob(new BenefitBatchable(), 200, 'benefit fields'));
            jobs.add(new ChainedJob(new PolicyBatchable(), 200, 'policy fields'));
        }
        if (isInstalledVersionBefore(new Version(3, 2))) {
            jobs.add(new ChainedJob(new PaidBatchable(), 100, 'paid fields'));
        }
        if (isInstalledVersionBefore(new Version(3, 29))) {
            jobs.add(new ChainedJob(new IncurredBatchable(), 100, 'incurred fields'));
        }
        ChainedJob.executeAsChain(jobs);
    }
    
    private Boolean isInstalledVersionBefore(Version version) {
        return previousVersion != null && previousVersion.compareTo(version) < 0;
    }
}

The ChainedJob class implements Database.Stateful so that the values of its members are maintained through the series of transactions. Note that the class is hard coded to expect the batchable start method to return a Database.QueryLocator (which is the pattern that offers the highest governor limit capability).

public class ChainedJob implements Database.Batchable<SObject>, Database.Stateful {
    
    public String description {get; private set;}
    
    private Database.Batchable<SObject> delegate;
    private Integer scope;
    private ChainedJob nextJob;
    
    public ChainedJob(
            Database.Batchable<SObject> delegate,
            Integer scope,
            String description
            ) {
        this.delegate = delegate;
        this.scope = scope;
        this.description = description;
    }
    
    public Database.QueryLocator start(Database.BatchableContext context) {
        return (Database.QueryLocator) delegate.start(context);
    }
    
    public void execute(Database.BatchableContext context, List<SObject> scope) {
        delegate.execute(context, scope);
    }
    
    public void finish(Database.BatchableContext context) {
        try {
            delegate.finish(context);
        } finally {
            if (nextJob != null) {
                // Next job
                nextJob.execute();
            }
        }
    }
    
    // Create the list and then invoke this method
    public static Id executeAsChain(List<ChainedJob> jobs) {
        if (jobs.size() > 0) {
            for (Integer i = 0; i < jobs.size() - 1; i++) {
                jobs[i].nextJob = jobs[i + 1];
            }
            // First job
            return jobs[0].execute();
        } else {
            return null;
        }
    }
    
    private Id execute() {
        System.debug('>>> executing "' + this.description + '"');
        return Database.executeBatch(this, scope);
    }
}

Testing a Database.Batchable implementation

I have some processing code that makes use of batch Apex to avoid hitting governor limits. Running a representative test generates this error:

System.UnexpectedException: No more than one executeBatch can be called from within a testmethod. Please make sure the iterable returned from your start method matches the batch size, resulting in one executeBatch invocation.

Testability is an important feature but sadly missing in a few areas of Force.com including this one…

The normal work-around is suggested in the error message. But using that means that the code does not get tested across a batch boundary: such testing is particularly important for Database.Batchable implementations that also implement Database.Stateful to maintain state across batches.

My processing code also makes use of batch chaining where in the finish method of the Database.Batchable sometimes a further instance is created and executed. Unfortunately (but not surprisingly) that chained execution also generates the above error and there is no easy work-around.

Below is a small class I created to work-around both these problems. Instead of invoking Database.executeBatch, invoke BatchableExecutor.executeBatch. When called from a test, this method makes the start/execute(s)/finish pattern of calls synchronously itself and so avoids the above errors. As long as the test uses a small batch size e.g. 3 and makes sure a moderate number of records are returned by the start method e.g. 10 the Database.Batchable logic can be pretty fully tested without hitting any governor limits.

/*
 * Allows basic testing of a Database.Batchable using more than one batch.
 */
public class BatchableExecutor {
    
    private static final String KEY_PREFIX = AsyncApexJob.SObjectType.getDescribe().getKeyPrefix();
    
    public static Id executeBatch(Database.Batchable<SObject> batchable, Integer scopeSize) {
        
        if (!Test.IsRunningTest()) {
            return Database.executeBatch(batchable, scopeSize);
        } else {
            return executeBatchSynchronously(batchable, scopeSize);
        }
    }
    
    private static Id executeBatchSynchronously(Database.Batchable<SObject> batchable, Integer scopeSize) {
        
        // Fake implementation of this interface could be added as neeed
        Database.BatchableContext bc = null;
        
        // Invoke start (assumes QueryLocator is being used)
        Database.QueryLocator start = (Database.QueryLocator) batchable.start(bc);
        Database.QueryLocatorIterator iter = start.iterator();
        List<SObject> sobs = new List<SObject>();
        try {
            // Invoke execute
            while(iter.hasNext()) {
                sobs.add(iter.next());
                if (sobs.size() == scopeSize) {
                    // These calls could be wrapped in try/catch too for negative tests
                    batchable.execute(bc, sobs);
                    sobs.clear();
                }
            }
            if (sobs.size() > 0) {
                batchable.execute(bc, sobs);
            }
        } finally {
            // Invoke finish
            batchable.finish(bc);
        }
        
        // Fake id
        return KEY_PREFIX + '000000000000';
    }
}

Bear in mind that this code does an OK job of emulating the happy path only (as demonstrated by the test below). Also it obviously cannot emulate the transactions, asynchronous execution and object lifecycle of the real mechanism.

@isTest
private class BatchableExecutorTest {
    
    private class Fixture {
        List<Account> accounts = new List<Account>();
        Fixture addAccounts(Integer objectCount) {
            for (Integer i = 0; i < objectCount; i++) {
                accounts.add(new Account(Name = 'target-' + i, Site = null));
            }
            insert accounts;
            return this;
        }
        Fixture execute(Boolean useDatabaseExecuteBatch, Integer batchSize) {
            Test.startTest();
            // Test that both mechanisms produce the same results
            Id jobId = useDatabaseExecuteBatch
                    ? Database.executeBatch(new BatchableExecutorTestBatchable(), batchSize)
                    : BatchableExecutor.executeBatch(new BatchableExecutorTestBatchable(), batchSize)
                    ;
            Test.stopTest();
            System.assertNotEquals(null, jobId);
            return this;
        }
        Fixture assert(Integer expectedBatches) {
            System.assertEquals(1, [select Count() from Account where Name = 'start']);
            System.assertEquals(expectedBatches,  [select Count() from Account where Name = 'execute']);
            System.assertEquals(1,  [select Count() from Account where Name = 'finish']);
            System.assertEquals(accounts.size(), [select Count() from Account where Name like 'target-%' and Site = 'executed']);
            return this;
        }
    }
    
    @isTest
    static void batchableExecutorExecuteBatch() {
        new Fixture().addAccounts(10).execute(false, 3).assert(4);
    }
    
    @isTest
    static void databaseExecuteBatch() {
        new Fixture().addAccounts(10).execute(true, 10).assert(1);
    }
}
// Has to be a top-level class
public class BatchableExecutorTestBatchable implements Database.Batchable<SObject>, Database.Stateful {
    
    public Database.QueryLocator start(Database.BatchableContext bc) {
        Database.QueryLocator ql = Database.getQueryLocator([select Name from Account order by Name]);
    	insert new Account(Name = 'start');
        return ql;
    }
    
    public void execute(Database.BatchableContext bc, List<SObject> scope) {
    	for (SObject sob : scope) {
    		Account a = (Account) sob;
            a.Site = 'executed';
        }
        update scope;
        insert new Account(Name = 'execute');
    }
    
    public void finish(Database.BatchableContext bc) {
    	insert new Account(Name = 'finish');
    }
}