Avoiding “too many concurrent batch jobs” in an InstallHandler

A managed package application I work on has a data model that makes it necessary to calculate many values via triggers rather than via SObject formulas in response to specific data changes. The application has been developed incrementally meaning that the fields and logic were introduced in various versions. So InstallHandler code is needed to populate those fields when an upgrade is done.

The original approach to this was to make a series of calls to Database.executeBatch with a Database.Batchable passed in to each one which worked fine until there were 6 of these calls. Then when upgrading an early version (or running the install handler’s tests) this governor limit came in to play:

System.LimitException: Attempted to schedule too many concurrent batch jobs in this org (limit is 5).

So how to get the many benefits of using batch Apex while limiting the number of batch jobs created? The approach shown below makes use of batch chaining where in the finish method of one batchable kicks off the next batchable. To allow the existing batchables to be re-used without change, they are wrapped in a ChainedJob class. This class holds the reference to the next ChainedJob, and delegates to the batchable it wraps; it also holds the scope size value used to execute the batchable.

The install handler can then be coded like this where zero or more ChainedJobs are created and then executed in a chain:

public class XyzInstallHandler implements InstallHandler {

    private Version previousVersion;
    
    public void onInstall(InstallContext context) {
        
        previousVersion = context.previousVersion();
        
        List<ChainedJob> jobs = new List<ChainedJob>();
        if (isInstalledVersionBefore(new Version(2, 12))) {
            jobs.add(new ChainedJob(new ContactBatchable(), 200, 'contact fields'));
        }
        if (isInstalledVersionBefore(new Version(2, 19))) {
            jobs.add(new ChainedJob(new AccountBatchable(), 1000, 'account fields'));
        }
        if (isInstalledVersionBefore(new Version(2, 44))) {
            jobs.add(new ChainedJob(new BenefitBatchable(), 200, 'benefit fields'));
            jobs.add(new ChainedJob(new PolicyBatchable(), 200, 'policy fields'));
        }
        if (isInstalledVersionBefore(new Version(3, 2))) {
            jobs.add(new ChainedJob(new PaidBatchable(), 100, 'paid fields'));
        }
        if (isInstalledVersionBefore(new Version(3, 29))) {
            jobs.add(new ChainedJob(new IncurredBatchable(), 100, 'incurred fields'));
        }
        ChainedJob.executeAsChain(jobs);
    }
    
    private Boolean isInstalledVersionBefore(Version version) {
        return previousVersion != null && previousVersion.compareTo(version) < 0;
    }
}

The ChainedJob class implements Database.Stateful so that the values of its members are maintained through the series of transactions. Note that the class is hard coded to expect the batchable start method to return a Database.QueryLocator (which is the pattern that offers the highest governor limit capability).

public class ChainedJob implements Database.Batchable<SObject>, Database.Stateful {
    
    public String description {get; private set;}
    
    private Database.Batchable<SObject> delegate;
    private Integer scope;
    private ChainedJob nextJob;
    
    public ChainedJob(
            Database.Batchable<SObject> delegate,
            Integer scope,
            String description
            ) {
        this.delegate = delegate;
        this.scope = scope;
        this.description = description;
    }
    
    public Database.QueryLocator start(Database.BatchableContext context) {
        return (Database.QueryLocator) delegate.start(context);
    }
    
    public void execute(Database.BatchableContext context, List<SObject> scope) {
        delegate.execute(context, scope);
    }
    
    public void finish(Database.BatchableContext context) {
        try {
            delegate.finish(context);
        } finally {
            if (nextJob != null) {
                // Next job
                nextJob.execute();
            }
        }
    }
    
    // Create the list and then invoke this method
    public static Id executeAsChain(List<ChainedJob> jobs) {
        if (jobs.size() > 0) {
            for (Integer i = 0; i < jobs.size() - 1; i++) {
                jobs[i].nextJob = jobs[i + 1];
            }
            // First job
            return jobs[0].execute();
        } else {
            return null;
        }
    }
    
    private Id execute() {
        System.debug('>>> executing "' + this.description + '"');
        return Database.executeBatch(this, scope);
    }
}
Advertisements

2 thoughts on “Avoiding “too many concurrent batch jobs” in an InstallHandler

  1. I was getting ‘Attempted to schedule too many concurrent batch jobs in this org (limit is 5)’ error when I called Database.executeBatch(batchName, 100) the 6th time. (Not unexpected.)

    After implementing this post, lo-and-behold, no more errors and all 9 of my batch jobs appear to execute. Thank you!

    But I don’t understand the mechanism that allows this to work. Is it because each subsequent batch job’s execute method is called from the finish method and therefore, there is never more than 2 jobs processing – the one that’s ending and the one that it calls in its finish method?

    In any event, well-done.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s