Fixing a common cause of System.LimitException: Apex CPU time limit exceeded

When developing code, automated unit tests and interactive testing naturally tend to use small numbers of objects. As the number of objects increases, the execution time has to increase in proportion – linearly. But it is all too easy to introduce code where the execution time grows as the square of the number of objects or the cube of the number of objects. (See e.g. Time complexity for some background.) For example, even with 100 objects, code that takes 100ms for the linear case, takes 10s for the squared case and 1,000s for the cubed case. So for the squared or cubed cases

System.LimitException: Apex CPU time limit exceeded

exceptions can result with even modest numbers of objects.

Is this governor limit a good thing? On the positive side, it forces a bad algorithm to be replaced by a better one. But on the negative side, your customer is stuck unable to work until you can deliver a fix to them. Some sort of alarm and forgiveness from the platform for a while would be more helpful…

Here is a blatant example of the problem (assuming all the collections include large numbers of objects):

// Linear - OK
for (Parent__c p : parents) {
    // Squared - problem
    for (Child__c c : children) {
        // Cubed - big problem
        for (GrandChild__c bc : grandChildren) {
            // ...
        }
    }
}

So review all nested loops carefully. Sometimes the problem is hidden, with one loop in one method or class and another loop in another method or class, and so harder to find.

Often the purpose of the loops is just to find the objects/object in one collection that match an object in another collection. There are two contexts that require two different approaches (though in more complicated cases the approaches can be combined) to fix the problem or better to avoid it in the first place:

  • In e.g. a controller where the query is explicit, a parent and child can be queried together with direct relationship references available using a relationship query.
  • In a trigger, no __r collections are populated, so maps have to be used. A map allows a value to be looked up without the cost of going through every entry in a list.

Here is how to fix the problem for the two contexts in parent-child relationships:

// Explicit SOQL context
for (Parent__c p : [
        select Name, (select Name from Childs__r)
        from Parent__c
        ]) {
    // Loop is over the small number of related Child__c not all of the Child__c
    for (Child__c c : p.Childs__r) {
        // ...
    }
}


// Trigger context
Map<Id, List<Child__c>> children = new Map<Id, List<Child__c>>();
for (Child__c c : [
        select Parent__c, Name
        from Child__c
        where Parent__c in Trigger.newMap.keySet()
        ]) {
    List<Child__c> l = children.get(c.Parent__c);
    if (l == null) {
        l = new List<Child__c>();
        children.put(c.Parent__c, l);
    }
    l.add(c);
}
for (Parent__c p : Trigger.new) {
    // Loop is over the small number of related Child__c not all of the Child__c
    if (children.containsKey(p.Id)) {
        for (Child__c c : children.get(p.Id) {
            // ...
        }
    }
}

And for the two contexts in child-parent relationships:

// Explicit SOQL context
for (Child__c c : [
        select Name, Parent__r.Name
        from Child__c
        ]) {
    // The one parent
    Parent__c p = c.Parent__r;
    if (p != null) {
        // ...
    }
}


// Trigger context
Set<Id> parentIds = new Set<Id>();
for (Child__c c : Trigger.new) {
    if (c.Parent__c != null) {
        parentIds.add(c.Parent__c);
    }
}
if (parentIds.size() > 0) {
    Map<Id, Parent__c> parents = new Map<Id, Parent__c>([
            select Id, Name
            from Parent__c
            where Id in :parentIds
            ]);
    for (Child__c c : Trigger.new) {
        // The one parent
        if (c.Parent__c != null) {
            Parent__c p = parents.get(c.Parent__c);
            if (p != null) {
                // ...
            }
        }
    }
}

This fixed code will operate in close to linear time.

Advertisements

Avoiding “too many concurrent batch jobs” in an InstallHandler

A managed package application I work on has a data model that makes it necessary to calculate many values via triggers rather than via SObject formulas in response to specific data changes. The application has been developed incrementally meaning that the fields and logic were introduced in various versions. So InstallHandler code is needed to populate those fields when an upgrade is done.

The original approach to this was to make a series of calls to Database.executeBatch with a Database.Batchable passed in to each one which worked fine until there were 6 of these calls. Then when upgrading an early version (or running the install handler’s tests) this governor limit came in to play:

System.LimitException: Attempted to schedule too many concurrent batch jobs in this org (limit is 5).

So how to get the many benefits of using batch Apex while limiting the number of batch jobs created? The approach shown below makes use of batch chaining where in the finish method of one batchable kicks off the next batchable. To allow the existing batchables to be re-used without change, they are wrapped in a ChainedJob class. This class holds the reference to the next ChainedJob, and delegates to the batchable it wraps; it also holds the scope size value used to execute the batchable.

The install handler can then be coded like this where zero or more ChainedJobs are created and then executed in a chain:

public class XyzInstallHandler implements InstallHandler {

    private Version previousVersion;
    
    public void onInstall(InstallContext context) {
        
        previousVersion = context.previousVersion();
        
        List<ChainedJob> jobs = new List<ChainedJob>();
        if (isInstalledVersionBefore(new Version(2, 12))) {
            jobs.add(new ChainedJob(new ContactBatchable(), 200, 'contact fields'));
        }
        if (isInstalledVersionBefore(new Version(2, 19))) {
            jobs.add(new ChainedJob(new AccountBatchable(), 1000, 'account fields'));
        }
        if (isInstalledVersionBefore(new Version(2, 44))) {
            jobs.add(new ChainedJob(new BenefitBatchable(), 200, 'benefit fields'));
            jobs.add(new ChainedJob(new PolicyBatchable(), 200, 'policy fields'));
        }
        if (isInstalledVersionBefore(new Version(3, 2))) {
            jobs.add(new ChainedJob(new PaidBatchable(), 100, 'paid fields'));
        }
        if (isInstalledVersionBefore(new Version(3, 29))) {
            jobs.add(new ChainedJob(new IncurredBatchable(), 100, 'incurred fields'));
        }
        ChainedJob.executeAsChain(jobs);
    }
    
    private Boolean isInstalledVersionBefore(Version version) {
        return previousVersion != null && previousVersion.compareTo(version) < 0;
    }
}

The ChainedJob class implements Database.Stateful so that the values of its members are maintained through the series of transactions. Note that the class is hard coded to expect the batchable start method to return a Database.QueryLocator (which is the pattern that offers the highest governor limit capability).

public class ChainedJob implements Database.Batchable<SObject>, Database.Stateful {
    
    public String description {get; private set;}
    
    private Database.Batchable<SObject> delegate;
    private Integer scope;
    private ChainedJob nextJob;
    
    public ChainedJob(
            Database.Batchable<SObject> delegate,
            Integer scope,
            String description
            ) {
        this.delegate = delegate;
        this.scope = scope;
        this.description = description;
    }
    
    public Database.QueryLocator start(Database.BatchableContext context) {
        return (Database.QueryLocator) delegate.start(context);
    }
    
    public void execute(Database.BatchableContext context, List<SObject> scope) {
        delegate.execute(context, scope);
    }
    
    public void finish(Database.BatchableContext context) {
        try {
            delegate.finish(context);
        } finally {
            if (nextJob != null) {
                // Next job
                nextJob.execute();
            }
        }
    }
    
    // Create the list and then invoke this method
    public static Id executeAsChain(List<ChainedJob> jobs) {
        if (jobs.size() > 0) {
            for (Integer i = 0; i < jobs.size() - 1; i++) {
                jobs[i].nextJob = jobs[i + 1];
            }
            // First job
            return jobs[0].execute();
        } else {
            return null;
        }
    }
    
    private Id execute() {
        System.debug('>>> executing "' + this.description + '"');
        return Database.executeBatch(this, scope);
    }
}

Winter ’14 – Maximum CPU time woes

From an org that has multiple managed packages installed plus additional source code:

Number of code statements: 148143 out of 200000 ******* CLOSE TO LIMIT
Maximum CPU time: 37266 out of 10000 ******* CLOSE TO LIMIT

Looks like a situation where counting the governor limit across all namespaces rather than per namespace really hurts. See e.g. Script Limits, Begone for the background to this.

Winter ’14 – no more code statement limit

Having bumped up against the 200,000 code statement governor limit in some valid cases, it’s good to see that this safety mechanism is changing (presumably) for the better:

There is no more code statement limit. Although we’ve removed this limit, we haven’t removed the safety mechanism it provided. Instead we’ve limited the CPU time for transactions. The limits are now 10,000ms for synchronous Apex and 60,000ms for asynchronous Apex.

(From Winter ’14 Developer Preview; see the clarifying comments too.)

Edit: Also see Script Limits, Begone!

Some lessons learned delivering an enterprise application on Force.com

The app this is based on uses dozens of related custom objects. And pretty much always customers want to add further custom fields and customization logic. The app is also constantly having new features added. These characteristics obviously influence the lessons learned…

  1. Managed released packages work well and so do patch releases. At this point we have created hundreds of versions as we develop incrementally. (Managed beta packages are fairly useless as they don’t support upgrading.) The version compatibility constraints leave junk in the data model and don’t protect against all breaking changes. Customers expect it to be able to back out an upgrade but unfortunately that isn’t supported. There is a major pain point in that the upgrade process does not propagate new versions of things like layouts.
  2. Customer-specific extensions can be delivered using a further managed released package that builds on the core app one or via the Ant deploy task. Given that the former approach does not propagate many common changes on update, the latter approach is usually the best one to use.
  3. Version management just about works (for small teams). The managed packaged org is the definitive version: your Git or SVN copy is your best attempt to track it and doesn’t include information such as the exact set of components in the package. (We use the convention that everything in the org should be in the package.)
  4. The Eclipse based Force.com IDE is poor (as it has had only basic maintenance investment over the last few years) but necessary. It allows the code and other artifacts to be seen and searched as a whole and is the best place to see change and version information.
  5. Unit tests are the safety net that let your managed package change and grow. Use continuous integration server to make sure they are run frequently (and to make sure what you have in Git or SVN is complete).
  6. Governor limit violations mostly but not always result from a design error such as non-linear algorithms (hitting the code statements limit) or non-bulkified queries (hitting the number of SOQL queries limit). But they do come up in other situations. For example, your managed package controller starts some processing that consumes some of its governor limits directly and in its triggers; a trigger added by your customer also runs and updates some of your objects consuming from its own set of governor limits; this then causes triggers back in the managed package to consume more of the managed package governor limits exceeding a limit. Whatever the cause, hitting these limits is a brutal problem: where on other platforms performance might become poor, on Force.com your app is broken and your customer is left unable to work.
  7. There are two eras in how open for extension a Force.com app is. The boundary between these eras is the availability in Summer ’12 of Type.forName and Type.newInstance. These allows extension points to be built into an app and extension code to be added by supplying the names of classes that implement the required interfaces. JSON strings in static resources or directly in custom settings are also a help. Before this it was very hard to add external code.
  8. Embrace the use of Contact and Account. As well as helping your app share this information with other apps, some API calls require the Id value passed to be a Contact Id.
  9. Dynamic SOQL is inelegant (as you lose compile time checks) but necessary in some cases. An example is where you want to do a deep clone of part of your object graph: use describe calls to get all the field names for each object so that custom fields that your managed package does not know about – ones added for specific customers – are included.
  10. While the execution order of before triggers compared to after triggers is defined and important to know and make use of, the execution order of separate triggers for the same event e.g. “before update” on a Contact is not defined. Within one code base discipline and patterns must be applied to define the order. But this problem is hard to address across multiple code bases e.g. your app and extending code triggers.
  11. While it is tempting to write an Apex classes that deals with a single object on the grounds that the code will be cleaner and “it’ll never be used in a bulk situation”, sometime in the future it probably will be used in a bulk situation and result on a governor limit error that stops some important data migration or change from being possible. Always deal in sets of objects.
  12. Client-side JavaScript using e.g. jQuery is a must-have these days. It is easy to introduce into Visualforce pages but has to be hacked into standard layout based UI – see TehNrd’s Show and Hide Buttons on Page Layouts. See Using JavaScript with Force.com for a good overview of other JavaScript techniques.
  13. Sometimes you have the difficult choice between delivering poor usability or instead depending on platform implementation that may change e.g. hack to find field ids.
  14. Profiles are hugely painful to keep updated over time and are rarely tested well enough. Do everything you can to keep the number of them in use small.
  15. On custom settings, and particularly if you are using profile or user-specific hierarchical ones, you will eventually need to use code to keep them updated consistently.

Parsing CSV while staying within the “total number of executed script statements” governor limit

I have some sample data included as static resources in a Force.com application and a setup page that gives the user the option of loading this data into some custom objects. The files are Comma Separated Value (CSV) files exported by the Apex Data Loader.

(Incidentally this code supports automatic relationship id fixup and also self-referencing objects such as Contact that has a ReportsTo field that is a lookup to another Contact. So a graph of related objects can be loaded.)

Parsing a CSV file is easy if things like line feeds, quotes and escape characters embedded in values are ignored:

String[] fields = line.split(',', -1);

Note that the -1 argument stops trailing empty fields from being discarded.

But the logic to handle the more complicated cases is pretty messy and not easy to get right. So I ported the opencsv code from Java to Apex and this code worked well for a while. Then recently the amount of sample data increased and the code hit the “Total number of executed script statements” governor limit of 200,000. (Note that in Winter ’14 this limit is being replaced by a CPU timeout limit that should result in a higher number of statements being allowed but I have not yet seen any quantitative data about how much more…) Given that the compute capacity already appears to be throttled as part of the multi-tenant architecture, there is a natural limitation imposed by how long your users are prepared to wait. So this brick-wall limit that means that at best you have to optimize your code and at worst you have to throw your code away is really annoying. It reminds me of the bad old days when platform limitations often got in the way of getting functionality of value to the business implemented.

To optimize the code, I first used some unit test cases and Limits.getScriptStatements calls to discover that the CSV parsing was costing about 100 statements per value because the algorithm was working character by character. My first optimization attempts only got me about a 20% improvement in return for some pretty ugly code changes – not the way to go. The core problem was processing characters in Apex rather than whole values or whole lines. So before processing a line the code now checks for the embedded line feeds, quotes and escape characters. If they are found (the rare case for my data) the expensive opencsv parsing is used and if they are not found (the common case for my data) the cheap String.split as shown above is used. The result is a 90% improvement (ten times less script statements executed) which gives me plenty of headroom for the near future.