“AsyncApexTests Limit exceeded” blocker

I’ve just hit this “AsyncApexTests Limit exceeded” error when I try to run tests in an org. Running the tests in a different way reports this:

To protect all customers from excessive usage and Denial of Service attacks, we limit the number of long-running requests that are processed at the same time by an organization. Your request has been denied because this limit has been exceeded by your organization. Please try your request again later.

Googling reveals posts like this. And the limit is documented in Understanding Execution Governors and Limits.

But having to wait 24 hours before I can continue working? FUBAR.

Some lessons learned delivering an enterprise application on Force.com

The app this is based on uses dozens of related custom objects. And pretty much always customers want to add further custom fields and customization logic. The app is also constantly having new features added. These characteristics obviously influence the lessons learned…

  1. Managed released packages work well and so do patch releases. At this point we have created hundreds of versions as we develop incrementally. (Managed beta packages are fairly useless as they don’t support upgrading.) The version compatibility constraints leave junk in the data model and don’t protect against all breaking changes. Customers expect it to be able to back out an upgrade but unfortunately that isn’t supported. There is a major pain point in that the upgrade process does not propagate new versions of things like layouts.
  2. Customer-specific extensions can be delivered using a further managed released package that builds on the core app one or via the Ant deploy task. Given that the former approach does not propagate many common changes on update, the latter approach is usually the best one to use.
  3. Version management just about works (for small teams). The managed packaged org is the definitive version: your Git or SVN copy is your best attempt to track it and doesn’t include information such as the exact set of components in the package. (We use the convention that everything in the org should be in the package.)
  4. The Eclipse based Force.com IDE is poor (as it has had only basic maintenance investment over the last few years) but necessary. It allows the code and other artifacts to be seen and searched as a whole and is the best place to see change and version information.
  5. Unit tests are the safety net that let your managed package change and grow. Use continuous integration server to make sure they are run frequently (and to make sure what you have in Git or SVN is complete).
  6. Governor limit violations mostly but not always result from a design error such as non-linear algorithms (hitting the code statements limit) or non-bulkified queries (hitting the number of SOQL queries limit). But they do come up in other situations. For example, your managed package controller starts some processing that consumes some of its governor limits directly and in its triggers; a trigger added by your customer also runs and updates some of your objects consuming from its own set of governor limits; this then causes triggers back in the managed package to consume more of the managed package governor limits exceeding a limit. Whatever the cause, hitting these limits is a brutal problem: where on other platforms performance might become poor, on Force.com your app is broken and your customer is left unable to work.
  7. There are two eras in how open for extension a Force.com app is. The boundary between these eras is the availability in Summer ’12 of Type.forName and Type.newInstance. These allows extension points to be built into an app and extension code to be added by supplying the names of classes that implement the required interfaces. JSON strings in static resources or directly in custom settings are also a help. Before this it was very hard to add external code.
  8. Embrace the use of Contact and Account. As well as helping your app share this information with other apps, some API calls require the Id value passed to be a Contact Id.
  9. Dynamic SOQL is inelegant (as you lose compile time checks) but necessary in some cases. An example is where you want to do a deep clone of part of your object graph: use describe calls to get all the field names for each object so that custom fields that your managed package does not know about – ones added for specific customers – are included.
  10. While the execution order of before triggers compared to after triggers is defined and important to know and make use of, the execution order of separate triggers for the same event e.g. “before update” on a Contact is not defined. Within one code base discipline and patterns must be applied to define the order. But this problem is hard to address across multiple code bases e.g. your app and extending code triggers.
  11. While it is tempting to write an Apex classes that deals with a single object on the grounds that the code will be cleaner and “it’ll never be used in a bulk situation”, sometime in the future it probably will be used in a bulk situation and result on a governor limit error that stops some important data migration or change from being possible. Always deal in sets of objects.
  12. Client-side JavaScript using e.g. jQuery is a must-have these days. It is easy to introduce into Visualforce pages but has to be hacked into standard layout based UI – see TehNrd’s Show and Hide Buttons on Page Layouts. See Using JavaScript with Force.com for a good overview of other JavaScript techniques.
  13. Sometimes you have the difficult choice between delivering poor usability or instead depending on platform implementation that may change e.g. hack to find field ids.
  14. Profiles are hugely painful to keep updated over time and are rarely tested well enough. Do everything you can to keep the number of them in use small.
  15. On custom settings, and particularly if you are using profile or user-specific hierarchical ones, you will eventually need to use code to keep them updated consistently.

Winter ’12 Developer Console

I didn’t take much notice of the Winter ’12 Force.com Platform Release Developer Console as there was no mention of that important word “breakpoint”. But if you are in debug hell then the features that are provided to allow debugging without having to add System.debug calls – including the ability to capture and view the heap at points marked in your source code – are worth investigating.

This (hour long) presentation demonstrates how to use the features:

It also suggests that the Eclipse tooling (and 3rd party additions to that tooling) will eventually be able to use the same API. I find the current browser-based console fragile (using Chrome) and a bit awkward to use as it packs a lot of tabs and frames into a small space; hopefully the richer UI facilities in Eclipse will help. But worse, on the sort of pages and code that require debugging, i.e. non-trivial larger ones, the volumes of data that the tooling attempts to move from server to client are so large that the time delays make the tooling barely usable at all.

Note that while this tooling can be used as you develop your managed package, when that managed package is installed in a customer’s org all the code-related debug information is unavailable (by default but can be turned back on via a support request) so the ability to debug is also lost.

Patch organizations

Just a short note on the “Patch Organization” feature that I used today for first time. Having got this turned on in your packaging org by logging a case, this extra “Patch Organization” tab appears for your package:

This gives you the ability to clone the source code of any version of your package – 3.22 in this screen shot – into a new org (that keeps the same namespace) and then modify it to create new versions e.g. 3.22.1, 3.22.2 etc. These versions can then be packaged and distributed in the normal way. So you can provide small fixes to customers that are using one version of your package while isolating those customers from other changes as your package evolves. But note that there are limits on the changes allowed – this is not a full-blown branching mechanism.

For more information, see e.g. the ISVforce Guide.

“Check all” in permission set and profile UI

I just started using the new permission sets mechanism. The first setting I needed to make was to enable full create, read and edit permission on a custom object that has 34 fields. Naturally the permissions default to unchecked. Checking three check boxes for the read/create/edit object-level permissions was one thing. But having to check close to 68 check boxes for read/edit field-level permissions was another. Across the 30 or so custom objects we use there are close to 1000 fields and with several permission sets needed overall thats an awful lot of tedious clicks and a step towards RSI.

The problem is that there is no “check all” feature for the field-level permissions, even though the UI is brand new. Supporting large scale development requires specific consideration – here are some examples from Eclipse’s past: large-scale development issues – but that does not seem to be a priority in the Force.com web tooling. Another example of UI that is painful when the scale goes up is picklist value deletion. This involves several clicks per value, not a problem if you are deleting a few values but a while ago I needed to delete a few hundred values…

The work-around I am using for the permission sets is to use the Check All extension to Chrome. This adds a button to the right of the URL text box that displays a menu that includes “Check All” and “Uncheck All” options that apply to all check boxes in the current page. Not perfect but better than nothing.

Note that if you have a small number of profile or permission set changes to make across many profiles or permission sets this Editing Profiles Using Profile Lists mechanism should help.

Add “Deployment Status” to the check-list…

Here is another item to add to the check-list (see Always double check that managed packages are deployed for some others) when objects or fields are not being displayed.

Our problem was that while the application worked fine through the development process and when tested in the production org via a “System Administrator” user, for any other profile user, the fields of two of the objects did not display at all in their Visualforce pages.

In this case the cause was that the “Deployment Status” of the objects had been accidentally left as “In Development”. This status is represented as <deploymentStatus>InDevelopment</deploymentStatus> in the .object file and appears in the development web UI like this:

Knowing what to look for in the help, it is easy to find the explanation:

Choose “In Development” as the Deployment Status when first creating your custom object to hide it from users while you are designing and testing it. Making the status “In Development” hides the custom object tab, search results, related lists, and report data types from all users except those with the “Customize Application” permission.

Fair enough functionality if you are modifying a live environment but in any approach where the software is developed separately from the live environment it is not useful and in this case was harmful. So always sticking to “Deployed” seems the way to go. And of course testing earlier under the actual profile(s) that will be used…

Free Force.com Security Source Code Scanner Revisited

A Force.com IDE (Eclipse-based) alternative to the PDF output scan results has been available for a couple of months as described in this blog post Source Code Scanning. The “Scanning a project” section of the User Guide illustrates the benefit of having the scan results within the IDE: a perspective is provided that lets you click through the results and open the class or trigger at the lines that are significant.

A couple of things to note about the IDE-based version:

  • One step of the installation process took several minutes to complete and I was tempted to just cancel the installation so be patient.
  • The scan is still done remotely and takes many minutes to complete unlike similar tools for other languages in Eclipse. So it only makes sense to run the scan occasionally.

It is a few months since I have run the scanner and it identified about a dozen issues so was well worth the effort of running again. But the results do include a lot of false positives making it fairly time consuming to separate out the real issues. Here are two examples that caused many false positives.

The first is a pattern to automatically initialize custom settings to their default values by indirecting through a class and so avoid users having to click “Manage” on all the custom settings before the application can be used. The null guard ensures that there is only one DML operation performed but the scanner (understandably) sees this as a case of “DML statements inside loops” when the custom setting is referenced inside a loop in a trigger:

global with sharing class CustomSettings {
    global static DateConversionFactors__c getDateConversionFactors() {
        if (DateConversionFactors__c.getInstance() == null) {
            upsert new DateConversionFactors__c(SetupOwnerId = UserInfo.getOrganizationId());
        }
        return DateConversionFactors__c.getInstance();
    }
    // Other similar methods for other custom settings
}

The second is something that perhaps should be addressed in the scanner and concerns the “list or set iteration for loop”. In this example some SOQL is executed in the ProcessUtil.withoutApprovalProcess which is called once before the loop. But the scanner treats it as if it is within the loop and so incorrectly reports it as a “SOQL/SOSL statements inside loops” issue:

for (Id anId : ProcessUtil.withoutApprovalProcess(ids)) {
    // Other logic goes here
}

Pattern for handling multiple approvals at once in an approval process

I had a requirement to check for 8 different factors that could require an object to be sent for approval in an approval process. Also the approver needed to be told about all the factors that applied and approve or reject them in one go. This requirement does not fit to easily into the available approval process mechanisms. (If I’m wrong and there is a cleaner way to do this please comment.)

Here is the pattern that allows this behavior:

  • Add a text area custom field to the object called something like “ApprovalRequiredReasons”.
  • Put all the approval logic – in my case 8 checks on fields of the object and fields of parent objects – in an “Initial Submission Action” of type “Field Update” that updates the “ApprovalRequiredReasons” field. The formula builds a new-line separated text string explaining the factors that apply. Here is an example of the style of formula to use:
    IF(booleanExpression1, 'Factor 1 explanation' + BR(), '')
    + IF(booleanExpression2, 'Factor 2 explanation' + BR(), '')
    + IF(booleanExpression3, 'Factor 3 explanation' + BR(), '')
    + ...
  • Then the approval steps just reference the “ApprovalRequiredReasons” field and so avoid duplicating any of the logic. In the simplest case they can just check whether the string has a length greater than zero or not. But in my case a two step approval was sometimes required and I could achieve this by matching to two specific values in the string in two approval steps.

The email template that is sent can include the “ApprovalRequiredReasons” as can the approval page layout:

Quick summary of how to get started with the Metadata WSDL API via Java’s JAXB

The Metadata API is used to manipulate customization information in an org including Apex classes and Visualforce pages by code such as the Force IDE. I wanted to create an Ant task to automate the creation of a large’ish number of static resources from files via this API and so following on from the Quick summary of how to get started with the Enterprise WSDL API via Java’s JAXB here are few extra things I discovered relating to the Metadata API.

The first is that the Metadata WSDL obtained from Setup -> Develop -> API -> Generate Metadata WSDL had a schema element that was missing the attribute xmlns=”http://www.w3.org/2001/XMLSchema&#8221; that puts it in the correct namespace and so this had to be manually added before the JAXB code generation would work. Also, as in the previous post, a JAXB customization file that had the targetNamespace of its bindings element set to “http://soap.sforce.com/2006/04/metadata&#8221; needed to be created.

The second is that you need classes generated from the Partner WSDL or the Enterprise WSDL to be able to do the login and obtain the session id that is then set in the session header from the Metadata WSDL. I ended up with a project with both the Partner and Metadata WSDL code generated to be able to do this. A bit of care is then needed to use the classes from the correct package.

Lastly, when you cut and paste code that relates to the Partner WSDL or Enterprise WSDL, the endpoint that you use from the login response may accidentally end up being loginResponse.getResult().getServerUrl() instead of the one you need for the Metadata WSDL case of loginResponse.getResult().getMetadataServerUrl().

And that is it if a “quick summary” is what you are interested in. If you want to see some sample code read on…

This example of how to use the Metadata API is obfuscated a little by the code being in the form of an Ant task. All you need to know to understand the Ant part is that the entry point is the execute method and attributes that have set methods can be configured via XML from a build file.

This base class handles the login and connection leaving the specifics of Metadata API or Partner API operations to extending classes:

package com.claimvantage.ant;

import java.net.URL;

import javax.xml.namespace.QName;
import javax.xml.ws.BindingProvider;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;

import com.sforce.soap.metadata.MetadataPortType;
import com.sforce.soap.metadata.MetadataService;
import com.sforce.soap.partner.Login;
import com.sforce.soap.partner.LoginResponse;
import com.sforce.soap.partner.SforceService;
import com.sforce.soap.partner.Soap;

public abstract class ForceApiTaskBase extends Task {
    
    private String un;
    private String pw;
    
    private LoginResponse loginResponse;
    
    private MetadataPortType port;
    private com.sforce.soap.metadata.SessionHeader metaDataSessionHeader;
    
    private Soap soap;
    private com.sforce.soap.partner.SessionHeader partnerSessionHeader;

    public String getUn() {
        return un;
    }

    public void setUn(String un) {
        this.un = un;
    }

    public String getPw() {
        return pw;
    }

    public void setPw(String pw) {
        this.pw = pw;
    }

    public void execute() throws BuildException {
        try {
            doExecute();
        } catch (BuildException e) {
            throw e;
        } catch (Exception e) {
            throw new BuildException(e);
        }
    }
    
    protected abstract void doExecute() throws Exception;
    
    protected MetadataPortType getMetadataPortType() throws Exception {
        createMetadataSession();
        return port;
    }
    
    protected com.sforce.soap.metadata.SessionHeader getMetadataSessionHeader() throws Exception {
        createMetadataSession();
        return metaDataSessionHeader;
    }
    
    protected Soap getPartnerPortType() throws Exception {
        createPartnerSession();
        return soap;
    }
    
    protected com.sforce.soap.partner.SessionHeader getPartnerSessionHeader() throws Exception {
        createPartnerSession();
        return partnerSessionHeader;
    }
    
    private void createMetadataSession() throws Exception {
        
        if (port == null) {
            
            // Login done here
            createPartnerSession();
            
            MetadataService service = new MetadataService(getUrl("metadata.wsdl"), new QName("http://soap.sforce.com/2006/04/metadata", "MetadataService"));
            port = service.getMetadata();
            
            BindingProvider b = (BindingProvider) port;
            b.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, loginResponse.getResult().getMetadataServerUrl());
            
            metaDataSessionHeader = new com.sforce.soap.metadata.SessionHeader();
            metaDataSessionHeader.setSessionId(loginResponse.getResult().getSessionId());
        }
    }

    private void createPartnerSession() throws Exception {
        
        if (loginResponse == null) {
            
            SforceService service = new SforceService(getUrl("partner.wsdl"), new QName("urn:partner.soap.sforce.com", "SforceService"));
            soap = service.getSoap();
            
            // Login
            log("connecting as user " + getUn());
            Login login = new Login();
            login.setUsername(getUn());
            login.setPassword(getPw());
            
            loginResponse = soap.login(login, null, null);
            
            BindingProvider b = (BindingProvider) soap;
            b.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, loginResponse.getResult().getServerUrl());
            
            partnerSessionHeader = new com.sforce.soap.partner.SessionHeader();
            partnerSessionHeader.setSessionId(loginResponse.getResult().getSessionId());
        }
    }
    
    private URL getUrl(String name) throws Exception {
        
        // WSDL must be packaged with code
        URL url = getClass().getResource(name);
        if (url != null) {
            return url;
        } else {
            throw new Exception("Could not access WSDL from " + name);
        }
    }
}

And here is an example of an Ant task that makes use of the Metadata API. (The polling is needed to deal with the asynchronous nature of the API.) The Ant task transfers a set of files to an org as static resources:

package com.claimvantage.ant;

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Project;

import com.sforce.soap.metadata.AsyncRequestStateType;
import com.sforce.soap.metadata.AsyncResultType;
import com.sforce.soap.metadata.CheckStatus;
import com.sforce.soap.metadata.Create;
import com.sforce.soap.metadata.Delete;
import com.sforce.soap.metadata.StaticResourceCacheControlType;
import com.sforce.soap.metadata.StaticResourceType;

/**
 * Transfer files into an org as static resources.
 */
public class LoadStaticResources extends ForceApiTaskBase {
    
    // Text between this and the extension (if present) used as the description and stripped from the resource name
    private static final String DESCRIPTION_SEPARATOR = "-";
    
    // Text after this stripped
    private static final String EXTENSION_SEPARATOR = ".";
    
    private String contentType;
    private File folder;

    public void setContentType(String contentType) {
        this.contentType = contentType;
    }

    // All files in this folder pushed to server
    public void setFolder(File folder) {
        this.folder = folder;
    }

    protected void doExecute() throws Exception {
        
        if (contentType == null) {
            throw new BuildException("contentType must be set");
        }
        if (folder == null) {
            throw new BuildException("folder must be set");
        }
        if (!folder.isDirectory()) {
            throw new BuildException("folder must be a directory");
        }
        
        File[] filesArray = folder.listFiles();
        if (filesArray == null || filesArray.length == 0) {
            log(folder + " is empty");
            return;
        }
        List<File> files = Arrays.asList(filesArray);

        // API limits to this many at a time
        final int max = 10;
        for (int batch = 0; true; batch++) {
            
            int from = max * batch;
            int to = Math.min(files.size(), from + max);
            
            doBatch(files.subList(from, to));

            if (to >= files.size()) {
                break;
            }
        }
        log("Done");
    }
    
    private void doBatch(List<File> files) throws Exception {
        
        Delete delete = new Delete();
        for (File file : files) {
            StaticResourceType sr = new StaticResourceType();
            sr.setFullName(resourceName(file.getName()));
            delete.getMetadata().add(sr);
        }
        waitForCompletion(files, getMetadataPortType().delete(delete, getMetadataSessionHeader(), null).getResult(), "delete");
        
        Create create = new Create();
        for (File file : files) {
            StaticResourceType sr = new StaticResourceType();
            sr.setFullName(resourceName(file.getName()));
            sr.setDescription(resourceDecription(file.getName()));
            sr.setContent(read(file));
            sr.setContentType(contentType);
            sr.setCacheControl(StaticResourceCacheControlType.PRIVATE);
            create.getMetadata().add(sr);
        }
        waitForCompletion(files, getMetadataPortType().create(create, getMetadataSessionHeader(), null).getResult(), "create");
    }
    
    private void waitForCompletion(List<File> files, List<AsyncResultType> results, String operation) throws Exception {
        
        CheckStatus check = new CheckStatus();
        for (AsyncResultType result : results) {
            check.getAsyncProcessId().add(result.getId());
        }

        int sleepSeconds = 1;
        boolean notAllDone = true;
        while(notAllDone) {
            notAllDone = false;

            log("waiting for async result - sleeping for " + sleepSeconds + " s");
            Thread.sleep(sleepSeconds * 1000L);
            if (sleepSeconds < 64) {
                sleepSeconds *= 2;
            }

            int i = 0;
            for (AsyncResultType result : getMetadataPortType().checkStatus(check, getMetadataSessionHeader(), null).getResult()) {
                int level = result.getState() == AsyncRequestStateType.ERROR ? Project.MSG_ERR : Project.MSG_INFO;
                log(operation + " " + result.getState() + " " + files.get(i).getName() + " " + (result.getMessage() != null ? result.getMessage() : ""), level);
                if (result.getState() == AsyncRequestStateType.QUEUED || result.getState() == AsyncRequestStateType.IN_PROGRESS) {
                    notAllDone = true;
                }
                i++;
            }
        }
    }

    private String resourceName(String s) {
        
        int slash = s.indexOf(DESCRIPTION_SEPARATOR);
        int dot = s.lastIndexOf(EXTENSION_SEPARATOR);
        
        if (slash == -1) {
            slash = s.length();
        }
        if (dot == -1) {
            dot = s.length();
        }
        return s.substring(0, Math.min(slash, dot));
    }
    
    private String resourceDecription(String s) {
        
        int slash = s.indexOf(DESCRIPTION_SEPARATOR);
        int dot = s.lastIndexOf(EXTENSION_SEPARATOR);
        
        if (slash == -1) {
            return "";
        }
        if (dot == -1) {
            dot = s.length();
        }
        
        return s.substring(slash + DESCRIPTION_SEPARATOR.length(), dot);
    }
    
    private byte[] read(File file) throws IOException {
        
        log("reading " + file);
        
        // This is assuming that the data is < 5M bytes as that is the Force.com limit
        byte[] buffer = new byte[5 * 1024 * 1024];
        BufferedInputStream is = new BufferedInputStream(new FileInputStream(file));
        int size = is.read(buffer);
        
        byte[] result = new byte[size];
        System.arraycopy(buffer, 0, result, 0, size);
        return result;
    }
}

An Ant task that required the Partner API would just call the getPartnerPortType/getPartnerSessionHeader methods instead of the getMetadataPortType/getMetadataSessionHeader methods.