Continuous integration performance problem and solution

The performance of one of the orgs that we use for Continuous integration had never been good and as we added unit tests became more and more unacceptable. With 224 tests added, each “build” (each deploy and test run) was taking 75 minutes, about 10x longer than expected. The performance problem could also be repeated when just interactively running a single test.

Thanks to Salesforce Partner Premier Support we now have the problem worked around. The cause of the problem was the pattern we use to access custom settings in our Apex code. We reference custom settings via a class that initializes each setting to its default values if Manage/New has not been used on the custom setting in the org:

global with sharing class CustomSettings {
    global static DateConversionFactors__c getDateConversionFactors() {
        if (DateConversionFactors__c.getInstance() == null) {
            upsert new DateConversionFactors__c(SetupOwnerId = UserInfo.getOrganizationId());
        return DateConversionFactors__c.getInstance();
    // Other similar methods for other custom settings

This approach allows code that references custom settings to be cleaner because that code will never be supplied with a null object; it also avoids customers having to manually set up all the custom settings.

In a normal org (where the custom settings have not had Manage/New used on them) the upsert only happens the first time the method is called. But if the method is called in a unit test, the upsert is automatically rolled back at the end of the test method so that the upsert overhead is incurred every time the method is called. With a typical unit test creating multiple objects, the triggers on those objects referencing multiple custom settings, and each custom setting upsert taking about 3 seconds to upsert (for whatever reason), this overhead was killing performance.

The work-around was simple – just use Manage/New interactively on every custom setting in the org. The result: 10 minutes to run the build instead of 75 minutes.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s