Managed packages can take hours to become available

The “upload” of a managed package – the creation of a new version – typically takes a few minutes to complete. But there is a second step – some sort of queued replication – that must also complete before the package version becomes available on other instances (e.g. na14 or eu2 or ap1).

Recently we’ve seen delays of up to 5 hours. The delay also applies to actions like “undeprecating” a version: a customer deployment was stalled for over 2 hours waiting for that change to propagate.

For those of us using continuous integration and the newly introduced automation for installing packages (see e.g. Andrew Fawcett’s post “Look ma, no hands!” : Automating Install and Uninstall of Packages!) these delays can result in a series of broken builds until the version becomes available.

I’ve checked with salesforce support and they have responded that:

At present there is no SLA on the installation process as it is an async process that depends on the server availability.

which is a clear position. I guess it’s a case of “hope for the best and plan for the worst” as far as these delays are concerned.

Managed package upgrade fail

Software development is always an iterative process and repeatability is key. So multiple deployments to internal or external test teams or to production environments should produce predictable results.

The good news is that a managed package can achieve this for most (but unfortunately not all) parts of a app if fresh installs are done. But in reality managed package upgrades are far more common than fresh installs because they can be done without disturbing existing data: test teams have test data setup and want to log bugs that link to examples of problems; production environments contain (large volumes of) production data. The bad news is that by design the managed package upgrade process chooses to not update layouts and picklist entries. So as an app evolves, layout changes such as new fields or re-arrangements of existing fields or new custom buttons and links are lost by default. And so are new picklist entries meaning functionality based on the new values is also lost by default.

I understand the reason for this approach is that layouts and picklist entries can be edited after they are installed and so an upgrade process that updates will overwrite any local changes and I agree that that would be a bad thing. But based on my experience over the last year with multiple customers, issuing lists of manual update steps and dealing with bugs about fields missing in layouts and values missing in picklists is also a bad thing. And with customers wanting to operate in multiple orgs – development, QA, UAT, production – and work iteratively with updates every one or two weeks the cost (in dollars, in risk to your reputation, and emotional drudgery) of trying to manually keep consistency is large.

I suggest that there are some apps where end-user customization is normal and expected and there are other apps where no customization or more controlled customization (including version control) is normal. For the latter type of app, having a managed package upgrade option to update layout and picklist entries seems like an important and obvious feature.

My attempts to influence salesforce on this directly have failed as have several postings on the ideas forum. If you are a managed package developer or are thinking of becoming one, I urge you to raise this issue with salesforce. Or be prepared to suffer poor quality and high costs for the foreseeable future.

Continuous integration performance problem and solution

The performance of one of the orgs that we use for Continuous integration had never been good and as we added unit tests became more and more unacceptable. With 224 tests added, each “build” (each deploy and test run) was taking 75 minutes, about 10x longer than expected. The performance problem could also be repeated when just interactively running a single test.

Thanks to Salesforce Partner Premier Support we now have the problem worked around. The cause of the problem was the pattern we use to access custom settings in our Apex code. We reference custom settings via a class that initializes each setting to its default values if Manage/New has not been used on the custom setting in the org:

global with sharing class CustomSettings {
    global static DateConversionFactors__c getDateConversionFactors() {
        if (DateConversionFactors__c.getInstance() == null) {
            upsert new DateConversionFactors__c(SetupOwnerId = UserInfo.getOrganizationId());
        return DateConversionFactors__c.getInstance();
    // Other similar methods for other custom settings

This approach allows code that references custom settings to be cleaner because that code will never be supplied with a null object; it also avoids customers having to manually set up all the custom settings.

In a normal org (where the custom settings have not had Manage/New used on them) the upsert only happens the first time the method is called. But if the method is called in a unit test, the upsert is automatically rolled back at the end of the test method so that the upsert overhead is incurred every time the method is called. With a typical unit test creating multiple objects, the triggers on those objects referencing multiple custom settings, and each custom setting upsert taking about 3 seconds to upsert (for whatever reason), this overhead was killing performance.

The work-around was simple – just use Manage/New interactively on every custom setting in the org. The result: 10 minutes to run the build instead of 75 minutes.

Identifying Apex tests to run using wildcards in sf:deploy

When developing code including tests that builds on a managed package it is typically necessary to exclude the managed package tests from the test runs. For example, the new code might add an additional data constraint that causes a managed package test to fail because its data setup violates the constraint. (There is also the secondary issue that the managed package tests may slow the development cycle if they take many minutes to run.)

The sf:deploy Ant task supports a flag to run all tests or alternatively accepts the names of the test classes to run. If the number of tests involved is small, the latter option works fine. But if you have a large number of tests it starts to become tedious and error prone to maintain the list of tests. This is the same problem that Java projects using the JUnit Ant task face and there the solution is to allow the set of tests that are run to be driven by file name matches in the source tree via the batchtest nested element.

I’ve added this mechanism to the DeployWithXmlReportTask (an extension of sf:deploy) that is available in the force-deploy-with-xml-report-task Google code project. Below is an example of how to use it in a project where all test classes follow the convention of having their file names end in “Test”.

<path id="ant.additions.classpath">
    <fileset dir="ant"/>
<target name="deployAndTestAndReport">
    <delete dir="test-report-xml" quiet="true"/>
    <echo message="deploying to ${sf.username}"/>
        <!-- Run all tests that match the include file name pattern (so avoiding running managed package tests) -->
            <fileset dir="src/classes">
                <include name="*Test.cls"/>

Closed vs Open Source

I have just spent the day trying to understand and work-around this error:

Error: objects/User.object(User):Entity cannot be untracked.

that (since this morning) comes up when I deploy our application using Ant and causes the deployment process to abort.

I could find nothing helpful on Developer Force or via Google. No one has commented on my Developer Force post. I have no access to additional log files. So I have created a support case that in two working days time will reach the top of the support queue and sometime later may get escalated to an appropriate person. Pity about an upcoming deadline we have.

So right now I am dreaming of past days working with open source products with active communities and the option of downloading the source and finding out exactly what the cryptic error message means when all else fails. While that did not necessarily make it easy to fix a problem, at least it kept the resolution of the problem in the hands of the person for whom it was priority #1 – me.

Continuous integration

This developerforce article introduces the subject in the context of

But it neglects to mention that some changes fail to deploy and that there is no way to “clean” an existing environment. (See and vote for the sf:clean — new task for ant-saleforce.jar idea.) So every few days occasionally when we make such a change we have to go and create a new account and switch to that. This is 10 minutes of annoying and unnecessary manual intervention.

We use Hudson Jenkins and have made a small extension to the com.salesforce.ant.DeployTask (available as a Google Code project) so that it outputs a basic JUnitReport format XML file of the test and coverage results. This integrates nicely with Hudson’s Jenkins’ test reporting – providing details of each test plus code coverage data – including with the test result trend graph: