Scripting the Apex Data Loader via Ant

The Apex Data Loader can be scripted using Spring bean XML definitions. Where the configuration needs to vary for each object that approach makes sense. But I had a need to (repeatedly) export many different objects with largely the same configuration and so wanted to use Ant and keep the configuration in Ant.

The resulting Ant build.xml file is shown here, where a macro is used that accepts a file name, an object name and a SOQL string, making it simple to add additional object exports as needed:

<?xml version="1.0" encoding="UTF-8"?>
<project name="Export" default="all">
    <macrodef name="export">
        <attribute name="file"/>
        <attribute name="object"/>
        <attribute name="soql"/>
        <sequential>
            <echo message="Exporting @{object}"/>
            <mkdir dir="exports"/>
            <copy file="config/template-process-conf.xml" tofile="config/process-conf.xml" overwrite="true" failonerror="true"/>
            <replace file="config/process-conf.xml">
                <replacefilter token="_object_" value="@{object}"/>
                <replacefilter token="_soql_" value="@{soql}"/>
                <replacefilter token="_file_" value="exports/@{file}.csv"/>
            </replace>
            <java classname="com.salesforce.dataloader.process.ProcessRunner" classpath="lib/DataLoader.jar" failonerror="true">
                <sysproperty key="salesforce.config.dir" value="config"/>
                <arg line="process.name=@{object}"/>
            </java>
        </sequential>
    </macrodef>
    <target name="all">
        <export file="SampleAccounts" object="Account" soql="Select Id, Name, ... From Account"/>
        <export file="SampleContacts" object="Contact" soql="Select Id, Name, ... From Contact"/>
        ...
    </target>
</project>

The Ant macro is replacing tokens in this template-process-conf.xml file to produce a process-conf.xml file for each object export:

<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
    <bean id="_object_" class="com.salesforce.dataloader.process.ProcessRunner" singleton="false">
        <description>TemplatedCsvExtract extracts to a CSV file."</description>
        <property name="name" value="TemplatedCsvExtract"/>
        <property name="configOverrideMap">
            <map>
                <entry key="sfdc.endpoint" value="..."/>
                <entry key="sfdc.username" value="..."/>
                <entry key="sfdc.password" value="..."/>
                <entry key="sfdc.debugMessages" value="false"/>
                <entry key="sfdc.timeoutSecs" value="600"/>
                <entry key="sfdc.extractionRequestSize" value="500"/>
                <entry key="process.operation" value="extract"/>
                <entry key="dataAccess.type" value="csvWrite"/>
                <entry key="sfdc.entity" value="_object_"/>
                <entry key="sfdc.extractionSOQL" value="_soql_"/>
                <entry key="dataAccess.name" value="_file_"/>
            </map>
        </property>
    </bean>
</beans>

The directories required are:

  • “config” containing the template-process-conf.xml file, an empty config.properties file and a log-conf.xml file
  • “lib” containing a copy of DataLoader.jar so that Ant can invoke its com.salesforce.dataloader.process.ProcessRunner class

The output CSV files are written to the “exports” folder.

PS I was working with this again recently on a different machine than the encrypt utility was run on. So as well as setting “sfdc.password” to be the encrypted version of the password and security token I also needed to set the “process.encryptionKeyFile” to reference a file containing the key used to encrypt the password.

Advertisements

3 thoughts on “Scripting the Apex Data Loader via Ant

  1. Hy Keith,

    great article about ant but the code is not working. The Ant process stops after the first call to
    <export file=.
    The next call of the java call which creates a Process Runner spring bean throws a
    Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'Account' is defined.

    Any idea why this happens? Doesn't it happen on your maschine?

    Best regards,

    Robert

    • Robert,

      Yes it works fine for me. The error message is saying that the bean with id=”Account” (that is in the file conf/process-conf.xml) was not loaded. The most likely reason is a problem with the relative locations. The above assumes that you are executing the Ant script from a directory that has the conf directory within it with the salesforce.config.dir system property specifying to look in the conf directory for the magically named process-conf.xml file.

      Best of luck,
      Keith

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s