2000 metadata JDBC driver downloads

OK so downloading isn’t the same as actively using, but nevertheless I’m pleased that my force-metadata-jdbc-driver JAR that allows SchemaSpy to produce ERD output like the sample below for a Force.com org has been downloaded 1000 times (as of September 2013) 2000 times (as of September 2015).

sample-schemaspy-page

Advertisements

Creating an Attachment SObject complete with its body using the REST API from Java

Exporting the body of an Attachment using the REST API is fairly straightforward: the “Body” field of the Attachment JSON contains the URL to GET the raw bytes for the Attachment body from. But it took me a while to get the importing of an Attachment and its body to work because PATCH or POST to the equivalent body URL is not allowed.

This documentation Insert or Update Blob Data explains generically what needs to be done. But it took me a bit of time to work out how to do this in Java using the Apache HttpClient. Here is a stripped down version of what I ended up with that might help you if you are writing similar Java code:

package com.claimvantage.ant;

import java.io.File;
import java.io.IOException;
import java.io.OutputStream;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.multipart.FilePart;
import org.apache.commons.httpclient.methods.multipart.MultipartRequestEntity;
import org.apache.commons.httpclient.methods.multipart.Part;
import org.apache.commons.httpclient.methods.multipart.PartBase;

public class AttachmentExample {
    
    private class JsonPart extends PartBase {
        
        private byte[] bytes;
        
        public JsonPart(String name, String json) throws IOException {
            super(name, "application/json", "UTF-8", null);
            this.bytes = json.getBytes("UTF-8");
        }
        
        @Override
        protected void sendData(OutputStream os) throws IOException {
            os.write(bytes);
        }

        @Override
        protected long lengthOfData() throws IOException {
            return bytes.length;
        }
    }
    
    private String baseUrl;     // Initialization not shown here
    private String sessionId;   // Initialization not shown here
    
    /**
     * Create attachment SObject from its JSON populating its Body from a file at the same time.
     */
    public void createAttachment(String attachmentJson, File attachmentFile) throws Exception {
        
        PostMethod post = new PostMethod(baseUrl + "/services/data/v23.0/sobjects/Attachment");
        post.setRequestHeader("Authorization", "OAuth " + sessionId);
        Part[] parts = new Part[] {
                new JsonPart("Json", attachmentJson),
                new FilePart("Body", attachmentFile)
                };
        post.setRequestEntity(new MultipartRequestEntity(parts, post.getParams()));
        try {
            new HttpClient().executeMethod(post);
            if (post.getStatusCode() == HttpStatus.SC_CREATED) {
                // Logic for OK
            } else {
                // Error handling logic
            }
        } finally {
            post.releaseConnection();
        }
    }
}

Creating and running a fake web service from a WSDL

I needed to develop some Apex web service callout code – never a pleasant experience see e.g. When “Generate from WSDL” fails – hand-coding web service calls – but did not have access to a running instance of the service. Java 6 includes web service technology by default and I was pleasantly surprised how easy it is to create and run a fake web service to develop against using this…

Having run wsimport to generate the web service classes from the WSDL, creating a server implementation class is simply a matter of extending the service interface and using Eclipse’s “Add unimplemented methods” to create the server skeleton then adding a couple of annotations:

package com.claimvantage.test;

import javax.jws.WebService;
import javax.xml.ws.BindingType;

// Other imports removed

@WebService(
        serviceName = "FNCEWS40Service",
        portName = "FNCEWS40MTOMPort",
        targetNamespace = "http://www.filenet.com/ns/fnce/2006/11/ws/MTOM/wsdl",
        wsdlLocation = "com/claimvantage/test/filenet.wsdl",
        endpointInterface = "com.claimvantage.filenet.FNCEWS40PortType"
        )
@BindingType(value="http://java.sun.com/xml/ns/jaxws/2003/05/soap/bindings/HTTP/")
public class ServiceFakeImpl implements FNCEWS40PortType {

    @Override
    public ObjectSetType executeSearch(SearchRequestType request, Localization header) throws FaultResponse {
        // TODO fake implementation goes here
        return null;
    }

    // Other methods removed
}

The Endpoint class makes running this service very easy (with no separate web server or XML configuration required):

package com.claimvantage.test;

import javax.xml.ws.Endpoint;

public class ServiceRunner {

    private static final String URL = "http://localhost:9080/wsi/FNCEWS40MTOM/";

    public static void main(String[] args) {
        Endpoint endpoint = Endpoint.create(new ServiceFakeImpl());
        endpoint.publish(URL);
        System.out.println("Service running at " + URL);
    }
}

Then make the fake service accessible from the internet (and so from Force.com) by e.g. setting up firewall port forwarding.

Debug output and fake responses can be added to the fake server methods. When combined with debug logging in the Apex and a tool such as tcpmon to view the request and response headers and XML this gives full visibility to what is going on across the whole request/response cycle.

SchemaSpy Org ERD with easier setup

The code that presents a Force.com org via the JDBC metadata interfaces has been updated to make it simpler to use. This code allows tools like SchemaSpy and Open ModelSphere to work with a Force.com org.

In the case of SchemaSpy, a set of diagrams like this (and accompanying HTML descriptions of all the objects and fields) are produced:

Details at force-metadata-jdbc-driver, specifically UsageForV2.

Quick summary of how to get started with the Metadata WSDL API via Java’s JAXB

The Metadata API is used to manipulate customization information in an org including Apex classes and Visualforce pages by code such as the Force IDE. I wanted to create an Ant task to automate the creation of a large’ish number of static resources from files via this API and so following on from the Quick summary of how to get started with the Enterprise WSDL API via Java’s JAXB here are few extra things I discovered relating to the Metadata API.

The first is that the Metadata WSDL obtained from Setup -> Develop -> API -> Generate Metadata WSDL had a schema element that was missing the attribute xmlns=”http://www.w3.org/2001/XMLSchema” that puts it in the correct namespace and so this had to be manually added before the JAXB code generation would work. Also, as in the previous post, a JAXB customization file that had the targetNamespace of its bindings element set to “http://soap.sforce.com/2006/04/metadata” needed to be created.

The second is that you need classes generated from the Partner WSDL or the Enterprise WSDL to be able to do the login and obtain the session id that is then set in the session header from the Metadata WSDL. I ended up with a project with both the Partner and Metadata WSDL code generated to be able to do this. A bit of care is then needed to use the classes from the correct package.

Lastly, when you cut and paste code that relates to the Partner WSDL or Enterprise WSDL, the endpoint that you use from the login response may accidentally end up being loginResponse.getResult().getServerUrl() instead of the one you need for the Metadata WSDL case of loginResponse.getResult().getMetadataServerUrl().

And that is it if a “quick summary” is what you are interested in. If you want to see some sample code read on…

This example of how to use the Metadata API is obfuscated a little by the code being in the form of an Ant task. All you need to know to understand the Ant part is that the entry point is the execute method and attributes that have set methods can be configured via XML from a build file.

This base class handles the login and connection leaving the specifics of Metadata API or Partner API operations to extending classes:

package com.claimvantage.ant;

import java.net.URL;

import javax.xml.namespace.QName;
import javax.xml.ws.BindingProvider;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;

import com.sforce.soap.metadata.MetadataPortType;
import com.sforce.soap.metadata.MetadataService;
import com.sforce.soap.partner.Login;
import com.sforce.soap.partner.LoginResponse;
import com.sforce.soap.partner.SforceService;
import com.sforce.soap.partner.Soap;

public abstract class ForceApiTaskBase extends Task {
    
    private String un;
    private String pw;
    
    private LoginResponse loginResponse;
    
    private MetadataPortType port;
    private com.sforce.soap.metadata.SessionHeader metaDataSessionHeader;
    
    private Soap soap;
    private com.sforce.soap.partner.SessionHeader partnerSessionHeader;

    public String getUn() {
        return un;
    }

    public void setUn(String un) {
        this.un = un;
    }

    public String getPw() {
        return pw;
    }

    public void setPw(String pw) {
        this.pw = pw;
    }

    public void execute() throws BuildException {
        try {
            doExecute();
        } catch (BuildException e) {
            throw e;
        } catch (Exception e) {
            throw new BuildException(e);
        }
    }
    
    protected abstract void doExecute() throws Exception;
    
    protected MetadataPortType getMetadataPortType() throws Exception {
        createMetadataSession();
        return port;
    }
    
    protected com.sforce.soap.metadata.SessionHeader getMetadataSessionHeader() throws Exception {
        createMetadataSession();
        return metaDataSessionHeader;
    }
    
    protected Soap getPartnerPortType() throws Exception {
        createPartnerSession();
        return soap;
    }
    
    protected com.sforce.soap.partner.SessionHeader getPartnerSessionHeader() throws Exception {
        createPartnerSession();
        return partnerSessionHeader;
    }
    
    private void createMetadataSession() throws Exception {
        
        if (port == null) {
            
            // Login done here
            createPartnerSession();
            
            MetadataService service = new MetadataService(getUrl("metadata.wsdl"), new QName("http://soap.sforce.com/2006/04/metadata", "MetadataService"));
            port = service.getMetadata();
            
            BindingProvider b = (BindingProvider) port;
            b.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, loginResponse.getResult().getMetadataServerUrl());
            
            metaDataSessionHeader = new com.sforce.soap.metadata.SessionHeader();
            metaDataSessionHeader.setSessionId(loginResponse.getResult().getSessionId());
        }
    }

    private void createPartnerSession() throws Exception {
        
        if (loginResponse == null) {
            
            SforceService service = new SforceService(getUrl("partner.wsdl"), new QName("urn:partner.soap.sforce.com", "SforceService"));
            soap = service.getSoap();
            
            // Login
            log("connecting as user " + getUn());
            Login login = new Login();
            login.setUsername(getUn());
            login.setPassword(getPw());
            
            loginResponse = soap.login(login, null, null);
            
            BindingProvider b = (BindingProvider) soap;
            b.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, loginResponse.getResult().getServerUrl());
            
            partnerSessionHeader = new com.sforce.soap.partner.SessionHeader();
            partnerSessionHeader.setSessionId(loginResponse.getResult().getSessionId());
        }
    }
    
    private URL getUrl(String name) throws Exception {
        
        // WSDL must be packaged with code
        URL url = getClass().getResource(name);
        if (url != null) {
            return url;
        } else {
            throw new Exception("Could not access WSDL from " + name);
        }
    }
}

And here is an example of an Ant task that makes use of the Metadata API. (The polling is needed to deal with the asynchronous nature of the API.) The Ant task transfers a set of files to an org as static resources:

package com.claimvantage.ant;

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Project;

import com.sforce.soap.metadata.AsyncRequestStateType;
import com.sforce.soap.metadata.AsyncResultType;
import com.sforce.soap.metadata.CheckStatus;
import com.sforce.soap.metadata.Create;
import com.sforce.soap.metadata.Delete;
import com.sforce.soap.metadata.StaticResourceCacheControlType;
import com.sforce.soap.metadata.StaticResourceType;

/**
 * Transfer files into an org as static resources.
 */
public class LoadStaticResources extends ForceApiTaskBase {
    
    // Text between this and the extension (if present) used as the description and stripped from the resource name
    private static final String DESCRIPTION_SEPARATOR = "-";
    
    // Text after this stripped
    private static final String EXTENSION_SEPARATOR = ".";
    
    private String contentType;
    private File folder;

    public void setContentType(String contentType) {
        this.contentType = contentType;
    }

    // All files in this folder pushed to server
    public void setFolder(File folder) {
        this.folder = folder;
    }

    protected void doExecute() throws Exception {
        
        if (contentType == null) {
            throw new BuildException("contentType must be set");
        }
        if (folder == null) {
            throw new BuildException("folder must be set");
        }
        if (!folder.isDirectory()) {
            throw new BuildException("folder must be a directory");
        }
        
        File[] filesArray = folder.listFiles();
        if (filesArray == null || filesArray.length == 0) {
            log(folder + " is empty");
            return;
        }
        List<File> files = Arrays.asList(filesArray);

        // API limits to this many at a time
        final int max = 10;
        for (int batch = 0; true; batch++) {
            
            int from = max * batch;
            int to = Math.min(files.size(), from + max);
            
            doBatch(files.subList(from, to));

            if (to >= files.size()) {
                break;
            }
        }
        log("Done");
    }
    
    private void doBatch(List<File> files) throws Exception {
        
        Delete delete = new Delete();
        for (File file : files) {
            StaticResourceType sr = new StaticResourceType();
            sr.setFullName(resourceName(file.getName()));
            delete.getMetadata().add(sr);
        }
        waitForCompletion(files, getMetadataPortType().delete(delete, getMetadataSessionHeader(), null).getResult(), "delete");
        
        Create create = new Create();
        for (File file : files) {
            StaticResourceType sr = new StaticResourceType();
            sr.setFullName(resourceName(file.getName()));
            sr.setDescription(resourceDecription(file.getName()));
            sr.setContent(read(file));
            sr.setContentType(contentType);
            sr.setCacheControl(StaticResourceCacheControlType.PRIVATE);
            create.getMetadata().add(sr);
        }
        waitForCompletion(files, getMetadataPortType().create(create, getMetadataSessionHeader(), null).getResult(), "create");
    }
    
    private void waitForCompletion(List<File> files, List<AsyncResultType> results, String operation) throws Exception {
        
        CheckStatus check = new CheckStatus();
        for (AsyncResultType result : results) {
            check.getAsyncProcessId().add(result.getId());
        }

        int sleepSeconds = 1;
        boolean notAllDone = true;
        while(notAllDone) {
            notAllDone = false;

            log("waiting for async result - sleeping for " + sleepSeconds + " s");
            Thread.sleep(sleepSeconds * 1000L);
            if (sleepSeconds < 64) {
                sleepSeconds *= 2;
            }

            int i = 0;
            for (AsyncResultType result : getMetadataPortType().checkStatus(check, getMetadataSessionHeader(), null).getResult()) {
                int level = result.getState() == AsyncRequestStateType.ERROR ? Project.MSG_ERR : Project.MSG_INFO;
                log(operation + " " + result.getState() + " " + files.get(i).getName() + " " + (result.getMessage() != null ? result.getMessage() : ""), level);
                if (result.getState() == AsyncRequestStateType.QUEUED || result.getState() == AsyncRequestStateType.IN_PROGRESS) {
                    notAllDone = true;
                }
                i++;
            }
        }
    }

    private String resourceName(String s) {
        
        int slash = s.indexOf(DESCRIPTION_SEPARATOR);
        int dot = s.lastIndexOf(EXTENSION_SEPARATOR);
        
        if (slash == -1) {
            slash = s.length();
        }
        if (dot == -1) {
            dot = s.length();
        }
        return s.substring(0, Math.min(slash, dot));
    }
    
    private String resourceDecription(String s) {
        
        int slash = s.indexOf(DESCRIPTION_SEPARATOR);
        int dot = s.lastIndexOf(EXTENSION_SEPARATOR);
        
        if (slash == -1) {
            return "";
        }
        if (dot == -1) {
            dot = s.length();
        }
        
        return s.substring(slash + DESCRIPTION_SEPARATOR.length(), dot);
    }
    
    private byte[] read(File file) throws IOException {
        
        log("reading " + file);
        
        // This is assuming that the data is < 5M bytes as that is the Force.com limit
        byte[] buffer = new byte[5 * 1024 * 1024];
        BufferedInputStream is = new BufferedInputStream(new FileInputStream(file));
        int size = is.read(buffer);
        
        byte[] result = new byte[size];
        System.arraycopy(buffer, 0, result, 0, size);
        return result;
    }
}

An Ant task that required the Partner API would just call the getPartnerPortType/getPartnerSessionHeader methods instead of the getMetadataPortType/getMetadataSessionHeader methods.

Quick summary of how to get started with the Enterprise WSDL API via Java’s JAXB

With JAX-WS and JAXB part of Java 6, using JAXB to handle the marshalling needed to access an org through the Enterprise WSDL API makes sense because no JARs in addition to the JRE are needed. I found this excellent series of articles Salesforce.com Partner SOAP API JAX-WS Tutorial by Marshall Pierce that covered most of what I needed (including the use of compression – very nice).

There were only two extra things to deal with:

  • The technique for setting the session id didn’t work for me so I used the wsimport tool’s -XadditionalHeaders option that allows SOAP headers (including the SessionHeader that holds the session id) to be passed in the method calls.
  • Unlike the Partner WSDL where a generic interface is used to access the SObject fields, using the Enterprise WSDL results in a class with getters and setters being generated for each SObject (strongly typed access). By default the setters and getters for primitive values like Strings were using a JAXBElement generic making for tedious coding and a crazily large ObjectFactory. Setting a generateElementProperty=”false” customization at generation time resulted in the more normal direct setters and getters.

Here is the summary part; there are just three pieces of more or less boilerplate needed…

One – the code generation (here just invoking the wsimport command-line tool through Ant):

<target name="wsimport" description="JAXB code generation">
    <exec executable="/usr/bin/wsimport">
        <arg value="-XadditionalHeaders"/>
        <arg value="-Xnocompile"/>
        <arg value="-b"/>
        <arg value="jaxb-customization.xml"/>
        <arg value="-verbose"/>
        <arg value="-d"/>
        <arg value="src-generated-jaxb"/>
        <arg value="-p"/>
        <arg value="com.sforce.soap.enterprise"/>
        <arg value="src/com/claimvantage/force/client/xml/enterprise.wsdl"/>
    </exec>
</target>

Two – the jaxb-customization.xml customization file used by the code generation:

<bindings xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
	xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
	xmlns="http://java.sun.com/xml/ns/jaxws">
	<bindings
		node="//xsd:schema[@targetNamespace='urn:enterprise.soap.sforce.com']">
		<jaxb:globalBindings
		      underscoreBinding="asCharInWord"
		      generateElementProperty="false"
		      />
		<jaxb:schemaBindings>
			<jaxb:nameXmlTransform>
				<jaxb:typeName suffix="Type" />
			</jaxb:nameXmlTransform>
		</jaxb:schemaBindings>
	</bindings>
	<enableWrapperStyle>false</enableWrapperStyle>
	<enableAsyncMapping>false</enableAsyncMapping>
</bindings>

Three – an example of how to login and use the API (imports left out);

public class Processor {
    private Soap soap;
    private SessionHeader sessionHeader;
    private QueryOptions queryOptions;
    public Processor(String un, String pw, String st) throws Exception {
        connect(un, pw, st);
    }
    public void process() throws Exception {
        // Implement your logic here. In general case use Query/QueryMore to handle large sets of data
        List<SObject> sobs = soap.query(createQuery("select Birthdate from Contact where LastName = 'Smith'"),
                sessionHeader, queryOptions, null, null).getResult().getRecords();
        for (SObject sob : sobs) {
            Contact contact = (Contact) sob;
            System.out.println(contact.getBirthdate());
        }
    }
    private Query createQuery(String soql) {
        Query query = new Query();
        query.setQueryString(soql);
        return query;
    }
    private void connect(String un, String pw, String st) throws Exception {
        // WSDL must be packaged with code
        String wsdlPath  = "enterprise.wsdl";
        URL url = this.getClass().getResource(wsdlPath);
        if (url == null) {
            throw new Exception("Could not access WSDL from path " + wsdlPath);
        }
        // This triggers a lot of class loading
        SforceService service = new SforceService(url, new QName("urn:enterprise.soap.sforce.com", "SforceService"));
        soap = service.getSoap();
        Login login = new Login();
        login.setUsername(un);
        login.setPassword(pw + st);
        LoginResponse loginResult = soap.login(login, new LoginScopeHeader());
        BindingProvider b = (BindingProvider) soap;
        b.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, loginResult.getResult().getServerUrl());
        // Use compression for speed
        Map<String, List> httpHeaders = new HashMap<String, List>();
        httpHeaders.put("Content-Encoding", Collections.singletonList("gzip"));
        httpHeaders.put("Accept-Encoding", Collections.singletonList("gzip"));
        b.getRequestContext().put(MessageContext.HTTP_REQUEST_HEADERS, httpHeaders);
        // This must be passed with each call
        sessionHeader = new SessionHeader();
        sessionHeader.setSessionId(loginResult.getResult().getSessionId());
        // Up to this many will be returned but may be less
        queryOptions = new QueryOptions();
        queryOptions.setBatchSize(2000);
    }
}

Running the Apex Data Loader on a Mac

Even though the Apex Data Loader is written in Java it doesn’t work on a Mac for two main reasons:

  • It is packaged in a Windows-only installer
  • Instead of using Swing for its GUI that does not require any platform-specific code to be included in the application it uses SWT that does require platform-specific code

The first problem can be worked around by installing on a Windows machine first and then copying the installation across to the Mac.

The second problem can be worked around using these steps (assuming you have a working copy of the Eclipse IDE on your Mac perhaps to make use of the Force IDE):

  • Copy the files that start org.eclipse.swt (e.g. org.eclipse.swt_3.6.1.v3655c.jar and org.eclipse.swt.cocoa.macosx.x86_64_3.6.1.v3655c.jar) from eclipse/plugins to the directory that contains DataLoader.jar (that you copied across from Windows).
  • Use this command-line (using the file names from the previous step) to run the DataLoader:
    java -Dsalesforce.config.dir=conf -XstartOnFirstThread -cp org.eclipse.swt_3.6.1.v3655c.jar:org.eclipse.swt.cocoa.macosx.x86_64_3.6.1.v3655c.jar:DataLoader.jar com.salesforce.dataloader.process.DataLoaderRunner

The -D argument specifies the location of the configuration files, the -X argument appears to be required for this version of SWT to work, the -cp argument gives priority to the Mac version of the SWT classes ahead of the Windows versions included in DataLoader.jar and the last argument is the name of the class that launches the GUI.

If you choose to run the data loader like this obviously Salesforce will not support you and you should be cautious and check your results carefully. Also consider using LexiLoader – see comments below.