Simplicity and Utility, or, Why SOAP Lost

tldr: Simplicity and utility trump large corporate backing.

In the middle of 2000, SOAP - an XML-based messaging protocol - started to rise in popularity as a communication protocol for distributed computing. By 2004, there were dozens of SOAP implementations in as many languages, including all of the most popular ones, such as Java, C/C++, .NET, Perl, and PHP. The term Web service became popular, describing a new model of interaction: APIs over the Web.

I was on the team that shipped various SOAP and Web services products at Microsoft. This included various product management tasks, designing features, working across companies on interoperability testing, chairing the first WS-I committee, etc. I helped build and popularize SOAP. And I watched as it lost.

Ten years later SOAP is gone. Very few, if any, developers, are creating new SOAP-based Web services. But Web services, APIs over HTTP, are more popular than ever. SOAP was popular for a while, but it lacked longevity.

What happened? #

SOAP was part of an ever expanding set of protocols that ended up getting called WS-*. There was WSDL, WS-Security, WS-SecureConversation, WS-ReliableMessaging, WS-Discovery, and so on. Each of these protocols worked with SOAP to build an entire messaging and distributed computing infrastructure. And, for many people, this looked like overkill. Sending some XML over HTTP was simple enough. But each of these protocols added complexity to that simple idea. The motivation for WS-* was sound, but it was getting marketed far beyond its usefulness.

Microsoft, in particular, was building a brand-new distributed computing framework called the Windows Communication Foundation (WCF), which they wanted to become the replacement for DCOM, MSMQ, and all of the other remoting APIs they previously offered. So, they needed a protocol that could handle not just plain RPC, but all of the distributed work you could do with the Windows platform.

In the absence of an alternative, maybe the tooling required to build WS-* APIs would have gained more traction. But Microsoft was the only major framework developer to build those tools. Simultaneously, two things started happening:

• Ruby on Rails and other open source framework became very popular
• JSON started replacing XML as a serialization technology

Developers needed a way to provide data to mobile clients. This was simple enough using many Web frameworks: look something up in a database, return the rows in a serialized format. Accept POSTs of the same kind of data. SOAP was doomed to failure for such simple, but very popular scenarios. It was overkill. Consider a simple Web service to return customer information.

SOAP vs JSON/HTTP #

First, using SOAP to request a specific customer record. The request will be an HTTP POST and look like this:

POST /customers HTTP/1.1
Host: www.example.org
Content-Type: application/soap+xml; charset=utf-8

<?xml version="1.0"?>
<soap:Envelope 
          xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
  <soap:Body>
    <m:GetCustomer 
          xmlns:m="http://www.example.org/customers">
      <m:CustomerId>43456</m:CustomerId>
    </m:GetCustomer>
  </soap:Body>
</soap:Envelope>

After looking up the customer, the response will look like this:

HTTP/1.1 200 OK
Content-Type: application/soap+xml; charset=utf-8

<?xml version='1.0' ?>
<env:Envelope 
        xmlns:env="http://www.w3.org/2003/05/soap-envelope" >
 <env:Body>
    <m:GetCustomerResponse 
          xmlns:m="http://www.example.org/customers">
       <m:Customer>Foobar Quux, inc</m:Customer>
    </m:GetCustomerResponse>
 </env:Body>
</env:Envelope>

Now, contrast that to using JSON over HTTP, with a slightly more RESTful orientation. The request for the customer record is an HTTP GET:

GET /customers/43456 HTTP/1.1
Host: www.example.org

And the response:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{‘Customer’: ‘Foobar Quux, inc’}

Both of these calls are equivalent. The SOAP version sends a lot more data in both the request and response. And, because it uses XML, it’s much harder to read with your eyes. Even worse, because it uses POSTs both ways, it can’t be quickly tested in the browser.

The JSON over HTTP version is limited in utility compared to the SOAP version. Securing the request and response inside the message itself, for instance, is possible with SOAP in an extensible manner. But this isn’t a feature most developers need. The SOAP version would work over a non-http transport. But that also isn’t a feature most developers need.

What We Can Learn From SOAP’s Failure #

There are three lessons I learned from SOAP’s failure:

• Market leaders can’t force the market. Microsoft, IBM, and many others supported SOAP. But the market (developers everywhere) chose something else.
• Simple is important. Perhaps the most important thing. Simple means different things to different people, but I think Rich Hickey has the best definition in this context.
• YAGNI (You Ain’t Gonna Need It) trumps future extensibility.

Thanks to David Jarvis, Bryan Costanich, Ian Geoghegan, and Rohit Rajan for their reviews.