XML/SOAP is kicking my ass

I got thrown into a customer project involving e-commerce using XML and SOAP, and it's proven to be a real bear to work with.

XML has been described as "HTML on crack", and it allows the encoding of data in a much more self-descriptive format than regular HTML. Instead of just encoding the presentation of data, it encodes the meaning of the data:

<person>
  <firstname> Steve </firstname>
  <lastname> Friedl </lastname>
  <email> steve@nospam </email>
</person>

XML has a more rigid syntax (no overlapping nesting, all tags must be closed, etc.), but it's tolerably readable by a person (though that's not really the main goal).

The tags actually required in an interchange can be defined by an XSD (a "schema"), and this says that a "person" has three elements: a "firstname", a "lastname", and an "email". It's straightforward enough to describe most hierarchical data structures. XSDs themselves are written in XML, and there are plenty of tools out there that will take an XML file and its associated XSD and validate the file against the schema.

Personally I use XML Spy Suite v4.4 , which is an excellent IDE for working with XML and related files. Their graphical schema design tools are really outstanding, as XSDs can be hard to "skim" to get the big-picture overview.

So far, so good: I've done this before, and generating XML from flat-file inputs in perl is really straightforward.

But then we step up to SOAP, which is the Simple Object Access Protocol, and it's essentially remote procedure calls over HTTP with XML used as the serializing mechanism instead of XDR. This is where it gets a lot trickier.

Let's say that I wish to offer a service that provides the current temperature in Tustin California via the web. Obviously a web page that updates now and then would work for a person sitting at a web browser, but to automate this is a bit more work. Clearly we could do "screen scraping" of the web page, but by using SOAP we can make the request look like a function call.

In practice, the remote client often sees it exactly this way (shown in pseudo-perl, drastically oversimplified):

my $service = new SOAPService("http://www.unixwiz.net/soap/");

my $temp = $service->getTemp("F");

Here, the SOAPService object knows how to package up the parameters ("F", for Fahrenheit) via XML, wrap everything in special SOAP wrappers and headers, make an HTTP connection to my web server and submit the request. My web server would decode the request and fetch the local temperature somehow. Then it serializes the answer and sends it back to the client where it's deserialized and presented back to the caller. A tremendous amount of work is done behind the scenes by that "getTemp" function call.

OK, so how do you describe the services being offered by a web service provider? One answer is "WSDL", Web Services Description Language. This is an XML file that describes each service being offered: the name, the parameter(s), the return value(s). This specification seems to be a little bit in flux, and unless one is really up to speed on all this technology, it is fairly dense reading.

I am using the SOAP::Lite toolkit in perl, which seems very full featured. The problem is that I keep getting "undefined" errors from deep within the SOAP::Lite package file itself, and at 5000 lines (I keep running into modules like this) it's just impenetrable. The code itself is only lightly documented internally.

I can create the XML files just fine, but talking to the web service provider has been no joy, so I figured I'll just try XML Spy's SOAP debugger. I load all the XSD files and create a SOAP session, and it barfs while parsing the vendor's WSDL file. There are bits and pieces that I'm able to hack my way through, but at some point the offending parts just look too important to rip out.

A seriously complicating factor is the the web services provider requires that larger XML documents be compressed and sent as attachments, and this isn't really provided for by SOAP::Lite due to the way MIME attachments are handled. I have a feeling that none of the other toolkit providers do either.

The vendor did provide a wrapper for SOAP::Lite that handles the attachments and the like, but since I can't even get it to work right in the non-attachment case, I'm pretty much stuck.

To put the cherry on top of my ice cream scoop of woe, I bought the O'Reilly book Programming Web Services with SOAP hoping it would help get me over my hump. Considering that they had specific coverage of Java and Perl, and that one of the authors is the fellow who wrote SOAP::Lite, I figured this would be a hit. Wrong.

The book gives a pretty decent high-level overview of the whole technology space, and the writing style is pretty good. But's it's very thin on getting into the real nitty-gritty of SOAP::Lite. What a disappointment.

So I'm sitting here completely over my head, with deadlines fast approaching, other projects delayed, and I can't even think about running my clock for all this burned time.

Why did I take this project?

Posted by Steve at August 19, 2002 01:29 PM