Moxad Enterprises Inc. ...blog

back to Moxad Enterprises

Living without WSDL files in a SOAP infested world

May 7th, 2014

The problem

About five years ago, I had a requirement of developing a SOAP (Simple Object Access Protocol) service on some Linux systems. It was actually a SOAP interface needed to an existing service which till then was fielded by XML/RPC requests. Not knowing much about SOAP, I poked around on the Web and talked to people, and there seemed to be a strong consensus that one should be using a WSDL (Web Services Description Language) file. I looked at some example WSDL files, and thought: "Are you kidding me?!". What I wanted to do was really simple. I wanted to describe it in about a dozen lines of text. An access point, some scalar inputs and some scalar outputs. Simple. But a typical WSDL file, being almost incomprehensible to read, was typically hundreds of lines. You'd need some tool or complex development environment to produce it. And then it got worse. At the time, my choices of acceptable languages for that environment were Perl and Java. I was already a competent Perl programmer. But there didn't seem to be much WSDL support in SOAP modules available for Perl. I didn't know Java. And I didn't want to know Java. Yet as soon as you mention SOAP to anyone, you hear Java in response. When I asked Java programmers for an example simple Java SOAP client, I'd get a minimum of 500 lines of code. I thought again: "Are you kidding me?!". I want to see about a dozen lines of code. Not 500. So, what to do?

The solution

Well, it's five years later. I've written several SOAP web-services, in Perl, some that have complex inputs and outputs that are beyond anything any Perl module can currently handle when it comes to WSDL support (that I know of), and at least one of those services handles thousands of requests a day in a production environment for a Telco, handling on-demand diagnostics for call-centers and service technicians in the field.

Now this is not about making a case for Perl. It just happened to be an approved language in my environment, and one that I knew. I also wrote test clients for those some of those services in Ruby, PHP, and Python, to prove that the service was language agnostic, as it should be. Each of those clients is just a few lines of code to handle the SOAP call. Not 500. It was also very little time to figure out how to write those clients as well.

So I'm going to demonstrate here how to set up a trivial SOAP service on a Linux system. The actual service is a bit dopey. It just provides a filename as input and gets back the mode of the file and the size in bytes of that file. The service is unimportant. What I want to demonstrate is the framework of the service and I wanted at least one input (the filename) and more than one output (in this case two - the mode of the file and the file size).

I'll do the server side in Perl. Not only is it a trivial amount of code, but once in place, it allows you to add other services (a new module) at any time without any additional code to field the incoming SOAP request to that new module.

Apache web-server changes

First, lets create a entry point and tell our Apache web-server about it. We'll handle the SOAP requests with a Perl CGI (Common Gateway Interface) program. Here's what we'll add to our Apache config file, which on a Linux system may be under /etc/apache2/sites-enabled/:

      ScriptAlias SoapWS /var/www/cgi-bin/SOAP_WS.cgi

This says that a URL of http://server:port/SoapWS will actually call the program SOAP_WS.cgi. You'd then typically restart your Apache service with "/etc/init.d/apache2 restart"

The server-side CGI program

And here is the SOAP_WS.cgi program:

        #!/usr/bin/env perl
  
        use strict;
        use warnings;
        use SOAP::Transport::HTTP;
        use version; our $VERSION = qv('1.0.0');
  
        # dynamically load any modules requested from this directory
        my $module_dir = '/usr/local/SOAP-WS' ;
  
        SOAP::Transport::HTTP::CGI
           ->dispatch_to( $module_dir )
           ->handle;
        

This says that for incoming SOAP requests that Apache gets, to look in the directory /usr/local/SOAP-WS for the module to handle whatever service your client asks for. There would be a separate module (file) for each different service you make available. We only care about setting up one right now. The above program would require the SOAP::Lite package which you can get from CPAN (Comprehensive Perl Archive Network).

The server-side module for our service

Here is our server-side module for a service we will call TestWS (Test Web Service). It has a single method called fetchdata. This module will be placed in the directory /usr/local/SOAP-WS. The client will make a call to the TestWS service using the fetchdata method. The Apache web-service will call the CGI program SOAP_WS.cgi, which will look in the /usr/local/SOAP-WS directory for this file called TestWS.pm

        package TestWS ;
  
        use strict ;
        use warnings ;
        use Exporter;
  
        BEGIN {
            use Exporter ;
            our ( @ISA, @EXPORT );
  
            @ISA = qw( Exporter );
            @EXPORT = qw( TestWS );
        }
  
        sub fetchdata {
            my $either   = shift ;   # instance or class
            my $filename = shift ;
  
            die SOAP::Fault
                ->faultstring( "No such file: $filename" )
                ->faultcode( "Err-01" ) if ( ! -f $filename ) ;
  
            # get the mode and size of the file in bytes
            # see: http://perldoc.perl.org/functions/stat.html
            my $mode  = (stat($filename))[2] ;
            my $size  = (stat($filename))[7] ;
  
            # create SOAP data structures with XML tag to use
            my $mode_data = SOAP::Data
                ->name( "Mode" )
                ->value( $mode ) ;
  
            my $size_data = SOAP::Data
                ->name( "Size" )
                ->value( $size ) ;
  
            return( $mode_data, $size_data  );
        }
  
        1;
        

All this does is grab the mode and size of the filename using stat() in two lines of code. The rest is just wrapping this data into a SOAP object with the name being the XML tag that will be used. It returns two values. There is a check for the existence of the file and if it does not exist, then a error code and error string is set and the module returns.

The client program (in Perl)

Finally, here is a client program. This example is in Perl (don't worry, I'll provide examples in other languages). Like the server side code, you would need to install the CPAN Soap::Lite module for this example. The real meat of this is really just one line of Perl code to call the SOAP service, split up over 4 lines to make it readable. That one line of Perl is just:

        my $response = SOAP::Lite
           ->uri( "http://${ip_address}:${port_number}/${service}" )
           ->proxy( "http://${ip_address}:${port_number}/${endpoint}" )
           ->$method( "$filename" );
        

We've already talked about the service name (TestWS), the endpoint (http://server:port/SoapWS), and the method (fetchdata). Instead of hard-coding them into the above statement, they were made variables, which can be seen below in the entire client program:

        #!/usr/bin/env perl
  
        # Get the file mode and size, in bytes, of a filename via SOAP call
  
        use warnings;
        use strict;
        use SOAP::Lite;
  
        my $filename    = $ARGV[0] || '/etc/motd';
        my $ip_address  = '127.0.0.1' ;
        my $port_number = '80' ;
        my $service     = 'TestWS' ;    # module name
        my $endpoint    = 'SoapWS' ;    # see Apache config
        my $method      = 'fetchdata' ; # method in service
  
        my $response = SOAP::Lite
           ->uri( "http://${ip_address}:${port_number}/${service}" )
           ->proxy( "http://${ip_address}:${port_number}/${endpoint}" )
           ->$method( "$filename" );
  
        if ( $response->fault ) {
            print {*STDERR} "You got a error dude!:\n" .
                "\tFault code:\t" .   $response->faultcode . "\n" .
                "\tFault string:\t" . $response->faultstring . "\n";
            exit 1;
        }
  
        my @results = $response->paramsall();
        my $mode = $results[0];
        my $mode_octal = sprintf( "%o", $mode ) ;
        my $size = $results[1];
        print "$filename is mode ${mode_octal} and ${size} bytes\n";
  
        exit 0;
        

The output is "/etc/motd is mode 100644 and 253 bytes"

Almost all of the above was setting variables, making the SOAP call, checking for any errors, and then just printing out the result. We are printing out the mode in octal. (If you're a Unix/Linux sys-admin, then seeing permissions shown as an octal number makes sense to you). And that's it. If we only care about having the client in Perl, then we'd be done. But we'll try a few other languages for the client to make sure there are no surprises.

XML request and response

Before we get to clients in other languages though, let's take a look at the XML for both the request and the response. We can do this easily with the above Perl client if we replace the:

      use strict;       use SOAP::Lite;

instead with this:
      use strict "refs";       use strict "vars";       use SOAP::Lite +trace;

With the regular output , it will also give us the XML as well for both the request and response. It will be difficult to read without formatting, so the easiest thing to do, if the amount of data is small like in our example web-service, is to just cut and paste the XML into a file and use a formatting program against it. I used a program called xmlformat. This will give us a XML request of:

        <?xml version="1.0" encoding="UTF-8"?>
        <soap:Envelope
            soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
            xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
            xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
            xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <soap:Body>
               <fetchdata xmlns="http://127.0.0.1:80/TestWS">
                  <c-gensym3 xsi:type="xsd:string">/etc/motd</c-gensym3>
               </fetchdata>
            </soap:Body>
        </soap:Envelope>
        

Notice that for this client XML, the XML tag is that generated for the filename string /etc/motd is c-gensym3. This is because our client program did not bother to specify a XML tag to use. We're being lazy, because we know that the server side is written in Perl and is forgiving and doesn't care. The server side specifies the XML tags to return, but doesn't care what they are in the request. And here is our XML response:

        <?xml version="1.0" encoding="UTF-8"?>
        <soap:Envelope
           soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
           xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
           xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
           xmlns:xsd="http://www.w3.org/2001/XMLSchema"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
           <soap:Body>
              <fetchdataResponse xmlns="http://127.0.0.1:80/TestWS">
                 <Mode xsi:type="xsd:int">33188</Mode>
                 <Size xsi:type="xsd:int">253</Size>
              </fetchdataResponse>
           </soap:Body>
        </soap:Envelope>
        

You'll notice that here we have the XML tags around the data that we specified in the server: Mode and Size.

A client in PHP

Just to make sure our service is language agnostic, lets write some clients in other languages.
Here it is in PHP:

        <?php
        $IP        = "127.0.0.1" ;
        $port      = "80" ;
        $URL       = "/SoapWS" ;
        $service   = "TestWS" ;
        $filename  = "/etc/motd" ;
  
        try {
            $client = new SoapClient(
                null,
                array( 'location' => "http://$IP:$port/$URL",
                        'uri'     => "http://$IP:$port/$service"
                    )
            );
            $response = $client->fetchdata( $filename ) ;
        }
        catch( Exception $e ) {
            echo $e->getMessage(), "\n" ;
            exit(1);
        }
  
        $mode = $response[ "Mode" ] ;
        $octal_mode = sprintf( "%o", $mode ) ;
        $size = $response[ "Size" ] ;
        echo $filename, " is mode ", $octal_mode, " and ", $size, " bytes\n"
        ?>
        

And like the Perl version, the output is: "/etc/motd is mode 100644 and 253 bytes".

Again, it is a trivial amount of code for the actual SOAP call.

A client in Python

Below is a Python client:

        #!/usr/bin/env python
  
        from SOAPpy import SOAPProxy
        import SOAPpy.Types ;
        import sys
  
        ip          = '127.0.0.1'
        port        = '80'
        service     = 'TestWS'      # module name
        endpoint    = 'SoapWS'      # see Apache config
        method      = 'fetchdata'
        filename    = '/etc/motd'
  
        url_prefix = 'http://' + ip + ':' + port + '/'
        url        = url_prefix + endpoint
        urn        = url_prefix + service
        action     = urn + '#' + method
  
        server = SOAPProxy( url, namespace=urn, soapaction=action )
        try:
            result = server.fetchdata( filename )
        except SOAPpy.Types.faultType, e:
            print e
            sys.exit(1)
        except:
            print "Unexpected error:", sys.exc_info()[0]
            sys.exit(2)
  
        print "{0} is mode {1:o} and {2:d} bytes".format(
            filename, result[0], result[1] )
        

Like the Perl and PHP versions, this prints out: "/etc/motd is mode 100644 and 253 bytes"

When I first ran this on my slightly old favorite development system (Linux Mint 14 Nadia), everything was fine. However, to be safe, I then set up a stock virtual Linux Mint 16 Petra using virtualbox and tried it there. I ran into a few issues. As you can see from the program, you need to import the SoapPy module. But, I found that some include files were missing for the build to succeed. So I installed a development package of Python, specifically python2.7dev which seemed to fix this. Then I needed PyXML which was failing because of "memmove" not existing. Apparently, a hack to get this to work is to append:

      #define HAVE_MEMMOVE 1

to the file /usr/include/python2.7/pyconfig.h, in my case since I had version 2.7 of Python. Along the way, I had to install git and pip and then finally install SoapPY, by doing:

      pip install -e \         "git+http://github.com/pelletier/SOAPpy.git@develop#egg=SOAPpy"

The following URLs helped me with this:

PyXML install memmove does not exist
installing packages with pip
PyXML module

A client in Ruby

And finally a client in Ruby:

        #!/usr/bin/ruby -w
        
        require 'soap/rpc/driver'
        
        filename   = ARGV[0] || '/etc/motd'
        NAMESPACE  = 'TestWS'
        URL        = 'http://127.0.0.1:80/SoapWS'
        
        begin
            driver = SOAP::RPC::Driver.new(URL, NAMESPACE)
        
            # Add remote sevice methods
            driver.add_method('fetchdata', 'Filename' )
        
            # Call remote service methods
            values = driver.fetchdata( filename )
            mode = sprintf( "%o", values[0] )   # in octal
        
            puts "#{filename} is mode #{mode} and #{values[1]} bytes"
            
        rescue => err
            puts err.message
        end
        

Like the Python example, I ran into a few issues when I tried a fresh install on a virtual Linux Mint 16 Petra using virtualbox. The example above worked fine on an older Linux Mint system where I had ruby 1.8.7 (2012-02-08 patchlevel 358). But once I had version 1.9.3 installed, I found that the soap driver had been removed from the standard library. If I install that driver with:

      gem install soap4r-ruby1.9

then the program will work but will produce a billion warnings. Ok, not a billion. 118 to be exact. If I remove the -w warnings flag (which I do not recommend doing), it will remove all warnings except one:

        /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36: in `require':
            iconv will be deprecated in the future,
            use String#encode instead.
        

But it will still produce the correct output of: "/etc/motd is mode 100644 and 253 bytes"

If you are a Ruby person, I suggest you check out a modern Soap package, such as savon, which I have not used, but it seems to have the most downloads if you look at SOAP Clients

It's also worth checking out The best way to use SOAP with Ruby where it says "Ruby has basically dropped SOAP in favor of REST"

Conclusion

So we ended up implementing a SOAP service with a small amount of code and created clients in several high-level scripting languages (Perl, Python, PHP, and Ruby), all of which were a tiny amount of code to handle calling the web service, all without requiring a WSDL file.

The service we set up may have been trivial in inputs and outputs, but you can still have complex data structures for the request and response. What I often do with a new web-service is to have the last argument in the request be a optional options associative array (hash). Then I have a way of adding new features later to the service without breaking the existing older API. It simply has a new keyword = value in the optional options hash. For example, lets say you had a SNMP (Simple Network Management Protocol) service available via SOAP. You might have a "version" = "1", in the options to use SNMP version 1 instead of a default 2c or 3. So you might have in a call, the following structure which contains a newly implemented Newfeature:

        {
          'Version'    =>  1,
          'Timeout'    =>  2,
          'NewFeature' =>  'some-value',
        }
        

where Version and Timeout were pre-existing options. As far as your service is concerned though, the above 3 options are in a single structure (argument) in the request - an associative array. You can add as many elements as you wish in that structure in the request.

Although using WSDL files would give you the ability to use SOAP's huge flexibility to handle complex inputs and outputs beyond just a basic RPC call of ordered arguments, I have so far been able to handle any unusual cases that come along without requiring the complexity and verbosity of WSDL files. Admittedly, I've encountered some cases where it took a bit of work to figure out how to do some things (like tweak required XML tags, namespaces, etc, for some specific inflexible service), but once I figured out how, the resulting code was trivial. Hopefully, some of the above will be useful to you or encourage you to consider using a high-level scripting language without WSDL files instead of what seems to be the default of using Java and WSDL.

Source files used in the above examples is available here at Github.

 

Update (2019)

Perl install on recent Linux Mint

It's now 5 years (Sept 2019) since I originally wrote this article in 2014. I just installed this on a Linux Mint 19.2 Tina system. I ran into problems when I went to install the CPAN module SOAP::Lite, which gave me an error of:

******** WARNING ************* Your Perl is configured to link against libgdbm, but libgdbm.so was not found. You could just symlink it to /usr/lib/x86_64-linux-gnu/libgdbm.so.5.0.0

which I did by creating a symlink to it as /usr/local/lib/libgdbm.so. Running the CPAN install again and it informed me it was now installed correctly. I tried the Perl client program and it worked.

 

specify specific request XML tags

In the section XML request and response , I stated that we let the XML tag c-gensym3 be generated from laziness because we knew the server side was written in Perl and was forgiving on the request XML tags. Suppose it wasn't though. Suppose that the server required that the request XML tag for the input filename to be 'File'. We can do that by changing the last line in the SOAP request from:

->$method( "$filename" );
to:
->call( $method, SOAP::Data->name( 'File' )->value( $filename ));

This would cause the XML to change from:

<c-gensym3 xsi:type="xsd:string">/etc/motd</c-gensym3>
to:
<File xsi:type="xsd:string">/etc/motd</File>

 


You can contact RJ by e-mail at rj.white@moxad.com