Getting RAW Soap Data from a Web Reference Client running in ASP.net

I'm trying to trouble shoot a web service client in my current project. I'm not sure of the platform of the Service Server (Most likely LAMP). I believe there is a fault on their side of the fence as i have eliminated the potential issues with my client. The client is a standard ASMX type web reference proxy auto generated from the service WSDL.

What I need to get to is the RAW SOAP Messages (Request and Responses)

What is the best way to go about this?

I made following changes in web.config to get the SOAP (Request/Response) Envelope. This will output all of the raw SOAP information to the file trace.log.

<system.diagnostics>
  <trace autoflush="true"/>
  <sources>
    <source name="System.Net" maxdatasize="1024">
      <listeners>
        <add name="TraceFile"/>
      </listeners>
    </source>
    <source name="System.Net.Sockets" maxdatasize="1024">
      <listeners>
        <add name="TraceFile"/>
      </listeners>
    </source>
  </sources>
  <sharedListeners>
    <add name="TraceFile" type="System.Diagnostics.TextWriterTraceListener"
      initializeData="trace.log"/>
  </sharedListeners>
  <switches>
    <add name="System.Net" value="Verbose"/>
    <add name="System.Net.Sockets" value="Verbose"/>
  </switches>
</system.diagnostics>

You can implement a SoapExtension that logs the full request and response to a log file. You can then enable the SoapExtension in the web.config, which makes it easy to turn on/off for debugging purposes. Here is an example that I have found and modified for my own use, in my case the logging was done by log4net but you can replace the log methods with your own.

public class SoapLoggerExtension : SoapExtension
{
	private static readonly ILog log = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
	private Stream oldStream;
	private Stream newStream;

	public override object GetInitializer(LogicalMethodInfo methodInfo, SoapExtensionAttribute attribute)
	{
		return null;
	}

	public override object GetInitializer(Type serviceType)
	{
		return null;
	}

	public override void Initialize(object initializer)
	{

	}

	public override System.IO.Stream ChainStream(System.IO.Stream stream)
	{
		oldStream = stream;
		newStream = new MemoryStream();
		return newStream;
	}

	public override void ProcessMessage(SoapMessage message)
	{

		switch (message.Stage)
		{
			case SoapMessageStage.BeforeSerialize:
				break;
			case SoapMessageStage.AfterSerialize:
				Log(message, "AfterSerialize");
					CopyStream(newStream, oldStream);
					newStream.Position = 0;
				break;
				case SoapMessageStage.BeforeDeserialize:
					CopyStream(oldStream, newStream);
					Log(message, "BeforeDeserialize");
				break;
			case SoapMessageStage.AfterDeserialize:
				break;
		}
	}

	public void Log(SoapMessage message, string stage)
	{

		newStream.Position = 0;
		string contents = (message is SoapServerMessage) ? "SoapRequest " : "SoapResponse ";
		contents += stage + ";";

		StreamReader reader = new StreamReader(newStream);

		contents += reader.ReadToEnd();

		newStream.Position = 0;

		log.Debug(contents);
	}

	void ReturnStream()
	{
		CopyAndReverse(newStream, oldStream);
	}

	void ReceiveStream()
	{
		CopyAndReverse(newStream, oldStream);
	}

	public void ReverseIncomingStream()
	{
		ReverseStream(newStream);
	}

	public void ReverseOutgoingStream()
	{
		ReverseStream(newStream);
	}

	public void ReverseStream(Stream stream)
	{
		TextReader tr = new StreamReader(stream);
		string str = tr.ReadToEnd();
		char[] data = str.ToCharArray();
		Array.Reverse(data);
		string strReversed = new string(data);

		TextWriter tw = new StreamWriter(stream);
		stream.Position = 0;
		tw.Write(strReversed);
		tw.Flush();
	}
	void CopyAndReverse(Stream from, Stream to)
	{
		TextReader tr = new StreamReader(from);
		TextWriter tw = new StreamWriter(to);

		string str = tr.ReadToEnd();
		char[] data = str.ToCharArray();
		Array.Reverse(data);
		string strReversed = new string(data);
		tw.Write(strReversed);
		tw.Flush();
	}

	private void CopyStream(Stream fromStream, Stream toStream)
	{
		try
		{
			StreamReader sr = new StreamReader(fromStream);
			StreamWriter sw = new StreamWriter(toStream);
			sw.WriteLine(sr.ReadToEnd());
			sw.Flush();
		}
		catch (Exception ex)
		{
			string message = String.Format("CopyStream failed because: {0}", ex.Message);
			log.Error(message, ex);
		}
	}
}

[AttributeUsage(AttributeTargets.Method)]
public class SoapLoggerExtensionAttribute : SoapExtensionAttribute
{
	private int priority = 1; 

	public override int Priority
	{
		get { return priority; }
		set { priority = value; }
	}

	public override System.Type ExtensionType
	{
		get { return typeof (SoapLoggerExtension); }
	}
}

You then add the following section to your web.config where YourNamespace and YourAssembly point to the class and assembly of your SoapExtension:

<webServices>
  <soapExtensionTypes>
    <add type="YourNamespace.SoapLoggerExtension, YourAssembly" 
       priority="1" group="0" />
  </soapExtensionTypes>
</webServices>

Not sure why all the fuss with web.config or a serializer class. The below code worked for me:

XmlSerializer xmlSerializer = new XmlSerializer(myEnvelope.GetType());

using (StringWriter textWriter = new StringWriter())
{
    xmlSerializer.Serialize(textWriter, myEnvelope);
    return textWriter.ToString();
}

Try Fiddler2 it will let you inspect the requests and response. It might be worth noting that Fiddler works with both http and https traffic.

It looks like Tim Carter's solution doesn't work if the call to the web reference throws an exception. I've been trying to get at the raw web resonse so I can examine it (in code) in the error handler once the exception is thrown. However, I'm finding that the response log written by Tim's method is blank when the call throws an exception. I don't completely understand the code, but it appears that Tim's method cuts into the process after the point where .Net has already invalidated and discarded the web response.

I'm working with a client that's developing a web service manually with low level coding. At this point, they are adding their own internal process error messages as HTML formatted messages into the response BEFORE the SOAP formatted response. Of course, the automagic .Net web reference blows up on this. If I could get at the raw HTTP response after an exception is thrown, I could look for and parse any SOAP response within the mixed returning HTTP response and know that they received my data OK or not.

Later ...

Here's a solution that does work, even after an execption (note that I'm only after the response - could get the request too):

namespace ChuckBevitt
{
    class GetRawResponseSoapExtension : SoapExtension
    {
        //must override these three methods
        public override object GetInitializer(LogicalMethodInfo methodInfo, SoapExtensionAttribute attribute)
        {
            return null;
        }
        public override object GetInitializer(Type serviceType)
        {
            return null;
        }
        public override void Initialize(object initializer)
        {
        }

        private bool IsResponse = false;

        public override void ProcessMessage(SoapMessage message)
        {
            //Note that ProcessMessage gets called AFTER ChainStream.
            //That's why I'm looking for AfterSerialize, rather than BeforeDeserialize
            if (message.Stage == SoapMessageStage.AfterSerialize)
                IsResponse = true;
            else
                IsResponse = false;
        }

        public override Stream ChainStream(Stream stream)
        {
            if (IsResponse)
            {
                StreamReader sr = new StreamReader(stream);
                string response = sr.ReadToEnd();
                sr.Close();
                sr.Dispose();

                File.WriteAllText(@"C:\test.txt", response);

                byte[] ResponseBytes = Encoding.ASCII.GetBytes(response);
                MemoryStream ms = new MemoryStream(ResponseBytes);
                return ms;

            }
            else
                return stream;
        }
    }
}

Here's how you configure it in the config file:

<configuration>
     ...
  <system.web>
    <webServices>
      <soapExtensionTypes>
        <add type="ChuckBevitt.GetRawResponseSoapExtension, TestCallWebService"
           priority="1" group="0" />
      </soapExtensionTypes>
    </webServices>
  </system.web>
</configuration>

"TestCallWebService" shoud be replaced with the name of the library (that happened to be the name of the test console app I was working in).

You really shouldn't have to go to ChainStream; you should be able to do it more simply from ProcessMessage as:

public override void ProcessMessage(SoapMessage message)
{
    if (message.Stage == SoapMessageStage.BeforeDeserialize)
    {
        StreamReader sr = new StreamReader(message.Stream);
        File.WriteAllText(@"C:\test.txt", sr.ReadToEnd());
        message.Stream.Position = 0; //Will blow up 'cause type of stream ("ConnectStream") doesn't alow seek so can't reset position
    }
}

If you look up SoapMessage.Stream, it's supposed to be a read-only stream that you can use to inspect the data at this point. This is a screw-up 'cause if you do read the stream, subsequent processing bombs with no data found errors (stream was at end) and you can't reset the position to the beginning.

Interestingly, if you do both methods, the ChainStream and the ProcessMessage ways, the ProcessMessage method will work because you changed the stream type from ConnectStream to MemoryStream in ChainStream, and MemoryStream does allow seek operations. (I tried casting the ConnectStream to MemoryStream - wasn't allow.)

So ..... Microsoft should either allow seek operations on the ChainStream type or make the SoapMessage.Stream truly a read-only copy as it's supposed to be. (Write your congressman, etc...)

One further point. After creating a way to retreive the raw HTTP response after an exception, I still didn't get the full response (as determined by a HTTP sniffer). This was because when the development web service added the HTML error messages to the beginning of the response, it didn't adjust the Content-Length header, so the Content-Length value was less than the size of the actual response body. All I got was the Content-Length value number of characters - the rest were missing. Obviously, when .Net reads the response stream, it just reads the Content-Length number of characters and doesn't allow for the Content-Length value possibily being wrong. This is as it should be; but if the Content-Length header value is wrong, the only way you'll ever get the entire response body is with a HTTP sniffer (I user HTTP Analyzer from http://www.ieinspector.com).

Here's a simplified version of the top answer. Add this to the <configuration> element of your web.config or App.config file. It will create a trace.log file in your project's bin/Debug folder. Or, you can specify an absolute path for the log file using the initializeData attribute.

  <system.diagnostics>
    <trace autoflush="true"/>
    <sources>
      <source name="System.Net" maxdatasize="9999" tracemode="protocolonly">
        <listeners>
          <add name="TraceFile" type="System.Diagnostics.TextWriterTraceListener" initializeData="trace.log"/>
        </listeners>
      </source>
    </sources>
    <switches>
      <add name="System.Net" value="Verbose"/>
    </switches>
  </system.diagnostics>

It warns that the maxdatasize and tracemode attributes are not allowed, but they increase the amount of data that can be logged, and avoid logging everything in hex.

I would prefer to have the framework do the logging for you by hooking in a logging stream which logs as the framework processes that underlying stream. The following isn't as clean as I would like it, since you can't decide between request and response in the ChainStream method. The following is how I handle it. With thanks to Jon Hanna for the overriding a stream idea

public class LoggerSoapExtension : SoapExtension
{
    private static readonly string LOG_DIRECTORY = ConfigurationManager.AppSettings["LOG_DIRECTORY"];
    private LogStream _logger;

    public override object GetInitializer(LogicalMethodInfo methodInfo, SoapExtensionAttribute attribute)
    {
        return null;
    }
    public override object GetInitializer(Type serviceType)
    {
        return null;
    }
    public override void Initialize(object initializer)
    {
    }
    public override System.IO.Stream ChainStream(System.IO.Stream stream)
    {
        _logger = new LogStream(stream);
        return _logger;
    }
    public override void ProcessMessage(SoapMessage message)
    {
        if (LOG_DIRECTORY != null)
        {
            switch (message.Stage)
            {
                case SoapMessageStage.BeforeSerialize:
                    _logger.Type = "request";
                    break;
                case SoapMessageStage.AfterSerialize:
                    break;
                case SoapMessageStage.BeforeDeserialize:
                    _logger.Type = "response";
                    break;
                case SoapMessageStage.AfterDeserialize:
                    break;
            }
        }
    }
    internal class LogStream : Stream
    {
        private Stream _source;
        private Stream _log;
        private bool _logSetup;
        private string _type;

        public LogStream(Stream source)
        {
            _source = source;
        }
        internal string Type
        {
            set { _type = value; }
        }
        private Stream Logger
        {
            get
            {
                if (!_logSetup)
                {
                    if (LOG_DIRECTORY != null)
                    {
                        try
                        {
                            DateTime now = DateTime.Now;
                            string folder = LOG_DIRECTORY + now.ToString("yyyyMMdd");
                            string subfolder = folder + "\\" + now.ToString("HH");
                            string client = System.Web.HttpContext.Current != null && System.Web.HttpContext.Current.Request != null && System.Web.HttpContext.Current.Request.UserHostAddress != null ? System.Web.HttpContext.Current.Request.UserHostAddress : string.Empty;
                            string ticks = now.ToString("yyyyMMdd'T'HHmmss.fffffff");
                            if (!Directory.Exists(folder))
                                Directory.CreateDirectory(folder);
                            if (!Directory.Exists(subfolder))
                                Directory.CreateDirectory(subfolder);
                            _log = new FileStream(new System.Text.StringBuilder(subfolder).Append('\\').Append(client).Append('_').Append(ticks).Append('_').Append(_type).Append(".xml").ToString(), FileMode.Create);
                        }
                        catch
                        {
                            _log = null;
                        }
                    }
                    _logSetup = true;
                }
                return _log;
            }
        }
        public override bool CanRead
        {
            get
            {
                return _source.CanRead;
            }
        }
        public override bool CanSeek
        {
            get
            {
                return _source.CanSeek;
            }
        }

        public override bool CanWrite
        {
            get
            {
                return _source.CanWrite;
            }
        }

        public override long Length
        {
            get
            {
                return _source.Length;
            }
        }

        public override long Position
        {
            get
            {
                return _source.Position;
            }
            set
            {
                _source.Position = value;
            }
        }

        public override void Flush()
        {
            _source.Flush();
            if (Logger != null)
                Logger.Flush();
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            return _source.Seek(offset, origin);
        }

        public override void SetLength(long value)
        {
            _source.SetLength(value);
        }

        public override int Read(byte[] buffer, int offset, int count)
        {
            count = _source.Read(buffer, offset, count);
            if (Logger != null)
                Logger.Write(buffer, offset, count);
            return count;
        }

        public override void Write(byte[] buffer, int offset, int count)
        {
            _source.Write(buffer, offset, count);
            if (Logger != null)
                Logger.Write(buffer, offset, count);
        }
        public override int ReadByte()
        {
            int ret = _source.ReadByte();
            if (ret != -1 && Logger != null)
                Logger.WriteByte((byte)ret);
            return ret;
        }
        public override void Close()
        {
            _source.Close();
            if (Logger != null)
                Logger.Close();
            base.Close();
        }
        public override int ReadTimeout
        {
            get { return _source.ReadTimeout; }
            set { _source.ReadTimeout = value; }
        }
        public override int WriteTimeout
        {
            get { return _source.WriteTimeout; }
            set { _source.WriteTimeout = value; }
        }
    }
}
[AttributeUsage(AttributeTargets.Method)]
public class LoggerSoapExtensionAttribute : SoapExtensionAttribute
{
    private int priority = 1;
    public override int Priority
    {
        get
        {
            return priority;
        }
        set
        {
            priority = value;
        }
    }
    public override System.Type ExtensionType
    {
        get
        {
            return typeof(LoggerSoapExtension);
        }
    }
}

You haven't specified what language you are using but assuming C# / .NET you could use SOAP extensions.

Otherwise, use a sniffer such as Wireshark

This is the command that gets your raw message in c#:

OperationContext.Current.RequestContext.RequestMessage

I wasted 3 days trying with soap extention and it was way too simpler than I thought. Soap extentions are deprecated . I recommend reading these How To: Inspect or Modify Messages on the Client on MSDN. and Using the Message Class

I realize I'm quite late to the party, and since language wasn't actually specified, here's a VB.NET solution based on Bimmerbound's answer, in case anyone happens to stumble across this and needs a solution. Note: you need to have a reference to the stringbuilder class in your project, if you don't already.

 Shared Function returnSerializedXML(ByVal obj As Object) As String
    Dim xmlSerializer As New System.Xml.Serialization.XmlSerializer(obj.GetType())
    Dim xmlSb As New StringBuilder
    Using textWriter As New IO.StringWriter(xmlSb)
        xmlSerializer.Serialize(textWriter, obj)
    End Using


    returnSerializedXML = xmlSb.ToString().Replace(vbCrLf, "")

End Function

Simply call the function and it will return a string with the serialized xml of the object you're attempting to pass to the web service (realistically, this should work for any object you care to throw at it too).

As a side note, the replace call in the function before returning the xml is to strip out vbCrLf characters from the output. Mine had a bunch of them within the generated xml however this will obviously vary depending on what you're trying to serialize, and i think they might be stripped out during the object being sent to the web service.