Ning Chat Capture Overview

(a.k.a. HOWTO create a transcript from Ning Chat.)

Some of you wanted to understand a little better how I captured the Jeff Kulczycki chat. Here is an overview.

I used a web programmers diagnostic tool named Fiddler. This program logs all traffic sent to and from your web browser. This helps programmers understand the communications between the browser and the server.

Since Ning Chat is hosted in a browser, I found I could use Fiddler to log all of the traffic sent from my browser to Ning's IM server (ningim.com). Lo and behold, each chat message is transmitted from ningim.com back to the browser on line at a time!

Unfortunately, there is a whole lot of information included with each message from the server that we don't need. (See Figure 1.) Wading through all that information, we get to the information we want. (See Figure 2.)

Now, I export the raw text of all of the chat message package to a text file. (See Figure 3, and sample raw browser to server communications log.)

I tried manually stripping out the bulk of the junk, after an hour I was maybe 5% of the way through the log. So, I decided to try using a magical search and replace trick called Regular Expressions. Using Regular Expressions I can find text that "looks like a date" and replace it with "nothing". Using tricks like that get 80-90% of the nonsense stripped.

At this point I am left with a much more manageable log file. The last 10% is a lot of manual formatting.

I think next chat I will set my computer up to log the chat. That way I only have to read the chat while I am editing it. ;-)

The first time took me 2.5 hours after the chat ended; plus the duration of the chat; plus 3-4 hours of research to find a way to log the chat. All told I probably spent 7-8 hours.

However, I am going to try to provide the transcripts for at least one to two more chats. It is my hope that I refine my technique down to where I can take a chat log and clean it up in 1.5 hours. I think 1.5 hours is a reasonable amount of time to spend on a chat transcript. If all goes well, I may even be able to get it down to an hour by writing an AppleScript to do some of the initial log cleanup for me. :-)

Thank you for your support and encouragement everyone. I hope I didn't bore you too much. :-)

Brian Hanifin