Streaming CGI

Every now and again I come across the need to build a CGI/web page which has streaming output. That is, I want the page to start rendering before the CGI stream has finished – the way that the LookingGlass traceroute and ping tools do.

In this case, it’s for automating both the host and the Mac builds. Between yesterday and today I’ve completely automated the Mac build process along with all kinds of command switches and safeties and fallbacks. A little collection of perl scripts on the mac dev box and the patch build box.

What it needs now is a web interface (because we do everything else thru a web interface).

The easiest route is just to write a page with all the options that calls a CGI that runs the build process and finally renders when it’s done, but I’d much rather have it updating on the fly.

What I’m thinking is that perhaps when you click submit it simply issues an async Ajax query to get the results, and I could fill out a <status> division with the log results and do my own bit of cleverness to catch highlights from the output to set a title so you can see it updating, until it’s finished.

The other alternative would be to use an IFRAME, but then I need to remember how to tell the browser to display stuff as soon as it arrives and to continue waiting until the end of the stream without timing out.

Google is not helping me this time :)


Something like this? or

Perl & Python & Jsps has similar constructs.

What about javascript?

A lot of people just fake it and do a meta refresh every few seconds, just keep appending to the page and refreshing it.

I can’t figure why the mac buildbox is called “asshat2” :)

Maybe you used Bilton’s builder money to buy it?

(I kid ;)

IIRC the browser will always render an incomplete stream, as best it can. By flushing your output stream periodically you should be able to get what you want.

If the output stream is compressed, it won’t render until the entire page is done.

In case of tables, it will not render the table until the final tag is seen.

I think I’ll try the ajax approach. I’m fairly sure that I’ll receive data in the async handler on the fly so I can process it as it comes in.

I believe partial table rendering is a browser-specific thing. Some browsers try to render the table halfway to make it look like the page is loading faster.

One thing I used to do with Perl CGI’s was to turn on “hot pipes” with $|=1; but I’m sure Oli’s familiar with that one. The compression will definitely be a problem, although I would still expect the browser to be able to render intermediate chunks once it has them.

Even though before the flushing-output-stream+browser-trying-its-best was what most people used, I think the AJAX solution would be the more sensible nowadays.

You will be able to have your page completely rendered and then update just the part that you want. Even though one could try the approach known as Comet, server updating the client through the same HTTP channel that is kept open, a polling approach with some AJAX would scale probably better and it is not that bleeding edge so you’ll find good docs and libraries that support it.

The web part in fact shouldn’t be too difficult, the “toughest” part sometimes is getting the background process to inform you of its status, as some processes just asnwer back when finished, but if you have that part solved, then you are good to go.

The output I’m sending isn’t going to be complex HTML, it’s going to be pretty raw things like

-Connected to SCM Repository
-Fetching updated files
-Starting Build Process
-Pushing files to Build Box

The process can take upto 5 minutes, and I don’t want people clicking multiple times or thinking it’s timed out; I’ll have an atomic locking mechanism to stop multiple concurrent runs or allow me to kill an earlier run, but I just don’t want them to be blind.

That sounds relatively simple, a short JSON answer should do, interpreted in the client to update the HTML and make it look nicer.
Regarding the clicking multiple times etc, you could simply disable the ability to send another request after the first has been sent and provide some feedback to let the user know something is going on, like the typical animations, an aprox countdown…

Leave a Reply

Name and email address are required. Your email address will not be published.

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <s> <strike> <strong> 

%d bloggers like this: