Strategy: Asynchronous Queued Virus Scanning

Atif Ghaffar has a nice strategy to deal with virus checking uploads:

Upload item into a safe area. If necessary, the uploader blocks waiting for a result.

Queue a work order into a job system so all the work can be distributed throughout your cluster.

A service in your cluster performs the virus scan and informs the uploader of the result.

Move the vetted item into your system.

This removes the CPU bottleneck from your web servers and distributes it through your cluster. Keep your web servers providing prompt service to users. Let your cluster do the heavy lifting. This minimizes response time and maximizes throughput. A similar system can be used for creating thumbnails, transcoding, copyright checks, updating indexes, event notification or any other kind of intensive work.

Comments

[atif.ghaffar]

Thanks for the heads-up Todd,

One more comment,

The same can be applied for your file uploads.
Dont set your frontend webservers to wait for a max_execution_time of 60 seconds etc.
Anything that does not finish in 1 second is a bug.

Set up a different server that handles only uploads and knows nothing else.
The upload server can respond back with a temp URL that can be used to move/copy the uploaded file to the permanent area.

This can be used for image/whatever.

best regards