Response.Write/Flush hangs after many calls

Question

DISCLAIMER: I posted the same question in Stackoverflow in English, I will carry the valid answer from here to there or from there to here if necessary, but I know that not all users here access Stackoverflow in English, so I intended to expand the answer radius :)

Problem

I am trying to solve this problem for days. We have a process of importing records from a file csv to the database, through an administrative page that resides in a project ASP.NET Web Forms (.NET 4.0).The process was slow and I was responsible for making it more agile. I started changing the internal logic, which has already given a significant gain in performance.

However, if I upload large files (well, relatively large, of at most 3MB), I have to wait until the upload process is over until I start importing, and I don’t return any progress information to the client while I do so. The process itself is not so time-consuming, it takes 5 to 10 seconds to complete, and yes, I’ve thought about creating a Task separate and use Polling to drip the server, but I found this a cannon to kill mosquito.

---

What I’ve done so far?

So, to fix this problem, I decided to read the stream request and import values while doing so. I created a Handler generic (.ashx), and I put the following code inside the method void ProcessRequest(HttpContext context):

using (var stream = context.Request.GetBufferlessInputStream())
{
}

First I remove the headers (request headers), and then I read the stream (through a StreamReader) until I find a CRLF ( r n), convert the line to my model, and keep reading the CSV. When I get 200 files (or until the end of the file), I update them all at once in the database. Then, I continue the process getting more records until the end of the file.

That seems to work, but then I decided to respond via stream also. First, I disabled the Bufferoutput:

context.Response.BufferOutput = false;

And then, I added these headers to my reply (sponse)

context.Response.AddHeader("Keep-Alive", "true");
context.Response.AddHeader("Cache-Control", "no-cache");
context.Response.ContentType = "application/X-MyUpdate";

Then, after sending the 200 records to the base, I write on sponse:

response.Write(s);
response.Flush();

s is a fixed size string of 256 chars. I know 256 chars don’t always equal 256 bytes, but I just wanted to make sure I didn’t write text walls at once and spoil something.

Here is the format:

| pipeline (delimitador)
1 or 0 sucesso ou falha
; delimitador
mensagem de erro (se houver)
| pipeline (próximo delimitador)

Example:

|0;"Preço inválido na linha 123"|1;|1;|0;"Id inválido na linha 127"|

In the client-side, I have it here (just the snippet that makes the request):

function import(){
    var formData = new FormData();
    var file = $('[data-id=file]')[0].files[0];
    formData.append('file', file);

    var xhr = new XMLHttpRequest();
    var url = "/Site/Update.ashx";

    xhr.onprogress = updateProgress;

    xhr.open('POST', url, true);
    xhr.setRequestHeader("Content-Type", "multipart/form-data");
    xhr.setRequestHeader("X-File-Name", file.name);
    xhr.setRequestHeader("X-File-Type", file.type);
    xhr.send(formData);
}

function updateProgress(evt){
    debugger;
}

---

What happened :(

• The data is not sent immediately to the client when I call response.Flush. I understand that there is buffering on the customer’s side, but it still doesn’t seem to work, even when I send a bunch of junk together to go over that buffer.
• After a while, after writing a lot on Response.Write, the method will become progressively slow until it crashes. The same thing may occur in the Response.Flush. I believe I’m letting something slip through here...
• I created a simple webform project to test what I’m trying to do. It has a Handler generic that will return one number per second for 10 seconds. It actually updates (not always at the rate of 1 second per update) and I can see progress.
• When I write a few lines on sponse, progress is displayed (the event is called), but ALWAYS after the process is almost over. The problem is that when I get errors, I write these errors in Sponse. That is, the message gets bigger than when everything returns success, because they contain the error messages.

I’m assuming if I write Response.Flush is not a 100% guarantee that what I wrote will immediately go to the client, right? Or the problem is itself client? If it is the client, why the server hangs when I call the Response.Write often?

I will be happy to provide further information if you deem it necessary.

Answer 1

Conrad,

I think you may have already solved the problem but worth the reflection.

I found your solution very clever and do not understand why the flush is giving problem. I don’t think a 3MB file is large but I see that we need to re-evaluate the purpose of the functionality. The solution you are using now is genuinely complex: you are establishing a connection to the server, transmitting the file, inserting it into the database and responding to progress all in one operation at the same time. It’s a lot at once. The risks of mistakes and dark situations are remarkable.

Pooling solutions are interesting, at least those based on the creation of temporary sessions on the server for the client keep asking the status of the processing. In the image below is my view of a standard pooling architecture for asynchronous processes.

It is worth reflecting that if we are talking about a utility app, with multiple users, it will be better to use something very responsive and careful. Solutions that make a good impression on the user are expensive to develop and there is no way.

But if you consider this "a mosquito-killing cannon," then I encourage you to use an even simpler solution. For example: A process where your server receives the file and processes the information asynchronously with detailed right of reply at the end. This log could be sent by email or appear in a specific area.