[TYPO3-dam] Indexing freezes browser

[Florian Schaeffer]

Hi Kristian,

Kristian Domke schrieb:
> Thomas Murphy schrieb:
> 
>> Hello everyone,
>>
>> in a large project we're indexing about 2000 files a day.
>> I wrote a service which extracts some metadata from the filenames and 
>> copies them from an "incoming" folder to an archive.
>> We skipped the indexing for some days now, having a total of about 8000.
>> Indexing these freezes the browser by using too much virtual memory.
>> I searched in my service for a solution and erased all suspicious 
>> looking devlog statements, but nothing changes.
>> Even when indexing around 1000 files, the browser freezes and the 
>> indexing stops at approximately 500 files.
>>
> 
> Is it the same File everytime? Maybe it is corrupt in one way or another...
> 
> Kristian

as I'm working with Murphy on that problem I feel free to answer the 
question for him:

No it is not the same file, it is only the huge amount of files to index 
in one step.

As Murphy said, we had about 8000 images in one "incoming"-folder and 
tried to index the whole folder in one step. That was not possible, the 
*browser* freezed all the time we tried (we used Firefox 1.0.7 on WinXP, 
IE 6 on WinXP and IE 5.5 on Win2000, all browsers were very slow, they 
needed huge amounts of virtual memory (around 240 MB) and finally (after 
around 130 files) crashed), so we removed files from the folder. then we 
tried 1000 files in one step, that was damn slow but after waiting a 
while it worked. The progress bar was nevertheless not moving from lets 
say around 500 files until the state "finished" appeared.

Then I tried to index around 500 files in one step and it had the same 
effect, after 470 files the progress bar stopped, and then switched to 
finished.

Then I tried around 400 files, and everything was perfect, the indexing 
process was much faster and the progress bar was counting perfectly up 
through 100%.

So there seems to be a problem when indexing huge file amounts in one 
step, or at least the progress bar isn't working as expected...

Florian