[TYPO3-english] problem with External Data Import ext

[Nicola Cerchiari]

Hi Françcois,
thank you very much for ur advices, they have been very beautiful. In fact i
could find out that the "problem" is into the method:
$tce->process_datamap();
Moreover I also found out that it's just (or mainly) a matter of execution
time in fact rising it from 30 to 90 seconds makes the execution not to
fail.

This matter, as i suppose, happens only in the manual sync coz it uses ajax?

May I ask you also a suggestion? We are importing a huge amount of data,
just to give u an estimate 10 to 20 fields from a 1000 to 150.000 records
per connector.
Do you have any "best-practice" to import such an amount of data? I.e.
importing less field per time creating more connectors.

Bests and thank you very much again,
Nicola

PS if u would like i can send u the t3x about the connector for db I made,
but I think ur solution is much better without even seeing it ;-)


2011/3/7 François Suter <fsu-lists at cobweb.ch>

> Hi Nicola,
>
>
>  UPDATE:
>> I found it (stupid me), i didn't really understand that
>> "additional_fields"
>> do not have to be declared if u also want to import them ;).
>>
>
> Correct :-)
>
>
>  So now the import goes smooth but still  it gives me "syncronization
>> failed!" even if the data are all correctly imported.
>> It always stops at the debug message "TCEmain data".
>>
>> Never get to reach the following steps "New IDs" and "Synchronisation of
>> table..." that sometimes (and I didn't get how) it reached.
>>
>
> Hmm, this is really weird. The situation with the "Synchronization failed"
> message is a bit of a bother. It may be that a PHP error happened.
> Unfortunately you never get to see the error message in the TYPO3 BE,
> because the output is flushed at some point and whatever error output there
> was is lost (that has to do with the fact that the synchronisation occurs
> via an AJAX call). OTOH you should be able to see something in your PHP logs
> if an error really happened.
>
> There's another situation, also related to the use of AJAX: it may be that
> the response object is too large. I could not figure out what is the size
> limit, but if the response is too large, it is simply dropped. Thus you
> actually get a successful HTTP response (code 200), but an empty response
> and thus a "Synchronization failed" message. The response may be too large
> if the actual data storage generated many errors (I have a development
> version which fixes that, but it's not ready for release yet). However in
> this case you should see error messages in the devlog if you have the debug
> mode activated. So I would rather goes with the first hypothesis.
>
> Other than that, it means the problem is more peculiar and is probably
> related to a specificity of your system, so I can only advise you to dig
> into the source code and place additional debug statements. In your case the
> problem would probably be happening in the
> tx_externalimport_importer::storeData() method.
>
> I hope this helps a bit.
>
>
>  1) I made a new connector service as I need to retrieve data from db ->
>>> works fine;
>>>
>>
> Interesting. I worked on that too and am actually close to releasing such
> an extension. I just need to write the manual. See:
>
> http://forge.typo3.org/projects/show/extension-svconnector_sql
>
> Cheers
>
> --
>
> Francois Suter
> Cobweb Development Sarl - http://www.cobweb.ch
>
> _______________________________________________
> TYPO3-english mailing list
> TYPO3-english at lists.typo3.org
> http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-english
>