[TYPO3-english] problem with External Data Import ext

[François Suter]

Hi Nicola,

> Moreover I also found out that it's just (or mainly) a matter of execution
> time in fact rising it from 30 to 90 seconds makes the execution not to
> fail.
>
> This matter, as i suppose, happens only in the manual sync coz it uses ajax?

I think, yes. AFAIK there's no timeout when executed on the command line 
(i.e. via the Scheduler).

I guess you did notice that you can raised the execution time just for 
external_import in the extension's configuration.

> May I ask you also a suggestion? We are importing a huge amount of data,
> just to give u an estimate 10 to 20 fields from a 1000 to 150.000 records
> per connector.
> Do you have any "best-practice" to import such an amount of data? I.e.
> importing less field per time creating more connectors.

I have experience with up to 50 fields for about 15000 records and the 
update still happens in a couple of minutes. The longest update I have 
is around 20 minutes (although it's not the largest, I don't know why).

What I would advise, if you have so much data is to split it into 
several chunks, for example 10000 records at a time. To achieve this you 
need to have as many external_import configurations, which is a hassle 
to set up, but it may be worth the effort.

Maybe you can also filter out records which were changed or not in the 
external data. If your conector can issue a query based on some time 
stamp or "new" flag, you could maybe have 2 imports only:

- one that imports new or changed stuff, and which runs regularly
- on that imports everything and which is only run occasionally.

> PS if u would like i can send u the t3x about the connector for db I made,
> but I think ur solution is much better without even seeing it ;-)

Why not. Just send it along. My connector is pretty advanced yet, but 
it's not perfect either, so there may be ideas to take from yours :-)

Cheers

-- 

Francois Suter
Cobweb Development Sarl - http://www.cobweb.ch