[TYPO3-english] problem with External Data Import ext

[Nicola Cerchiari]

Hi again,
i went on with my develop and made some trials with the scheduled tasks, but
i'm stuck again (sorry).
The first run  complain that the cli user had not enough rights to update
tt_address (table that i need to update).
Then I updated cli_scheduler user rights to everything i found that could
make sense with tt_address table and in fact second run went smooth and it
inserted every record, but the third run and on complain that "Attempt to
modify record ########### without permission. Or non-existing page."

I checked again and again cli_scheduler user  rights and I also added db
mount of the page where addresses are, but nothing change.

Any help?

Nicola


2011/3/8 François Suter <fsu-lists at cobweb.ch>

> Hi Nicola,
>
>
>  Moreover I also found out that it's just (or mainly) a matter of execution
>> time in fact rising it from 30 to 90 seconds makes the execution not to
>> fail.
>>
>> This matter, as i suppose, happens only in the manual sync coz it uses
>> ajax?
>>
>
> I think, yes. AFAIK there's no timeout when executed on the command line
> (i.e. via the Scheduler).
>
> I guess you did notice that you can raised the execution time just for
> external_import in the extension's configuration.
>
>
>  May I ask you also a suggestion? We are importing a huge amount of data,
>> just to give u an estimate 10 to 20 fields from a 1000 to 150.000 records
>> per connector.
>> Do you have any "best-practice" to import such an amount of data? I.e.
>> importing less field per time creating more connectors.
>>
>
> I have experience with up to 50 fields for about 15000 records and the
> update still happens in a couple of minutes. The longest update I have is
> around 20 minutes (although it's not the largest, I don't know why).
>
> What I would advise, if you have so much data is to split it into several
> chunks, for example 10000 records at a time. To achieve this you need to
> have as many external_import configurations, which is a hassle to set up,
> but it may be worth the effort.
>
> Maybe you can also filter out records which were changed or not in the
> external data. If your conector can issue a query based on some time stamp
> or "new" flag, you could maybe have 2 imports only:
>
> - one that imports new or changed stuff, and which runs regularly
> - on that imports everything and which is only run occasionally.
>
>
>  PS if u would like i can send u the t3x about the connector for db I made,
>> but I think ur solution is much better without even seeing it ;-)
>>
>
> Why not. Just send it along. My connector is pretty advanced yet, but it's
> not perfect either, so there may be ideas to take from yours :-)
>
>
> Cheers
>
> --
>
> Francois Suter
> Cobweb Development Sarl - http://www.cobweb.ch
> _______________________________________________
> TYPO3-english mailing list
> TYPO3-english at lists.typo3.org
> http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-english
>