[TYPO3-Solr] How to handle changes on related values

[Ingo Renner]

Am 27.12.12 02:44, schrieb Stephan Schuler:
> Hi Ingo.
>
>
> Thank you for answering my question. For I minute, I thought you would just ignore this because my "workaround" is the way to go.
>
>
> On the one hand I think ref index is kind of unreliable.

True, that's also why we went for the Index Queue back then.

> I don't know if things have changed since extbase rose because I didn't touch the ref index since 4.2 or so.

I think it got better through IRRE and especially FAL. Still would need 
to check that tough.

> But when I tried to use ref index the last time, I had several extensions that modified database tables without updating the ref index table. So using ref index should work when you deal with records getting modified by backend only, but everything else feels kind of unpredictable to me. I know that this situation is related to bad programming of foreign extensions and we should not care about stuff that works against the stable API, but if this means a certain percentage of use cases conflicts and behaves unpredictable crazy, we should do the best to avoid this. A not properly managed ref index should be really hard to discover when the error message is "some records are only indexed partially".

Agreed.

> And on the other hand: I don't know if we should restrict the "foreign record tracking" behavior to records that have TCA relations to a record.
> Think about a relation from Tx_Myext_Record to tt_content. That's something the ref index will cover.
> But think about the related tt_content being either a templavoila record or of type "core records" (don't know its exact name, but there definitely is a default core tt_content which collects a couple of UIDs and passes them to the cObject "RECORD" when being rendered). Then you will keep track of relations to the tt_content container record, but this doesn't necessarily change when child records get changed.
> As soon as the child record has relations itself and gets passed to the indexer by default core rendering mechanisms (RECORDS/CONTENT), you completely lose the child-child records that might be important for the rendering output. They influence the rendering output, but you don't know about them and therefore cannot track them. I guess very often we will end up collecting almost every record in the database when we rely on ref index.

Right, that's where the flexibility of page rendering in TYPO3 bites us. 
That's also why we chose for FE indexing for pages... it's just too 
flexible.

> Another thing: Not all child record update actions really do influence the rendering output. I think the very most relations a record has don't go into solr.

True, but not really a problem when indexing records, as that's usually 
fast.

>
> And the last thing is very technical: Management of indexing queue filled with both, indexing records and relation records. The indexing queue currently holds the lastIndexTime, then it decides by a simple sql query if a record is younger then the lastIndexTime. When we add relations to the tracking mechanism, we either add them to the indexing_queue table as well and introduce a type field or we create a dedicated relations table. But in both cases, we need some place to track the lastIndexeTime for the related record.

Uhh true! That's the case for pages already. If you change a tt_content 
record it'll change the page's last updated time to the most recent 
change time of the tt_content elements on that page.

> So, it's up to you what you want to do. But as you can see, I really would not go ref index but a custom relation tracker.

A relation tracker in general sounds like a good idea. In the end I 
would still like to follow a principle that we have followed until know: 
For the regular integrator it should be very easy and fast to achieve 
results. So whenever possible I would like to avoid having people to 
touch PHP and do most(/all) things through TS configuration.


Ingo

-- 
Ingo Renner
TYPO3 Core Developer, Release Manager TYPO3 4.2, Admin Google Summer of Code

TYPO3 - Open Source Enterprise Content Management System
http://typo3.org

Apache Solr for TYPO3 -
Open Source Enterprise Search meets Open Source Enterprise CMS
http://www.typo3-solr.com