[TYPO3-Solr] Indexing custom Items

[Stephan Schuler]

Hi Dmitry.


This is a good point on one hand. If you consider Extbase as privileged frontend mechanism to be covered by Solr, then we should do the same for the old fashioned way.

But the other hand is: Using plain pi_base has several other drawbacks that are really unfriendly to integration of data and handling of them inside of TYPO3. Think about the "tstamp" field, the sys_history, the sys_refindex or the the local representation field of a mm field, which usually contains the number of related records.

The point is: If there is a persistence-related API that handles those things. If there is an internal mechanism that takes care of integrity of data and that keeps track of internal storage things, then Solr should hook to that API and make itself aware of all of those changes that are made by the official persistence API stuff.

If Extbase currently does not update sys_history, sys_refindex, tstamp or whatever other stuff usually gets done when a be_user handles records, then that's a problem and failure of Extbase and needs to get fixed in Extbase.

Using pi_base usually means: You need to do everything by hand. You take care of creating record, deleting it. Putting it on a PID where it is allowed to be (pages, storage folders, rootpage). Updating the "type" field if there is a record type. Make sure to only have data inside of the database that matches the TCA in terms of validation.

If there is any persistence API that handles those things, that's the point where Solr can rely on -- and should.
If there is another persistence API that only provides raw access to any storage backend, then it's the task of whoever uses this API to make Solr deal with it.
It's kind of "general record post processing".

Would be nice if all of those things provided a nice an easy onRecordUpdateHook that only notifying about something like "array($tableName => array($uid1, $uid2, $uid3))". This way Solr could rely on that and then you're done.

Another option, completely independent from this would be: Create a Solr task that internally compares and handles inconsistencies between the solr index and the original records.
I'm thinking about something like this:

> SELECT
>       *
> FROM
>       tx_news_domain_model_news AS record
> WHERE
>       record.uid NOT IN (SELECT item_uid FROM tx_solr_indexqueue_item WHERE item_type = "tx_news_domain_model_news")
>       AND (
>               record.sys_language_uid IN (0, -1)
>               OR record.l10n_parent = 0
>       )
>       AND record.deleted != 1
>       AND record.hidden != 1

I'm not talking about the "Force Re-Index" task here obviously. I don't want to make already indexed records being reindexed and I don't want to clear Solr content here. I just want to provide a mechanism to attach newly created records to the indexqueue. This should take the table to be indexed as parameter, and maybe it should take an array of storage folders optionally.

It's not nice, but it should work with any persistence mechanism one choses.
And I would clearly make it optional and require the administrator of an installation and configure this specific integrity task.

Of course it's far away from completed because it doesn't handle workspaces. Maybe I just wanted to prove my point with this snippet.


Regards,




Stephan Schuler
Web-Entwickler

Telefon: +49 (911) 539909 - 0
E-Mail: Stephan.Schuler at netlogix.de
Website: media.netlogix.de



--
netlogix GmbH & Co. KG
IT-Services | IT-Training | Media
Neuwieder Straße 10 | 90411 Nürnberg
Telefon: +49 (911) 539909 - 0 | Fax: +49 (911) 539909 - 99
E-Mail: info at netlogix.de | Internet: http://www.netlogix.de

netlogix GmbH & Co. KG ist eingetragen am Amtsgericht Nürnberg (HRA 13338)
Persönlich haftende Gesellschafterin: netlogix Verwaltungs GmbH (HRB 20634)
Umsatzsteuer-Identifikationsnummer: DE 233472254
Geschäftsführer: Stefan Buchta, Matthias Schmidt



-----Ursprüngliche Nachricht-----
Von: typo3-project-solr-bounces at lists.typo3.org [mailto:typo3-project-solr-bounces at lists.typo3.org] Im Auftrag von Dmitry Dulepov
Gesendet: Mittwoch, 2. Oktober 2013 09:01
An: typo3-project-solr at lists.typo3.org
Betreff: Re: [TYPO3-Solr] Indexing custom Items

Hi!

Stephan Schuler wrote:
> If Extbase really doesn't trigger TCEMain hooks (which is quite likely), then Solr should take care of this itself and register to Extbase persistence manager signals instead.
> It's not your applications concern (nor your controllers nor the concern of one of your services) to call Solr. Imho it's completely the task of Solr itself make itself called.

What about extensions that are not based on Extbase? How should Solr work with them?

--
Dmitry Dulepov

Today is a good day to have a good day.
_______________________________________________
TYPO3-project-solr mailing list
TYPO3-project-solr at lists.typo3.org
http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-project-solr