[TYPO3-dev] Exception, "hidden" and sorting_foreign field of sys_file_reference not honored for pages.media?

[Christian Reiter]

Hi list,

In TYPO3 6.1.7 I've seen some undesired behaviour with FAL.

It appears that in some circumstances,

* The "hidden" field of sys_file_refeence is not checked,
   and this kills the frontend  (Exception)

* The "sorting" available for inline references in the BE
   (stored in sys_file_reference.sorting_foreign) is not checked,
   leading to unexpected results when using e.g. listNum=0
   (in Typoscript)

Scenario:
---------

Editors are adding banner images to "pages" records (Tab "Resources") as 
they have done forever.

The references are read by a typoscript cointaing an IMAGE with

  file {
        import.data = levelfield:1, media
        treatIdAsReference = 1
        import.listNum = 0
        }
(exact conf may vary, no matter if slide, etc..)

Now editors sometimes get a new banner. They like the idea that they can 
"hide" the old reference and add the new one to test it. If the new 
banner is disapproved they delete the new one and unhide the old one 
(without having to pick the reference again)

However whenever there is a hidden old reference as well as a visible 
new one, there is ALWAYS an exception thrown
'No fileusage (sys_file_reference) found for given UID'

But when there is a hidden NEW reference and a visible OLD one the 
exception always disappears.

"Old" meaning "Lower UID entry in sys_file_reference table".

Also when there are several banners, results are confusing, instead of 
just showing "the top one" as expected from listNum=0

Now  the Typoscript import.listNum = 0 could be interpreted that if the 
editor sorts the references in the backend so that the hidden one is at 
bottom, and a visible one at top, he does not get the exception. Because 
listNum=0 should return the first (0 in array) one and that is visible.

This does not help however.

Sorting the visible reference to top does *not* influence what is picked 
as the UID from sys_file_reference by listNum=0 (this is shown directly 
in the exception screen. It is always the same uid picked from 
sys_file_reference no matter what the sorting of several references in 
the BE).

I tried to trace fully how this is rendered. (see below for details)
The result was that I found in class

	TYPO3\CMS\Core\Utility\RootlineUtility

function

	enrichWithRelationFields

the line 267

	$rows = $this->databaseConnection->exec_SELECTgetRows(
		'uid', $table, $whereClause);

which for this case has as $whereClause

	uid_foreign = 15 AND fieldname = 'media'
	AND tablenames = 'pages' AND sys_file_reference.deleted=0

No enableFields.
No sorting.
This will by default tend to return the rows ordered as they were created.

So the old, hidden one first (since it is not deleted)

Reordering in the backend changes the sorting_foreign field in 
sys_file_references but that is not checked for at all.

If the query was

	uid_foreign = 15 AND fieldname = 'media'
	AND tablenames = 'pages' AND sys_file_reference.deleted=0
	and sys_file_reference.hidden=0
	order by sys_file_reference.sorting_foreign

it would change two things:

* Hidden references in pages.media wouldn't kill the frontend

* If you have several visible references and use listNum=0,
   it would actually deliver the top one.

I believe this could be considered a bug, for the reason:

* Unexpected behaviour of implicit logic in Typoscript - listnum=0 
should deliver top reference

* Enablefields not honored in what is effectively a core function

* A legal backend operation (hide reference) plus reference.compliant TS 
leads to a total failure of the page - at worst if a hidden reference is 
sorted to the top it should deliver empty image/warning, but really, 
HIDDEN should be skipped in the frontend - isn't that the point?

----

I think part of the solution would be to read more from TCA.

   TYPO3\CMS\Core\Utility
   \ExtensionManagementUtility::getFileFieldTCAConfig

already does seem to set
	'foreign_sortby' => 'sorting_foreign'

by default but that isn't being read in

   TYPO3\CMS\Core\Utility\RootlineUtility::enrichWithRelationFields

in fact there seems to be no provision for sorting or hidden or TYPO3 
enableFields there whatsoever.

In 4.x this worked because media was simply a comma-separated list and 
listnum would work on that - there as also no hidden status, either a 
file was listed in pages.media or it wasn't.

Using sorting_foreign here would at least enable the workaround of 
escaping the exception by sorting the hidden references to the bottom.

If it is necessary that the rootline record contains hidden file 
references then at least 
\TYPO3\CMS\Core\Resource\ResourceFactory::getFileReferenceObject should 
fail gracefully if it can't resolve due to a hidden UID.

   \TYPO3\CMS\Core\Resource\ResourceFactory::getFileReferenceObject

does

... if (is_object($GLOBALS['TSFE'])) {
$fileReferenceData = 
$GLOBALS['TSFE']->sys_page->checkRecord('sys_file_reference', $uid);
}

and

   TYPO3\CMS\Frontend\Page\PageRepository::checkRecord

does

   $res = $GLOBALS['TYPO3_DB']->exec_SELECTquery('*', $table, 'uid = ' . 
$uid . $this->enableFields($table));

This means:

enableFields are *ignored* when putting sys_file_reference entries into 
the rootline.

Since sorting is ignored also, any hidden references always win first 
place if they were the first ones created.

However when the reference stored in the rootline is resolved, 
enableFIelds are *not* ignored.

Therefore the exception.

If enableFields were honored while adding to the rootline it would go away.

Greetings to all,

Christian


-----

My attempt to trace:

TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::getImgResource

does


if ($fileArray['import.']) {
$importedFile = trim($this->stdWrap('', $fileArray['import.']));
if (!empty($importedFile)) {
$file = $importedFile;
}

At this point it gets the ID of the hidden sys_file_reference converted 
from the import. conf

This is done in stdWrap

stdWrap_data delivers BOTH the IDs from  sys_file_reference - the hidden 
one first

stdWrap_listNum then returns only the first

stdWrap_data calls getData which does

switch ($type) {
...
case 'levelfield':
$keyP = \TYPO3\CMS\Core\Utility\GeneralUtility::trimExplode(',', $key);
$nkey = $this->getKey($keyP[0], $GLOBALS['TSFE']->tmpl->rootLine);
$retVal = $this->rootLineValue($nkey, $keyP[1], strtolower($keyP[2]) == 
'slide');
break;

So I need to look how it gets int $GLOBALS['TSFE']->tmpl->rootLine

which I find in TYPO3\CMS\Core\Utility\RootlineUtility
function enrichWithRelationFields