[TYPO3] Search indexing with crawler not finding dynamic pages

[Anna Nees]

We are currently trying to implement the search functionality on our Typo3 website. 
 
The default is working fine (when a page gets indexed it can then be found via the search).
 
However we are trying to set up the crawler extension to do this automatically, but it doesn't seem to work.  We have followed the extension manual but still no luck.  Basically when we press "Crawl URLs" it appears to index a few and the stop - the status of all the others remaining as "..".
 
Our second problem is that have have a lot of pages which have php include files in them, which pull in dynamic data on the fly as needed.  It's really important that these are indexed, however Typo3 is not currently finding them (even if we visit a specific page which such content - to ensure it's indexed - and then search for it).
 
Any help with either of these issues would be greatly appreciated!
 
Anna