[TYPO3-doc] Experiences with ReSTructured Text

[François Suter]

Hi Sebastian,

Thanks a lot for your detailed feedback. Sorry for the slight delay of 
my answer, but I'm pretty busy these days and also had to thoroughly 
read and think about the whole stuff.

(Thanks also to all the others who already contributed to this debate. 
This post includes answers to points raised in other parts of this thread.)

First of all it's great that you made a positive experience. The fact 
that the Python community is managing all its documentation with ReST is 
certainly proof enough that the format is a worthy contender in our bid 
to move away from OpenOffice.

BTW there were remarks that OpenOffice was definitely easy to use for 
editors, but we *are* moving away from this format anyway, unless we 
face a full-fledged riot. As was mentioned by Sebastian the fact that 
OpenOffice is a binary format is huge disadvantage for collaborating. 
Furthermore I don't quite agree with the ease of use: try editing a long 
document (like the TSref) and see all your styles being arbitrarily 
change and you won't be friend with OpenOffice for long. Additionally 
the fact that it's easy to style documents in OpenOffice also means that 
it's easy to mess them up...

> * Although it is plain text, it is practically unreadable with a normal
> text editor, as it is very verbose and contains many many tags.

I must admit that this has been our main worry so far while trying out 
and performing sample migrations to DocBook. We had come to the 
conclusion that it would probably be okay for official doc maintainers, 
as we can expect them to make the learning effort and get used to it, 
but it was a worry for extension developers: could we really force them 
to use DocBook too? This was an unanswered question so far.

I would say that this is ReST's best selling point: it is clearly easy 
to write. It may be less so that a word processor, but it's very close 
to wiki syntax and I would say we can clearly expect extension 
developers to be able to write in a wiki-like format.

> * While there are numerous solutions to cross-linking between documents
> (OLinks etc), they make the build process considerably more complex.

This is also an area which was still a bit fuzzy, as it seems to be very 
complicated indeed. I must say that this is an extremely important item 
in the migration agenda for two reasons:

- we want to render manuals in HTML with unchanging URLs (as much as 
possible, i.e. as long as the document structure is not strongly 
modified). Currently we have terrible URLs, full of numbers, which 
change upon each rendering. This makes it impossible to have "permanent" 
URLs and many manuals, not to mention mailing list archives, are full of 
broken links
- cross-linking should be as easy as possible for manual writers. After 
all, the web is all about cross-linking. Heavily and properly 
cross-linked manuals are a true added value.

> However, the main problem in my opinion was the following: While we
> thought a lot about how we can adjust DocBook for the needs of the TYPO3
> project, we somehow forgot that documentation writing must be *fun* and
> *easy* for contributors, as that's the only chance they will do it IMHO.
> I mean, we are mostly programmers, and that's why to most of us (myself
> included), documentation writing is a *necessary evil*.

I'll take your word on that. I'm not sure how fun documentation-writing 
can really be made. For me it's a real conviction: a piece of software 
is not complete without documentation. Others don't feel this way. 
However it helps - of course - if the entry barrier to writing a manual 
is as low as possible.

> Also, the Sphinx / ReST converter gives much feedback to the user if the
> user used some unsupported syntax etc.

That's good. That's actually one of the big advantages we would have 
with DocBook: strict validation rules.

> In my opinion, the main benefit of ReST vs other formats (Markdown,
> Textile, ...) is that they have thought a lot about extensibility. There
> is a syntax available for creating custom code-blocks and custom inline
> markup, and with a little bit of python code, we can extend ReST.
>
> We could use this for example to create custom, semantic markup for
> *TypoScript* or *Fluid*.

This is very important. Indeed one very important manual that we have 
and need to migrate is the TSref (but also TSconfig and TCA reference). 
The current format - in tables in the OpenOffice document - makes it 
impossible to reuse that information in a different format, in 
particular to turn it into a true online reference, which would be 
hugely useful to everyone.

While researching on the migration to DocBook I came to a satisfying 
format for representing the TSref with DocBook [1]. With appropriate 
rendering that structure could lead to have an HTML page per TS 
property, all "keyed" logically by the use of id attributes respecting 
strict naming conventions.

If we should choose to use ReST we need to make sure that we can achieve 
something similar.

> Graphviz + UML support
> ----------------------
> [snip]
> That, I think, makes a lot easier to illustrate certain concepts with a
> well-written diagram; and we can also define a common look-and-feel for
> diagrams.

Could be nice although it's not a crucial point IMO.

> Cross-Linking
> =============
>
> Crosslinking between documents is easily possible by an extension called
> "intersphinx". When rendering a Sphinx document, an index of all
> cross-reference targets is automatically generated. When wanting to link
> to a certain document, one needs to *import* this document under a
> prefix, and then use this prefix in links. As an example, in order to
> link to the *Python* documentation, one needs to define the following
> mapping::
>
>      intersphinx_mapping = {'python': ('http://docs.python.org/3.2', None)}
>
> Now, when using ``python:comparisons`` as link, we are linking to the
> target ``comparisons`` in the Python manual.
>
> Thus, the only thing we need to implement linking between documents are
> stable and well-thought URLs to our (rendered) documents.

That seems to be clean and easy indeed. Is it actually possible to view 
the index of targets?

> Output Formats
> ==============
>
> Sphinx supports quite some output formats, among them HTML, PDF, EPub,
> ..... Here we'll only focus on HTML and PDF.
>
> The generation of the generated HTML pages can be easily adjusted, as it
> is using a simple template engine for the markup, quite similar to Fluid.

OK. That's typically the place where I would not expect much trouble ;-)

> For generating PDF, there are two possibilities: One with a direct
> conversion, the other one with LaTeX in between. I did not try the
> direct conversion yet. The conversion via LaTeX can be triggered with a
> simple command, and is giving reasonable and well-looking output.

Rendering DocBook to PDF seems to be quite convoluted. It could also be 
a big selling point for ReST if it were easier. Could you go in a bit 
more detail about how it compares?

>
> Translating ReST documents
> ==========================
>
> While translation of ReST documents had to be done manually, without
> tool support, for quite some time, this has changed last year thanks to
> a GSoC project which integrates Sphinx and ReST:
> http://gsoc.robertlehmann.de/making-ponies-fly/
>
> So, basically, we can use Pootle to translate our ReST documents.

I saw that and it's rather good news.

> I have not yet tried this toolchain, so I don't know yet how stable it
> is or how well it works.

Indeed this needs to be tested. The sure thing is that Pootle is not 
appropriate for translation of DocBook format. Every tag gets converted 
to an individual string, making it near impossible to translate stuff 
when there are too many nested tags. This is based on a test set up by 
Laurent Cherpit.

> Build chain
> ===========

How would you rate the complexity of setting up and tuning the build 
chain? One thing which kept me worried about DocBook is the use of XSLT, 
which I find really inconvenient to use.

Other issues
============

About the need for WYSIWYG I would mitigate it. I think it's ok to rely 
on plain text editing as long as there's an easy way to preview. This is 
true both for DocBook and ReST. Indeed some DocBook editors give an 
impression of WYSIWYG but it may actually be misleading (as was pointed 
at by Tom Schraitle quite a while ago). So whatever format we adopt 
eventually, we will need to provide an online service to easily preview 
the HTML format, complete with the skin that applies on typo3.org.

About the workflow the idea is to deliver only the documentation source 
(DocBook or ReST) with an extension. Rendering would happen on the 
server. This is the only way to ensure a coherent look and feel across 
all manuals. It also solves issues like having the right fonts installed 
for rendering (in case of the PDF format).

Cheers

-- 

Francois Suter
Cobweb Development Sarl - http://www.cobweb.ch

[1] 
https://svn.typo3.org/TYPO3v4/Documentation/official_template/trunk/DocBook/reference.xml