[TYPO3-doc] Experiences with ReSTructured Text

[Xavier Perseguers]

Hi Sebastian,

> as you might have heard, we (the FLOW3 / Phoenix team) have tried out
> ReSTructured text as a documentation format. In this (long) posting, I
> want to share my experiences with you.

Thanks.

> this is a writeup of my experiences with Sphinx and ReST, based on two
> use cases:
>
> [...]
>
> Writing documentation always was somehow "hard" inside the TYPO3
> Project. When we used *OpenOffice*, it was hard to produce well-looking
> and readable documentation; the documentation rendering on
> http://typo3.org is somehow hard to predict and is not very legible.

True!

> This was IMHO mainly due to our custom-baked conversion process from
> OpenOffice to DocBook to HTML, which was not well-maintained.

True.

> While it was easy for the average user to write OpenOffice
> documentation, it was difficult for collaboratively maintained documents
> like the TSRef. This was because the SXW format is a binary format, so
> we could not use our standard diff / patch / gerrit toolchain for
> reviewing documentation changes.

True.

> *DocBook* set out to create the standard in technical documentation
> publishing. They created an XML based format and a rendering toolchain
> based on XSL Transformations. We have worked with DocBook for quite some
> documentation, and I personally have set up the rendering toolchain
> quite a few times, both for clients and for my work. I also wrote my
> "Großer Beleg" at university using DocBook.
>
> Being an XML based format, there is a rich toolchain for processing
> these documents. There are editors available; we mostly used `XMLMind
> XMLEditor<http://www.xmlmind.com/xmleditor/>`_. For conversion to
> output formats, we usually used custom bash or PHP scripts, which ran
> the multi-step conversion.
>
> For my "Großer Beleg", I used `dblatex
> <http://dblatex.sourceforge.net>`_ which makes rendering DocBook ->
> LaTeX ->  PDF a lot easier and less error-prone.

Good to know.

> Still, there are numerous difficulties with DocBook:
>
> * Although it is plain text, it is practically unreadable with a normal
> text editor, as it is very verbose and contains many many tags.

Problem with XML, as always...

> * When using XMLEditor, a plain-text diff is unreadable, as XMLEditor
> automatically wraps long lines.

You have to use special graphical diff tool that can deal with XML I guess.

> * When one extends DocBook, one essentially has to "fork" it, again
> making it more difficult to use XMLEditor etc. The main
> * While there are numerous solutions to cross-linking between documents
> (OLinks etc), they make the build process considerably more complex.
>
> However, the main problem in my opinion was the following: While we
> thought a lot about how we can adjust DocBook for the needs of the TYPO3
> project, we somehow forgot that documentation writing must be *fun* and
> *easy* for contributors, as that's the only chance they will do it IMHO.
> I mean, we are mostly programmers, and that's why to most of us (myself
> included), documentation writing is a *necessary evil*.

I agree, but I take the opportunity to add something here! For many 
people, writing documentation is easier with a WYSIWYG editor. Moving to 
plain text editing and some "compilation" process must really something 
in the end. I say that because if you get a nice HTML output as well as 
really pretty PDF output out of the compilation process, then you have 
much chance to make people change their mind and like writing 
documentation in plain text format.

I say that because until now I still did not find anything that could 
output a document as pretty as when created with LaTeX. Of course, we 
don't want to write documentation in LaTeX but still, we should have a 
really nice PDF output if we want people to switch from their most-liked 
WYSIWYG editor.

Not to blame you, you know me! But the PDF you show us are what I 
consider a very basic output of some documentation you wrote. I really 
like the HTML output of the FLOW3 documentation. It is nice, functional, 
and can easily be enhanced with some colours if we change the CSS but 
the PDF are as ugly (sorry!) as what you get after playing a bit with 
PDFlib. Content is there, and basic features as well (bullet lists, 
references, ...) but it is so far from what you get automatically when 
you create exactly the same content without anything more than pure 
standard packages when you use LaTeX...

> After having expressed my thoughts about DocBook, I'll now go deeper
> into my experiences with ReST and Sphinx.
>
> Writing Experience
> ==================
>
> My first impression of using ReST was, that it was really *easy*. You
> don't need to remember a whole bunch of XML tags, keyboard shortcuts of
> your XML editor, etc. You only need to fire up a text editor, and write
> something. Then, with a simple command, you'll get a preview in your
> browser showing you the produced markup.
>
> There are quite some neat things in ReST: For example, you when using
> "such quotes", they are automatically converted into opening and closing
> quotes. When you write two or three dashes (-), you get an en- and
> em-dash like this: -- ---.

This is depending on the language you write in... did you try with 
French (using "Chevrons") and German (with "double-comma" at the 
beginning)? What about additional typographic features such as quotes 
inside quotes which behave differently? => single quotes 
(opening/closing) in English, "English" quotes in French, ...

> For bold markup, just wrap your text in *stars*. The only thing which
> needed some time to get accustomed to was the syntax for code, which is
> the use of ``two backticks``.

What about emphasized word within an emphasized paragraph? => should be 
(back to) regular in common typographic rules but the editor should not 
bother with that...

> In order to write code blocks, just end the last sentence with two
> double-colons, and indent the code::
>
>      $this->myCode();
>
> With the correct setup, you even get automatic syntax highlighting for
> code :-)

What about additional features such as highlighting lines? Or 
integrating links within a code block or making some part of the code 
"bold"?

Just a few ideas of what we can easily do nowadays with OOo...

> So, to sum it up, writing ReST is much much easier than writing DocBook.
> Rens, one of our team members, once said "I've never thought that
> writing documentation can actually be fun" - and that's also my feeling
> about it.

I can easily take that for true, writing plain text is easier than 
writing XML. But is it easier than writing a documentation with 
OpenOffice or Word (:D) ? Because once you know the pitfalls in those 
editors, you still can do real good documents and very easily (I'm 
definitively not a Word adept, nor an OpenOffice adept but I must say 
that this has some good features and relatively good UX anyway).

> Extensibility
> =============
>
> In my opinion, the main benefit of ReST vs other formats (Markdown,
> Textile, ...) is that they have thought a lot about extensibility. There
> is a syntax available for creating custom code-blocks and custom inline
> markup, and with a little bit of python code, we can extend ReST.

Very nice being able to extend it but if we start using it for 
documentation and replacing the OOo format, we have to make sure that:

- it's straightforward to install and start writing (download, install, 
use, period)
- output is already preconfigured to render as well as the current 
OpenOffice output (if not better)
- anybody may without taking hours, reproduce any common needs of 
current documentation (picture, code block, enumerated lists, links, 
references) - just take current TSref with all missing intra-links, it 
should be dead easy to do that

and once again, the output should really be pretty! I don't ask for the 
same polish as in LaTeX but PDFlib is a no-go for user acceptance.

> Output Formats
> ==============
>
> Sphinx supports quite some output formats, among them HTML, PDF, EPub,
> ..... Here we'll only focus on HTML and PDF.
>
> The generation of the generated HTML pages can be easily adjusted, as it
> is using a simple template engine for the markup, quite similar to Fluid.

As said, HTML is not a big deal.

> For generating PDF, there are two possibilities: One with a direct
> conversion, the other one with LaTeX in between. I did not try the
> direct conversion yet. The conversion via LaTeX can be triggered with a
> simple command, and is giving reasonable and well-looking output.

Did you try with real-world documents? I tried such conversion once and 
you have to make sure it's bullet-proof because you can easily come into 
blocking stuff in LaTeX if you have a syntax error somewhere.

And we cannot expect standard users to install LaTeX on their machines.

What is your idea for extension authors? Should they publish a ReST 
source and some server will create the PDF and HTML out of it or should 
they distribute, let's say, the PDF as well? Allowing them to output the 
PDF with any workflow they like best, if wanted?

> Translating ReST documents
> ==========================
>
> While translation of ReST documents had to be done manually, without
> tool support, for quite some time, this has changed last year thanks to
> a GSoC project which integrates Sphinx and ReST:
> http://gsoc.robertlehmann.de/making-ponies-fly/
>
> So, basically, we can use Pootle to translate our ReST documents.
>
> I have not yet tried this toolchain, so I don't know yet how stable it
> is or how well it works.

Looking forward to hearing other thoughts!

Thanks for this post.

-- 
Xavier Perseguers
Release Manager TYPO3 4.6

TYPO3 .... inspiring people to share!
Get involved: http://typo3.org