NeoDoc - blog - docs - tests - todo
@neodoc.style

NOTE: This was original posted on the atmosphere.community discourse server.

Notes on a Plain Text Field

Definitely a Good Idea

[quote="adamfortuna, post:9, topic:1042"] I think we should make plaintext field mandatory, and have content included in multiple formats. Maybe a plaintext version then an array of contentType versions provided by the platform? [/quote]

First off, couldn't agree more with the first part. A plain text field feels absolutely necessary. We should strive to make content in the Atmosphere as portable as possible. Plain text distribution can go a long way in that direction. The specific contents of the field are worthy of discussion, though.

From the surrounding context, I read adamfortuna's perspective to mean no explicit formatting for the plain text (i.e. it's not markdown, just un-formatted strings of text with blank lines representing paragraph breaks). I believe that's the wrong direction. Defining a spec with effectively no formatting severally limits the practical portability of the content.

For example, say we have two apps:

The developers of App Bravo have a few options when it comes to content from App Alfa. They can:

  1. Create a transformation process that converts text/x-app-alfa-book-review content into their desired output.

  2. Ingest and distribute the plain text version of the App Alfa content with a generic converter that turns empty lines into paragraph breaks.

  3. Not include the content from App Alfa.

Option 1 takes work, obviously. But, if App Bravo wants the content from App Alfa, it's really the only option.

If App Bravo tried the second option to just output the plain text content then the reviews would either be incomplete (e.g. if App Alfa removed the strikethrough text when making the plain text copy), or they would be a mess (e.g. if the strikethrough text gets included, there would be no way to identify what should be struck and what shouldn't). Either way, the view of the content that App Bravo provides diverges from the original every time strikethrough text is used.

So, if App Bravo wants the content from App Alfa in a presentable style, they have to do the work to ingest the text/x-app-alfa-book-review format. Doable, but it could end up being a lot of work depending on the complexity of the format. The work would also have to be done for every other app that App Bravo wants to add to their reader.

If the plain text content App Alfa distributes is in a portable format, App Bravo can use pre-existing packages to handle the transformations with minimal workload. Even better. They only have to do that work once to cover ingesting content from every other app that also uses the format.

That leads us to Markdown.

Markdown

It's entirely possible I misunderstood the context and adamfortuna was talking about storing Markdown content in the mandatory plain text field. Things get a little better if that's the case.

With Markdown, App Alfa can still store their data in their custom text/x-app-alfa-book-review format. But, the plain text Markdown version they ship has known formatting rules to apply to it. All App Bravo needs to do to render the content is put it through one of the many available Markdown processors and render the results.

Mark down has a native syntax built in for strikethrough. So, App Bravo can fully reproduce the content from App Alfa in their review reader. That's an improvement, but it leads back to a new version of the original problem: Instead of limiting content to the least common features of plain text, content is now limited to the least common features of Markdown. Anything beyond basic blog post formatting is out of scope.

For example, say App Alfa provides a spoiler feature that let's folks hide content behind a spoiler cover with a label on it. There's no native way to represent that in Markdown. The feature needs a custom content type defined which is not something Markdown can do.

[aside: Embedding HTML in Markdown is possible, but that locks content into the specific tags used for the embed. I consider that a non-starter for portable content.]

App Bravo can always implement the text/x-app-alfa-book-review engine. But, we're back to the same issue of effort. If the formats get complicated and/or the review reader wants to ingest content from lots of other apps, the workload becomes a limiting factor (and the effort would be better spent building social features rather than ETLs). Dealing with custom formats also add significant complexity to the system since both sides must stay in sync if there are any changes.

We also haven't mentioned the increased burden on App Alfa. They must expend effort to generate and maintain the text/x-app-alfa-book-review spec and outputs. If apps aren't in place to use the format, the appeal of doing that work is limited. Which leads to the a chicken/egg/cold-start problem because building an App Bravo reader app based on custom formats which might exist someday isn't super appealing either.

I expect the practical reality of all this is that if the plaintext field is Markdown, then content from any app that does more than what Markdown can represent will effectively be locked into the original app. Trying to use it elsewhere will either be too much work, to broken, or both.

Given the creativity of the Atmosphere, that'll happen a lot.

I believe instead of expanding the number of content formats (i.e. each app being responsible for defining their own format/content-type), we should focus down to standardize on a smaller number of formats that are better at what they do. Markdown will certainly play a role. And, it's fine, as long as all you want to do is basic paragraph content. But, having to truncate all the content we want to be portable down to Markdown's limited feature set would be a huge miss.

That leads us to NeoDoc.

NeoDoc

NeoDoc is a content format born out of frustrations with Markdown's limitations. (And, full disclosure, it was me. I was the one who got so frustrated with Markdown's limitations that I designed the format to use on my various sites.)

Here's a quick example of how to use it to make spoiler content for a book review:

-- title

Dungeon Crawler Carl

A bunch of words about the book
go here.

-- spoiler/
-- label: Only read this if 
you're done with chapter 27.

If the doomsday device isn't a 
Chekhov's Gun, I don't know
what is.

-- /spoiler

And back to more writing
about the book that's not
under the spoiler.

NeoDoc produces a standardized AST designed for use in a templating system (which is also standardized). NeoDoc provides a default set of templates with the expectation that apps will customize them to their liking. App Alfa (where the content originated) would have title and spoiler templates responsible for rendering the -- title and -- spoiler sections of the content. The spoiler being set up to cover the text with an opaque box and the label.

App Bravo can make it's own versions of those templates. Or, it can use the original ones from App Alfa. (The idea being that App Alfa stores its templates in a lexicon that App Bravo has access to. More details on that to come in future discussions.) Or, it can fall all the way back to the defaults built into NeoDoc.

This gives each app complete control over how it display content without having to interact with each other through custom formats. Other apps can use the content just as easily and just as independently. All without limiting features beyond basic paragraphs.

The net result is a portable content format that doesn't sacrifice creativity to the limitations of Markdown's format.

Outro

There's a little more to NeoDoc than you see here, but not much. It's designed to keep the mental model small so you can focus on making content. You can check out the NeoDoc site for more info. Or, ping me or the neodoc accounts on bluesky if you've got question or feedback beyond the scope of this thread.

-a

Postscript