-- title
NeoDoc.Style TODO List
-- comment
Head's Up:
This post is written in NeoDoc.
At press time, the NeoDoc parser
hasn't been wired into this site yet.
So, you're seeing it without its HTML
web site formatting. That's okay, though.
It's designed to be read as plain-text.
It'll also give you an idea of what
the format looks like under the hood.
[Something cool is that the default
output hides ``-- comment`` sections like
this one. So, this message will disappear
as soon as the HTML formatting is applied]
-- h2
Brain Dump
I've just started work on making the
lexicon. There's a ton of stuff I don't
know yet. This is the pile of things
I'm investigating.
-- checklist/
-- h3: For Consideration
-- prelude: These are very much
work-in-progress/scratch notes.
Contradictions and unclear thinking
are to be expected. The order doesn't
involve any type of grouping or
organization. I'm simply throwing
stuff at the top of the list as they
come do mind.
[] Add `highlight()` filter that
includes `wrapper=false` if you
don't want the `pre` tag wrapper.
[] Consider making ``comment``
it's own top level and setting
the expectation that anything
inside should not be show in
out but make it clear that it's
still available? I worry that'll
catch some folks off guard, but
comments need to exist and
will at the secondary level
so maybe it makes sense to
move them to the top?
That would let you do
``-- custom-type comment``.
But, the more I think about that
it just adds complexity. Probably
best to stick with comment just
being a raw parse with
a template that has no output.
[] Use ``data.`` for the data being
passed into the template.
Use ``metadata`` for the stuff
that comes from the ``neo-metadata``
section.
Use ``site.`` for stuff that comes
from the overall site.
[] Add a default ``comma`` flag
to the footnote so you can
put commas behind them if
you have more than one.
[] Use ``-- neo-vars`` to store
key values that go int ``::VAR::``
shorthands.
The section should process as
JSON to allow variables to be
defined that way.
It should also allow them to be
stored as basic section attrs.
(e.g. ``-- VAR: some value``)
Shorthand variables in the content
are replaced with the content.
Additional flags/attrs passed
into the var are ignored
(i.e. ``::VAR::`` and ``::VAR|whatever::``
produce the same thing).
[] Make the inline templates on
the page ``-- neo-template``
with an ``-- id`` that matches
the section name that it
should apply to
[] Create ``-- audio`` and
``-- video``, and ``-- loop``
that work the same way as image
where they pull files by name
only.
[] Use json5 for the json sections.
[] Use ``-- template: neo-page``
for a default render with a full
HTML headers, etc...
[] Create ``[[ ]]`` sections in the
parser that are for instruction blocks.
Pulled is a pre formatted text,
the idea is that an engine will
process anything in them directly
in NeoJinja without introducing
other blocks into a section
for output.
The default is two brackets for the
opening and closing. More brackets
can be used to make multiple passes
during processing (e.g. ``[[[ ]]]``
would be processed on the second
template pass). This lets
sites load data from existing pages
during one pass that can be
included in across the site
in future passes.
[] Use ``html-ATTR`` namespace in section
and span attributes. Anything with
the leading ``html-`` will be added into
the output of the corresponding element
as an attribute. (Some things like
``class`` and ``id`` without the
``html-`` prefix are already added
by default. But other things like
``template`` aren't. The ``html-``
prefix allows for adding anything.
It works with both flag and attribute
(e.g. ``-- html-checked`` will add
a ``checked`` boolean flag/attr to
an element.
[] Use ``-- neo-`` as a prefix for
neo related sections (e.g.
``-- neo-plugin`` and
``-- neo-template``).
[] Set up a follow up processors for
if content fails to parse that
goes as far as it can and returns
that payload then appends what it
can't do after that.
[] Use either JSON5 or JSONC or whatever
that makes JSON less fiddly.
[] Set up a `section.` object in each section
that has access to its flags and attrs.
[] Define the templates for spoilers
and content warnings as details elements
with the summary being the label.
[] Document template storage as a lexicons
so that any app can get the templates from
other apps they use for rendering.
[] Add predefined section for ``metatag``
that's attributes that if a full page
is rendered from it can be used
to generate arbitrary meta tags
in the head of the document.
[] The default output from NeoDoc is
a full HTML page. Show how to override
that with a template so the content
can be used as just a section and
without script and style tags as an
example.
[] Create a pre-defined ``-- refs`` section
that holds attributes that can be accessed
by doing ``:attr-name`` as either a flag
or an attribute value in other content.
TBD on if refs should be the only thing
that goes through on an attribute. That is
can the ref be inside a longer string.
Feels like the answer should be no.
If there's a ref, it's 100% of either
the flag of the value of the attr.
[] Maybe the ``-- status: `` attr in metadata
should be ``published`` and then everything
else which is left up to the app. That's
probably better than ``done``. It's more
explicit and ties in with the ``-- published: ``
attr.
Another options would be to simply
reply on the ``-- published: `` attr
as the indicator. That probably makes
even more sense. Stuff can still be sent up
but it's considered some type of draft
until ``-- published: DATETIME`` is set.
Apps could choose to implement custom
``-- status`` attrs. Or, they could
rely on the native ``-- template:``
attr to make other determinations.
Though, that would still likely lead
to having to change two things in order
to move from draft to published.
Unless, of course, the app just treated
everything that's not published as
a draft.
[] Define how text in raw sections should
be handled when it comes to leading whitespace.
For example, when a pre section is nested
it would be nice if the content could be
nested as well. This would mean looking
looking at all the lines to figure out which
ones is the least padded over and removing
that much whitespace from the entire.
There would need to be a way to turn that
feature off, but it feels like having
the space trimmed should be the default.
Maybe the way to turn it off is a
``no-trim`` flag.
[] Include ability to have alternate
version of the content in different
languages in the lexicon.
There should always be a default
one that has an identifier
of what the language is and
then slots for alternate ones.
[] Add spoiler and content warnings
sections and spans.
[] Trim leading and trailing whitespace
in spans.
[] Provide an escape character for
whitespace to add it to spans
(this allows adding space at
the front and end of a span
as well as adding non-collapsing
whitespace in general)
[] What if the NeoJinja templates
were included directly in the
content file instead of separately
in the lexicon. That probably makes
a ton of sense as the keeps everything
in one place which provides for
total portability (minus external
calls to images, etc...) via just the
text file itself. And, dealing
with external calls is the
responsibility of the App.
[] The generator feature of NeoDoc
needs to include a file object
and a site object for things the
individual output needs to reference
(e.g. images or other data).
Thinking about that, it should
also have the capability of
transforming multiple files in one
shot so that it doesn't have to
spin up the environment each
time from multiple files in the
same site.
That's leading me to the idea
of accepting a few different types
of input, but always providing
the same output, but that output
is at the site level. If an
stand-alone, individual piece
of NeoDoc content get transformed
it'll just come back as the
first item in the output slot.
Accepting an input payload for sites
has always been in play. The new
idea here is that the output
will always have the shape of a site
even if there's only one thing in
it.
[] Spec out a ``-- table`` section
that's a subType of CSV. It should
include something like
``-- headerRows: #`` to define
the number of rows to use as headers.
It shouldn't try to do much more
than that. Styling can be done
by adding a standard ``-- class``
but anything more advanced should
jump out to using a JSON section
with a render (e.g.
if different cells should have
independent classes on them).
[] Figure out an approach to using
plugins. Possibly: https://fair.pm/
Though, I'm not sure plugins
should be in scope. If the only use
case for NeoDoc was feeds, it
feels less likely, but since the
goal is generally to have websites
built off it too, it lends to
the idea that plugins should
be included.
[] Thinking more about the use
case. I do want to make sure
that full websites can be
built from a collection of
NeoDoc files. You shouldn't
have to keep somethings in NeoDoc
and something in another location,
or format, or whatever to
build a site off it.
Also, the output generation will
have to be able to reference other
content in order for image calls
to work with only the name
which is a priority.
[] Change section keys in the
AST so it's ``type`` and ``subType``
to help clarify instead of ``kind``
and ``type`` which is confusing
about which one is the parent.
Or, you could do it as an array?
``"type": ["p", "title"]`` but that
feels weird. Also, it's less explicit
so don't do that.
[] Make a Markdown to NeoDoc
conversion tool.
[] Figure out how to do something
so you can have references to links
that are stored elsewhere in the file
so the URLs can be split apart
from the content. Maybe a ``-- links``
section that uses attrs to store
they keys and URLs that get matched
in the file for links.
But, maybe not ``-- links`` since
that seems like a section that might
get some usage. There's already
``-- ref`` so maybe ``-- refs``.
Or, just ``-- urls``.
This would require multiple step
processing, but that's fine. It's
no different than the same requirement
for markdown. The key is that
everything has to be on the page.
[] Maybe that opens templates back up?
If there is gonna be some processing
why not let it process across all
the data in the file? Difference
from prior thoughts is that it
would only be what's in the file
itself and not across the
larger set of files. Though....
If you're gonna do the processing
maybe it is worth opening that up?
I think this is gonna come down
to needing a prototype to make
a decision.
[] If you hit an empty line that closes a
block while a span is still open, then
close it automatically.
[] Send a signal up that warns if
any errors occurred that were recoverable
and didn't cause a complete break of the
parser. (Goal is to render as much as
possible)
[] Set up so that multiple start/ending sections
can be smushed together without
requiring an empty line between them.
For example, instead of this:
-- code/
-- neodoc
-- div/
-- div/
Content
-- /div
-- div/
More
-- /div
-- /div
-- /code
You could do this:
-- code/
-- neodoc
-- div/
-- div/
Content
-- /div
-- div/
More
-- /div
-- /div
-- /code
[] Store all the templates for each
app, author, and content piece
in a single record with key
values pairs for the name so they
can be fetched with a single call.
[] Probably don't include the ability
to add template instructions in the
first version. That adds a ton
of complexity and requirement for
the parsing and processing.
By dropping it, there's no need
to do the reprocessing and loading
external data, etc.
The goal with adding that is to make
it easier to build full web sites
off the storage. That should
be a secondary goal/project.
(Templates will still be applied
to the neodoc files. They just
won't be able to have neojinja
instructions in them directly)
[] If there are errors on a page it
should do everything possible to render
as much as possible including
a solid error message.
[] Errors across all files are
be collected and linked to when
a full site is built
[] Errors are cascaded back. For example,
if a function that doesn't exist
is called in a for loop the first
thing that's reported is the function.
An internal version of the page
is generated up to the point where
the error occurred and another attempt
is made. Since the end of the for
loop will have been removed by
truncating the file it will case
a new error. This is the second
one reported. And so on until
some portion of the page
renders and the error stack
can be displayed.
TBD on the default output for
errors. Probably a ``details``
section that's collapsed
with a list of the errors inside
with the content of each one
inside another ``details``
section so it's not as
overwhelming when you first
open it.
[] Standard sections to consider
beyond the ones I already use:
``-- postscript``.
[] Talk with other folks across
disciplines to see what would make
sense for standard/default templates.
[] Define the template methodology
for doing the thing like markdown
where you identify a link by an
ID and then use a reference later
on the page to get the actual
link to use.
[] Thinking about the ``-- status:``
key for the ``-- metadata`` section:
maybe the guidance is to split between
``done`` and everything else. It
amounts to a binary indicator where
``done`` is ``true`` and everything else
equates to ``false``. So, apps get a
known indication of when things are
ready to include in feeds/etc. while
allowing folks to pick their own
status for app specific behavior.
[] Default set of section attributes
includes ``-- prelude`` which places
text inside the section but before
lists/checklist/etc. start. Allows
for adding content before the lists
that's inside the section tag without
being part of the list itself.
There's also one for text that
comes after the main portion of the
section. I'm not sure what the name of
that is yet. ``-- postscript`` probably,
but TBD.
[] Default output templates have
``<section class="neo-SECTION">``
wrappers so CSS can be applied without
having to mess with templating.
[] Whitespace gets collapsed
to a single space (e.g. if there's a
new line followed by more whitespace
before text starts again it gets
collapsed).
[] Define all the shorthand characters
and their mappings then update
the SINGLE_CHARACTER const in ``lib.rs``.
[] Standardise on camelCase
(slightly easier to type than
hitting underscore or dash and
goes with the JSON theme.
[] Going back and forth on this,
but what about a duplicate set
of the core lexicon under
an ``encrypted`` sub section
that has all the same things
but stores the data as binary
blobs with the expectation
that apps will handle the
security/encryption.
[] How does moderation work
across the network.
[] Include moderation into
the lexicon (not sure how
this will work yet, but it's
a primary focus).
[] Set up a way to define standard
template expectations (e.g. book-review
could require title and have an
optional author, etc.).
The definitions would need to be
name spaced.
[] Include ``hr`` as a section in
the default templates.
[] Include ``br`` as an inline
tag in default templates.
[] Provide the default set of
required templates that cover
all the possible fallbacks
if a post and the parent
creator don't have specific
templates that cover
the section or span.
[] Include something in the
lexicon for notification for
likes and follows and comments.
[] provide for image storage.
Images should be able to be referenced
from other content.
[] PROBABLY NOT: provide for drafts which
should be stored encrypted
with a password. (The more I think
about it there shouldn't be an
explicit handling of drafts. Apps
can do that as they see fit and
either store the data in the Atmosphere
or keep it locally based on
their requirements)
[] Determine if there's a standard
way to encrypt where the same
password can be used by different
apps with relative safely.
[] Determine if there's a way that
you can encrypt a piece of content
so that it can be decoded with
multiple different private keys.
(That's for way down the path
and encryption might be out of scope based
on security considerations)
Like, would it be possible to set
up something that artists could
use to distribute their works
for money. Not sure if this
is really in scope, but worth
considering.
[] Look at this proposal
for metadata - https://tangled.org/chrisshank.com/at-tags/
[] PROBABLY NOT: Maybe do do a ``status`` but with
only two values "draft" and "done"?
or is it better to do that with tags?
status is more explicit, but again,
probably not since there's no way
to control what apps will do with
it (or if they ignore it)
[] Define the requirements for the
``-- metadata`` section. (All
of which will be optional, but
providing known locations for
specific values).
[] Add a ``flags`` key as
a known thing for processing. It's
a way to pass data to apps.
The specific handling of any given
flag value is up to the app
itself.
Flags can also be set up as
key/value pairs by separating
the key and the value with a pipe
character.
[] Custom/Arbitrary key/value pairs
can be used in the ``-- metadata``
block beyond the ones specifically
defined. Handling them is
the responsibility of each app and
there's no guarantee that apps
will have the same behavior.
[] The rust parser is available for
use of parsing. It's the reference that
other parsers can be built off of.
[] A portion of the lexicon will hold
javascript. it's up to the app to
decided if it's allowed or not.
JavaScript wouldn't be included if
neodoc.style was only ever going to
be used for feeds. Since a goal
is to be able to run entire web sites
off it it effectively has to be included.
The goal of splitting it out in the
lexicon is to hopefully make switching
it on and off easier (e.g. to turn
it off, nuke an script tags and
don't output anything from the
script section)
[] The lexicon provides these for the
documents:
-- list/
- The content in Neopolitan format
- Metadata about the file
- Optional CSS styles to apply to the
document. These styles are
always added after an apps
default styles and the
publication specific styles
(i.e. the ones that are applied
to all posts from a given account)
- Optinal templates to supplement
or override the default
templates.
-- /list
[] iFrames can be used to include other content.
This is up to the display app to
determine if they're allowed.
[] There's a default set of neopolitan sections
and templates defined as part of the lexicon.
[] The ``-- metadata`` section is special. It's never
rendered directly on the page. It's content is used
for preview text in open graph style handling.
TODO: Update the rest of these notes to
convert from the blurb to the metadata.
It's not rendered by default. It's
only used to preview the content
when it's linked to from another page.
The blurb section is processed as a
raw section. The output is converted
to a single HTML escaped string (so,
text spacing and new lines don't
matter)
The metadata can be multiple paragraphs.
TODO: Set a size limit expectation for
the blurb. (This won't be enforced
in the content itself, but the
blurb is included in the lexicon
and when it's pulled from the
content it's truncated if necessary.
It's only used to store info
about the page. Apps aren't expected to use
this data. It's more for the tools that publish
the AT Proto entries to generate a valid
entry (e.g. ``created`` from the ``-- metadata``
section gets used as the created datetime in
the JSON object of the entry.
[] The parser always return a specific AST
for a given input, but the AST is not stored.
Only the original text is. It's the
responsibility of the app to parse the document.
[] Optional templates are in the MiniJinja
format with the `{{ }}` replaced with
`[x@ @x]` (TODO: Add the other two token
replacements here).
[] There are two sized thumbnails
for preview content: large and small,
The large is the current standard:
1200x600. The small one is a 1:1 ratio
with a recommended min size of 600px.
[] Each publication has a root URL
for images.
[] Image calls in posts can
either be full link (e.g. https://...
or relative to a doc root (in
which case the URL from the
publication is used for the base).
Relative paths aren't permitted.
(again, someone can put them in
but if they do it's up to the
parser to ignore them or figure
out what to do with them, but
they should be considered out
of spec)
[] There are no comment type
sections in the prose. The
expectation is that everything
in the document is visible
[] There are two image formats
for open graph style images.
This first is the same as
the current de facto standard
(which I think is 1200x600)
The second is a square size.
TODO: figure out the expected
display size. Something smaller
than the full unfurl of
the larger images. The idea
being the square image is
for going with the text on
smaller cards.
[] The publication level
has avatar images. There
are two one square and one
circular. They are both 1:1 ratio
images with the expectation
that the circular one will
be cropped into a circle.
[] Template fallback is:
-- list/
- Start with the specific section
name template in the post
itself.
- Fall back to the specifi section
name from the publication.
- Fall back to the specific
section name from the app.
- Fall back to the generic
section type from the post
- Fall back to the generic
section type from the publication
- Fall back to the generic
section type from the app
-- /list
[] Define how ``tags`` should be processed
in the metadata. Specifically, they
should be split on commas. If a
comma needs to be included in a tag
it can be escaped with a backslash.
If a backslash needs to be escaped
another backslash can be used. Those
are the only two escape characters.
[] All neodocs are UTF-8.
[] Define standard metadata keys:
-- list/
- title: STRING
- created: RFC3339
- completed: RFC3339
- updated: RFC3339
- PROBABLY NOT: status: ENUM (TBD on strings, but case insentive)
-- /list
[] Should there be a status? Maybe not.
The idea being that if it's out there, it's
out there and there's no way to control
apps to do things like only show "done"
and not "draft" posts. Also, trying
to spec the possible keys would be
an exercise in futility.
[] Define the cascade for looking
for a title. Probably start with the
first block from `-- title` if it
exists. Otherwise, use the ``title``
key from ``-- metadata``.
Actually, maybe there isn't a title
field in the metadata. If you want
a title, add a title section. That
way the methodology is always the
same.
[] Note that titles are not required.
[] Set up the parser so the first
section is automatically a paragraph
without requiring ``-- p`` to
start the string. The idea being you can
just type a paragraph by itself
and it's valid.
[] Everything that can be optional
should be. Specifically, you should
be able to write a single string
of text with no sections at all
and it be valid. That probably means
auto generating some things
in the lexicon metadata.
[] Try to require all fields in the
JSON and provide defaults (e.g. ``null``)
if their values don't exist. Goal
being to not have to check for
undefined for the shape of a JSON.
[] Should there be versioning in the
format? Could be done and done in
a way where users don't see it
if they don't need it. But, then
that could be surprising, so
probably not? And, adding versioning
would add complexity.
[] Add a canonical URL for linking
out somewhere.
[] Convert to using ssb for the time
being. Eventually the site will be
built with Neopoligen.
[] Define the default section types
that go into each section category.
[] Accept a serialized JSON string
as a config.
[] Accept a path to a config file.
[] Provide for a default config location.
[] Don't have any expectation for the
sections (e.g. ``page`` or ``metadata``
sections are not required. Everything
just loads in a standard manner. It's
up to the external processes to
determine what to do with the content)
[] Provide for inline definition of
section categories. (e.g. ``-- custom-section raw``)
[] Define the possible shortcodes and
what they represent. The specifics are
backed into the AST to ensure consistency.
[] Create a way to add plugins so other things
can extend it when the templates are being
processed. There is no expectation that
any given processor will have a certain
plugin.
[] All the data (e.g. JSON) and data from
the `-- metadata` section is available
for use in the templates on the page.
[] TBD on including other files. Probably
want to make that happen (e.g. if
you're going to build a website off it
you should be able to have all the files
without having to store anything on your
server.
That also means it's probably necessary
to do some of the ``filesInFolder`` type
stuff to do things like make lists of
files? That may be trying to push things
too far. Maybe includes are fine, but
anything else should be a plugin. That
would allow for different tools to
use different caching strategies, etc.
Lots to dig into there, but probably they should
only be able to be included off the
same PDS in the neodoc lexicon.
[] MiniJinja is the default template engine
that comes installed with the binary.
There's a set of tests that can be
used to validate the inputs and outputs
if other processes are built.
[] There are not actual files paths. Everything
is stored at the root. Files an have a
name attribute that can look like a path.
The expectation is that if there are tools
like ``filesInFolder()`` all the folder
level stuff is based of splitting on the `/`
character.
[] File paths can have white space between
the separators (e.g. ``some / file / path``
is the same as ``some/file/path``).
[] The ``formatDate(DATE, FORMAT)`` takes
an RFC3339 date and a format to transforms
it into the requested output.
[] TODO: Look through other functions that
should be built in beyond the defaults
provided by minijinja.
-- /checklist
-- list
-- h2: Research Links
- >>https://mu.social/>> via:
>>https://bsky.app/profile/did:plc:fivmz34azxgjafrk6ogns7k5>>
- >>https://semble.so/>> via:
>>https://bsky.app/profile/did:plc:k7wclckeajmuibxbamtbejjg>>
- >>https://aturi.to/>> via:
>>https://bsky.app/profile/did:plc:qntsxa2i4sb24noi45fx4np2>>
>>https://bsky.app/profile/did:plc:6teuhlkizzebk6wdp42633el>>
- >>https://leaflet.pub/>>
via: >>https://bsky.app/profile/did:plc:btxrwcaeyodrap5mnjw2fvmz>>
- >>https://frontpage.fyi/>>
via: >>https://bsky.app/profile/did:plc:klmr76mpewpv7rtm3xgpzd7x>>
- >>https://pckt.blog/>>
via: >>https://bsky.app/profile/did:plc:revjuqmkvrw6fnkxppqtszpv>>
- >>https://standard.site/>>
- >>https://offprint.app/>>