How to regenerate `docs/`?

There are "Update docs" commits (https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/commits/master/docs) but there doesn't seem to be a script to do this in one fell swoop for all crates, had to come up with my own contraption and added a mv (+ documented it) when doing 75ba9eaf.

Not to mention these inconsistencies:

• Only few crates have doc_target_path: the rest ends up in vendor.md;
  • If this is set however, the output path becomes relative to the config directory (so where Gir.toml is found) instead of docs/<crate name>/docs.md;
• Gir requires an output directory to be set with -o but ignores it, stuff ends up in the wrong place.

Have I missed something or were the previous commits fixed up by hand?

changed the description

Maybe a more interesting question: Why isn't the documentation generated in-line during a "normal" gir run? Why is it generated into separate (still checked in instead of generated on the fly from Gir xml files) markdown files that are later overlaid on top of the code? Is it an LGPL thing? Seems to me the documentation might be interesting to see/read while developing on these crates, especially with rust-analyzer and intra-doc-links to quickly navigate around.

Have I missed something or were the previous commits fixed up by hand?

I created/updated the docs stuff by running gir manually and copying files into place. Automating that would be useful, yes.

Is it an LGPL thing?

Yes, the docs are LGPL so you don't want them inside the bindings code or otherwise the bindings would have to be LGPL too.

I created/updated the docs stuff by running gir manually and copying files into place. Automating that would be useful, yes.

Would be cool to make the setup (doc_target_path etc) uniform and piece that together in a script (perhaps generator.py since that iterates Gir.toml already) for automated updating and testing.
Asking because I'm playing around with intra-doc-links ^^

Is there any reason to keep (outdated!) markdown docs checked in to the repo at that point, if such a script (in the CI) would generate the markdown (to a tmp dir) followed by building the docs with embed-lgpl-docs?

Yes, the docs are LGPL so you don't want them inside the bindings code or otherwise the bindings would have to be LGPL too.

Ack, I thought GStreamer in its entirety was LGPL, but the crates are of course following Rusts Apache/MIT. The bindings (symbols/names, signatures) are free to use a different name (most are transformed anyway) but copying the documentation text needs LGPL.

Btw @slomo the main reason I'm creating this issue instead of sending an MR for it like most other cases is that I don't really know how this is supposed to work. Should -o be involved anywhere? Should we set doc_target_path to "something sensible" (I don't really know what) and/or should gir be adjusted to use more sensible paths (ie. joining to the target vs config dir)?

For now setting doc_target_path = "../docs/gstreamer/docs.md" seems to work best; -o has to be passed but remains unused.

Is there any reason to keep (outdated!) markdown docs checked in to the repo at that point, if such a script (in the CI) would generate the markdown (to a tmp dir) followed by building the docs with embed-lgpl-docs?

I'd like to ideally have them updated as part of generator.py :) Having them in the repo is useful so that the docs crate can be uploaded at release time.

Btw @slomo the main reason I'm creating this issue instead of sending an MR for it like most other cases is that I don't really know how this is supposed to work. Should -o be involved anywhere? Should we set doc_target_path to "something sensible" (I don't really know what) and/or should gir be adjusted to use more sensible paths (ie. joining to the target vs config dir)?

For now setting doc_target_path = "../docs/gstreamer/docs.md" seems to work best; -o has to be passed but remains unused.

I don't know either :) IMHO setting doc_target_path in the Gir.tomls is the way to go (just like with the other things), but that -o must be provided anyway seems strange and unneeded.

Also thanks for looking into these kind of things, they're the parts that are not really interesting for myself so the way how it works right now is the minimum to make anything work. Improvements to that are definitely welcome :)

Having them in the repo is useful so that the docs crate can be uploaded at release time.

Isn't it something the CI could generate on the fly? And right, the text files are embedded in the doc lib so that build.rs scripts calling it can easily find it. Is the docs crate also published separately, or only used to generate docs that are subsequently uploaded elsewhere?

I don't know either :) IMHO setting doc_target_path in the Gir.tomls is the way to go (just like with the other things), but that -o must be provided anyway seems strange and unneeded.

We could possibly reuse -o to point to docs/, then get docs/<config path>/<file name> as output. Not sure if we really want to override every crate from vendor.md to docs.md?

Also thanks for looking into these kind of things, they're the parts that are not really interesting for myself so the way how it works right now is the minimum to make anything work. Improvements to that are definitely welcome :)

Not really interesting for me either, but rustdoc is just too cool and it is hard to resist not fixing something when coming across it :)

Is the docs crate also published separately, or only used to generate docs that are subsequently uploaded elsewhere?

It is, yes. So that ideally docs.rs can also use it (but that failed for the last release).

We could possibly reuse -o to point to docs/, then get docs/<config path>/<file name> as output. Not sure if we really want to override every crate from vendor.md to docs.md?

That's best to be discussed with @GuillaumeGomez

It is, yes. So that ideally docs.rs can also use it (but that failed for the last release).

How is that supposed to work? A crate that only contains text in markdown? Seems more beneficial to just have complete docs for all the individual crates published to docs.rs, while the sources on crates.io exclude it to adhere to the license.

See https://crates.io/crates/gstreamer-rs-lgpl-docs/0.16.0 and https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/blob/d69f8793d51ef49fa926142e351423e51eb8d778/gstreamer/Cargo.toml#L57-58

But there are no docs readable from that crate. I guess it is for crates that pull all dependencies from crates.io, in such a way that they can also build the documentation locally with embed-lgpl-docs without git-referencing the entire source?

Yes, exactly like that :) It's all ugly

Found why it works without -o on gtk-rs: 95779657. In our case the default comes from generator.py passing the directory containing Gir.toml.

Anyway, gtk-rs simply invokes gir a bunch of times with hardcoded paths: https://github.com/gtk-rs/gtk-rs/blob/master/.github/workflows/docs.yml.

Would be nice for us to move that into the generator, but we have a two-deep Gir.toml (gstreamer-gl/{egl,wayland,x11}) and the path passed to --doc-target-path is relative to that :/
In other words, we need to pass ../docs/gstreamer-gl/docs.md, or ../../docs/gstreamer-gl/{egl,wayland,x11}/docs.md.

So:

• Implement extra hackery in generator.py to fix up the relative paths;
• Change gir (collectively with gtk-rs) to not use a relative path, be smarter about how it is combined with -o, something like that? I don't really know yet what/how it should look like...;

• Having them in the repo is useful so that the docs crate can be uploaded at release time.

  Revise this statement - gtk-rs doesn't check them in either but generates it on the fly and publishes the results to GH-pages. GStreamer-rs is not using GH pages but we should be able to make a concession where the docs are either rebuilt or pulled from a CI artifact at deploy/release-time? The less outdated autogenerated cruft checked in, the better, IMO.
  We'll still need to deal with path mess, but only in the CI or a script - I wouldn't even go as far as adding it to generator.py then.

Have to revise that statement: gtk-rs does check them in, not in a docs/ folder but straight inside <crate>/docs.md where the generator puts them by default. Erm no, they actually don't - those files were generated locally and I didn't even validate in git.

Best to discuss this with @GuillaumeGomez, but either seems fine to me.

Perhaps a good idea to continue that discussion on IRC/Matrix. For now the MR addressing this entire issue is merged :)

Fixed in !739 (merged)

closed

mentioned in merge request !739 (merged)