Not an Issue: General Discussion #35

Open
opened 4 months ago by trholding · 14 comments

Place Holder and Purpose

I hope you don't mind that I opened this is issue as a form of General Discussion so that it could be kept open and people could key in their thoughts, plans and ideas... ie. Talk about FeatherWiki stuff without opening issues for that. Good catches from this Non Issue thread could be converted to issues by you.

Place Holder and Purpose I hope you don't mind that I opened this is issue as a form of General Discussion so that it could be kept open and people could key in their thoughts, plans and ideas... ie. Talk about FeatherWiki stuff without opening issues for that. Good catches from this Non Issue thread could be converted to issues by you.
Poster

So here I am writing some opinions and stuff that don't fit anywhere else.

Here is a wish list:

  • Comprehensive Documentation - Let's create detailed dev and user docs with Feather Wiki itself (Eat our own dog food (even if you are a cat person) :) )

  • A document/list about all stylable elements in FW, and ability to override

  • A quick over view about FW functions and files so one knows where to start hacking

If you create a doc sub dir or git, contributors could contribute snarkdown/snarky formated texts for docs.

So here I am writing some opinions and stuff that don't fit anywhere else. Here is a wish list: * Comprehensive Documentation - Let's create detailed dev and user docs with Feather Wiki itself (Eat our own dog food (even if you are a cat person) :) ) * A document/list about all stylable elements in FW, and ability to override * A quick over view about FW functions and files so one knows where to start hacking If you create a doc sub dir or git, contributors could contribute snarkdown/snarky formated texts for docs.
Owner

Especially as I come closer to releasing extension support, I've been wanting to gather up everything in documentation as well! I want it to eventually make its way into the main feather.wiki site, but I really like your idea of making a separate docs directory so others can contribute markdown-formatted documentation and guides. Paired with an extension like what was requested in #29, this would make compiling more comprehensive documentation much easier!

It might take a little bit of setup to write a quick guide for expected formatting, but I think it would be very worth it.

Especially as I come closer to releasing extension support, I've been wanting to gather up everything in documentation as well! I want it to eventually make its way into the main feather.wiki site, but I really like your idea of making a separate docs directory so others can contribute markdown-formatted documentation and guides. Paired with an extension like what was requested in #29, this would make compiling more comprehensive documentation much easier! It might take a little bit of setup to write a quick guide for expected formatting, but I think it would be very worth it.
Poster

I'll contribute to the docs. You need to provide some hints.

I'll contribute to the docs. You need to provide some hints.
Alamantus added the
discussion
label 4 months ago

Since it's general discussion, here's one thought I'd like to share...

Having the builds checked into the git repo probably isn't the best way to manage them. It would be nice to have the builds created on release and available as artifacts for download, rather than have them in the repo directly.

E.g. see https://github.com/sigstore/cosign/releases . The assets there are built and available for download. I'm not sure how that works, but I expect you can create automation in Github to do it. Perhaps codeberg has something similar.

Even though the builds are technically html and javascript, because of the compression they're actually more like binaries, and of course they are entirely reproducible from their source, so IMO it doesn't make sense to git add them.

On top of that it produces a lot of noise in the git history, and when hacking you can end up with big ugly diffs that you never want to commit. This would be avoided if ./builds was .gitignored.

Edit: I see you have https://codeberg.org/Alamantus/FeatherWiki/releases already. The source zip and tarball aren't very useful there. Imagine if all the Feather Wiki builds were there instead, and that was the primary way for people to download them.

Since it's general discussion, here's one thought I'd like to share... Having the builds checked into the git repo probably isn't the best way to manage them. It would be nice to have the builds created on release and available as artifacts for download, rather than have them in the repo directly. E.g. see https://github.com/sigstore/cosign/releases . The assets there are built and available for download. I'm not sure how that works, but I expect you can create automation in Github to do it. Perhaps codeberg has something similar. Even though the builds are technically html and javascript, because of the compression they're actually more like binaries, and of course they are entirely reproducible from their source, so IMO it doesn't make sense to git add them. On top of that it produces a lot of noise in the git history, and when hacking you can end up with big ugly diffs that you never want to commit. This would be avoided if ./builds was .gitignored. Edit: I see you have https://codeberg.org/Alamantus/FeatherWiki/releases already. The source zip and tarball aren't very useful there. Imagine if all the Feather Wiki builds were there instead, and that was the primary way for people to download them.
Owner

Yeah @sbaird, I completely agree! I will remove the builds folder and strive to put them as separate downloadable files on the releases individually. I think that will ultimately be a better way for people to know what exactly the release is as well.

Yeah @sbaird, I completely agree! I will remove the builds folder and strive to put them as separate downloadable files on the releases individually. I think that will ultimately be a better way for people to know what exactly the release is as well.
Owner

Also as a note, I am working on creating templates for the Issues tab in the hopes that we can use them as a makeshift forum for general conversation like this but so it's doable with separate topics. The templating system is pretty nice and makes it so that as long as you choose the right template, it will label the issue correctly.

I believe with some explanation (and some better signposting on my part) that it will be a viable way to prevent people from emailing me directly from my profile website's contact form 😅

Also as a note, I am working on creating templates for the Issues tab in the hopes that we can use them as a makeshift forum for general conversation like this but so it's doable with separate topics. The templating system is pretty nice and makes it so that as long as you choose the right template, it will label the issue correctly. I believe with some explanation (and some better signposting on my part) that it will be a viable way to prevent people from emailing me directly from my profile website's contact form 😅
Poster

I am working on a tiny polyglot server for FeatherWiki in my freeime - if all works well the FeatherWiki itself could run to serve itself...

I need to understand something about the put saver.

As far as I understand the PUT Saver versions works like this:

  1. SENDS "OPTIONS"
  2. LOOKS FOR "DAV" in reply
  3. PUTS CONTENT

Is there more to it?

I am working on a tiny polyglot server for FeatherWiki in my freeime - if all works well the FeatherWiki itself could run to serve itself... I need to understand something about the put saver. As far as I understand the PUT Saver versions works like this: 1. SENDS "OPTIONS" 2. LOOKS FOR "DAV" in reply 3. PUTS CONTENT Is there more to it?

That's pretty much it. The options request is used to decide if the "put save" button should be visible or not. So in a little more details it's like this:

  1. On startup, if the current url starts with http, send an options request
  2. If "dav" header is present in the response, assume the server might be able to handle a put save, and make the put save button visible.

Then:

  1. On put save button click, do a put request with the current content
  2. If it succeeds then assume the save worked and clear the "save needed" flag
That's pretty much it. The options request is used to decide if the "put save" button should be visible or not. So in a little more details it's like this: 1. On startup, if the current url starts with http, send an options request 2. If "dav" header is present in the response, assume the server might be able to handle a put save, and make the put save button visible. Then: 1. On put save button click, do a put request with the current content 2. If it succeeds then assume the save worked and clear the "save needed" flag
Poster
  1. On put save button click, do a put request with the current content
  2. If it succeeds then assume the save worked and clear the "save needed" flag

Thanks a ton for this.

> 1. On put save button click, do a put request with the current content > 2. If it succeeds then assume the save worked and clear the "save needed" flag Thanks a ton for this.
Poster

Just a little update / sneak peek.

FeatherWiki now can be self served by itself! (Not yet ready for release. Still hacking on it)

When you download this version of the wiki, you can:

  • Open it in a browser and view it
  • Execute it as a shell script and it will serve the wiki (current default: http://localhost:8000)
  • View the locally served wiki

I'll be implementing a couple of features including solution for issue #31

Today I'll be adding the PUT saving option.

This is experimental / unsafe, hence I call it Gorilla Serv / And Gorilla's Nest wrt to FeatherWiki. Ideal to host it on a RasPi Zero in a educational setting where security doesn't matter.

I'll release this once I've done the PUT DAV saving and cleaning up stuff.

Next goal is to harden. Signature check. Add Encryption. Extract, Insert tools. Compression. IPFS publishing and some fancy stuff as long as it stays small.

Size is 3.5kb extra. If I remove all place holder comments and reduce text, its around 1.5kb. Compressed it takes some 400 bytes. But I think the final version with all bells and whistles / tools will be around 5.5kb.

Changes to the build system that need to be made is use this as template for selfserving builds where you add all the html into this marker {{WEBAPP}}

Just a little update / sneak peek. FeatherWiki now can be self served by itself! (Not yet ready for release. Still hacking on it) When you download this version of the wiki, you can: * Open it in a browser and view it * Execute it as a shell script and it will serve the wiki (current default: http://localhost:8000) * View the locally served wiki I'll be implementing a couple of features including solution for issue #31 Today I'll be adding the PUT saving option. This is experimental / unsafe, hence I call it Gorilla Serv / And Gorilla's Nest wrt to FeatherWiki. Ideal to host it on a RasPi Zero in a educational setting where security doesn't matter. I'll release this once I've done the PUT DAV saving and cleaning up stuff. Next goal is to harden. Signature check. Add Encryption. Extract, Insert tools. Compression. IPFS publishing and some fancy stuff as long as it stays small. Size is 3.5kb extra. If I remove all place holder comments and reduce text, its around 1.5kb. Compressed it takes some 400 bytes. But I think the final version with all bells and whistles / tools will be around 5.5kb. Changes to the build system that need to be made is use this as template for selfserving builds where you add all the html into this marker {{WEBAPP}}
Owner

Holy smokes, a self executable Feather Wiki is an extremely cool idea! I honestly like that a lot better than having a separate server like I planned. Haha I love the "gorilla" name too. Probably will change that to a bird again, but there are some pretty big birds so it won't be hard to find a good one. Falcon or Eagle could be a good fit depending on how big it needs to get.

One important thing to ask is if you've checked the current dev branch. I've got a bunch of features and improvements that I'm working on finalizing for the 1.3.0 release, and I want to make sure the changes you're making don't get too disrupted by that stuff!

I can definitely help with scaling things down when you're comfortable sharing it!

Holy smokes, a self _executable_ Feather Wiki is an extremely cool idea! I honestly like that a lot better than having a separate server like I planned. Haha I love the "gorilla" name too. Probably will change that to a bird again, but there are some pretty big birds so it won't be hard to find a good one. Falcon or Eagle could be a good fit depending on how big it needs to get. One important thing to ask is if you've checked the current `dev` branch. I've got a bunch of features and improvements that I'm working on finalizing for the 1.3.0 release, and I want to make sure the changes you're making don't get too disrupted by that stuff! I can definitely help with scaling things down when you're comfortable sharing it!
Owner

This is a Node script rather than a shell script, but in case it helps, here's an extremely basic implementation of PUT-save in the Feather Wiki test-build script (there's no password protection, but I have an idea of a very simple way to implement it): https://codeberg.org/Alamantus/FeatherWiki/src/branch/main/scripts/test-build.js

Another edit so I don't flood too many comments, but just a note to say that a ton of hardening beyond password protection might not be super necessary. I always recommend people only ever run apps like these behind a reverse proxy, and hardening that (and the server itself) is the most important part.

This is a Node script rather than a shell script, but in case it helps, here's an extremely basic implementation of PUT-save in the Feather Wiki `test-build` script (there's no password protection, but I have an idea of a very simple way to implement it): https://codeberg.org/Alamantus/FeatherWiki/src/branch/main/scripts/test-build.js Another edit so I don't flood too many comments, but just a note to say that a ton of hardening beyond password protection might not be super necessary. I always recommend people only ever run apps like these behind a reverse proxy, and hardening _that_ (and the server itself) is the most important part.
Poster

Holy smokes, a self executable Feather Wiki ... Falcon or Eagle could be a good fit depending on how big it needs to get.

Make it a big fat dangerous bird :) Goal is for this to be like a guerrilla server - spread wikified information simple and fast. Also as tool to manage the wiki / add external features (such as encryption / publish to IPFS) until FeatherWiki gets the plugin support...

One important thing to ask is if you've checked the current dev branch. ... too disrupted by that stuff!

Not checked yet, will do. Do not worry, this can theoretically serve any HTML file / SPA even TiddlyWiki, so you need not worry or slow down. :) We just encapsulate the HTML file with a shell script that serves stuff the UNIX way.

I can definitely help with scaling things down when you're comfortable sharing it!

Probably only a new target such as FeatherWiki_Vulture/Ostrich and this template to which you output a FeatherWiki

This is the layout to understand:

[Bash Server Stub - Ignored by browser]
[Your FeatherWiki - Any Version pasted into the middle]
[Bash Closing with signatures - Ignored by browser]

This is a Node script rather than a shell script, but in case it helps, here's an extremely basic implementation of PUT-save in the Feather Wiki ...

Thanks a ton for this. Good idea to provide a nodejs version!. I hope, I can use parts of the logic.

Edit:

        'Content-Type': 'text/html',
        'dav': 1,

Thanks, that part saved me a lot of hairs and bytes. I was actually stupid and implementing a flow! Very clever to just respond with dav always!

Another edit so I don't flood too many comments ... reverse proxy, and hardening that (and the server itself) is the most important part.

True, I usually use caddy for that. I have to harden this as its a shell script, a bypass would mean shell access.

Info wanted:

One part that I am thinking of is how to parse posts so that I could extract, insert.
Eg: $ FeatherWiki_Vulture.html -i "New Post" "This will show up in the wiki!"
I need to study the format, would have been awesome easy if FW stored it in TSV/CSV! instead of JSON:).

> Holy smokes, a self _executable_ Feather Wiki ... Falcon or Eagle could be a good fit depending on how big it needs to get. Make it a big fat dangerous bird :) Goal is for this to be like a guerrilla server - spread wikified information simple and fast. Also as tool to manage the wiki / add external features (such as encryption / publish to IPFS) until FeatherWiki gets the plugin support... > One important thing to ask is if you've checked the current `dev` branch. ... too disrupted by that stuff! Not checked yet, will do. Do not worry, this can theoretically serve any HTML file / SPA even TiddlyWiki, so you need not worry or slow down. :) We just encapsulate the HTML file with a shell script that serves stuff the UNIX way. > I can definitely help with scaling things down when you're comfortable sharing it! Probably only a new target such as FeatherWiki_Vulture/Ostrich and this template to which you output a FeatherWiki This is the layout to understand: ``` [Bash Server Stub - Ignored by browser] [Your FeatherWiki - Any Version pasted into the middle] [Bash Closing with signatures - Ignored by browser] ``` > This is a Node script rather than a shell script, but in case it helps, here's an extremely basic implementation of PUT-save in the Feather Wiki ... Thanks a ton for this. Good idea to provide a nodejs version!. I hope, I can use parts of the logic. Edit: ``` 'Content-Type': 'text/html', 'dav': 1, ``` Thanks, that part saved me a lot of hairs and bytes. I was actually stupid and implementing a flow! Very clever to just respond with dav always! > Another edit so I don't flood too many comments ... reverse proxy, and hardening _that_ (and the server itself) is the most important part. True, I usually use caddy for that. I have to harden this as its a shell script, a bypass would mean shell access. Info wanted: One part that I am thinking of is how to parse posts so that I could extract, insert. Eg: $ FeatherWiki_Vulture.html -i "New Post" "This will show up in the wiki!" I need to study the format, would have been awesome easy if FW stored it in TSV/CSV! instead of JSON:).
Poster

Sorry for the cliff hanger / ghosting, was busy, apologies for this lengthy post in case it clogs up your mailbox...

This is the small progress with gorilla:

  • Serves latest Featherwiki
  • Generates Static Pages
  • Serves the FeatherWiki itself + static singlepage (content size doubled)
  • Current size 6.2 KB added to FW
Sizes on disk:

148K - Feather Wiki Website (JS Dove)
154K - Feather Wiki Website wrapped with Gorilla Serv
210K - Feather Wiki Website + static content wrapped with Gorilla Serv

Size on disk/network/wire with compression:

 58K - Feather Wiki Website (brotli)
 60K - Feather Wiki Website + wrapped Gorilla Serv (No static output) (brotli)
 62K - Feather Wiki Website + static + wrapped Gorilla serv (brotli)
 
Inference: With compression turned on and complete compatibility, the overhead is around + ~4k from compressed baseline and a minus ~86k from uncompressed baseline. This inference sort of prevents selfguilt / sadness that KBs were added. :)
 

Makes Feather Wiki compatible with:

  • Browsers with Javascript disabled such as TOR Browser in strict
  • Supports vintage and Text browsers such as lynx and w3m
  • Kindle Browsers

Included here:

  • Sample output of static gen
  • Sample test screenshots

TODO / In progress:

  • HTTP gz and brotli compression (in progress)
  • Basic HTTP Auth and HTTPS (SSL)
  • Better parsing and generation / wikilinks / images / nested pages (in progress)
  • PUT should be reliable (it isn't now in test version)
  • Inject posts to wiki via CLI / import to wiki
  • Implement a POST Form for adding posts to wiki on non js browsers
  • Wraping and saving has to be reliable (it isn't now in certain cases)
  • Clean up and minification (such as repeated string components) / halve size
  • Code signing for integrity + content signing with minisig and gpg (probably a world first for a wiki - people can be sure that code is genuine and content is authentic)
  • Publish to ipfs -> Infura / Pinata
  • Cloudlfare ipfs dnslink to publish to domain
  • Ask Alamantus if he is planning to provide a featherwiki service that maps to ipfs ie myfunwiki.FWN.wiki.org or feather.wiki/myfunwiki etc so that users can instant publish

I'll release this as v0.0.1 when I have overcome the reliability issues with saving.

I kindly request you to add wanted features here so that I can implement them in one go when I get the Zen Enso Flow...

Sorry for the cliff hanger / ghosting, was busy, apologies for this lengthy post in case it clogs up your mailbox... This is the small progress with gorilla: * Serves latest Featherwiki * Generates Static Pages * Serves the FeatherWiki itself + static singlepage (content size doubled) * Current size 6.2 KB added to FW ``` Sizes on disk: 148K - Feather Wiki Website (JS Dove) 154K - Feather Wiki Website wrapped with Gorilla Serv 210K - Feather Wiki Website + static content wrapped with Gorilla Serv Size on disk/network/wire with compression: 58K - Feather Wiki Website (brotli) 60K - Feather Wiki Website + wrapped Gorilla Serv (No static output) (brotli) 62K - Feather Wiki Website + static + wrapped Gorilla serv (brotli) Inference: With compression turned on and complete compatibility, the overhead is around + ~4k from compressed baseline and a minus ~86k from uncompressed baseline. This inference sort of prevents selfguilt / sadness that KBs were added. :) ``` Makes Feather Wiki compatible with: * Browsers with Javascript disabled such as TOR Browser in strict * Supports vintage and Text browsers such as lynx and w3m * Kindle Browsers Included here: * Sample output of static gen * Sample test screenshots TODO / In progress: * HTTP gz and brotli compression (in progress) * Basic HTTP Auth and HTTPS (SSL) * Better parsing and generation / wikilinks / images / nested pages (in progress) * PUT should be reliable (it isn't now in test version) * Inject posts to wiki via CLI / import to wiki * Implement a POST Form for adding posts to wiki on non js browsers * Wraping and saving has to be reliable (it isn't now in certain cases) * Clean up and minification (such as repeated string components) / halve size * Code signing for integrity + content signing with minisig and gpg (probably a world first for a wiki - people can be sure that code is genuine and content is authentic) * Publish to ipfs -> Infura / Pinata * Cloudlfare ipfs dnslink to publish to domain * Ask Alamantus if he is planning to provide a featherwiki service that maps to ipfs ie myfunwiki.FWN.wiki.org or feather.wiki/myfunwiki etc so that users can instant publish I'll release this as v0.0.1 when I have overcome the reliability issues with saving. I kindly request you to add wanted features here so that I can implement them in one go when I get the Zen Enso Flow...
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Reference: Alamantus/FeatherWiki#35
Loading…
There is no content yet.