New Issue

[Feature]: Custom KaTeX Macro #9759

Closed

opened by naskya · 20 comments

naskya commented

The original idea came from @five_seven@misskey.io (post).

What feature would you like implemented?

Allow users to set up custom KaTeX macros (as a user setting item) to use in their posts.

Why should we add this feature?

The fact that you can write mathematical expressions on Calckey is so lovely. Now that Misskey v13 eliminated this feature, I think only the forks of Misskey v12 officially support that (yeah, I know some instances like Mathtodon implement it independently).

So I think Calckey has aspect of being a service that accepts people who want to write mathematical expressions, which is why I propose this feature.

You have to type a lot to write mathematical expressions using LaTeX. For example, this is how I get the Hamilton–Jacobi equation and the Schrödinger equation:

\frac{\partial}{\partial t} S(q, t) = -H \left(q, \frac{\partial}{\partial q} S(q, t), t \right)

\mathrm{i} \hbar \, \frac{\mathrm{d}}{\mathrm{d} t} \left| \Psi(t) \right\rangle = \hat{H} \left| \Psi(t) \right\rangle

these \left, \right, \frac{\partial ... always annoy me.

LaTeX has many useful packages to ease this. For example, using the physics package, you can replace what's written above with the following:

\pdv{t} S(q, t) = -H \qty(q, \pdv{q} S(q, t), t)

\mathrm{i} \hbar \, \dv{t} \ket{\Psi(t)} = \hat{H} \ket{\Psi(t)}

This is way more concise and takes little effort to write and read, and I can only live with these clever packages. (edit: I've just realized that KaTeX provides \ket function, so you don't need to type \left| \right\rangle. But this doesn't negate the main point.)

Sadly, such a package is unavailable in KaTeX, but macros allow you to achieve most of the same things (at least, I believe so). So I want to have custom KaTeX macros in Calckey.

Version

13.2.0-beta2

Instance

No response

What browser are you using?

Firefox

Relevant log output

No response

Contribution Guidelines

I agree to follow this project's Contribution Guidelines

Let me say this first: I understand that only a limited number of people need this feature. So this issue asks you if you want this feature in Calckey (rather than a feature request). If I find this feature is desired, I'll try to implement this (I wonder if I can do that, though). The original idea came from `@five_seven@misskey.io` ([post](https://misskey.io/notes/9azvjfxyna)). ### What feature would you like implemented? Allow users to set up custom [KaTeX macros](https://katex.org/docs/supported.html#macros) (as a user setting item) to use in their posts. ### Why should we add this feature? The fact that you can write mathematical expressions on Calckey is so lovely. Now that Misskey v13 eliminated this feature, I think only the forks of Misskey v12 officially support that (yeah, I know some instances like [Mathtodon](https://mathtod.online/public) implement it independently). So I think Calckey has aspect of being a service that accepts people who want to write mathematical expressions, which is why I propose this feature. You have to type a lot to write mathematical expressions using LaTeX. For example, this is how I get the Hamilton–Jacobi equation and the Schrödinger equation: ![](https://cdn.discordapp.com/attachments/823878222897741868/1088473939367428226/2023-03-23_234624.png) ``` \frac{\partial}{\partial t} S(q, t) = -H \left(q, \frac{\partial}{\partial q} S(q, t), t \right) \mathrm{i} \hbar \, \frac{\mathrm{d}}{\mathrm{d} t} \left| \Psi(t) \right\rangle = \hat{H} \left| \Psi(t) \right\rangle ``` these `\left`, `\right`, `\frac{\partial ...` always annoy me. LaTeX has many useful packages to ease this. For example, using the [physics package](https://ctan.org/pkg/physics), you can replace what's written above with the following: ``` \pdv{t} S(q, t) = -H \qty(q, \pdv{q} S(q, t), t) \mathrm{i} \hbar \, \dv{t} \ket{\Psi(t)} = \hat{H} \ket{\Psi(t)} ``` This is way more concise and takes little effort to write and read, and I can only live with these clever packages. (edit: I've just realized that KaTeX provides `\ket` function, so you don't need to type `\left| \right\rangle`. But this doesn't negate the main point.) Sadly, such a package is unavailable in KaTeX, but macros allow you to achieve most of the same things (at least, I believe so). So I want to have custom KaTeX macros in Calckey. ### Version 13.2.0-beta2 ### Instance _No response_ ### What browser are you using? Firefox ### Relevant log output _No response_ ### Contribution Guidelines - [x] I agree to follow this project's Contribution Guidelines

thatonecalculator commented

Owner

I don't use KaTeX that often, do you have a good idea on how that could be implemented, especially from UI perspective?

naskya commented

Poster

I only have a brief picture, but we should expand the macro entirely in the backend so that we don't have to do much work on the frontend, different users' macros don't get mixed up in the timeline, etc. The only UI change is to add a single text area to the settings page.

naskya commented

Poster

And I don't know how to count the characters as you can configure \a to expand to aaaaaaaaaaaaa.

Maybe first allow a post attempt if the apparent character count is within the limit, then expand macros internally and check the character count again? (I feel there is a better way...)

And I don't know how to count the characters as you can configure `\a` to expand to `aaaaaaaaaaaaa`. Maybe first allow a post attempt if the apparent character count is within the limit, then expand macros internally and check the character count again? (I feel there is a better way...)

panos added the

✨Feature

label

naskya commented

Poster

we should expand the macro entirely in the backend so that we don't have to do much work on the frontend

No, I forgot to think about the preview. I'll think of another way.

> we should expand the macro entirely in the backend so that we don't have to do much work on the frontend No, I forgot to think about the preview. I'll think of another way.

Unattributed commented

I think this is a cool suggestion, but I do have a question:

Does it really need to be built in? Or could it be better implemented as a plugin?

The reason I ask is that it would be one more thing that would need to be parsed for every message coming into the server. If the admin of an instance thinks it's widely used enough to want it, okay. But for those that don't find it necessary, does it just add more overhead that they may feel is unwarranted?

I think this is a cool suggestion, but I do have a question: Does it really need to be built in? Or could it be better implemented as a plugin? The reason I ask is that it would be one more thing that would need to be parsed for every message coming into the server. If the admin of an instance thinks it's widely used enough to want it, okay. But for those that don't find it necessary, does it just add more overhead that they may feel is unwarranted?

naskya commented

Poster

@Unattributed

could it be better implemented as a plugin?

I'm unfamiliar with plugins, so it slipped my mind. But that's definitely something I had to think about.
However, having this feature would have another value in giving Calckey a direction to accept people who want to write formulas. Conversely, there is a fear that this feature may not fit the project's philosophy, which is one of the reasons I opened this issue.

need to be parsed for every message coming into the server

We can convert KaTeX source to HTML (https://katex.org/docs/api.html#server-side-rendering-or-rendering-to-a-string). So formulas can be delivered as HTML.

Also, though I forgot that, we can already use macros in posts (not very useful). So implementing this doesn't introduce a new problem, I guess... (but I must consider more because macros are not used so much)

@Unattributed > could it be better implemented as a plugin? I'm unfamiliar with plugins, so it slipped my mind. But that's definitely something I had to think about. However, having this feature would have another value in giving Calckey a direction to accept people who want to write formulas. Conversely, there is a fear that this feature may not fit the project's philosophy, which is one of the reasons I opened this issue. > need to be parsed for every message coming into the server We can convert KaTeX source to HTML (https://katex.org/docs/api.html#server-side-rendering-or-rendering-to-a-string). So formulas can be delivered as HTML. Also, though I forgot that, we can already use macros in posts (not very useful). So implementing this doesn't introduce a new problem, I guess... (but I must consider more because macros are not used so much) ![](https://cdn.discordapp.com/attachments/823878222897741868/1088622312485552208/2023-03-24_093554.png)

Unattributed commented

I like the idea of converting it to HTML when the post is created - less overhead. But, there might be a problem with that scenario: editing the post. Would the author just have to edit the HTM? Replace the HTML with re-written KaTeX? Or would the HTML be converted back into KaTeX?

I'm sure most people would want the last option instead of the first two (and might even complain if it was implemented in one of the first two methods). How easy is to convert back from HTML to KaTeX? That's probably something else to consider.

Yeah, I forgot that macros are available already as well. That makes it less of a stretch to implement KaTeX... Although I am still on the fence about the usefulness of the feature. It just seems somewhat narrowly focused, being primarily for those who are in the scientific, mathematical or technical fields... This is why I was thinking about the idea of doing this as a plugin. It seems to have a relatively narrow audience (although admittedly a high percentage of the Fediverse folks who have been around for quite some time are in those fields). I like the idea of converting it to HTML when the post is created - less overhead. But, there might be a problem with that scenario: editing the post. Would the author just have to edit the HTM? Replace the HTML with re-written KaTeX? Or would the HTML be converted back into KaTeX? I'm sure most people would want the last option instead of the first two (and might even complain if it was implemented in one of the first two methods). How easy is to convert back from HTML to KaTeX? That's probably something else to consider.

naskya commented

Poster

@Unattributed

Would the author just have to edit the HTM? Replace the HTML with re-written KaTeX? Or would the HTML be converted back into KaTeX?

How easy is to convert back from HTML to KaTeX? That's probably something else to consider.

You pointed out a significant problem.

Honestly, when I opened this issue, I (& the person who came up the original idea) thought we could expand macros without doing anything else (like what gcc -E does for the C language). But I found that it's an internal API we shouldn't access, as I looked into how to implement it. That's the biggest obstacle, and I regret that I should've known about it before creating this issue.

By the way, it isn't necessary to provide all KaTeX macros (though it'd be best if we could). Providing only a subset of macros that can be implemented by just replacing strings would be helpful enough (indeed, it's super helpful). The conversion I mentioned first (e.g., partial derivative operator \pdv{x} → \frac{\partial}{\partial x}, parenthesis \qty(x) → \left( x \right)) can be realized with this range of capabilities. KaTeX offers more high-end macros, but there should be little need for that, especially for use in social networking services.

I'll look into KaTeX macros a bit more, but if it isn't possible to fully realize them, I'd like to create a simple converter to provide the subset of the features. I believe that this works great as a KaTeX helper (I admit that the range of people I interact with is very biased, but I see many people who want to post formulas on a daily basis) and is not a nuisance to those who don't write formulas.

edit: I mean, if we use the limited (text replacing only) macro feature, we can just expand macros (without converting to HTML) and deliver/store them.

What do you think about this 🤔

@Unattributed > Would the author just have to edit the HTM? Replace the HTML with re-written KaTeX? Or would the HTML be converted back into KaTeX? > How easy is to convert back from HTML to KaTeX? That's probably something else to consider. You pointed out a significant problem. Honestly, when I opened this issue, I (& the person who came up the original idea) thought we could expand macros without doing anything else (like what `gcc -E` does for the C language). But I found that it's an internal API we shouldn't access, as I looked into how to implement it. That's the biggest obstacle, and I regret that I should've known about it before creating this issue. By the way, it isn't necessary to provide all KaTeX macros (though it'd be best if we could). Providing only a subset of macros that can be implemented by just replacing strings would be helpful enough (indeed, it's super helpful). The conversion I mentioned first (e.g., partial derivative operator `\pdv{x}` → `\frac{\partial}{\partial x}`, parenthesis `\qty(x)` → `\left( x \right)`) can be realized with this range of capabilities. KaTeX offers more high-end macros, but there should be little need for that, especially for use in social networking services. I'll look into KaTeX macros a bit more, but if it isn't possible to fully realize them, I'd like to create a simple converter to provide the subset of the features. I believe that this works great as a KaTeX helper (I admit that the range of people I interact with is very biased, but I see many people who want to post formulas on a daily basis) and is not a nuisance to those who don't write formulas. edit: I mean, if we use the limited (text replacing only) macro feature, we can just expand macros (without converting to HTML) and deliver/store them. What do you think about this 🤔

naskya commented

Poster

Progress: I implemented this. I'll test for a while to see if there are any bugs. (and maybe the description that I wrote is long so try to make it short 😂)

I initially thought of storing the macro information in the database and processing them in the backend, but now I think it's better to save this as a client setting and do the entire macro expansion process on the browser, so I chose that method.

Progress: I implemented this. I'll test for a while to see if there are any bugs. (and maybe the description that I wrote is long so try to make it short 😂) I initially thought of storing the macro information in the database and processing them in the backend, but now I think it's better to save this as a client setting and do the entire macro expansion process on the browser, so I chose that method. ![](https://s3.arkjp.net/misskey/ee3e7abb-0078-4b8b-a103-fbf2f820ef5f.png) ![](https://s3.arkjp.net/misskey/054dbefa-61dc-4e83-adf4-1f9d86a15a67.png) ![](https://s3.arkjp.net/misskey/97858bdb-b81b-4715-a239-691fc0a89af3.png)

Unattributed commented

IMO - I think you are on the right path with this... It will make it easier for the user to determine if the macro expansion becomes too large for the message they are writing.

But, I was just thinking: what are the chances of getting stuck in an infinite loop during the expansion? What about recursion in the macro?

IMO - I think you are on the right path with this... It will make it easier for the user to determine if the macro expansion becomes too large for the message they are writing. But, I was just thinking: what are the chances of getting stuck in an infinite loop during the expansion? What about recursion in the macro?

naskya commented

Poster

@Unattributed

I think you are on the right path with this

Thanks!

what are the chances of getting stuck in an infinite loop during the expansion?

When the macro expands 200 times, the expansion process is terminated, and the text at that point is returned (I don't know if 200 is the best, but I think it's enough for regular use and not computational-heavy).

@Unattributed > I think you are on the right path with this Thanks! > what are the chances of getting stuck in an infinite loop during the expansion? When the macro expands 200 times, the expansion process is terminated, and the text at that point is returned (I don't know if 200 is the best, but I think it's enough for regular use and not computational-heavy).

naskya commented

Poster

It will make it easier for the user to determine if the macro expansion becomes too large for the message they are writing

The character counter takes the text as expanded, so it's okay (btw thank you for sharing your opinion!)

> It will make it easier for the user to determine if the macro expansion becomes too large for the message they are writing The character counter takes the text as expanded, so it's okay (btw thank you for sharing your opinion!) ![](https://cdn.discordapp.com/attachments/823878222897741868/1089419218144993340/2023-03-26_142051.png) ![](https://cdn.discordapp.com/attachments/823878222897741868/1089419218828677241/2023-03-26_142109.png)

Unattributed commented

Looks good. I was just thinking about what could happen if the macro expansion tried to write beyond the message space. Anytime it's possible to end up with an out of bounds issue (or a pointer going past it's data space) it's worth being careful.

👍 1

naskya commented

Poster

I opened an instance to test this, so please contact me (@naskya@misskey.io) if you are curious/want to debug this.

I opened an instance to test this, so please contact me ([@naskya@misskey.io](https://misskey.io/@naskya)) if you are curious/want to debug this.

naskya changed title from ~~[Feature / Discussion]: Per-user custom KaTeX macros~~ to [Feature / Discussion]: Custom KaTeX Macro

panos commented

Owner

I tested naskya's implementation, looks great to me! I think we can merge it. Well done!

naskya commented

Poster

@panos Thank you, I'm happy to hear that. I've tried to implement it to fit in with the current Calckey UI, and I hope you feel that.

naskya commented

Poster

I initially said

If I find this feature is desired, I'll try to implement this

but I now recognize this feature isn't really desired by many people (and I guess that's why Misskey v13 eliminated the KaTeX feature), but a limited range of people strongly wants it (the proposer of the original idea was very pleased!). Thus this feature is "narrow and deep", so to speak.

I made this feature harmless for those who don't use this (e.g., don't modify the backend, turn it off by default). So I'll create a pull request, even though it contradicts what I initially said.

BTW there may be unknown bugs, as I wrote all the programs to parse the syntax and expand macros from scratch. I'll test this for a few more days.

I initially said > If I find this feature is desired, I'll try to implement this but I now recognize this feature isn't really desired by many people (and I guess that's why Misskey v13 eliminated the KaTeX feature), but a limited range of people strongly wants it (the proposer of the original idea was very pleased!). Thus this feature is "narrow and deep", so to speak. I made this feature harmless for those who don't use this (e.g., don't modify the backend, turn it off by default). So I'll create a pull request, even though it contradicts what I initially said. BTW there may be unknown bugs, as I wrote all the programs to parse the syntax and expand macros from scratch. I'll test this for a few more days.

panos commented

Owner

Yes, it's also something I would never use, like most users, but I think that your implementation is indeed unobtrusive, so I don't mind that it's there since some people will find it useful - the rest of us can just ignore it.
Don't worry about extended testing, even if it has some bug then someone will report it at some point, it's not the end of the world if something is found and fixed a little later!

Yes, it's also something I would never use, like most users, but I think that your implementation is indeed unobtrusive, so I don't mind that it's there since some people will find it useful - the rest of us can just ignore it. Don't worry about extended testing, even if it has some bug then someone will report it at some point, it's not the end of the world if something is found and fixed a little later!

👍 2

naskya changed title from ~~[Feature / Discussion]: Custom KaTeX Macro~~ to [Feature]: Custom KaTeX Macro

Unattributed commented

I'd say go ahead and merge it. Sure maybe there aren't a lot of users for it, but it's not something that is going to detract from others' using Calckey. Plus, if it's merged now in the beta cycle, it can get tested by enough people to find any issues before a stable release.