Performance issues / 504 #266

Open
opened 8 months ago by tacerus · 4 comments

Hello!

Thanks for the great software!
We operate reddit.lol, and seem to encounter some performance issues and crashes.

We have teddit installed manually from a recent git clone in an NVM with Node 16.13.0 on a KVM VM with four virtual cores and 20GB of allocated memory, along with nginx. Originally Redis was connected to a seperate machine, however thinking the Fast Ethernet link might be a bottleneck, it was moved to the same machine teddit runs on.

Occasionally we see teddit crash and restart in systemd. At other times it does not crash, but makes nginx serve a 504, logging a "Connection refused while connecting to upstream". I cannot see any errors in the teddit log at the timestamps nginx logs the connection error. I can however see teddit processing other requests.
Whilst Redis does consume most of the resources, especially during bgsave operations, it does not seem to overload the machine.

We set the Redis maxmemory policy to 13GB with allkeys-lru, as suggested in the Wiki - there are no other Redis instances running on the machine.

Are there any other recommendations to improve the performance and stability?

Here is the running systemd service:
https://pasta.lysergic.dev/?47209e9c77c5c0d9#9ddSmc6eQ44HcQWCrtAnFmovD1rF2tDZ9NmYiskDaF4f

Here is the running teddit configuration:
https://pasta.lysergic.dev/?bab4e0f23211751e#4uEapBu2jccpgmEBUoPakZn6mz9ZQXPkmg6vy9e3zFYF

Here is the running nginx configuration block:
https://pasta.lysergic.dev/?4360b6f37f36d56e#5PdJgLLRD4h4p6STNpHndJiCr7ZqP1Qs9Rp6WeopvvU6

Best,
Georg

Hello! Thanks for the great software! We operate reddit.lol, and seem to encounter some performance issues and crashes. We have teddit installed manually from a recent git clone in an NVM with Node 16.13.0 on a KVM VM with four virtual cores and 20GB of allocated memory, along with nginx. Originally Redis was connected to a seperate machine, however thinking the Fast Ethernet link might be a bottleneck, it was moved to the same machine teddit runs on. Occasionally we see teddit crash and restart in systemd. At other times it does not crash, but makes nginx serve a 504, logging a "Connection refused while connecting to upstream". I cannot see any errors in the teddit log at the timestamps nginx logs the connection error. I can however see teddit processing other requests. Whilst Redis does consume most of the resources, especially during bgsave operations, it does not seem to overload the machine. We set the Redis maxmemory policy to 13GB with allkeys-lru, as suggested in the Wiki - there are no other Redis instances running on the machine. Are there any other recommendations to improve the performance and stability? Here is the running systemd service: https://pasta.lysergic.dev/?47209e9c77c5c0d9#9ddSmc6eQ44HcQWCrtAnFmovD1rF2tDZ9NmYiskDaF4f Here is the running teddit configuration: https://pasta.lysergic.dev/?bab4e0f23211751e#4uEapBu2jccpgmEBUoPakZn6mz9ZQXPkmg6vy9e3zFYF Here is the running nginx configuration block: https://pasta.lysergic.dev/?4360b6f37f36d56e#5PdJgLLRD4h4p6STNpHndJiCr7ZqP1Qs9Rp6WeopvvU6 Best, Georg
Owner

Hi!

Have you experienced performance issues right from the start or is this a brand new issue?

This sounds weird:

I can however see teddit processing other requests.

If you are getting 504s from nginx, but teddit is processing requests in the background, it sounds like there's something wrong between the connection between teddit and nginx. Maybe teddit crashes, and systemd brings it up again, but nginx somehow fails to "re-bind" itself to the new process, or something like that. How do you usually solve these 504 issues, do you restart nginx?

And by the way, if you see any errors which are crashing teddit, please open up a new issue - I usually try to fix crashing bugs asap.

Hi! Have you experienced performance issues right from the start or is this a brand new issue? This sounds weird: >I can however see teddit processing other requests. If you are getting 504s from nginx, but teddit is processing requests in the background, it sounds like there's something wrong between the connection between teddit and nginx. Maybe teddit crashes, and systemd brings it up again, but nginx somehow fails to "re-bind" itself to the new process, or something like that. How do you usually solve these 504 issues, do you restart nginx? And by the way, if you see any errors which are crashing teddit, please open up a new issue - I usually try to fix crashing bugs asap.
teddit added the
question
label 8 months ago
Poster

Hi,

I would say this only occured recently, after the fresh installation on either of the servers it worked perfectly, but then again, we only deployed this instance a few weeks ago.

We usually don't "resolve" them, but just wait til it works again, given that it seemingly looks like it is working for others (or could the requests that are logged just be some sort of backlog or buffer?). I now noticed however (unsure why I did not test this before), that I am unable to reach it locally either. I.e. a curl to the binding address and port on the server it runs on will just get stuck.

Sorry to not have any more information - it feels kind of like an issue on our end, but I don't get it yet.

I will report should I see any errors!

Hi, I would say this only occured recently, after the fresh installation on either of the servers it worked perfectly, but then again, we only deployed this instance a few weeks ago. We usually don't "resolve" them, but just wait til it works again, given that it seemingly looks like it is working for others (or could the requests that are logged just be some sort of backlog or buffer?). I now noticed however (unsure why I did not test this before), that I am unable to reach it locally either. I.e. a curl to the binding address and port on the server it runs on will just get stuck. Sorry to not have any more information - it feels kind of like an issue on our end, but I don't get it yet. I will report should I see any errors!
Poster

Aha! Just after I posted this I received more useful output from teddit.

-- Logs begin at Fri 2021-12-10 03:29:29 CET. --
Dec 10 16:30:28 selene npm[4523]: Fetched the short URL for post from Reddit.
Dec 10 16:30:30 selene npm[4523]: Fetched the JSON from reddit.com/r/perfectlycutscreams/comments/xxx.
Dec 10 16:30:31 selene npm[4523]: Fetched the JSON from reddit.com/r/AmItheAsshole/xxx
Dec 10 16:30:32 selene npm[4523]: Something went wrong while fetching data from Reddit. 403 – Forbidden
Dec 10 16:30:32 selene npm[4523]: Seems like your instance is either blocked (e.g. due to API rate limiting), reddit is currently down, or your API key is expired and not renewd properly. This can also happen for other reasons.
Dec 10 16:30:38 selene npm[4523]: Fetched sidebar for fuckoffrebecca from reddit API
Dec 10 16:31:02 selene npm[4523]: Got short URL for post key from redis.
Dec 10 16:31:25 selene npm[4523]: Something went wrong while fetching data from Reddit. 403 – Forbidden
Dec 10 16:31:25 selene npm[4523]: Seems like your instance is either blocked (e.g. due to API rate limiting), reddit is currently down, or your API key is expired and not renewd properly. This can also happen for other reasons.
Dec 10 16:31:32 selene npm[4523]: Fetched the JSON from reddit.com/r/Eldenring.

But is it normal that it is able to process some requests in the middle and after that?

Aha! Just after I posted this I received more useful output from teddit. ``` -- Logs begin at Fri 2021-12-10 03:29:29 CET. -- Dec 10 16:30:28 selene npm[4523]: Fetched the short URL for post from Reddit. Dec 10 16:30:30 selene npm[4523]: Fetched the JSON from reddit.com/r/perfectlycutscreams/comments/xxx. Dec 10 16:30:31 selene npm[4523]: Fetched the JSON from reddit.com/r/AmItheAsshole/xxx Dec 10 16:30:32 selene npm[4523]: Something went wrong while fetching data from Reddit. 403 – Forbidden Dec 10 16:30:32 selene npm[4523]: Seems like your instance is either blocked (e.g. due to API rate limiting), reddit is currently down, or your API key is expired and not renewd properly. This can also happen for other reasons. Dec 10 16:30:38 selene npm[4523]: Fetched sidebar for fuckoffrebecca from reddit API Dec 10 16:31:02 selene npm[4523]: Got short URL for post key from redis. Dec 10 16:31:25 selene npm[4523]: Something went wrong while fetching data from Reddit. 403 – Forbidden Dec 10 16:31:25 selene npm[4523]: Seems like your instance is either blocked (e.g. due to API rate limiting), reddit is currently down, or your API key is expired and not renewd properly. This can also happen for other reasons. Dec 10 16:31:32 selene npm[4523]: Fetched the JSON from reddit.com/r/Eldenring. ``` But is it normal that it is able to process some requests in the middle and after that?
Owner

Those log entries seem perfectly normal. Time to time Reddit just gives 403 errors, can't do anything about those really.

I don't think those log entries are related to this particular nginx issue, when nginx is giving 504 error.

Those log entries seem perfectly normal. Time to time Reddit just gives 403 errors, can't do anything about those really. I don't think those log entries are related to this particular nginx issue, when nginx is giving 504 error.
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: teddit/teddit#266
Loading…
There is no content yet.