Codeberg CI not executing any tasks for the past days

Andre601 commented

2023-10-15 11:20:12 +00:00

Comment

I (And according to chats on matrix some others) encounter an issue right now with the Codeberg CI instance.

The issue is, that the CI does start a run, but not execute any of its set jobs... No matter how long it has been running.
As an example, a recent run I started has not yet even cloned the repository 12 hours after it has been triggered.

I would apreciate if the Codeberg team could look into this as I believe there is a backend-issue in the CI causing jobs to be queued but never run...

### Comment I (And according to chats on matrix some others) encounter an issue right now with the Codeberg CI instance. The issue is, that the CI does start a run, but not execute any of its set jobs... No matter how long it has been running. As an example, a recent run I started has not yet even cloned the repository **12 hours** after it has been triggered. I would apreciate if the Codeberg team could look into this as I believe there is a backend-issue in the CI causing jobs to be queued but never run...

6543 was assigned by fnetX

2023-10-15 11:22:21 +00:00

SamWhited commented

2023-11-25 13:22:06 +00:00

I've been having the same thing for a month or so; all CI triggers appear to be running on the repo but are getting back a 410 from CI. I can manually run jobs fine.

SamWhited commented

2023-11-25 13:23:32 +00:00

Hitting "Repair Repository" in the CI settings appears to re-create the webhook on the repo and then everything works fine. It's a bit annoying to have to do this for every repo, but for anyone finding this it should fix your builds. If that doesn't work you may have to manually remove the webhook in the repo settings then hit repair repo in CI.

fnetX commented

2023-11-25 13:33:56 +00:00

Owner

I would also appreciate this to be fixed, because breaking changes like this lead to very poor user experience. But there is a known workaround, and I heard that it was related to an actual edge case / bug that the "Repair all" action failed on our CI instance.

I suppose we can close this now.

I would also appreciate this to be fixed, because breaking changes like this lead to very poor user experience. But there is a known workaround, and I heard that it was related to an actual edge case / bug that the "Repair all" action failed on our CI instance. I suppose we can close this now.

fnetX closed this issue

2023-11-25 13:33:57 +00:00

SamWhited commented

2023-11-25 13:36:06 +00:00

Am I to understand that this won't be fixed? I really don't want to go through every single repo (there are a lot of them) and hit the repair button. I know it sounds silly, but it would be very tedious and there must be a way to just update the webhook for affected repos? Can we at least get a description of what the issue was?

Thanks for your understanding; I'm sorry if I sound frustrated but this has been really hurting for a month now so the abrupt closure of this issue is a bit disheartening. @fnetX

Am I to understand that this *won't* be fixed? I really don't want to go through every single repo (there are a lot of them) and hit the repair button. I know it sounds silly, but it would be very tedious and there must be a way to just update the webhook for affected repos? Can we at least get a description of what the issue was? Thanks for your understanding; I'm sorry if I sound frustrated but this has been really hurting for a month now so the abrupt closure of this issue is a bit disheartening. @fnetX

fnetX commented

2023-11-25 13:39:23 +00:00

Owner

Pinging @pat-s and @6543 again who are maintaining the Woodpecker CI. It is probably tracked elsewhere though …

Andre601 commented

2023-11-25 14:23:21 +00:00

Poster

Am I to understand that this won't be fixed? I really don't want to go through every single repo (there are a lot of them) and hit the repair button. I know it sounds silly, but it would be very tedious and there must be a way to just update the webhook for affected repos? Can we at least get a description of what the issue was?

My personal guess is, that when Codeberg updated their CI/infrastructure, it caused the currently used webhooks, or probs more the tokens, to expire and no longer be valid, since the webhook and webhook secret contain a token for validation.

> Am I to understand that this *won't* be fixed? I really don't want to go through every single repo (there are a lot of them) and hit the repair button. I know it sounds silly, but it would be very tedious and there must be a way to just update the webhook for affected repos? Can we at least get a description of what the issue was? My personal guess is, that when Codeberg updated their CI/infrastructure, it caused the currently used webhooks, or probs more the tokens, to expire and no longer be valid, since the webhook and webhook secret contain a token for validation.

pat-s commented

2023-11-25 16:28:01 +00:00

I've hit the "repair all" button again on the admin interface which again failed. Unfortunately, I don't have access to the VM and the logs yet, so I can't debug further.

But maybe this already helped for some repos? (I don't know whether if does some partial work or fails for all right from the start).

I've hit the "repair all" button again on the admin interface which again failed. Unfortunately, I don't have access to the VM and the logs yet, so I can't debug further. But maybe this already helped for some repos? (I don't know whether if does some partial work or fails for all right from the start).

Codeberg CI not executing any tasks for the past days #1316

Comment