Codeberg
/
Community
56
226
Issues 283 Activity

tasks not executed on push (email, notificatons, webhooks, user activity list etc) #1330

New Issue
Open
opened 2023-11-05 16:51:28 +00:00 by derat · 9 comments
derat commented 2023-11-05 16:51:28 +00:00

Comment

I just observed a webhook failing to execute in response to a push to https://codeberg.org/derat/nup.

https://codeberg.org/derat/nup/settings/hooks/3653 is configured to run in response to Create, Delete, and Push events.

The latest delivery listed at the bottom of the page is 8e5dac62-9c3c-4275-b941-8b9874cbeb94 at 2023-11-04 18:58 UTC, triggered by commit 49c5e94c8f6b82bdb045906d693c3b316e61be9a.

I pushed commit 898500c4cc016bba154563fbe91b0d0f854194ca to the repo at 2023-11-05 16:12 UTC (30 minutes ago), but as far as I can tell, the webhook did not execute.

I haven't made any recent changes to this repository's settings and don't think I've seen the webhook fail to execute before. I haven't yet tried pushing another commit or clicking the "Test Delivery" button to see if this was a spurious failure.

#1308 may be related, but it's unclear to me whether the incidents described there involved webhooks not running at all or whether they were executed but failed for some other reason.

### Comment I just observed a webhook failing to execute in response to a push to https://codeberg.org/derat/nup. https://codeberg.org/derat/nup/settings/hooks/3653 is configured to run in response to Create, Delete, and Push events. The latest delivery listed at the bottom of the page is 8e5dac62-9c3c-4275-b941-8b9874cbeb94 at 2023-11-04 18:58 UTC, triggered by commit 49c5e94c8f6b82bdb045906d693c3b316e61be9a. I pushed commit 898500c4cc016bba154563fbe91b0d0f854194ca to the repo at 2023-11-05 16:12 UTC (30 minutes ago), but as far as I can tell, the webhook did not execute. I haven't made any recent changes to this repository's settings and don't think I've seen the webhook fail to execute before. I haven't yet tried pushing another commit or clicking the "Test Delivery" button to see if this was a spurious failure. #1308 may be related, but it's unclear to me whether the incidents described there involved webhooks not running at all or whether they were executed but failed for some other reason.
👍 1
derat commented 2023-11-05 23:46:08 +00:00
Poster

I just pushed another commit (52e0ec03e8517fa11bd4e82eecc214675886668d at 2023-11-05 23:39 UTC) and the webhook again did not execute.

When I triggered the webhook manually using the "Test Delivery" button, it executed successfully (delivery a6184864-3282-4891-b8a7-89a4581668a3).

I just pushed another commit (52e0ec03e8517fa11bd4e82eecc214675886668d at 2023-11-05 23:39 UTC) and the webhook again did not execute. When I triggered the webhook manually using the "Test Delivery" button, it executed successfully (delivery a6184864-3282-4891-b8a7-89a4581668a3).
Aetherum commented 2023-11-06 02:35:42 +00:00

Hey) I think I am experiencing the same issue with Discord webhooks. The webhooks worked perfectly yesterday, but now nothing is sent to Discord. Even in the "Recent Deliveries", the last message is from yesterday and no mentions of today's commits.

Hey) I think I am experiencing the same issue with Discord webhooks. The webhooks worked perfectly yesterday, but now nothing is sent to Discord. Even in the "Recent Deliveries", the last message is from yesterday and no mentions of today's commits.
derat commented 2023-11-06 11:36:53 +00:00
Poster

The issue doesn't seem to happen consistently. This morning, my hook ran (delivery 52225a20-19f3-4461-b0ef-97cd340249ad) in response to my commit 5ca3ad20d52d34abc13e77ec02e377312a96e616.

The issue doesn't seem to happen consistently. This morning, my hook ran (delivery 52225a20-19f3-4461-b0ef-97cd340249ad) in response to my commit 5ca3ad20d52d34abc13e77ec02e377312a96e616.
fnetX commented 2023-11-06 14:28:04 +00:00
Owner

@derat Can you confirm that your activity view at https://codeberg.org/derat?tab=activity is also missing the activity? I cannot find commit 52e0ec03e8517fa11bd4e82eecc214675886668d from about 12 hours ago 2023-11-05 23:39 UTC.

@Andre601 can you also check your activity log by chance if you can find missing activity? Since we haven't found logs, any pointers to where a failure in the code occurs will be helpful.

@derat Can you confirm that your activity view at https://codeberg.org/derat?tab=activity is also missing the activity? I cannot find commit `52e0ec03e8517fa11bd4e82eecc214675886668d` from about 12 hours ago 2023-11-05 23:39 UTC. @Andre601 can you also check your activity log by chance if you can find missing activity? Since we haven't found logs, any pointers to where a failure in the code occurs will be helpful.
derat commented 2023-11-06 15:12:33 +00:00
Poster

@fnetX Thanks for taking a look! Yes, these two pushes are missing from my activity log.

Here's what I see when I print the remote reflog as described at https://stackoverflow.com/a/48086403:

% TZ=UTC git reflog show origin/main --pretty='%h %gd %gs' --date=iso-local
a43311b origin/main@{2023-11-06 11:47:32 +0000} update by push
5ca3ad2 origin/main@{2023-11-06 10:55:08 +0000} update by push
52e0ec0 origin/main@{2023-11-05 23:40:43 +0000} update by push
898500c origin/main@{2023-11-05 16:15:19 +0000} update by push
49c5e94 origin/main@{2023-11-04 18:58:44 +0000} update by push
...

The activity log is missing 52e0ec0 and 898500c but includes newer and older pushes. I looked back to 2023-08-29 but didn't find any other missing pushes.

@fnetX Thanks for taking a look! Yes, these two pushes are missing from my activity log. Here's what I see when I print the remote reflog as described at https://stackoverflow.com/a/48086403: ``` % TZ=UTC git reflog show origin/main --pretty='%h %gd %gs' --date=iso-local a43311b origin/main@{2023-11-06 11:47:32 +0000} update by push 5ca3ad2 origin/main@{2023-11-06 10:55:08 +0000} update by push 52e0ec0 origin/main@{2023-11-05 23:40:43 +0000} update by push 898500c origin/main@{2023-11-05 16:15:19 +0000} update by push 49c5e94 origin/main@{2023-11-04 18:58:44 +0000} update by push ... ``` The activity log is missing `52e0ec0` and `898500c` but includes newer and older pushes. I looked back to 2023-08-29 but didn't find any other missing pushes.
Andre601 commented 2023-11-06 16:04:33 +00:00

@Andre601 can you also check your activity log by chance if you can find missing activity? Since we haven't found logs, any pointers to where a failure in the code occurs will be helpful.

The PR merge itself is listed in my activity tab. However, my notifications-tab does not list the PR in question. Only the PR I in received a mail for in another repository.

Activity tab:
grafik

Notifications (bottom one is last week to show that there is no notification for blog-source):
grafik

Also, yes I do follow the repo since it's my own:
grafik

> @Andre601 can you also check your activity log by chance if you can find missing activity? Since we haven't found logs, any pointers to where a failure in the code occurs will be helpful. The PR merge itself is listed in my activity tab. However, my notifications-tab does not list the PR in question. Only the PR I in received a mail for in another repository. Activity tab: ![grafik](/attachments/1befd480-9bf3-4f9b-a3da-643be5b19d0e) Notifications (bottom one is last week to show that there is no notification for blog-source): ![grafik](/attachments/d8a57ee6-6b30-4b45-b558-a6f7daf6518e) Also, yes I do follow the repo since it's my own: ![grafik](/attachments/248947bb-c668-4305-b883-d14eac68d918)
grafik.png
34 KiB
grafik.png
30 KiB
grafik.png
7.6 KiB
Andre601 commented 2023-11-06 16:14:09 +00:00

Also, here are my last commit activities on my blog-source repository from the past 2 days based on @derat's aproach:

786a380 origin/master@{2023-11-06 13:52:02 +0000} update by push
def7111 origin/master@{2023-11-06 12:39:54 +0000} pull: fast-forward
6585b93 origin/master@{2023-11-05 22:44:44 +0000} update by push
a0962c3 origin/master@{2023-11-05 22:44:33 +0000} pull: fast-forward

Comparing it with my activities tab, the last two commits (6585b93 and a0962c3) don't appear in the activity tab... Tho I don't know how accurate this is? Because coincidentally are there commits listed in the activity tab that aren't listed by the log command used here, namely fb261ce615 and f865873032 which both are part of a PR (One committing the change, the other from the PR merge itself)... So idk if that log can be fully trusted here...

Also, here are my last commit activities on my blog-source repository from the past 2 days based on @derat's aproach: ``` 786a380 origin/master@{2023-11-06 13:52:02 +0000} update by push def7111 origin/master@{2023-11-06 12:39:54 +0000} pull: fast-forward 6585b93 origin/master@{2023-11-05 22:44:44 +0000} update by push a0962c3 origin/master@{2023-11-05 22:44:33 +0000} pull: fast-forward ``` Comparing it with my activities tab, the last two commits (`6585b93` and `a0962c3`) don't appear in the activity tab... Tho I don't know how accurate this is? Because coincidentally are there commits listed in the activity tab that aren't listed by the log command used here, namely `fb261ce615` and `f865873032` which both are part of a PR (One committing the change, the other from the PR merge itself)... So idk if that log can be fully trusted here...
fnetX commented 2023-11-06 22:34:48 +00:00
Owner

Relevant:

2023/11/05 21:33:40 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /api/internal/hook/post-receive/shuriken/pages for [::1]:59850, elapsed 3930.6ms @ private/hook_post_receive.go:25(private.HookPostReceive)
2023/11/05 21:33:41 ...hook_post_receive.go:79:HookPostReceive() [E] Failed to Update: shuriken/pages Total Updates: 1
2023/11/05 21:33:41 ...hook_post_receive.go:81:HookPostReceive() [E] Failed to Update: shuriken/pages Update: 0/1: Branch: main
2023/11/05 21:33:41 ...hook_post_receive.go:83:HookPostReceive() [E] Failed to Update: shuriken/pages Error: context deadline exceeded
2023/11/05 21:33:41 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/internal/hook/post-receive/shuriken/pages for [::1]:59850, 500 Internal Server Error in 5000.9ms @ private/hook_post_receive.go:25(private.HookPostReceive)

https://matrix.to/#/%23codeberg-ci%3Aobermui.de/%244jFfAdaLMvugMTpf-WrXczo-jkQxfPBtN13Gg0YMNas?via=matrix.tu-berlin.de&via=matrix.org&via=fsociety.social&via=catgirl.cloud

Relevant: ~~~ 2023/11/05 21:33:40 ...eb/routing/logger.go:68:func1() [W] router: slow POST /api/internal/hook/post-receive/shuriken/pages for [::1]:59850, elapsed 3930.6ms @ private/hook_post_receive.go:25(private.HookPostReceive) 2023/11/05 21:33:41 ...hook_post_receive.go:79:HookPostReceive() [E] Failed to Update: shuriken/pages Total Updates: 1 2023/11/05 21:33:41 ...hook_post_receive.go:81:HookPostReceive() [E] Failed to Update: shuriken/pages Update: 0/1: Branch: main 2023/11/05 21:33:41 ...hook_post_receive.go:83:HookPostReceive() [E] Failed to Update: shuriken/pages Error: context deadline exceeded 2023/11/05 21:33:41 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/internal/hook/post-receive/shuriken/pages for [::1]:59850, 500 Internal Server Error in 5000.9ms @ private/hook_post_receive.go:25(private.HookPostReceive) ~~~ https://matrix.to/#/%23codeberg-ci%3Aobermui.de/%244jFfAdaLMvugMTpf-WrXczo-jkQxfPBtN13Gg0YMNas?via=matrix.tu-berlin.de&via=matrix.org&via=fsociety.social&via=catgirl.cloud
fnetX changed title from Webhook didn't execute in response to push event to tasks not executed on push (email, notificatons, webhooks, user activity list etc) 2023-11-06 22:35:49 +00:00
fnetX commented 2023-11-06 22:41:50 +00:00
Owner

I can confirm the same occured for the two commits mentioned in #1330 (comment)

I can confirm the same occured for the two commits mentioned in https://codeberg.org/Codeberg/Community/issues/1330#issuecomment-1323407
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
Labels
Clear labels   
bug

Something is not working   
Codeberg

This issue is specific to Codeberg's downstream modifications or settings and must be addressed here   
contribution welcome

Please join the discussion and consider contributing a PR!   
docs

No bug, but an improvement to the docs or UI description will help   
duplicate

This issue or pull request already exists   
enhancement

New feature   
infrastructure

Involves changes to the server setups   
legal

An issue directly involving legal compliance   
licence / ToS

involving questions about the ToS, especially licencing compliance   
public relations

Things related to Codeberg's external communication   
question

More information is needed   
s/Gitea/Forgejo

Related to Forgejo or Gitea. Please also check Gitea's issue tracker   
s/Pages

Issues related to the Codeberg Pages feature   
s/Weblate

Issue is related to the Weblate instance at https://translate.codeberg.org   
s/Woodpecker

Woodpecker CI related issue   
security

involves improvements to the sites security   
service

Add a new service to the Codeberg ecosystem (instead of implementing into Gitea)   
spam
   
upstream

This issue is reported upstream   
wontfix

This won't be fixed
No Label
bug
Codeberg
contribution welcome
docs
duplicate
enhancement
infrastructure
legal
licence / ToS
public relations
question
s/Gitea/Forgejo
s/Pages
s/Weblate
s/Woodpecker
security
service
spam
upstream
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Codeberg/Community#1330
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?