New Issue

[CI] inconsitent tagging of releases #344

Closed

opened by dachary · 17 comments

dachary commented

Owner

The tag is no set to the right commit (8e8037a64d), it should be at 4e5be58493 which is where it was copied from.

The content of the files are correct: they are identical to those found in https://codeberg.org/forgejo-integration/forgejo/releases/tag/v1.18.3-1

The following releases show the same inconsistencies:

The SHA they must be set to are found in the integration repository:

The recommended action to fix past releases is to force push the tag with the SHA found in the integration organization.

![image](/attachments/3107b815-98ba-4732-9855-24c8498a7e87) The tag is no set to the right commit (8e8037a64d), it should be at 4e5be58493 which is where it was copied from. ![image](/attachments/6cc79919-e7c8-4847-b5a4-5d237a2c8f9d) The content of the files are correct: they are identical to those found in https://codeberg.org/forgejo-integration/forgejo/releases/tag/v1.18.3-1 The following [releases](https://codeberg.org/forgejo/forgejo/releases) show the same inconsistencies: * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.1-0 * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.2-0 * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.2-1 * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.3-0 * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.3-1 The SHA they must be set to are found in the [integration repository](https://codeberg.org/forgejo-integration/forgejo/releases): * https://codeberg.org/forgejo-integration/forgejo/releases/tag/v1.18.1-0 * https://codeberg.org/forgejo-integration/forgejo/releases/tag/v1.18.2-0 * https://codeberg.org/forgejo-integration/forgejo/releases/tag/v1.18.2-1 * https://codeberg.org/forgejo-integration/forgejo/releases/tag/v1.18.3-0 * https://codeberg.org/forgejo-integration/forgejo/releases/tag/v1.18.3-1 The recommended action to fix past releases is to force push the tag with the SHA found in the integration organization.

image.png

83 KiB

image.png

81 KiB

dachary added this to the Forgejo v1.18.3-1 milestone

dachary added the

bug

forgejo/release

labels

dachary self-assigned this

dachary commented

Poster

Owner

The solution seems to be force pushing the tag to have the value it should have.

I tried to force push a tag on an existing release on the forgejo-integration repository and check if that fixes it
https://codeberg.org/forgejo-integration/forgejo/releases/tag/v200.0.0-3 currently is at 695f6f9ce0.

And now the tag v200.0.0-3 is at a055570b60 and the files from the release can be downloaded. The same could be done for the release 1.18.3-1.

The question is... how did that happen?

The solution seems to be force pushing the tag to have the value it should have. I tried to force push a tag on an existing release on the forgejo-integration repository and check if that fixes it https://codeberg.org/forgejo-integration/forgejo/releases/tag/v200.0.0-3 currently is at 695f6f9ce0. And now the tag v200.0.0-3 is at a055570b601664c3a5f7243f20bb33f211523687 and the files from the release can be downloaded. The same could be done for the release 1.18.3-1. The question is... how did that happen?

dachary changed title from ~~[RELEASE] inconsitent tag for 1.18.3-1~~ to [CI] inconsitent tagging of releases

dachary commented

Poster

Owner

So the solution to this bug it to set the tag prior to uploading. It will not trigger any CI job because the experimental & the forgejo main repositories are not set to run on tags.

https://codeberg.org/forgejo/forgejo/src/commit/forgejo/releases/binaries-pull-push.sh is the script that copies the binary files from integration to forgejo And when it does here the tag is implicitly set as a side effect of creating the release. I have no clue where and how it picks the commit to set the tag with. But it does pick one. So the solution to this bug it to set the tag prior to uploading. It will not trigger any CI job because the experimental & the forgejo main repositories are not set to run on tags.

crystal commented

Owner

I'm pretty sure it's just using the current HEAD of the so-called "default" branch as defined in the branches settings. We should create the tag at the HEAD that we know was used to build the binaries before uploading the release.

dachary commented

Poster

Owner

Working theory at this point: if the API is required to create a release given a tag that does not exist, it will create one with the tip of the default branch. Creating the tag before uploading the release should fix the problem.

crystal commented

Owner

Does the tag actually get pushed to Codeberg before the release is created? It looks like the tags in forgejo-integration are correct, for the record.

Does the tag actually get pushed to Codeberg before the release is created? [It looks like the tags in forgejo-integration are correct](https://codeberg.org/forgejo-integration/forgejo/releases), for the record.

dachary commented

Poster

Owner

No, the tag is not pushed, it is a bug in the release scripts.

dachary referenced this issue

[CI] the tag SHA in the uploaded repository must match #345

dachary commented

Poster

Owner

I force pushed https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.1-0 which is now at dd1486af80 as it should have been. I also updated the description of the issue with a detailed and hopefully exhaustive description of the problem and the recommended action. There also is a PR expecting review to fix the root cause.

Before I proceed with force pushing the remaining tags, I'll wait for review.

@crystal @Gusted ?

I force pushed https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.1-0 which is now at dd1486af80 as it should have been. I also updated the description of the issue with a detailed and hopefully exhaustive description of the problem and the recommended action. There also is a PR expecting review to fix the root cause. Before I proceed with force pushing the remaining tags, I'll wait for review. @crystal @Gusted ?

dachary commented

Poster

Owner

There has been no observable negative side effect of force pushing the tag for v1.18.1-0 so I force pushed the others. The current situation is wrong anyway so I felt it was best to not wait for too long: * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.1-0 => dd1486af80 * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.2-0 => 1df08aab97fffc6b7bd696b03c1a08b8d644425f * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.2-1 => 480a0244914c872588da8df6a60386ecb2baff6a * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.3-0 => 7fa34500c9a000ffa20a21d0b63dd5017053aef7 * https://codeberg.org/forgejo/forgejo/releases/tag/v1.18.3-1 => 4e5be584934eef53e5529112075a9c8b70f88fae

👍 1

dachary closed this issue

dachary commented

Poster

Owner

I'm inclined to think this issue is information enough in case someone ran into the same problem before it was fixed. Or if someone wonders why the SHA of the tags moved.

If you think this should be more prominently advertised, please speak up.

I'm inclined to think this issue is information enough in case someone ran into the same problem before it was fixed. Or if someone wonders why the SHA of the tags moved. If you think this should be more prominently advertised, please speak up.

👍 1

fsologureng commented

Just to thank @crystal for discovering this bug

❤️ 2

crystal commented

Owner

It has come to my attention (thanks to tomgus1 in the Forgejo Matrix Space, which everyone should join btw), that this issue caused Arch Linux to distribute defective Forgejo packages for a few weeks, which are now preventing users who took the upgrade to the latest (fixed) package from starting the service because they're technically downgrading. I suppose the next step is to get in touch with the Arch team so they can publicize the problem and propose solutions for their users.

👍 1

dachary reopened this issue

dachary commented

Poster

Owner

Re-opening to work on how to properly communicate this problem and potential impact.

👍 1

dachary commented

Poster

Owner

https://floss.social/@forgejo/109854218136296565

You can skip this warning if you run a Forgejo instance using the binaries or container images downloaded from https://codeberg.org/forgejo/

⚠️ The tags on the Forgejo repository for v1.18.1-0, v1.18.2-0, v1.18.2-1, v1.18.3-0 and v1.18.3-1 were incorrectly set to the v1.19 development branch. The problem was fixed 11 February 2023 ~7am UTC.

If you built Forgejo from sources or got a package built by a third party using these tags, you are running v1.19.

A blog post will follow.

https://floss.social/@forgejo/109854218136296565 > You can skip this warning if you run a Forgejo instance using the binaries or container images downloaded from https://codeberg.org/forgejo/ > > ⚠️ The tags on the Forgejo repository for v1.18.1-0, v1.18.2-0, v1.18.2-1, v1.18.3-0 and v1.18.3-1 were incorrectly set to the v1.19 development branch. The problem was fixed 11 February 2023 ~7am UTC. > > If you built Forgejo from sources or got a package built by a third party using these tags, you are running v1.19. > > A blog post will follow.

dachary commented

Poster

Owner

Compiled by @crystal

Manjaro will be affected, unless they opt to keep the old broken package in the repo until v1.19 is available.
the ynh package downloads the binary so it's unaffected
The NixOS package actually does use downloaded sources, so it is also unaffected.
The Helm chart uses the docker image...
The Gentoo repo hasn't been updated to an affected version
Nor has the FreeBSD ports tree patch
The APKBUILD downloads the sources from https://codeberg.org/forgejo/forgejo/releases/download/v1.18.3-1/forgejo-src-1.18.3-1.tar.gz,

Compiled by @crystal * Manjaro will be affected, unless they opt to keep the old broken package in the repo until v1.19 is available. * the ynh package downloads the binary so it's unaffected * The NixOS package actually does use downloaded sources, so it is also unaffected. * The Helm chart uses the docker image... * The Gentoo repo hasn't been updated to an affected version * Nor has the FreeBSD ports tree patch * The APKBUILD downloads the sources from https://codeberg.org/forgejo/forgejo/releases/download/v1.18.3-1/forgejo-src-1.18.3-1.tar.gz,

👍 1

dachary commented

Poster

Owner

Associated blog post at forgejo/website#95

Associated blog post at https://codeberg.org/forgejo/website/pulls/95

crystal commented

Owner

I merged the blog post as I thought it would be good to get it out before a large number of users start experiencing issues.

👍 1

dachary commented

Poster