plans for automatic "release" uploads or "pages" builds? #3

Open
opened 1 year ago by ccoenen · 12 comments
ccoenen commented 1 year ago

currently, I do not see a way to get anything out of woodpecker easily. I saw all kinds of setups where people rsynced their stuff out of the containers, but that setup is tedious, and you also need to store a private key in woodpecker.

Two use cases that will probably come up often are:

  • build a static site from a static site generator and host the result on codeberg pages
  • build a binary and host the result from the project's releases page

both of which are common enough that there should be a "short" way to do it.

Is something like this in the works? (or perhaps even finished, but I just missed it?)

currently, I do not see a way to get anything out of woodpecker _easily_. I saw all kinds of setups where people `rsynced` their stuff out of the containers, but that setup is tedious, and you also need to store a private key in woodpecker. Two use cases that will probably come up often are: - build a static site from a static site generator and host the result on codeberg pages - build a binary and host the result from the project's releases page both of which are common enough that there should be a "short" way to do it. Is something like this in the works? (or perhaps even finished, but I just missed it?)
ccoenen changed title from plans for automatic "release" uploads? to plans for automatic "release" uploads or "pages" builds? 1 year ago

I also feel the need for such a thing. Therefore I asked for info on the fediverse which got boosted and favoured but no hints so far:
https://hofra.rocks/web/statuses/106665478927100781

I also feel the need for such a thing. Therefore I asked for info on the fediverse which got boosted and favoured but no hints so far: https://hofra.rocks/web/statuses/106665478927100781

There are plugins for drone, that might work: http://plugins.drone.io/

Upstream (Woodpecker) has to decide how to handle plugins. It might not be the best idea to depend on drone plugins.

https://woodpecker-ci.github.io/docs/usage/plugins/plugins

There are plugins for drone, that might work: http://plugins.drone.io/ - http://plugins.drone.io/appleboy/drone-git-push/ - http://plugins.drone.io/drone-plugins/drone-gitea-release/ Upstream (Woodpecker) has to decide how to handle plugins. It might not be the best idea to depend on drone plugins. https://woodpecker-ci.github.io/docs/usage/plugins/plugins
Poster

I was also referred to
https://github.com/woodpecker-ci/woodpecker/issues/335 by @6543. I will take a look at this sometime in the future. But I thought it would be good to collect all the info in one ticket for now for easier reference.

I was also referred to https://github.com/woodpecker-ci/woodpecker/issues/335 by @6543. I will take a look at this sometime in the future. But I thought it would be good to collect all the info in one ticket for now for easier reference.
Owner

yes this will be handled as now by plugins - but eventualy with a better ui ...

also ref to plugin "market place" https://github.com/woodpecker-ci/woodpecker/issues/315

yes this will be handled as now by plugins - but eventualy with a better ui ... also ref to plugin "market place" https://github.com/woodpecker-ci/woodpecker/issues/315
6543 added the
Upstream
Documentation
labels 1 year ago
  • build a static site from a static site generator and host the result on codeberg pages

I did something like that with Drone CI before: Codeberg/Community#498

The issue has links to Hexo, Hugo and Jekyll as examples. They work basically with appleboy/drone-git-push to push the generated static site content back to repo and has some tricks to stop seeing Drone commits as changes so that it doesn't run the build again. This worked perfectly with Pages v1, so I assume it should also work with v2. .drone.yml on those projects has the details.

I'm trying to port these to woodpecker and Pages v2, but I'm getting:

...
+ git remote add deploy git@codeberg.org:hexo-test/pages.git
+ git push deploy HEAD:pages
load pubkey "/root/.ssh/id_rsa": invalid format
Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts.
Load key "/root/.ssh/id_rsa": invalid format
git@codeberg.org: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
exit status 128 

I have something like this in .woodpecker.yml:

...
  push_commit:
    image: appleboy/drone-git-push
    settings:
      branch: pages
      remote: git@codeberg.org:hexo-test/pages.git
      force: false
      commit: true
      commit_message: "Woodpecker build ${CI_COMMIT_SHA:0:7}"
      author_name: "Adnan ___"
      author_email: "adnan360@___.com"
    secrets:
      - source: ssh_key
        target: plugin_ssh_key
    # To show green check in case previous step exits.
    when:
      status:
        - success

I tried to store the ssh private key as secrect on Woodpecker UI (with new lines replaced with \n by using cat ~/.ssh/id_adnan360_codeberg_ci | awk '{printf "%s\\n", $0}') as both ssh_key and SSH_KEY but failed. I used ssh-keygen -m PEM -t rsa -P "" -C "adnan360@___.com" -f ~/.ssh/id_adnan360_codeberg_ci to generate the key because in case of Drone it only worked if I did it like this. Put the public key in Settings SSH / GPG keys. The key shows "No recent activity" under it. Which is a bit odd.

I can't cat to see what's inside /root/.ssh/id_rsa because Woodpecker doesn't let me mix settings and commands. Not sure what's wrong with my setup.

> - build a static site from a static site generator and host the result on codeberg pages I did something like that with Drone CI before: https://codeberg.org/Codeberg/Community/issues/498 The issue has links to Hexo, Hugo and Jekyll as examples. They work basically with `appleboy/drone-git-push` to push the generated static site content back to repo and has some tricks to stop seeing Drone commits as changes so that it doesn't run the build again. This worked perfectly with Pages v1, so I assume it should also work with v2. .drone.yml on those projects has the details. I'm trying to port these to woodpecker and Pages v2, but I'm getting: ``` ... + git remote add deploy git@codeberg.org:hexo-test/pages.git + git push deploy HEAD:pages load pubkey "/root/.ssh/id_rsa": invalid format Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts. Load key "/root/.ssh/id_rsa": invalid format git@codeberg.org: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. exit status 128 ``` I have something like this in `.woodpecker.yml`: ``` ... push_commit: image: appleboy/drone-git-push settings: branch: pages remote: git@codeberg.org:hexo-test/pages.git force: false commit: true commit_message: "Woodpecker build ${CI_COMMIT_SHA:0:7}" author_name: "Adnan ___" author_email: "adnan360@___.com" secrets: - source: ssh_key target: plugin_ssh_key # To show green check in case previous step exits. when: status: - success ``` I tried to store the ssh private key as secrect on Woodpecker UI (with new lines replaced with `\n` by using `cat ~/.ssh/id_adnan360_codeberg_ci | awk '{printf "%s\\n", $0}'`) as [both](https://github.com/woodpecker-ci/woodpecker/issues/466#issuecomment-962934933) `ssh_key` and `SSH_KEY` but failed. I used `ssh-keygen -m PEM -t rsa -P "" -C "adnan360@___.com" -f ~/.ssh/id_adnan360_codeberg_ci` to generate the key because in case of Drone it only worked if I did it like this. Put the public key in Settings SSH / GPG keys. The key shows "No recent activity" under it. Which is a bit odd. I can't `cat` to see what's inside `/root/.ssh/id_rsa` because Woodpecker doesn't let me mix settings and commands. Not sure what's wrong with my setup.

When I logged in to Woodpecker today, I noticed the secret value input field is multiline now. I deleted the old ones and created the secret again (with name ssh_key). This time without escaping with \n, and it seems to be working now.

...
+ git remote add deploy git@codeberg.org:hexo-test/pages.git
+ git push deploy HEAD:pages --force
load pubkey "/root/.ssh/id_rsa": invalid format
Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts.
remote:
remote: Create a new pull request for 'pages':
remote: https://codeberg.org/hexo-test/pages/compare/main...pages
remote:
remote: . Processing 1 references 
...

The issue seems to have been fixed.

EDIT: Include secret name

When I logged in to Woodpecker today, I noticed the secret value input field is multiline now. I deleted the old ones and created the secret again (with name `ssh_key`). This time without escaping with `\n`, and it seems to be working now. ``` ... + git remote add deploy git@codeberg.org:hexo-test/pages.git + git push deploy HEAD:pages --force load pubkey "/root/.ssh/id_rsa": invalid format Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts. remote: remote: Create a new pull request for 'pages': remote: https://codeberg.org/hexo-test/pages/compare/main...pages remote: remote: . Processing 1 references ... ``` The issue seems to have been fixed. EDIT: Include secret name
Collaborator

FYI, Codeberg/pages-server#35 contains mostly information relevant for this issue as well, it seems like it was a duplicate in the wrong place.

FYI, https://codeberg.org/Codeberg/pages-server/issues/35 contains mostly information relevant for this issue as well, it seems like it was a duplicate in the wrong place.

build a binary and host the result from the project's releases page

I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher)

It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20

> build a binary and host the result from the project's releases page I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher) It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20

build a binary and host the result from the project's releases page

I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher)

It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20

This is cool.

I am not yet convinced of the plugin idea, because if the plugin does not work (for me), i need (YET ANOTHER!) account (this time on docker hub) to publish the plugin, if i understand this correctly.

I have the same thing for releases although we are not using it now and push to a repo. The release publisher just uses curl and the gitea api (most likele like you)... remove old release:
https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/remove_old_nightly.sh

attach to a release: https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/publish_nightly.sh

> > build a binary and host the result from the project's releases page > > I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher) > > It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20 This is cool. I am not yet convinced of the plugin idea, because if the plugin does not work (for me), i need (YET ANOTHER!) account (this time on docker hub) to publish the plugin, if i understand this correctly. I have the same thing for releases although we are not using it now and push to a repo. The release publisher just uses curl and the gitea api (most likele like you)... remove old release: https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/remove_old_nightly.sh attach to a release: https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/publish_nightly.sh
Collaborator

I mean, ideally such plugins can become officially supported for Woodpecker itself. I would love to have an official plugin or docker container for the tea command, then a command like tea releases create --tag "v1.0" --title v1.0 --target main --asset whatever.bin could be used for the releases and we don't have to reinvent the wheel for each task there could be with Gitea.

I mean, ideally such plugins can become officially supported for Woodpecker itself. I would love to have an official plugin or docker container for the [`tea`](https://gitea.com/gitea/tea) command, then a command like `tea releases create --tag "v1.0" --title v1.0 --target main --asset whatever.bin` could be used for the releases and we don't have to reinvent the wheel for each task there could be with Gitea.

Not sure if this is helpful for anyone else, but I ended up doing the following:

  • Create .tar.gz archive in a create-archive step in Woodpecker
  • Call shellscript that creates a new release in Codeberg + uploads the tar as an attachment
  • Have local shellscript for doing deploy (obviously optional, mainly for services)

Details look something like this:

Relevant woodpecker definition for create-archive and create-release:

   create-archive:
    image: debian
    when:
      branch: master
      event: push
    commands:
      - apt-get update && apt-get install --yes binutils upx
      - export RELEASE_DIR=ditzes-$CI_COMMIT_SHA
      - mkdir $RELEASE_DIR
      - cp src-tauri/target/release/api $RELEASE_DIR/ditzes-api
      - cp src-tauri/target/release/cli $RELEASE_DIR/ditzes-cli
      - strip $RELEASE_DIR/ditzes-api
      - upx $RELEASE_DIR/ditzes-api
      - strip $RELEASE_DIR/ditzes-cli
      - upx $RELEASE_DIR/ditzes-cli
      - tar cvzf ditzes-$CI_COMMIT_SHA-linux.tar.gz $RELEASE_DIR
  create-release:
    image: debian
    when:
      branch: master
      event: push
    secrets: [ API_TOKEN ]
    commands:
      - apt-get update && apt-get --yes install jq curl
      - export RELEASE_TAR=ditzes-$CI_COMMIT_SHA-linux.tar.gz
      - ./codeberg-release.sh

6063dc53c6/.woodpecker.yml (L10-L35)

codeberg-release.sh is basically two curl calls (with API_TOKEN stored as a secret via the Woodpecker UI): 106405d0fa/codeberg-release.sh

deploy-release.sh can be called from any instance that has access to the instance you want to deploy to, with the commit as the argument. Downloads the archive, scps the files over and restarts the service: 106405d0fa/deploy-release.sh

A bit hacky solution, but works for my simple purposes.

Not sure if this is helpful for anyone else, but I ended up doing the following: - Create .tar.gz archive in a `create-archive` step in Woodpecker - Call shellscript that creates a new release in Codeberg + uploads the tar as an attachment - Have local shellscript for doing deploy (obviously optional, mainly for services) Details look something like this: Relevant woodpecker definition for `create-archive` and `create-release`: ``` create-archive: image: debian when: branch: master event: push commands: - apt-get update && apt-get install --yes binutils upx - export RELEASE_DIR=ditzes-$CI_COMMIT_SHA - mkdir $RELEASE_DIR - cp src-tauri/target/release/api $RELEASE_DIR/ditzes-api - cp src-tauri/target/release/cli $RELEASE_DIR/ditzes-cli - strip $RELEASE_DIR/ditzes-api - upx $RELEASE_DIR/ditzes-api - strip $RELEASE_DIR/ditzes-cli - upx $RELEASE_DIR/ditzes-cli - tar cvzf ditzes-$CI_COMMIT_SHA-linux.tar.gz $RELEASE_DIR create-release: image: debian when: branch: master event: push secrets: [ API_TOKEN ] commands: - apt-get update && apt-get --yes install jq curl - export RELEASE_TAR=ditzes-$CI_COMMIT_SHA-linux.tar.gz - ./codeberg-release.sh ``` https://codeberg.org/CapableWeb/ditzes/src/commit/6063dc53c61ffc2a73b2b20ceac678987500003a/.woodpecker.yml#L10-L35 **codeberg-release.sh** is basically two curl calls (with API_TOKEN stored as a secret via the Woodpecker UI): https://codeberg.org/CapableWeb/ditzes/src/commit/106405d0fa7893ab4e574e3dbacb8670584ed0f8/codeberg-release.sh *deploy-release.sh* can be called from any instance that has access to the instance you want to deploy to, with the commit as the argument. Downloads the archive, scps the files over and restarts the service: https://codeberg.org/CapableWeb/ditzes/src/commit/106405d0fa7893ab4e574e3dbacb8670584ed0f8/deploy-release.sh A bit hacky solution, but works for my simple purposes.

I faced some problems while setting up SSR deploy to codeberg-pages with codeberg-ci and trying to replicate what I used to have with gh-pages. Reading here helped but also made a bit of confusion at some point. At the end is working like a charm, so I'm here just to give a full positive feedback and clarify it's easily doable.

The major issue was solved by using two separate repositories (source and target like it's suggested in codeberg-pages docs) and then I was able to use the image appleboy/drone-git-push straightforwardly.

    deploy:
      image: appleboy/drone-git-push
      secrets:
        - git_ssh_key
      settings:
        remote_name: origin-git
        remote: git@codeberg.org:username/pages.git
        branch: main
        commit: true
        commit_message: '[skip ci] publish to pages'
        ssh_key:
          from_secret: git_ssh_key
I faced some problems while setting up SSR deploy to codeberg-pages with codeberg-ci and trying to replicate what I used to have with gh-pages. Reading here helped but also made a bit of confusion at some point. At the end is working like a charm, so I'm here just to give a full positive feedback and clarify it's easily doable. The major issue was solved by using two separate repositories (source and target like it's suggested in codeberg-pages docs) and then I was able to use the image appleboy/drone-git-push straightforwardly. ``` deploy: image: appleboy/drone-git-push secrets: - git_ssh_key settings: remote_name: origin-git remote: git@codeberg.org:username/pages.git branch: main commit: true commit_message: '[skip ci] publish to pages' ssh_key: from_secret: git_ssh_key ```
Sign in to join this conversation.
No Milestone
No project
No Assignees
10 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Codeberg-CI/feedback#3
Loading…
There is no content yet.