Bring in GCC (and some other stuff) into the initial stage of the build #123

Merged
hjacobs merged 3 commits from lbruno/kube-web-view:lbruno-patch-1 into master 12 months ago
lbruno commented 1 year ago

I needed to rebuild the image for arm64.

Among others, Poetry depends on pyrsistent, msgpack, and cryptography packages, and these don’t have wheels for arm64 -- so they need a compile step themselves -- but python:3.8-slim doesn’t include gcc.

My first attempt was to change the first-stage to RUN apt update && install -y –no-install-recommends build-essential libssl-dev libffi-dev python3-dev && pip3 install poetry && apt remove –y –purge build-essential libssl-dev libffi-dev python3-dev. I verified that this allowed Poetry itself to be built.

NOTE: for some reason apt install python3-dev selected python3.7-dev from the archives, although the base image is called python:3.8.

Invoking Poetry would then also need GCC for some other package, and at that point I figured a smaller and more maintainable change would be to use the non-slim image for the initial stage; this appears to include the libffi and openssl headers, along with GCC and whatever's needed to compile a wheel in arm64.

I reckon the disk space from apt install build-essential is in the ballpark of using the full python:3.8 image anyway, and doing this saves having to debug why apt install was selecting 3.7-dev and didn't have 3.8.

I needed to rebuild the image for arm64. Among others, Poetry depends on pyrsistent, msgpack, and cryptography packages, and these don’t have wheels for arm64 -- so they need a compile step themselves -- but python:3.8-slim doesn’t include gcc. My first attempt was to change the first-stage to RUN apt update && install -y –no-install-recommends build-essential libssl-dev libffi-dev python3-dev && pip3 install poetry && apt remove –y –purge build-essential libssl-dev libffi-dev python3-dev. I verified that this allowed Poetry itself to be built. NOTE: for some reason apt install python3-dev selected python3.7-dev from the archives, although the base image is called python:3.8. Invoking Poetry would then also need GCC for some other package, and at that point I figured a smaller and more maintainable change would be to use the non-slim image for the initial stage; this appears to include the libffi and openssl headers, along with GCC and whatever's needed to compile a wheel in arm64. I reckon the disk space from apt install build-essential is in the ballpark of using the full python:3.8 image anyway, and doing this saves having to debug why apt install was selecting 3.7-dev and didn't have 3.8.
Poster

Figured that your github is a mirror; this is a dup from https://github.com/hjacobs/kube-web-view/pull/3

On github the CI failed with flake8 complaining of existing lint errors.

Figured that your github is a mirror; this is a dup from https://github.com/hjacobs/kube-web-view/pull/3 On github the CI failed with flake8 complaining of existing lint errors.
Poster

Friendly ping?

Friendly ping?
Owner

How would the final build for arm64 look like? Would this change alone already make docker build on arm64 work?

How would the final build for arm64 look like? Would this change alone already make `docker build` on arm64 work?
Poster

It worked on my machine, a Sopine from pine64.org (https://www.pine64.org/baseboard/).

I don't understand what you mean by "final build look like", but I had errors in two of the stages:

  1. pip3 install poetry
  2. the poetry invocation

For the first one I was able to just install enough compiler as you see above. From my notes, pyrsistent, msgpack, and cryptography needed a build because they didn't have wheels available.

And then I facepalmed when apt install && pip3 install poetry && apt remove worked, but poetry install also started trying to compile stuff. That's the point at which I tried the non-slim base for the first step of the Dockerfile. It worked directly, first try.

So, this is a yes for your second question. An alternative method would be to have the upstream wheels also be available in arm64 releases.

As for the first question: most of the pip3 install poetry step is just downloading wheels except for those 3 packages. The poetry.lock has a bunch of wheels that are only i686 or x86_64, so I'll assume all those need a compile. Sorry I didn't keep better notes for the second step.

Did I answer both your questions?

It worked on my machine, a Sopine from pine64.org (https://www.pine64.org/baseboard/). I don't understand what you mean by "final build look like", but I had errors in two of the stages: 1. pip3 install poetry 2. the poetry invocation For the first one I was able to just install enough compiler as you see above. From my notes, pyrsistent, msgpack, and cryptography needed a build because they didn't have wheels available. And then I facepalmed when `apt install && pip3 install poetry && apt remove` worked, but `poetry install` also started trying to compile stuff. That's the point at which I tried the non-slim base for the first step of the Dockerfile. It worked directly, first try. So, this is a **yes** for your second question. An alternative method would be to have the upstream wheels also be available in arm64 releases. As for the first question: most of the `pip3 install poetry` step is just downloading wheels except for those 3 packages. The poetry.lock has a bunch of wheels that are only i686 or x86_64, so I'll assume all those need a compile. Sorry I didn't keep better notes for the second step. Did I answer both your questions?
Poster

Hey, did I end up answering your question, or should I reword? :)

Hey, did I end up answering your question, or should I reword? :)
hjacobs merged commit bee7482c4a into master 12 months ago
The pull request has been merged as bee7482c4a.
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This pull request currently doesn't have any dependencies.

Loading…
There is no content yet.