2.14 Reproducible builds #200

Open
opened 2 months ago by fynngodau · 8 comments
Owner

RBs are broken again.

From the F-Droid log (I also attached it):

DOES NOT VERIFY
ERROR: APK Signature Scheme v3 signer #1: APK integrity check failed. CHUNKED_SHA256 digest mismatch. Expected: <bd7600e252e9134d2c1bae7172de17e1196b408be9a9eeb0b15ed8295e896a22>, actual: <5d52942e08038047b06023070b3393091e15bb01dcd02664b767937ca0930064>
2021-12-03 21:38:17,453 ERROR: 
/tmp/tmpkt9080fq/sigcp_de.corona.tracing_2140101.apk: 
2021-12-03 21:38:17,453 INFO: ...NOT verified - /tmp/tmpkt9080fq/sigcp_de.corona.tracing_2140101.apk
2021-12-03 21:38:18,849 DEBUG: > diff -r /tmp/tmpkt9080fq/unsigned_binaries_de.corona.tracing_2140101.binary /tmp/tmpkt9080fq/_tmp_tmpkt9080fq_sigcp_de.corona.tracing_2140101
2021-12-03 21:38:19,082 DEBUG: removing unsigned/de.corona.tracing_2140101.apk
2021-12-03 21:38:19,090 DEBUG: removing unsigned/binaries/de.corona.tracing_2140101.binary.apk
2021-12-03 21:38:19,254 ERROR: Could not build app de.corona.tracing: compared built binary to supplied reference binary but failed
==== detail begin ====
Unexpected diff output:
Binary files /tmp/tmpkt9080fq/unsigned_binaries_de.corona.tracing_2140101.binary/content/classes2.dex and /tmp/tmpkt9080fq/_tmp_tmpkt9080fq_sigcp_de.corona.tracing_2140101/content/classes2.dex differ
Binary files /tmp/tmpkt9080fq/unsigned_binaries_de.corona.tracing_2140101.binary/content/classes3.dex and /tmp/tmpkt9080fq/_tmp_tmpkt9080fq_sigcp_de.corona.tracing_2140101/content/classes3.dex differ
==== detail end ====
2021-12-03 21:38:19,254 DEBUG: Error encoutered, stopping by user request.
==== detail end ====

Further investigation is needed, but it likely is connected to the change that we now build microG ourselves.

RBs are broken again. From the [F-Droid log](https://f-droid.org/repo/de.corona.tracing_2140101.log.gz) (I also attached it): ``` DOES NOT VERIFY ERROR: APK Signature Scheme v3 signer #1: APK integrity check failed. CHUNKED_SHA256 digest mismatch. Expected: <bd7600e252e9134d2c1bae7172de17e1196b408be9a9eeb0b15ed8295e896a22>, actual: <5d52942e08038047b06023070b3393091e15bb01dcd02664b767937ca0930064> 2021-12-03 21:38:17,453 ERROR: /tmp/tmpkt9080fq/sigcp_de.corona.tracing_2140101.apk: 2021-12-03 21:38:17,453 INFO: ...NOT verified - /tmp/tmpkt9080fq/sigcp_de.corona.tracing_2140101.apk 2021-12-03 21:38:18,849 DEBUG: > diff -r /tmp/tmpkt9080fq/unsigned_binaries_de.corona.tracing_2140101.binary /tmp/tmpkt9080fq/_tmp_tmpkt9080fq_sigcp_de.corona.tracing_2140101 2021-12-03 21:38:19,082 DEBUG: removing unsigned/de.corona.tracing_2140101.apk 2021-12-03 21:38:19,090 DEBUG: removing unsigned/binaries/de.corona.tracing_2140101.binary.apk 2021-12-03 21:38:19,254 ERROR: Could not build app de.corona.tracing: compared built binary to supplied reference binary but failed ==== detail begin ==== Unexpected diff output: Binary files /tmp/tmpkt9080fq/unsigned_binaries_de.corona.tracing_2140101.binary/content/classes2.dex and /tmp/tmpkt9080fq/_tmp_tmpkt9080fq_sigcp_de.corona.tracing_2140101/content/classes2.dex differ Binary files /tmp/tmpkt9080fq/unsigned_binaries_de.corona.tracing_2140101.binary/content/classes3.dex and /tmp/tmpkt9080fq/_tmp_tmpkt9080fq_sigcp_de.corona.tracing_2140101/content/classes3.dex differ ==== detail end ==== 2021-12-03 21:38:19,254 DEBUG: Error encoutered, stopping by user request. ==== detail end ==== ``` Further investigation is needed, but it likely is connected to the change that we now build microG ourselves.
fynngodau added the
reproducible builds
label 2 months ago
Owner

As we asked for help on mastodon and people have in turn been asking how to help, here's some rough todos:

Try and see if https://codeberg.org/corona-contact-tracing-germany/cwa-android/src/branch/main/docs/rebuilding.md encounters the same problems as f-droid.org building, or otherwise try and figure out how/if building the app again produces a different apk from the one hosted at bubu1.eu/cctg. Then try and figure out with diffoscope where those diffs might come from and try and fix them.

Just getting a different apk and running both through diffoscope would actually be a big help already..

As we asked for help on mastodon and people have in turn been asking how to help, here's some rough todos: Try and see if https://codeberg.org/corona-contact-tracing-germany/cwa-android/src/branch/main/docs/rebuilding.md encounters the same problems as f-droid.org building, or otherwise try and figure out how/if building the app again produces a different apk from the one hosted at bubu1.eu/cctg. Then try and figure out with diffoscope where those diffs might come from and try and fix them. Just getting a different apk and running both through diffoscope would actually be a big help already..

I built the app according to the instructions and indeed got an error that it differs from the one in the repo. However, there was no APK in the repo folder afterwards, probably because of the verification error.

So, I spawned a shell in the container, ran the build process manually and extracted the unsigned APK, as that was the only one I could find:

docker run -it --volume ${PWD}/repo:/repo cctg-builder /bin/bash
fdroid build --latest --no-tarball de.corona.tracing
cp ./build/de.corona.tracing/Corona-Warn-App/build/outputs/apk/device/release/Corona-Warn-App-device-release-unsigned.apk repo/

The sha256sum of this file is:
ae9956b34d88f62083def580804f8f0082e95e6370c15924c3ec2e9df3ec63cf

I repeated the process four times (spawning a fresh the container each time) and the resulting APK always has the same checksum, so it at least seems to be deterministic (on my system).

The output of diffoscope is attached. There are only a few places where small functions regarding NavGraphDirections are missing/different.

I hope this information is helpful. If you need further information that I can gather from the builds, please let me know.

PS: Thank you very much for your work on this project :-)

I built the app according to the instructions and indeed got an error that it differs from the one in the repo. However, there was no APK in the `repo` folder afterwards, probably because of the verification error. So, I spawned a shell in the container, ran the build process manually and extracted the unsigned APK, as that was the only one I could find: ``` docker run -it --volume ${PWD}/repo:/repo cctg-builder /bin/bash fdroid build --latest --no-tarball de.corona.tracing cp ./build/de.corona.tracing/Corona-Warn-App/build/outputs/apk/device/release/Corona-Warn-App-device-release-unsigned.apk repo/ ``` The sha256sum of this file is: `ae9956b34d88f62083def580804f8f0082e95e6370c15924c3ec2e9df3ec63cf` I repeated the process four times (spawning a fresh the container each time) and the resulting APK always has the same checksum, so it at least seems to be deterministic (on my system). The output of diffoscope is attached. There are only a few places where small functions regarding `NavGraphDirections` are missing/different. I hope this information is helpful. If you need further information that I can gather from the builds, please let me know. PS: Thank you very much for your work on this project :-)
Poster
Owner

Many thanks for your effort, @klonfish. According to your findings, it looks like the cause is pretty much the same as in #135 and like this is one or two places we missed taking care of working around the bug as described there.

It is also typical for this issue that it generates the same output on the same machine, apparently because there is a sort of race condition going on.

Again, this is a huge help, thank you very much.

Many thanks for your effort, @klonfish. According to your findings, it looks like the cause is pretty much the same as in #135 and like this is one or two places we missed taking care of working around the bug as described there. It is also typical for this issue that it generates the same output on the same machine, apparently because there is a sort of race condition going on. Again, this is a huge help, thank you very much.

Glad that I could help.

If necessary (and feasible), I can also test/try out some changes with the setup I have now. Just give me some directions on what to change and how to build the locally modified version (I have some basic experience on Android development, but only from a few years ago).

Glad that I could help. If necessary (and feasible), I can also test/try out some changes with the setup I have now. Just give me some directions on what to change and how to build the locally modified version (I have some basic experience on Android development, but only from a few years ago).
Bubu self-assigned this 2 weeks ago
Bubu referenced this issue from a commit 2 weeks ago
Owner

I just pushed 348de13241 which should definitely help here. I'm not yet fully sure if this will fix all the issues though.

I just pushed https://codeberg.org/corona-contact-tracing-germany/cwa-android/commit/348de13241f69e030e8e9b5afcee7ead926bd5aa which should definitely help here. I'm not yet fully sure if this will fix all the issues though.
Owner

Let's see how this goes: 7ae72fdabf

@klonfish not sure how much time you have on your hands at the moment, but you maybe you could give buildin 2.14.1.1 a try as well?

Let's see how this goes: https://gitlab.com/fdroid/fdroiddata/-/commit/7ae72fdabf127230f8734463693f562dd76e922b @klonfish not sure how much time you have on your hands at the moment, but you maybe you could give buildin 2.14.1.1 a try as well?

There is still one difference left, unfortunately.

The version built on my system now has one of those NavDirections functions, that yours doesn't have. Previously, it was the other way round. From its name, it may be related to the fragment node just below the part that you removed in the commit.

Diffoscope output is attached.

There is still one difference left, unfortunately. The version built on my system now has one of those `NavDirections` functions, that yours doesn't have. Previously, it was the other way round. From its name, it may be related to the `fragment` node just below the part that you removed in the commit. Diffoscope output is attached.
Owner

Thanks for the diff output. I'll try and take a look later if I can figure out where this is still coming from.

Thanks for the diff output. I'll try and take a look later if I can figure out where this is still coming from.
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.