WKD profile isn't working #135

Closed
opened 5 months ago by gonz0 · 25 comments
gonz0 commented 5 months ago

I'm getting "Something went wrong while generating the profile
No public keys could be fetched using WKD" with my email domain, which has a valid WKS setup according to https://metacode.biz/openpgp/web-key-directory

The failong profile page is https://keyoxide.org/wkd/gonzalob@gonz0.com.ar

I'm not seeing any errors in JS console or network inspector.

I'm getting "Something went wrong while generating the profile No public keys could be fetched using WKD" with my email domain, which has a valid WKS setup according to https://metacode.biz/openpgp/web-key-directory The failong profile page is https://keyoxide.org/wkd/gonzalob@gonz0.com.ar I'm not seeing any errors in JS console or network inspector.
Poster

This seems to be related to the instance at https://keyoxide.org/; running a local instance with docker works fine.

This seems to be related to the instance at https://keyoxide.org/; running a local instance with docker works fine.
Owner

It works fine with the keyoxide-cli:

PGP key fingerprint: 607d555b2b11b1b3916acbde31e5fe63e2fc4825
Verification results:
Gonzalo Bermúdez <gonzalob@gonz0.com.ar>
  ✓ gonz0.com.ar (dns)
  ✓ gonzalob (github)
  ✓ @gonz0comar (twitter)
null
  ✓ gonz0.com.ar (dns)
  ✓ gonzalob (github)
  ✓ @gonz0comar (twitter)
Gonzalo Bermúdez <gonzalo@yopapp.me>
  ✓ gonz0.com.ar (dns)
  ✓ gonzalob (github)
  ✓ @gonz0comar (twitter)

That null is a bit strange though but it appears to work. Continuing my investigation.

It works fine with the `keyoxide-cli`: ``` PGP key fingerprint: 607d555b2b11b1b3916acbde31e5fe63e2fc4825 Verification results: Gonzalo Bermúdez <gonzalob@gonz0.com.ar> ✓ gonz0.com.ar (dns) ✓ gonzalob (github) ✓ @gonz0comar (twitter) null ✓ gonz0.com.ar (dns) ✓ gonzalob (github) ✓ @gonz0comar (twitter) Gonzalo Bermúdez <gonzalo@yopapp.me> ✓ gonz0.com.ar (dns) ✓ gonzalob (github) ✓ @gonz0comar (twitter) ``` That `null` is a bit strange though but it appears to work. Continuing my investigation.
Poster

The null looks like the photo id. It's the only missing identity in that list ;-)

The null looks like the photo id. It's the only missing identity in that list ;-)
Owner

That took me a long time to figure out. It appears you are putting your website behind Cloudflare and it seems Cloudflare wants Keyoxide to solve a challenge which obviously it can't.

So everything seems to be working as intended. Cloudflare prevents Keyoxide from fetching your key via WKD.

A local Keyoxide installation will not cause an issue as I suspect Cloudflare "knows" your internet connection is regularly used by a normal internet user so Keyoxide's request will not stand out.

That took me a long time to figure out. It appears you are putting your website behind Cloudflare and it seems Cloudflare wants Keyoxide to solve a challenge which obviously it can't. So everything seems to be working as intended. Cloudflare prevents Keyoxide from fetching your key via WKD. A local Keyoxide installation will not cause an issue as I suspect Cloudflare "knows" your internet connection is regularly used by a normal internet user so Keyoxide's request will not stand out.
Poster

Oh, ok!

Thanks for the heads up! I did not connect my migration to Cloudflare with this issue. Sorry for the inconvenience.

I disabled a feature "Bot Fight mode" in Cloudflare and it's working again. Hope this helps anyone with the same setup avoid some headache.

Oh, ok! Thanks for the heads up! I did not connect my migration to Cloudflare with this issue. Sorry for the inconvenience. I disabled a feature "Bot Fight mode" in Cloudflare and it's working again. Hope this helps anyone with the same setup avoid some headache.
Owner

Nice, good to know there's a solution for it that I could mention to others affected by this.

I do wonder if there's something Keyoxide could do on its servers to alleviate any Cloudflare "bot suspicion" though in a sense, Keyoxide is acting like a bot. I do not know much about Cloudflare so I hope someone could shed some light on this some day.

Nice, good to know there's a solution for it that I could mention to others affected by this. I do wonder if there's something Keyoxide could do on its servers to alleviate any Cloudflare "bot suspicion" though in a sense, Keyoxide is acting like a bot. I do not know much about Cloudflare so I hope someone could shed some light on this some day.
Poster

They keep a list of "whitelisted bots" in https://radar.cloudflare.com/verified-bots, and provide an online form to be added to it: https://docs.google.com/forms/d/e/1FAIpQLSdqYNuULEypMnp4i5pROSc-uP6x65Xub9svD27mb8JChA_-XA/viewform

They are very opaque in describing how they detect a bot, I think intentionally.

I'm not sure Keyoxide can be considered a bot however. But I guess that's a matter of perspective :-)

They keep a list of "whitelisted bots" in https://radar.cloudflare.com/verified-bots, and provide an online form to be added to it: https://docs.google.com/forms/d/e/1FAIpQLSdqYNuULEypMnp4i5pROSc-uP6x65Xub9svD27mb8JChA_-XA/viewform They are very opaque in describing how they detect a bot, I think intentionally. I'm not sure Keyoxide can be considered a bot however. But I guess that's a matter of perspective :-)
gonz0 closed this issue 5 months ago

Funny thing that since you mentioned WKD checker works it means Cloudflare recognizes the checker as a "good bot" and I didn't do anything :)

I just set the User-Agent not sure if that influences their algorithm: https://gitlab.com/wiktor/wkd-checker/-/blob/master/src/lib.rs#L178

Funny thing that since you mentioned WKD checker works it means Cloudflare recognizes the checker as a "good bot" and I didn't do anything :) I just set the User-Agent not sure if that influences their algorithm: https://gitlab.com/wiktor/wkd-checker/-/blob/master/src/lib.rs#L178
Owner

I wanted to do the same thing, just in case it makes a difference but Keyoxide uses @openpgpjs/wkd-client [1] to make the requests. I suppose this could make a fine PR.

[1] https://github.com/openpgpjs/wkd-client/blob/master/src/wkd.js

I wanted to do the same thing, just in case it makes a difference but Keyoxide uses @openpgpjs/wkd-client [1] to make the requests. I suppose this could make a fine PR. [1] https://github.com/openpgpjs/wkd-client/blob/master/src/wkd.js

I suspect that node-fetch used by the wkd-client is using generic User-Agent that makes CloudFlare not-so-happy. I guess this could be tested with a CloudFlare user turning on and off the "bot detector".

I suspect that node-fetch used by the wkd-client is using generic User-Agent that makes CloudFlare not-so-happy. I guess this could be tested with a CloudFlare user turning on and off the "bot detector".
Owner

Well, if @gonz0 is still around, we can test this :)

Well, if @gonz0 is still around, we can test this :)
Poster

I just enabled back the Bot Detector feature in Cloudflare. It is failing again, ready for testing :-)

I just enabled back the Bot Detector feature in Cloudflare. It is failing again, ready for testing :-)
Owner

Thanks! Gonna work on it right away

Thanks! Gonna work on it right away

@yarmo I think you can capture the actual User-Agent header using https://requestbin.net/ or something similar and then replay that using curl -H User-Agent XYZ...

@yarmo I think you can capture the actual `User-Agent` header using https://requestbin.net/ or something similar and then replay that using `curl -H User-Agent XYZ`...
Owner

I'm confused...

So node-fetch's user agent is node-fetch. All the following requests were made from the server that runs keyoxide.org

For the "direct" URL:

curl -i https://gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob

# body => Cloudflare challenge page
HTTP/2 503
date: Thu, 14 Jul 2022 11:13:56 GMT
content-type: text/html; charset=UTF-8
permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GpY9iPX1ddcoh91aezVT%2B%2FcQ%2FV9QZ26%2FBEVwfRDa5JApwPPCxxLdNjC07z2P%2FySs2kCcYtz%2BTVJAUuxPyz06oTqwXIAYlT0p7YfXxvkdN406sfzNhbYurQ7am8PymyZWoVa2fkr9wP9cleY%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=15552000; includeSubDomains; preload
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 72a9d4f58f7c9a05-FRA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
curl -i -H "User-Agent: node-fetch" https://gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob

# body => empty
HTTP/2 301
date: Thu, 14 Jul 2022 11:12:30 GMT
location: https://www.gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob
cache-control: max-age=3600
expires: Thu, 14 Jul 2022 12:12:30 GMT
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=stdRBynjXQe2tj4mm%2BMGl82OIrHWxZbH9DCvYfEKxLMFT3FGeAr%2FKweJ88oCys4aG3NbGH5B3ub7rWZ0HVCie1GWYHVLWOfRa3%2FtugZcom3WY1sNRm44I89R0wxTHqvt7jK7llRAkdBVmHc%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=15552000; includeSubDomains; preload
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 72a9d2ddfc059b2b-FRA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
curl -i -L -H "User-Agent: node-fetch" https://gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob

# body => binary key
HTTP/2 200
date: Thu, 14 Jul 2022 11:18:33 GMT
content-type: application/octet-stream
content-length: 34490
accept-ranges: bytes
access-control-allow-origin: *
last-modified: Thu, 14 Jul 2022 04:17:29 GMT
strict-transport-security: max-age=15552000; includeSubDomains; preload
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=AUp%2BNjQ%2BvIXn%2BVh4YfjRCE80HI42X2lZ8O9k5tytxGpmer6sxJsL6e3U4nanswMcrvNZKD1S6ZXwGk87wAmIuA2JDeDx3oGpsWvMBX5ru0dt9ZuRoqDIsC0G9oaHzEaFn8BGtnniHSjQDFvjEBAt"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 72a9dbbbab0e9137-FRA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

For the "advanced" URL:

curl -i https://openpgpkey.gonz0.com.ar/.well-known/openpgpkey/gonz0.com.ar/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob

# body => Cloudflare challenge page
HTTP/2 503
date: Thu, 14 Jul 2022 11:16:35 GMT
content-type: text/html; charset=UTF-8
permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=gMUZvL5CpVT90VMUejFl5aDaPCwJB3Wx72o4VS%2Fs%2BvL6f7Z3kNO4gt7160bLjDmGKaSlLGJ5cAwsWZ3FDItZCKdWj2KJP3zOwUn2QKsq9HxC%2FafbruENaAY9%2BwaA84sY%2Fl8%2BdqflIDPJptTdmDaNoYGbPdNelA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=15552000; includeSubDomains; preload
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 72a9d8d86bf2bb97-FRA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
curl -i -H "User-Agent: node-fetch" https://openpgpkey.gonz0.com.ar/.well-known/openpgpkey/gonz0.com.ar/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob

# body => binary key
HTTP/2 200
date: Thu, 14 Jul 2022 11:15:42 GMT
content-type: application/octet-stream
content-length: 34490
accept-ranges: bytes
access-control-allow-origin: *
last-modified: Thu, 14 Jul 2022 04:17:29 GMT
strict-transport-security: max-age=15552000; includeSubDomains; preload
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=smwlCevRfRI3joM9SErG2eHQ31WbcE6GvORYs21C9VrRBvRjK%2Bt%2FZJiOKhiRa%2FecbumEvfoHubcHoVZxGrsQTuniAFILYuRnoeIARdGI18NvfXkELECMt3KJCGjArwt%2FbPYO7YVGRC2hXBFUgsK34KK%2BA8Q2Eg%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 72a9d78caca89140-FRA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

In conclusion

Adding the "node-fetch" User-Agent actually saves the day when it comes to both URLs — I only need to follow redirects with -L when it comes to the direct URL.

So... Why doesn't Cloudflare like the node-fetch call??? Hmmm gonna try some more.

BTW, according to requestbin, these are node-fetch's request headers (anonymized some values):

Host: XYZXYZXYZXYZ.X.requestbin.net
Connection: close
Accept: */*
Accept-Encoding: gzip, deflate, br
User-Agent: node-fetch
X-Request-Id: d3a7af13-XXXX-XXXX-XXXX-XXXXXXXXXXXX
X-Forwarded-For: XYZ.XYZ.XYZ.XYZ
X-Forwarded-Proto: http
X-Forwarded-Port: 80
Via: 1.1 vegur
Connect-Time: 0
X-Request-Start: 1657796265889
Total-Route-Time: 0
I'm confused... So node-fetch's user agent is `node-fetch`. All the following requests were made from the server that runs keyoxide.org **For the "direct" URL**: ```bash curl -i https://gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob # body => Cloudflare challenge page HTTP/2 503 date: Thu, 14 Jul 2022 11:13:56 GMT content-type: text/html; charset=UTF-8 permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 expires: Thu, 01 Jan 1970 00:00:01 GMT report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GpY9iPX1ddcoh91aezVT%2B%2FcQ%2FV9QZ26%2FBEVwfRDa5JApwPPCxxLdNjC07z2P%2FySs2kCcYtz%2BTVJAUuxPyz06oTqwXIAYlT0p7YfXxvkdN406sfzNhbYurQ7am8PymyZWoVa2fkr9wP9cleY%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} strict-transport-security: max-age=15552000; includeSubDomains; preload expect-ct: max-age=86400, enforce referrer-policy: same-origin x-content-type-options: nosniff x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block server: cloudflare cf-ray: 72a9d4f58f7c9a05-FRA alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 ``` ```bash curl -i -H "User-Agent: node-fetch" https://gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob # body => empty HTTP/2 301 date: Thu, 14 Jul 2022 11:12:30 GMT location: https://www.gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob cache-control: max-age=3600 expires: Thu, 14 Jul 2022 12:12:30 GMT report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=stdRBynjXQe2tj4mm%2BMGl82OIrHWxZbH9DCvYfEKxLMFT3FGeAr%2FKweJ88oCys4aG3NbGH5B3ub7rWZ0HVCie1GWYHVLWOfRa3%2FtugZcom3WY1sNRm44I89R0wxTHqvt7jK7llRAkdBVmHc%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} strict-transport-security: max-age=15552000; includeSubDomains; preload expect-ct: max-age=86400, enforce referrer-policy: same-origin x-content-type-options: nosniff x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block server: cloudflare cf-ray: 72a9d2ddfc059b2b-FRA alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 ``` ```bash curl -i -L -H "User-Agent: node-fetch" https://gonz0.com.ar/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob # body => binary key HTTP/2 200 date: Thu, 14 Jul 2022 11:18:33 GMT content-type: application/octet-stream content-length: 34490 accept-ranges: bytes access-control-allow-origin: * last-modified: Thu, 14 Jul 2022 04:17:29 GMT strict-transport-security: max-age=15552000; includeSubDomains; preload cf-cache-status: DYNAMIC report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=AUp%2BNjQ%2BvIXn%2BVh4YfjRCE80HI42X2lZ8O9k5tytxGpmer6sxJsL6e3U4nanswMcrvNZKD1S6ZXwGk87wAmIuA2JDeDx3oGpsWvMBX5ru0dt9ZuRoqDIsC0G9oaHzEaFn8BGtnniHSjQDFvjEBAt"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} expect-ct: max-age=86400, enforce referrer-policy: same-origin x-content-type-options: nosniff x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block server: cloudflare cf-ray: 72a9dbbbab0e9137-FRA alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 ``` **For the "advanced" URL**: ```bash curl -i https://openpgpkey.gonz0.com.ar/.well-known/openpgpkey/gonz0.com.ar/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob # body => Cloudflare challenge page HTTP/2 503 date: Thu, 14 Jul 2022 11:16:35 GMT content-type: text/html; charset=UTF-8 permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 expires: Thu, 01 Jan 1970 00:00:01 GMT report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=gMUZvL5CpVT90VMUejFl5aDaPCwJB3Wx72o4VS%2Fs%2BvL6f7Z3kNO4gt7160bLjDmGKaSlLGJ5cAwsWZ3FDItZCKdWj2KJP3zOwUn2QKsq9HxC%2FafbruENaAY9%2BwaA84sY%2Fl8%2BdqflIDPJptTdmDaNoYGbPdNelA%3D%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} strict-transport-security: max-age=15552000; includeSubDomains; preload expect-ct: max-age=86400, enforce referrer-policy: same-origin x-content-type-options: nosniff x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block server: cloudflare cf-ray: 72a9d8d86bf2bb97-FRA alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 ``` ```bash curl -i -H "User-Agent: node-fetch" https://openpgpkey.gonz0.com.ar/.well-known/openpgpkey/gonz0.com.ar/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob # body => binary key HTTP/2 200 date: Thu, 14 Jul 2022 11:15:42 GMT content-type: application/octet-stream content-length: 34490 accept-ranges: bytes access-control-allow-origin: * last-modified: Thu, 14 Jul 2022 04:17:29 GMT strict-transport-security: max-age=15552000; includeSubDomains; preload cf-cache-status: DYNAMIC report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=smwlCevRfRI3joM9SErG2eHQ31WbcE6GvORYs21C9VrRBvRjK%2Bt%2FZJiOKhiRa%2FecbumEvfoHubcHoVZxGrsQTuniAFILYuRnoeIARdGI18NvfXkELECMt3KJCGjArwt%2FbPYO7YVGRC2hXBFUgsK34KK%2BA8Q2Eg%3D%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} expect-ct: max-age=86400, enforce referrer-policy: same-origin x-content-type-options: nosniff x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block server: cloudflare cf-ray: 72a9d78caca89140-FRA alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 ``` **In conclusion** Adding the "node-fetch" User-Agent actually saves the day when it comes to both URLs — I only need to follow redirects with -L when it comes to the direct URL. So... Why doesn't Cloudflare like the node-fetch call??? Hmmm gonna try some more. BTW, according to requestbin, these are node-fetch's request headers (anonymized some values): ``` Host: XYZXYZXYZXYZ.X.requestbin.net Connection: close Accept: */* Accept-Encoding: gzip, deflate, br User-Agent: node-fetch X-Request-Id: d3a7af13-XXXX-XXXX-XXXX-XXXXXXXXXXXX X-Forwarded-For: XYZ.XYZ.XYZ.XYZ X-Forwarded-Proto: http X-Forwarded-Port: 80 Via: 1.1 vegur Connect-Time: 0 X-Request-Start: 1657796265889 Total-Route-Time: 0 ```

Very interesting research @yarmo! Maybe some other request header trips CloudFlare like Via or X-... ? Maybe try requesting that with node-fetch from your localhost and adjusting User-Agent? hmm... super interesting debugging story! :)

(One way or another I think the User-Agent should be adjusted for the WKD client)...

Very interesting research @yarmo! Maybe some other request header trips CloudFlare like `Via` or `X-...` ? Maybe try requesting that with node-fetch from your localhost and adjusting `User-Agent`? hmm... super interesting debugging story! :) (One way or another I think the `User-Agent` should be adjusted for the WKD client)...
Owner

Really strange. I added all those headers to a CURL command and still managed to get the binary data back.

Meanwhile, I got a basic NodeJS script running on the server and no luck, I get the Cloudflare challenge page.

FYI, here are node-fetch's default headers: https://www.npmjs.com/package/node-fetch#default-headers=

Trying a few more things.

Really strange. I added all those headers to a CURL command and still managed to get the binary data back. Meanwhile, I got a basic NodeJS script running on the server and no luck, I get the Cloudflare challenge page. FYI, here are node-fetch's default headers: https://www.npmjs.com/package/node-fetch#default-headers= Trying a few more things.
Owner

There's also no difference between running node directly on the machine and from a docker container.

There's also no difference between running node directly on the machine and from a docker container.
Owner

Maybe I tested too much, I must have angered some internet deity somewhere. The CURL requests that above fetched the binary key no longer do so.

curl -i -H "User-Agent: node-fetch" https://openpgpkey.gonz0.com.ar/.well-known/openpgpkey/gonz0.com.ar/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob

# body => empty
HTTP/2 500
date: Thu, 14 Jul 2022 12:24:16 GMT
content-type: text/plain; charset=utf-8
content-length: 1
strict-transport-security: max-age=15552000; includeSubDomains; preload
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=G4jrbI1%2FXm0ePREh0XqKnngoLaI6eN25eTitpKflkIC2SCaJi1RYweribOluonLjy8tw5hM42Hr7skhXX0I7pFhL6ILREuguU7MU%2BuTa6kGJDPKF1l3Y%2B6z%2Bq48i1SqnaENXEjVprpauRiemMJj9QyIpOw76ow%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 72aa3bfdbe2a9036-FRA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

Cloudflare being Cloudflare. They must somehow be tired of the requests coming from this server, and not your WKD checker server (yet).

If it's even this hard to debug because they keep changing the playing field, my recommendation will likely be "turn off bot detection". I'll give the server some rest, try the same request in an hour or so.

Maybe I tested too much, I must have angered some internet deity somewhere. The CURL requests that above fetched the binary key no longer do so. ```bash curl -i -H "User-Agent: node-fetch" https://openpgpkey.gonz0.com.ar/.well-known/openpgpkey/gonz0.com.ar/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re?l=gonzalob # body => empty HTTP/2 500 date: Thu, 14 Jul 2022 12:24:16 GMT content-type: text/plain; charset=utf-8 content-length: 1 strict-transport-security: max-age=15552000; includeSubDomains; preload cf-cache-status: DYNAMIC report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=G4jrbI1%2FXm0ePREh0XqKnngoLaI6eN25eTitpKflkIC2SCaJi1RYweribOluonLjy8tw5hM42Hr7skhXX0I7pFhL6ILREuguU7MU%2BuTa6kGJDPKF1l3Y%2B6z%2Bq48i1SqnaENXEjVprpauRiemMJj9QyIpOw76ow%3D%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} expect-ct: max-age=86400, enforce referrer-policy: same-origin x-content-type-options: nosniff x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block server: cloudflare cf-ray: 72aa3bfdbe2a9036-FRA alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 ``` Cloudflare being Cloudflare. They must somehow be tired of the requests coming from this server, and not your WKD checker server (yet). If it's even this hard to debug because they keep changing the playing field, my recommendation will likely be "turn off bot detection". I'll give the server some rest, try the same request in an hour or so.
Owner

And 53 minutes later, I once again get a valid response... with CURL. node-fetch still gets the Cloudflare challenge page.

And 53 minutes later, I once again get a valid response... with CURL. node-fetch still gets the Cloudflare challenge page.
Poster

I'd never expect to get 500 from any sort of limiting or whatsoever. Cloudflare reported a minor outage recently. Maybe your test was affected by this?

I'd never expect to get 500 from any sort of limiting or whatsoever. Cloudflare reported a minor outage recently. Maybe your test was affected by this?
gonz0 reopened this issue 5 months ago
Owner

Could be that then.

Be that as it may, I am at a loss. Somehow, Cloudflare knows what is a CURL request and what is a node-fetch request. Using a different library (got), I get the same result.

Using requestbin.io (I had some difficulties with the one @wiktor suggested), I determined there are basically no differences between CURL and node-fetch, except User-Agent and a few of the X- headers but they appear to be added by the requestbin service. Setting node-fetch's User-Agent to CURL's User-Agent fixes nothing.

I really don't know anymore.

Could be that then. Be that as it may, I am at a loss. Somehow, Cloudflare knows what is a CURL request and what is a node-fetch request. Using a different library (`got`), I get the same result. Using requestbin.io (I had some difficulties with the one @wiktor suggested), I determined there are basically no differences between CURL and node-fetch, except User-Agent and a few of the X- headers but they appear to be added by the requestbin service. Setting node-fetch's User-Agent to CURL's User-Agent fixes nothing. I really don't know anymore.
Poster

Trying to help and not make too much noise at the same time.

I found the bot fight log. Here's a sample event from an attempt to load my keyoxide profile. I can't make much of it in terms of finding a way to solve the issue, but maybe it sheds some light for you :smile

I've redacted some information I think shouldn't be public.

{
  "action": "jschallenge",
  "clientASNDescription": "HETZNER-AS",
  "clientAsn": "24940",
  "clientCountryName": "<REDACTED>",
  "clientIP": "<REDACTED>",
  "clientRequestHTTPHost": "gonz0.com.ar",
  "clientRequestHTTPMethodName": "GET",
  "clientRequestHTTPProtocol": "HTTP/1.1",
  "clientRequestPath": "/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re",
  "clientRequestQuery": "",
  "datetime": "2022-07-14T17:01:19Z",
  "rayName": "72abd1d2f95e8883",
  "ruleId": "bot_fight_mode",
  "rulesetId": "",
  "source": "botFight",
  "userAgent": "got (https://github.com/sindresorhus/got)",
  "matchIndex": 0,
  "metadata": [],
  "sampleInterval": 1
}

The full, uncensored request is encrypted to your key (using keyoxide!) for completeness.

-----BEGIN PGP MESSAGE-----

wcDMA3EWDTLLEgFtAQv9GSwnDyJ/EA9Rz2DJ7wtOV/tJdU2ctzquvJ2iXuCu
w1oKd3tlXq2ofascmq5YpywANGd3H6yvZ4ki77EMqsQuFZOA930zURHRwYDf
iPRtBRluOOChUYSe0Wx6cf0egFH7H3t/0f2hzQuKtKj5XUEPT7IadGA0xAi3
Y5GRMZuhvhKlcC0X7NR3EiHOdrbEuI+zBDCu1sT2/K/7ID4cICERhPyk62pl
9n/KG/U8wtSRu826vvWjEFP9/nTOF1ql4B52He3G6gfpEWCga6wBbBvRuJwb
PqXxa56Vf9q2s0Wq+ISVkRZbMD4h+rx9eddpgO5coGHH1YMxwg9bgr/DegJf
w0rPUoY+ezJyqrriN4+3JkjVpmK1xBIfqY5Wv7Hde62St+CbWVAyKepZThPW
/tlIPirR4ncLNqivaQP/Mj9Pe9iwFkq6abAQgwZ2WXqPyBNM+mk9DMdplLty
2FPqOKpHiyZXQiPWKVNRREzms/5C/dDvsKw9KCVD0ZBUf7KQZjEZ0sIXAZ+I
jez/qPEm8SJPjvDCiubQw3lTdKbv9xE48skYCKBqI4JA0dZXgWkXL8xrvmhK
Nm2SFstOninAkc455p3/j2V0ZK6fUAhsKgiZu8oU1UfFH6sD4Xhj6nKUx2k3
XNh6SCr4WooV7DJVNUtO+BV6tJ6uQcvqPw+7cm1XVDWdg1fUzAc/24WSjFr4
DwxvuTQIsZ+oYvr7WRAG6F4HTTKUWr1395CkwuJwHqPLfmjRay4/yf8sf5r+
xD4dsUxeSaK9mrc3nRzv2qO1iIb9ARggJSo8aKuYdfztvZHUYj+fLKlmRorA
zhNDIc82D8naYMVo9qpryysrW85kUf2ny01PrsQt+kRGJAxVno/C4HPei207
mcgGsycfsMG7zZ+lcAjXuikd1GFRbITOuElD4/SYcoUR++dPIWr8Xez6xUC8
BvGwdvSKQbBzSo4tdSi5HK68z2oaNS+o49vMUAywKGMh8BURfpIPu04+iG7T
7rvPiF9GVMv307gah9LAgxrT5pW7T5BON/UzDvMTfk20SRa5xpoP3e3RaUMs
j8CFOO49MDoAAsYI8pm+Kva5obGNMU9jwjcp1yTLg4SwVfJp/552UCHf9axr
7rEYppCFSL1jZGO+gqh8PKxHeDM4UNk6d2alHdLVZvgsDY7SX4vr+tkNnDOS
eV1t601Ez7Xqfs0eWVtb8yn4HMHZ9ohCOnWy/rhN0/TAVFRD+H5nTvDauicC
2lMx6S+venYiGeJCljgHTa5Tr+0t11EPNGyM+XRCHtEC00pGhKZ/QwlvSqHD
fCJtrtnHjf/ZxwD0PqlcB4CtsVq5IhHiqAlL1JRi/gMkHE0f1V74KUOUG3zd
nr7lXbpxNq8FuU23R5AzkluSX/L8Z7HPhEpBiorifuWh8WdGNFY+Ib7b+GNF
i0iwtauk8xwhCqPpNw3t5Td0RzLnPH+snUBerhUno4R17B3G1yiSnfega+KA
06GTdA==
=kryq
-----END PGP MESSAGE-----
Trying to help and not make too much noise at the same time. I found the bot fight log. Here's a sample event from an attempt to load my keyoxide profile. I can't make much of it in terms of finding a way to solve the issue, but maybe it sheds some light for you :smile I've redacted some information I think shouldn't be public. ``` { "action": "jschallenge", "clientASNDescription": "HETZNER-AS", "clientAsn": "24940", "clientCountryName": "<REDACTED>", "clientIP": "<REDACTED>", "clientRequestHTTPHost": "gonz0.com.ar", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/.well-known/openpgpkey/hu/h3ixzu5zn3w4qkwhrigg9mjbry1if6re", "clientRequestQuery": "", "datetime": "2022-07-14T17:01:19Z", "rayName": "72abd1d2f95e8883", "ruleId": "bot_fight_mode", "rulesetId": "", "source": "botFight", "userAgent": "got (https://github.com/sindresorhus/got)", "matchIndex": 0, "metadata": [], "sampleInterval": 1 } ``` The full, uncensored request is encrypted to your key (using keyoxide!) for completeness. ``` -----BEGIN PGP MESSAGE----- wcDMA3EWDTLLEgFtAQv9GSwnDyJ/EA9Rz2DJ7wtOV/tJdU2ctzquvJ2iXuCu w1oKd3tlXq2ofascmq5YpywANGd3H6yvZ4ki77EMqsQuFZOA930zURHRwYDf iPRtBRluOOChUYSe0Wx6cf0egFH7H3t/0f2hzQuKtKj5XUEPT7IadGA0xAi3 Y5GRMZuhvhKlcC0X7NR3EiHOdrbEuI+zBDCu1sT2/K/7ID4cICERhPyk62pl 9n/KG/U8wtSRu826vvWjEFP9/nTOF1ql4B52He3G6gfpEWCga6wBbBvRuJwb PqXxa56Vf9q2s0Wq+ISVkRZbMD4h+rx9eddpgO5coGHH1YMxwg9bgr/DegJf w0rPUoY+ezJyqrriN4+3JkjVpmK1xBIfqY5Wv7Hde62St+CbWVAyKepZThPW /tlIPirR4ncLNqivaQP/Mj9Pe9iwFkq6abAQgwZ2WXqPyBNM+mk9DMdplLty 2FPqOKpHiyZXQiPWKVNRREzms/5C/dDvsKw9KCVD0ZBUf7KQZjEZ0sIXAZ+I jez/qPEm8SJPjvDCiubQw3lTdKbv9xE48skYCKBqI4JA0dZXgWkXL8xrvmhK Nm2SFstOninAkc455p3/j2V0ZK6fUAhsKgiZu8oU1UfFH6sD4Xhj6nKUx2k3 XNh6SCr4WooV7DJVNUtO+BV6tJ6uQcvqPw+7cm1XVDWdg1fUzAc/24WSjFr4 DwxvuTQIsZ+oYvr7WRAG6F4HTTKUWr1395CkwuJwHqPLfmjRay4/yf8sf5r+ xD4dsUxeSaK9mrc3nRzv2qO1iIb9ARggJSo8aKuYdfztvZHUYj+fLKlmRorA zhNDIc82D8naYMVo9qpryysrW85kUf2ny01PrsQt+kRGJAxVno/C4HPei207 mcgGsycfsMG7zZ+lcAjXuikd1GFRbITOuElD4/SYcoUR++dPIWr8Xez6xUC8 BvGwdvSKQbBzSo4tdSi5HK68z2oaNS+o49vMUAywKGMh8BURfpIPu04+iG7T 7rvPiF9GVMv307gah9LAgxrT5pW7T5BON/UzDvMTfk20SRa5xpoP3e3RaUMs j8CFOO49MDoAAsYI8pm+Kva5obGNMU9jwjcp1yTLg4SwVfJp/552UCHf9axr 7rEYppCFSL1jZGO+gqh8PKxHeDM4UNk6d2alHdLVZvgsDY7SX4vr+tkNnDOS eV1t601Ez7Xqfs0eWVtb8yn4HMHZ9ohCOnWy/rhN0/TAVFRD+H5nTvDauicC 2lMx6S+venYiGeJCljgHTa5Tr+0t11EPNGyM+XRCHtEC00pGhKZ/QwlvSqHD fCJtrtnHjf/ZxwD0PqlcB4CtsVq5IhHiqAlL1JRi/gMkHE0f1V74KUOUG3zd nr7lXbpxNq8FuU23R5AzkluSX/L8Z7HPhEpBiorifuWh8WdGNFY+Ib7b+GNF i0iwtauk8xwhCqPpNw3t5Td0RzLnPH+snUBerhUno4R17B3G1yiSnfega+KA 06GTdA== =kryq -----END PGP MESSAGE----- ```
Poster

I don't mind closing the issue and disabling bot fight. Just let me know if you want to take on round 2 and I'll re-enable it.

Thanks for the time! It's been enlightning and fun

I don't mind closing the issue and disabling bot fight. Just let me know if you want to take on round 2 and I'll re-enable it. Thanks for the time! It's been enlightning and fun
gonz0 closed this issue 5 months ago
Owner

Thx for the info!

Well, if you can't see anything on your end that does or does not trigger the bot fight, then I'm not sure what our next move is.

It'll go into the documentation as a warning.

Thanks for all the help! As soon as someone thinks of something genius, I'm ready for round 2. Until then…


Last test: HTTP2 vs HTTP1.1. Also not the solution, curl --http1.1 [...] works fine.

Thx for the info! Well, if you can't see anything on your end that does or does not trigger the bot fight, then I'm not sure what our next move is. It'll go into the documentation as a warning. Thanks for all the help! As soon as someone thinks of something genius, I'm ready for round 2. Until then… --- Last test: HTTP2 vs HTTP1.1. Also not the solution, `curl --http1.1 [...]` works fine.
Sign in to join this conversation.
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: keyoxide/keyoxide-web#135
Loading…
There is no content yet.