Rule generator needs update due to uBO CNAME uncloaking enabled by default #816

Closed
opened 5 months ago by hexer33 · 6 comments
hexer33 commented 5 months ago

Hello.

With CNAME uncloaking feature of uBlock Origin a list of exception rules from the rule generator become inefficient.

For example whitelisted use.fontawesome.com from the LocalCDN's test page have a CNAME use.fontawesome.com.cdn.cloudflare.net which is actually blocked. So the test page looks like this (there is (0) at LocalCDN icon in extended menu):
изображение

The uBlock Origin logs then:
изображение

So I think that in general with cdn names, cdn CNAME should be present in the generated list of rules.

As far as I can see, disabling this feature of uBO or whitelisting domains by static filters can lead to skipping trackers or fingerprinting libraries blocked by static filters.

I think a Firefox dns API can be used while generating rules to extend them with CNAME's. Similar as uBO do. And it will give exactly the same result as uBlock Origin.

Also R.Hill say that uMatrix will receive this feature as well.

As I can see, this issue may address only Firefox. But it's my main browser.
https://github.com/uBlockOrigin/uBlock-issues/issues/780#issuecomment-552477648

Hello. With [CNAME uncloaking feature](https://github.com/gorhill/uBlock/wiki/Dashboard:-Settings#uncloak-canonical-names) of uBlock Origin a list of exception rules from the rule generator become inefficient. For example whitelisted `use.fontawesome.com` from the LocalCDN's test page have a CNAME `use.fontawesome.com.cdn.cloudflare.net` which is actually blocked. So the test page looks like this (there is (0) at LocalCDN icon in extended menu): ![изображение](/attachments/d3492e2a-d4a7-4c17-959a-d5a5de07ade1) The uBlock Origin logs then: ![изображение](/attachments/e914846c-e5a4-4b16-acb5-bea8ca4f333e) So I think that in general with cdn names, cdn CNAME should be present in the generated list of rules. As far as I can see, disabling this feature of uBO or whitelisting domains by static filters can lead to skipping trackers or fingerprinting libraries blocked by static filters. I think a Firefox dns API can be used while generating rules to extend them with CNAME's. Similar as [uBO do](https://github.com/uBlockOrigin/uBlock-issues/issues/780#issuecomment-552465524). And it will give exactly the same result as uBlock Origin. [Also R.Hill say that uMatrix will receive this feature as well.](https://github.com/uBlockOrigin/uBlock-issues/issues/780#issuecomment-552468547) As I can see, this issue may address only Firefox. But it's my main browser. https://github.com/uBlockOrigin/uBlock-issues/issues/780#issuecomment-552477648
Poster

Ok, I found a PowerShell aproach to resolve cdn CNAME's and add them to list which maybe will be useful for someone else.

Get-Content -Path $env:userprofile\documents\cdn-rules.txt |
ForEach-Object { $_.Replace('* ', '').Replace(' noop', '') |
Resolve-DnsName -DnsOnly -Type CNAME -ea Continue |
Where-Object { $_.Type -eq 'CNAME' } |
Select-Object -First 1 -ExpandProperty NameHost } |
ForEach-Object { Write-Host "* $_ * noop" }

And the result is:

* mscomajax.vo.msecnd.net * noop
* mscomajax.vo.msecnd.net * noop
* cds.s5x3j6q5.hwcdn.net * noop
* cdn.jsdelivr.net.cdn.cloudflare.net * noop
* apps.bdimg.jomodns.com * noop
* developer.n.shifen.com * noop
* iduwdjf.qiniudns.com * noop
* cdn.bootcss.com.maoyundns.com * noop
* mat1.gtimg.com.tegsea.tc.qq.com * noop
* lb.sae.sina.com.cn * noop
* vo.aicdn.com * noop
* use.fontawesome.com.cdn.cloudflare.net * noop
* akamai-webcdn.kgstatic.net.edgesuite.net * noop
* dualstack.osff.map.fastly.net * noop
* materialdesignicons.b-cdn.net * noop
* cdn.embed.ly.cdn.cloudflare.net * noop
* sdn.inbond.gslb.geekzu.org * noop
* gateway.cname.ustclug.org * noop
* cdnjs.loli.net.cdn.cloudflare.net * noop
* ajax.loli.net.cdn.cloudflare.net * noop
* fonts.loli.net.cdn.cloudflare.net * noop
* lib.baomitu.com.qh-cdn.com * noop
* cdn.bootcdn.net.maoyundns.com * noop
* gstaticadssl.l.google.com * noop

Also there is an error that the name cdn.css.net does not exist

This resolves the issue for me for now, but maybe I am missing something in adopting that PowerShell script.

Ok, I [found](https://community.spiceworks.com/topic/1597672-bulk-cname-lookup) a PowerShell aproach to resolve cdn CNAME's and add them to list which maybe will be useful for someone else. ```powershell Get-Content -Path $env:userprofile\documents\cdn-rules.txt | ForEach-Object { $_.Replace('* ', '').Replace(' noop', '') | Resolve-DnsName -DnsOnly -Type CNAME -ea Continue | Where-Object { $_.Type -eq 'CNAME' } | Select-Object -First 1 -ExpandProperty NameHost } | ForEach-Object { Write-Host "* $_ * noop" } ``` And the result is: ``` * mscomajax.vo.msecnd.net * noop * mscomajax.vo.msecnd.net * noop * cds.s5x3j6q5.hwcdn.net * noop * cdn.jsdelivr.net.cdn.cloudflare.net * noop * apps.bdimg.jomodns.com * noop * developer.n.shifen.com * noop * iduwdjf.qiniudns.com * noop * cdn.bootcss.com.maoyundns.com * noop * mat1.gtimg.com.tegsea.tc.qq.com * noop * lb.sae.sina.com.cn * noop * vo.aicdn.com * noop * use.fontawesome.com.cdn.cloudflare.net * noop * akamai-webcdn.kgstatic.net.edgesuite.net * noop * dualstack.osff.map.fastly.net * noop * materialdesignicons.b-cdn.net * noop * cdn.embed.ly.cdn.cloudflare.net * noop * sdn.inbond.gslb.geekzu.org * noop * gateway.cname.ustclug.org * noop * cdnjs.loli.net.cdn.cloudflare.net * noop * ajax.loli.net.cdn.cloudflare.net * noop * fonts.loli.net.cdn.cloudflare.net * noop * lib.baomitu.com.qh-cdn.com * noop * cdn.bootcdn.net.maoyundns.com * noop * gstaticadssl.l.google.com * noop ``` Also there is an error that the name `cdn.css.net` does not exist This resolves the issue for me for now, but maybe I am missing something in adopting that PowerShell script.
Owner

Thanks for your screenshots and links 👍

You are right, the CNAME problem has not been considered until now. There are currently two ways to solve this problem:

  1. Resolve addresses: Just like uBlock Origin, it's possible to resolve the addresses of the CDN and add the CNAME entries dynamically. The "DNS" permission is required for this. The CNAME entries will probably change rather rarely and I'm not a big fan of additional permissions if there is another way.

  2. Manually: I could maintain the CNAME entries manually in the source code.

I would try the second solution for now. If we notice later that the addresses change frequently, we could implement the first solution.

Ok, I found a PowerShell aproach to resolve cdn CNAME's and add them to list which maybe will be useful for someone else.

Well done. Linux users could (temporarily) use something like this to create the rules for uBlock Origin:

#!/bin/bash

array=()

for cdn in `sed 's/* noop//g; s/* //g' ublock-origin-rules.txt`; do
    cname=$(dig +short $cdn CNAME)
    if [[ $cname != "" ]]; then
        array+=("* ${cname::-1} * noop")
    fi
    array+=("* ${cdn} * noop")
done

printf "%s\n" "${array[@]}" > ./rules.txt

I added a complete list of all CDNs including CNAME entries to the repository. This allows me to see immediately if there are any changes.

Also there is an error that the name cdn.css.net does not exist

It looks like the provider cdn.css.net no longer exists. Now I know what he wanted to tell me in this issue 🤦‍♂️

Thanks for your screenshots and links 👍 You are right, the CNAME problem has not been considered until now. There are currently two ways to solve this problem: 1. Resolve addresses: Just like uBlock Origin, it's possible to resolve the addresses of the CDN and add the CNAME entries dynamically. The "DNS" permission is required for this. The CNAME entries will probably change rather rarely and I'm not a big fan of additional permissions if there is another way. 2. Manually: I could maintain the CNAME entries manually in the source code. I would try the second solution for now. If we notice later that the addresses change frequently, we could implement the first solution. > Ok, I found a PowerShell aproach to resolve cdn CNAME's and add them to list which maybe will be useful for someone else. Well done. Linux users could (temporarily) use something like this to create the rules for uBlock Origin: ```bash #!/bin/bash array=() for cdn in `sed 's/* noop//g; s/* //g' ublock-origin-rules.txt`; do cname=$(dig +short $cdn CNAME) if [[ $cname != "" ]]; then array+=("* ${cname::-1} * noop") fi array+=("* ${cdn} * noop") done printf "%s\n" "${array[@]}" > ./rules.txt ``` I added a complete list of all CDNs including CNAME entries to the repository. This allows me to see immediately if there are any changes. > Also there is an error that the name cdn.css.net does not exist It looks like the provider cdn.css.net no longer exists. Now I know what he wanted to tell me in [this issue](https://codeberg.org/nobody/LocalCDN/issues/786) 🤦‍♂️
nobody referenced this issue from a commit 5 months ago
nobody referenced this issue from a commit 5 months ago
nobody added the
enhancement
label 5 months ago
nobody added this to the v2.6.22 milestone 5 months ago
nobody referenced this issue from a commit 5 months ago
Poster

Yes, I agree.

Then I should note that in some conversations there about a "new dns permissions" required by uBlock Origin I came across this bug 1617861 (fixed two years ago). But I can't find them now. Ok, I found it, release notes of uBlock Origin 1.25 on Reddit says:

Permission notification has now been removed in Firefox 75 and Firefox 74 Beta 9 - https://bugzilla.mozilla.org/show_bug.cgi?id=1617861#c9

Anyway... Thanks for your efforts.

P.S. Is there a space missing in sed 's/* noop//g; s/* //g'? (sed 's/ * noop//g; s/* //g')

Yes, I agree. Then I should note that in some conversations there about a "new dns permissions" required by uBlock Origin I came across this bug ~~[1617861](https://bugzilla.mozilla.org/show_bug.cgi?id=1617861)~~ (fixed two years ago). ~~But I can't find them now.~~ Ok, I found it, [release notes of uBlock Origin 1.25 on Reddit](https://www.reddit.com/r/uBlockOrigin/comments/f8qnpc/ublock_origin_1250_with_cname_uncloaking_is_out/) says: > Permission notification has now been removed in Firefox 75 and Firefox 74 Beta 9 - https://bugzilla.mozilla.org/show_bug.cgi?id=1617861#c9 Anyway... Thanks for your efforts. P.S. Is there a space missing in `sed 's/* noop//g; s/* //g'`? (`sed 's/ * noop//g; s/* //g'`)
Owner

Then I should note that in some conversations there about a "new dns permissions" required by uBlock Origin I came across this bug 1617861 (fixed two years ago)

Thanks for the information. Yes, I can remember that conversation. Nevertheless, the permission has to be specified in the manifest and there I would like to keep the list as short as possible.

Is there a space missing in sed 's/* noop//g; s/* //g'?

Yes, but this is not relevant for the following commands. The result is as expected with only one space. With a simple space this would not work, if you want to do that you would have to escape it, e.g.
sed 's/\s\* noop//g; s/* //g' ublock-origin-rules.txt.

> Then I should note that in some conversations there about a "new dns permissions" required by uBlock Origin I came across this bug 1617861 (fixed two years ago) Thanks for the information. Yes, I can remember that conversation. Nevertheless, the permission has to be specified in the manifest and there I would like to keep the list as short as possible. > Is there a space missing in `sed 's/* noop//g; s/* //g'`? Yes, but this is not relevant for the following commands. The result is as expected with only one space. With a simple space this would not work, if you want to do that you would have to escape it, e.g. `sed 's/\s\* noop//g; s/* //g' ublock-origin-rules.txt`.
nobody closed this issue 5 months ago
Poster

One more question. Will the user be notified that they must update their rules? Because it can't be automated.

Thanks for the information. Yes, I can remember that conversation. Nevertheless, the permission has to be specified in the manifest and there I would like to keep the list as short as possible.

I understand.

Yes, but this is not relevant for the following commands. The result is as expected with only one space. With a simple space this would not work, if you want to do that you would have to escape it, e.g.
sed 's/\s\* noop//g; s/* //g' ublock-origin-rules.txt.

My fault. I'm not familiar with sed.

One more question. Will the user be notified that they must update their rules? Because it can't be automated. > Thanks for the information. Yes, I can remember that conversation. Nevertheless, the permission has to be specified in the manifest and there I would like to keep the list as short as possible. I understand. > Yes, but this is not relevant for the following commands. The result is as expected with only one space. With a simple space this would not work, if you want to do that you would have to escape it, e.g. > `sed 's/\s\* noop//g; s/* //g' ublock-origin-rules.txt`. My fault. I'm not familiar with sed.
Owner

Will the user be notified that they must update their rules?

Yes, the user will be notified if configured in the settings. There are no notifications by default (source code)

0 = Never (Silent Updates)
1 = Only if new CDNs and rules
2 = Always

screenshot option page

My fault. I'm not familiar with sed.

No problem 🙂

> Will the user be notified that they must update their rules? Yes, the user will be notified if configured in the settings. There are no notifications by default ([source code](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/constants.js#L145)) ``` 0 = Never (Silent Updates) 1 = Only if new CDNs and rules 2 = Always ``` ![screenshot option page](https://codeberg.org/attachments/d4227480-2884-4944-9356-33f2e60a5209) > My fault. I'm not familiar with sed. No problem 🙂
nobody referenced this issue from a commit 5 months ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.