#150 Check cdnjs' repo for missing frameworks

Closed

opened 6 months ago by fyjzjj3yva · 6 comments

Although LocalCDN has included many frameworks, it is not complete and many less popular frameworks are probably not included.

In addition, manually checking each website to find out whether a framework/CDN combination is missing and then reporting it to the developer requires much work and time.

To include as many frameworks as possible, I think we can check cdnjs' repo.
That being said, there are nearly 4000 entries in cdnjs/ajax/libs/, so including all of them may require a lot of storage space.

nobody added the

question/discussion

label 6 months ago

The intention of LocalCDN is not to integrate and replace all existing frameworks. It's to make a website readable without contacting the CDNs.

So, in order for a framework mapping to be included in LocalCDN...

Find a website using XXX framework
Make sure the website calls the framework via CDN (i.e. the website does not host the framework itself)
Block the framework request (e.g. by using uBlock Origin) to see whether it is really necessary. If the website breaks, looks weird, or some features cannot be used, then it can be reported to the developer.

Like this?

Yes, exactly. That' s a way to test it. I'm using the developer tools of Firefox. Menu -> Web Developer -> Network. There you can block single requests.

nobody referenced this issue 6 months ago

[clonezilla.org] Cookie Consent #153

By the way, when testing and reporting framework mappings, is it a good idea to enable the “block requests for missing resources” option?

You can do that. When I test something I use a new Firefox profile and the developer tools of Firefox. At the same time I load the extension in another profile as "Temporary Extension" and activate logging in LocalCDN.

Is that option helpful in cases like #147, when a website requests unnecessary frameworks from CDNs?

This is hard to say in general terms, which is better. I always block all non existing frameworks.

By the way, is cookieconsent eligible for inclusion?

I think so. It is not big and could be important for someone. I create a separate issue (#153)

How does that option work?

TL;DR:

The listener webRequest.onBeforeRequest depends on the CDNs in mappings.js. If a CDN isn't listed there, LocalCDN ignores this request.

Does that option block all requests to CDN domains like cdnjs.cloudflare.com, cdn.jsdelivr.net?

If this domain is listed in mappings.js, then yes.

Details

Example: cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js

First the URL split into host and path
host = cdnjs.cloudflare.com
path = ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js
The supported CDNs are stored here in the requestFilter and used here. Only matches with a CDN are processed

Non-existent frameworks

If the path part doesn't match in mapping.js, it returns false (here).
This false will be processed here
LocalCDN decides here what to do with non-existing frameworks
The final step for non-existent frameworks is here with a redirection from HTTP to HTTPS if necessary.

Existent frameworks

Example: cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js

Extract the version number of origin path jquery/2.1.4/jquery.min.js -> 2.1.4 (request-analyzer.js#L125)
Insert extracted version into the target path resources/jquery/{version}/jquery.min.jsm -> resources/jquery/2.1.4/jquery.min.jsm (request-analyzer.js#L142)
Determine and replace the target version resources/jquery/2.1.4/jquery.min.jsm -> resources/jquery/2.2.4/jquery.min.jsm (helpers.js#L427)

> By the way, when testing and reporting framework mappings, is it a good idea to enable the “block requests for missing resources” option? You can do that. When I test something I use a new Firefox profile and the developer tools of Firefox. At the same time I load the extension in another profile as "Temporary Extension" and activate logging in LocalCDN. > Is that option helpful in cases like #147, when a website requests unnecessary frameworks from CDNs? This is hard to say in general terms, which is better. I always block all non existing frameworks. > By the way, is cookieconsent eligible for inclusion? I think so. It is not big and could be important for someone. I create a separate issue (#153) > How does that option work? ## TL;DR: The listener [webRequest.onBeforeRequest](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onBeforeRequest) depends on the CDNs in [mappings.js](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/mappings.js). If a CDN isn't listed there, LocalCDN ignores this request. > Does that option block all requests to CDN domains like cdnjs.cloudflare.com, cdn.jsdelivr.net? If this domain is listed in mappings.js, then yes. --- ## Details Example: `cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js` 1. First the URL split into host and path host = cdnjs.cloudflare.com path = ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js 2. The supported CDNs are stored [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/state-manager.js#L140) in the requestFilter and used [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/state-manager.js#L148). Only matches with a CDN are processed #### Non-existent frameworks 3. If the path part doesn't match in [mapping.js](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/mappings.js#L88), it returns `false` ([here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L90)). 4. This `false` will be processed [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L59) 5. LocalCDN decides [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L79) what to do with non-existing frameworks 6. The final step for non-existent frameworks is [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L93) with a redirection from HTTP to HTTPS if necessary. #### Existent frameworks Example: `cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js` 3. Extract the version number of origin path `jquery/2.1.4/jquery.min.js` -> `2.1.4` ([request-analyzer.js#L125](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L125)) 4. Insert extracted version into the target path `resources/jquery/{version}/jquery.min.jsm` -> `resources/jquery/2.1.4/jquery.min.jsm` ([request-analyzer.js#L142](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L142)) 5. Determine and replace the target version `resources/jquery/2.1.4/jquery.min.jsm` -> `resources/jquery/2.2.4/jquery.min.jsm` ([helpers.js#L427](https://codeberg.org/nobody/LocalCDN/src/branch/main/modules/internal/helpers.js#L427))

👍 1

Many thanks for the detailed explanation!

fyjzjj3yva closed this issue 6 months ago

You're welcome 😊

question/discussion

No Milestone

No Assignees

2 Participants

Notifications

Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.