#150 Check cdnjs' repo for missing frameworks

Closed
opened 6 months ago by fyjzjj3yva · 6 comments

Although LocalCDN has included many frameworks, it is not complete and many less popular frameworks are probably not included.

In addition, manually checking each website to find out whether a framework/CDN combination is missing and then reporting it to the developer requires much work and time.

To include as many frameworks as possible, I think we can check cdnjs' repo.
That being said, there are nearly 4000 entries in cdnjs/ajax/libs/, so including all of them may require a lot of storage space.

Although LocalCDN has included many frameworks, it is not complete and many less popular frameworks are probably not included. In addition, manually checking each website to find out whether a framework/CDN combination is missing and then reporting it to the developer requires much work and time. To include as many frameworks as possible, I think we can check [cdnjs' repo](https://github.com/cdnjs/cdnjs/tree/master/ajax/libs). That being said, there are **nearly 4000 entries** in `cdnjs/ajax/libs/`, so including all of them may require a lot of storage space.
nobody added the
question/discussion
label 6 months ago
nobody commented 6 months ago
Owner

The intention of LocalCDN is not to integrate and replace all existing frameworks. It's to make a website readable without contacting the CDNs.

The intention of LocalCDN is not to integrate and replace all existing frameworks. It's to make a website readable without contacting the CDNs.
Poster

So, in order for a framework mapping to be included in LocalCDN...

  1. Find a website using XXX framework
  2. Make sure the website calls the framework via CDN (i.e. the website does not host the framework itself)
  3. Block the framework request (e.g. by using uBlock Origin) to see whether it is really necessary. If the website breaks, looks weird, or some features cannot be used, then it can be reported to the developer.

Like this?

So, in order for a framework mapping to be included in LocalCDN... 1. Find a website using XXX framework 2. Make sure the website calls the framework via CDN (i.e. the website does not host the framework itself) 3. Block the framework request (e.g. by using uBlock Origin) to see whether it is really necessary. If the website breaks, looks weird, or some features cannot be used, then it can be reported to the developer. Like this?
nobody commented 6 months ago
Owner

Yes, exactly. That' s a way to test it. I'm using the developer tools of Firefox. Menu -> Web Developer -> Network. There you can block single requests.

Yes, exactly. That' s a way to test it. I'm using the developer tools of Firefox. Menu -> Web Developer -> Network. There you can block single requests.
nobody commented 6 months ago
Owner

By the way, when testing and reporting framework mappings, is it a good idea to enable the “block requests for missing resources” option?

You can do that. When I test something I use a new Firefox profile and the developer tools of Firefox. At the same time I load the extension in another profile as "Temporary Extension" and activate logging in LocalCDN.

Is that option helpful in cases like #147, when a website requests unnecessary frameworks from CDNs?

This is hard to say in general terms, which is better. I always block all non existing frameworks.

By the way, is cookieconsent eligible for inclusion?

I think so. It is not big and could be important for someone. I create a separate issue (#153)

How does that option work?

TL;DR:

The listener webRequest.onBeforeRequest depends on the CDNs in mappings.js. If a CDN isn't listed there, LocalCDN ignores this request.

Does that option block all requests to CDN domains like cdnjs.cloudflare.com, cdn.jsdelivr.net?

If this domain is listed in mappings.js, then yes.


Details

Example: cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js

  1. First the URL split into host and path
    host = cdnjs.cloudflare.com
    path = ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js

  2. The supported CDNs are stored here in the requestFilter and used here. Only matches with a CDN are processed

Non-existent frameworks

  1. If the path part doesn't match in mapping.js, it returns false (here).

  2. This false will be processed here

  3. LocalCDN decides here what to do with non-existing frameworks

  4. The final step for non-existent frameworks is here with a redirection from HTTP to HTTPS if necessary.

Existent frameworks

Example: cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js

  1. Extract the version number of origin path jquery/2.1.4/jquery.min.js -> 2.1.4 (request-analyzer.js#L125)

  2. Insert extracted version into the target path resources/jquery/{version}/jquery.min.jsm -> resources/jquery/2.1.4/jquery.min.jsm (request-analyzer.js#L142)

  3. Determine and replace the target version resources/jquery/2.1.4/jquery.min.jsm -> resources/jquery/2.2.4/jquery.min.jsm (helpers.js#L427)

> By the way, when testing and reporting framework mappings, is it a good idea to enable the “block requests for missing resources” option? You can do that. When I test something I use a new Firefox profile and the developer tools of Firefox. At the same time I load the extension in another profile as "Temporary Extension" and activate logging in LocalCDN. > Is that option helpful in cases like #147, when a website requests unnecessary frameworks from CDNs? This is hard to say in general terms, which is better. I always block all non existing frameworks. > By the way, is cookieconsent eligible for inclusion? I think so. It is not big and could be important for someone. I create a separate issue (#153) > How does that option work? ## TL;DR: The listener [webRequest.onBeforeRequest](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onBeforeRequest) depends on the CDNs in [mappings.js](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/mappings.js). If a CDN isn't listed there, LocalCDN ignores this request. > Does that option block all requests to CDN domains like cdnjs.cloudflare.com, cdn.jsdelivr.net? If this domain is listed in mappings.js, then yes. --- ## Details Example: `cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js` 1. First the URL split into host and path host = cdnjs.cloudflare.com path = ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js 2. The supported CDNs are stored [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/state-manager.js#L140) in the requestFilter and used [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/state-manager.js#L148). Only matches with a CDN are processed #### Non-existent frameworks 3. If the path part doesn't match in [mapping.js](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/mappings.js#L88), it returns `false` ([here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L90)). 4. This `false` will be processed [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L59) 5. LocalCDN decides [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L79) what to do with non-existing frameworks 6. The final step for non-existent frameworks is [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L93) with a redirection from HTTP to HTTPS if necessary. #### Existent frameworks Example: `cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js` 3. Extract the version number of origin path `jquery/2.1.4/jquery.min.js` -> `2.1.4` ([request-analyzer.js#L125](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L125)) 4. Insert extracted version into the target path `resources/jquery/{version}/jquery.min.jsm` -> `resources/jquery/2.1.4/jquery.min.jsm` ([request-analyzer.js#L142](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L142)) 5. Determine and replace the target version `resources/jquery/2.1.4/jquery.min.jsm` -> `resources/jquery/2.2.4/jquery.min.jsm` ([helpers.js#L427](https://codeberg.org/nobody/LocalCDN/src/branch/main/modules/internal/helpers.js#L427))
Poster

Many thanks for the detailed explanation!

Many thanks for the detailed explanation!
fyjzjj3yva closed this issue 6 months ago
nobody commented 6 months ago
Owner

You're welcome 😊

You're welcome :blush:
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.