nobody
/
LocalCDN
15
150
Fork 9
Code Issues Pull Requests Releases 58 Wiki Activity
Labels Milestones
New Issue

Check cdnjs' repo for missing frameworks #150

Closed
opened 12 months ago by fyjzjj3yva · 6 comments
fyjzjj3yva commented 12 months ago

Although LocalCDN has included many frameworks, it is not complete and many less popular frameworks are probably not included.

In addition, manually checking each website to find out whether a framework/CDN combination is missing and then reporting it to the developer requires much work and time.

To include as many frameworks as possible, I think we can check cdnjs' repo.
That being said, there are nearly 4000 entries in cdnjs/ajax/libs/, so including all of them may require a lot of storage space.

Although LocalCDN has included many frameworks, it is not complete and many less popular frameworks are probably not included. In addition, manually checking each website to find out whether a framework/CDN combination is missing and then reporting it to the developer requires much work and time. To include as many frameworks as possible, I think we can check [cdnjs' repo](https://github.com/cdnjs/cdnjs/tree/master/ajax/libs). That being said, there are **nearly 4000 entries** in `cdnjs/ajax/libs/`, so including all of them may require a lot of storage space.
nobody added the
question/discussion
 label 12 months ago
nobody commented 12 months ago
Owner

The intention of LocalCDN is not to integrate and replace all existing frameworks. It's to make a website readable without contacting the CDNs.

The intention of LocalCDN is not to integrate and replace all existing frameworks. It's to make a website readable without contacting the CDNs.
fyjzjj3yva commented 12 months ago
Poster

So, in order for a framework mapping to be included in LocalCDN...

  1. Find a website using XXX framework
  2. Make sure the website calls the framework via CDN (i.e. the website does not host the framework itself)
  3. Block the framework request (e.g. by using uBlock Origin) to see whether it is really necessary. If the website breaks, looks weird, or some features cannot be used, then it can be reported to the developer.

Like this?

So, in order for a framework mapping to be included in LocalCDN... 1. Find a website using XXX framework 2. Make sure the website calls the framework via CDN (i.e. the website does not host the framework itself) 3. Block the framework request (e.g. by using uBlock Origin) to see whether it is really necessary. If the website breaks, looks weird, or some features cannot be used, then it can be reported to the developer. Like this?
nobody commented 12 months ago
Owner

Yes, exactly. That' s a way to test it. I'm using the developer tools of Firefox. Menu -> Web Developer -> Network. There you can block single requests.

Yes, exactly. That' s a way to test it. I'm using the developer tools of Firefox. Menu -> Web Developer -> Network. There you can block single requests.
nobody commented 12 months ago
Owner

By the way, when testing and reporting framework mappings, is it a good idea to enable the “block requests for missing resources” option?

You can do that. When I test something I use a new Firefox profile and the developer tools of Firefox. At the same time I load the extension in another profile as "Temporary Extension" and activate logging in LocalCDN.

Is that option helpful in cases like #147, when a website requests unnecessary frameworks from CDNs?

This is hard to say in general terms, which is better. I always block all non existing frameworks.

By the way, is cookieconsent eligible for inclusion?

I think so. It is not big and could be important for someone. I create a separate issue (#153)

How does that option work?

TL;DR:

The listener webRequest.onBeforeRequest depends on the CDNs in mappings.js. If a CDN isn't listed there, LocalCDN ignores this request.

Does that option block all requests to CDN domains like cdnjs.cloudflare.com, cdn.jsdelivr.net?

If this domain is listed in mappings.js, then yes.

Details

Example: cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js

  1. First the URL split into host and path
    host = cdnjs.cloudflare.com
    path = ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js

  2. The supported CDNs are stored here in the requestFilter and used here. Only matches with a CDN are processed

Non-existent frameworks

  1. If the path part doesn't match in mapping.js, it returns false (here).

  2. This false will be processed here

  3. LocalCDN decides here what to do with non-existing frameworks

  4. The final step for non-existent frameworks is here with a redirection from HTTP to HTTPS if necessary.

Existent frameworks

Example: cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js

  1. Extract the version number of origin path jquery/2.1.4/jquery.min.js -> 2.1.4 (request-analyzer.js#L125)

  2. Insert extracted version into the target path resources/jquery/{version}/jquery.min.jsm -> resources/jquery/2.1.4/jquery.min.jsm (request-analyzer.js#L142)

  3. Determine and replace the target version resources/jquery/2.1.4/jquery.min.jsm -> resources/jquery/2.2.4/jquery.min.jsm (helpers.js#L427)

> By the way, when testing and reporting framework mappings, is it a good idea to enable the “block requests for missing resources” option? You can do that. When I test something I use a new Firefox profile and the developer tools of Firefox. At the same time I load the extension in another profile as "Temporary Extension" and activate logging in LocalCDN. > Is that option helpful in cases like #147, when a website requests unnecessary frameworks from CDNs? This is hard to say in general terms, which is better. I always block all non existing frameworks. > By the way, is cookieconsent eligible for inclusion? I think so. It is not big and could be important for someone. I create a separate issue (#153) > How does that option work? ## TL;DR: The listener [webRequest.onBeforeRequest](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onBeforeRequest) depends on the CDNs in [mappings.js](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/mappings.js). If a CDN isn't listed there, LocalCDN ignores this request. > Does that option block all requests to CDN domains like cdnjs.cloudflare.com, cdn.jsdelivr.net? If this domain is listed in mappings.js, then yes. --- ## Details Example: `cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js` 1. First the URL split into host and path host = cdnjs.cloudflare.com path = ajax/libs/cookieconsent2/3.0.3/cookieconsent.min.js 2. The supported CDNs are stored [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/state-manager.js#L140) in the requestFilter and used [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/state-manager.js#L148). Only matches with a CDN are processed #### Non-existent frameworks 3. If the path part doesn't match in [mapping.js](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/mappings.js#L88), it returns `false` ([here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L90)). 4. This `false` will be processed [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L59) 5. LocalCDN decides [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L79) what to do with non-existing frameworks 6. The final step for non-existent frameworks is [here](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/interceptor.js#L93) with a redirection from HTTP to HTTPS if necessary. #### Existent frameworks Example: `cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js` 3. Extract the version number of origin path `jquery/2.1.4/jquery.min.js` -> `2.1.4` ([request-analyzer.js#L125](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L125)) 4. Insert extracted version into the target path `resources/jquery/{version}/jquery.min.jsm` -> `resources/jquery/2.1.4/jquery.min.jsm` ([request-analyzer.js#L142](https://codeberg.org/nobody/LocalCDN/src/branch/main/core/request-analyzer.js#L142)) 5. Determine and replace the target version `resources/jquery/2.1.4/jquery.min.jsm` -> `resources/jquery/2.2.4/jquery.min.jsm` ([helpers.js#L427](https://codeberg.org/nobody/LocalCDN/src/branch/main/modules/internal/helpers.js#L427))
👍 1
fyjzjj3yva commented 12 months ago
Poster

Many thanks for the detailed explanation!

Many thanks for the detailed explanation!
fyjzjj3yva closed this issue 12 months ago
nobody commented 12 months ago
Owner

You're welcome 😊

You're welcome :blush:
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
develop
main
Labels
Apply labels
Clear labels
bug
Something is not working Chromium incompatibility
Not compatible with chromium based browsers documentation
documentation duplicate
This issue or pull request already exists enhancement
Feature request/Enhancement firefox-bug
Firefox bug help wanted
Need some help invalid
Something is wrong missing framework/mapping
Missing framework or mapping need info
Need info from the reporter observation question/discussion
Question about the extension sop/cors
Same-Origin-Policy or Cross-Origin-Resource-Sharing testing
testing update framework
Update framework website
Own Website (www.localcdn.org) wontfix
This won't be fixed
No Label bug Chromium incompatibility documentation duplicate enhancement firefox-bug help wanted invalid missing framework/mapping need info observation question/discussion sop/cors testing update framework website wontfix
Milestone
Set milestone
Clear milestone
No items
No Milestone
Assignees
Assign users
Clear assignees
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch '%!s(MISSING)'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes