Efficiently dealing with dependency version ranges #22

New Issue

Open

opened 2023-06-28 02:06:35 +00:00 by philipmw · 1 comment

philipmw commented

2023-06-28 02:06:35 +00:00

I have a project where lein commands run instantly.

When I add [software.amazon.awscdk/aws-cdk-lib "2.85.0"] to :dependencies, Lein starts to take almost three minutes to prepare the program, consistently.

Here's an example lein test: https://pastebin.mozilla.org/joi8JKs8

I brought this up on Clojure Zulip: https://clojurians.zulipchat.com/#narrow/stream/151763-beginners/topic/.E2.9C.94.20Lein.20takes.20minutes.20to.20run.20any.20task

And a kind participant determined that this is caused by version ranges. The problem goes away if I specify versions of all ranged transitive dependencies. This is not tractable without a program, which the kind participant wrote as a proof of concept.

This behavior makes it very challenging to use Leiningen with AWS CDK and certainly other dependencies. (I don't know how common this problem is.) Since I cannot control dependencies, this makes it challenging to develop with Clojure.

This problem doesn't appear to exist with other tools like Maven or Gradle.

Is there any solution to the long runtime that Leiningen can offer?

I have a project where `lein` commands run instantly. When I add `[software.amazon.awscdk/aws-cdk-lib "2.85.0"]` to `:dependencies`, Lein starts to take almost three minutes to prepare the program, consistently. Here's an example `lein test`: https://pastebin.mozilla.org/joi8JKs8 I brought this up on Clojure Zulip: https://clojurians.zulipchat.com/#narrow/stream/151763-beginners/topic/.E2.9C.94.20Lein.20takes.20minutes.20to.20run.20any.20task And a kind participant determined that this is caused by version ranges. The problem goes away if I specify versions of all ranged transitive dependencies. This is not tractable without a program, which the kind participant wrote as a proof of concept. This behavior makes it very challenging to use Leiningen with AWS CDK and certainly other dependencies. (I don't know how common this problem is.) Since I cannot control dependencies, this makes it challenging to develop with Clojure. This problem doesn't appear to exist with other tools like Maven or Gradle. Is there any solution to the long runtime that Leiningen can offer?

technomancy commented

2023-06-29 00:03:31 +00:00

Owner

Ideally the best case issue would be to get the dependency to fix their declaration and stop using version ranges, but since it's coming from Amazon, that is probably unlikely.

As a stopgap measure, you can disable this check altogether by adding :pedantic? false to your project.clj file, however if you do this you will also miss out on legitimate warnings from other dependencies.

You may also be able to bypass this problem by adding exclusions, but it appears this dependency would need so many exclusions that this may be infeasible.

I think the ideal solution for Leiningen itself would be to make it so that you can disable pedantic checks on individual dependencies rather than making it an all-or-nothing setting. I would be happy to take a patch for this and can advise you if you want to take a shot at implementing it.

I've looked into this code several times, and I've had a really hard time trying to track down the worst-case performance issues. This problem doesn't occur with Maven or Gradle because those tools are not as concerned with repeatable builds, and thus they don't warn you when your dependencies have this specific problem. Ideally the best case issue would be to get the dependency to fix their declaration and stop using version ranges, but since it's coming from Amazon, that is probably unlikely. As a stopgap measure, you can disable this check altogether by adding `:pedantic? false` to your project.clj file, however if you do this you will also miss out on legitimate warnings from other dependencies. You may also be able to bypass this problem by adding exclusions, but it appears this dependency would need so many exclusions that this may be infeasible. I think the ideal solution for Leiningen itself would be to make it so that you can disable pedantic checks on individual dependencies rather than making it an all-or-nothing setting. I would be happy to take a patch for this and can advise you if you want to take a shot at implementing it.