rust: add hasLibraries and hasIncludes by bbigras · Pull Request #119 · numtide/devshell

bbigras · 2021-06-02T18:29:54Z

zimbatm

Sounds good on principle. It looks a lot like the language.c module so maybe it's better to import that instead. @Mic92 also said that LD_LIBRARY_PATH means that the binary won't work outside of the devshell if I am not mistaken.

Mic92 · 2021-06-03T06:47:51Z

extra/language/rust.nix

+    libraries = mkOption {
+      type = types.listOf strOrPackage;
+      default = [ ];
+      description = "Use this when another language dependens on a dynamic library";


Typo, but I still don't understand what this means.

Suggested change

description = "Use this when another language dependens on a dynamic library";

description = "Use this when another language depends on a dynamic library";

What would be another language? Can you make an example.

I think "another language" isn't really necessary (and a holdover from the equivalent C option), this should really be for any non-crate library dependencies (i.e. anything that would need pkg-config).

Mic92 · 2021-06-03T06:49:14Z

extra/language/rust.nix

+    includes = mkOption {
+      type = types.listOf strOrPackage;
+      default = [ ];
+      description = "Rust dependencies from nixpkgs";


Suggested change

description = "Rust dependencies from nixpkgs";

description = "Dependencies on C headers to provided by nixpkgs";

Mic92 · 2021-06-03T06:50:48Z

extra/language/rust.nix

-    }];
+    }]
+    ++
+    (lib.optionals hasLibraries [


Do we actually want to make the separation into libraries/includes? Would it not be easier to have just one package options that adds both?

Since includes were for C and specifically C_INCLUDE_PATH, I don't see how it's necessary to port that to the rust options.

Mic92 · 2021-06-03T06:52:11Z

extra/language/rust.nix

+        prefix = "$DEVSHELL_DIR/include";
+      }
+      {
+        name = "PKG_CONFIG_PATH";


Don't libraries also depend on PKG_CONFIG_PATH some time?

bbigras · 2021-06-03T06:53:53Z

@Mic92 it's a copy/paste from https://github.com/numtide/devshell/blob/84f1493776f44960e329f21843ffad1686b6c7a3/extra/language/c.nix btw.

extra/language/rust.nix

Mic92 · 2021-06-03T07:00:25Z

@Mic92 it's a copy/paste from https://github.com/numtide/devshell/blob/84f1493776f44960e329f21843ffad1686b6c7a3/extra/language/c.nix btw.

Ok. Opened issue here: #120

bbigras · 2021-06-03T20:15:21Z

@Mic92 also said that LD_LIBRARY_PATH means that the binary won't work outside of the devshell if I am not mistaken.

Does it differ from using a "normal" nix-shell with cargo as a Rust dev environment?

With a "normal" nix-shell it can happen that a crate is linked against a lib that gets cleared by the gc. I have no idea if there's a solution for this.

zimbatm · 2021-12-13T09:54:52Z

Oops, I just realized that PRs have stacked up. Is this PR still relevant?

happysalada · 2022-03-17T04:12:02Z

This PR is still relevant. Compilation on a rust project actually fails at the moment.
I tried to set up the environment variables in this PR however and it still failed.
There is something done in the hooks that makes darwin compilation work.
I should be able to get a reproducible example for anyone who has darwin. (probably this weekend)

happysalada · 2022-03-19T20:29:59Z

alright, I tested on x86_darwin and couldn't reproduce the problem on a public repo.
On a private repo on aarch64-darwin. I had to
set RUSTFLAGS="-L${pkgs.darwin.apple_sdk.frameworks.CoreFoundation}/Library/Frameworks"
(also had to add the Security and SystemConfiguration)
I also ran into trouble compiling curl-sys library, for that I had to set
CFLAGS="-F ${pkgs.darwin.apple_sdk.frameworks.SystemConfiguration}/Library/Frameworks -I${pkgs.darwin.apple_sdk.libs.libDER}/include"

it took me quite a while to figure those out. I don't know exactly in which hook those are getting set. Inside the rustflags variable are stuff for the linker. So those could have been set into a wrapper for ld. The cflags could have been set in another wrapper. Not sure exactly which wrapper those are.

If anyone has an idea, I'm happy to test things.

zimbatm · 2022-03-20T17:11:00Z

Thanks for taking the time and test things. Are there any lessons to integrate back into the module?

happysalada · 2022-03-20T19:26:19Z

I did some testing on linux and had to add glibc to the rustflags (just mentioning).

I quite like the api of this module, however I don't think I would implement it in this way. Let me try to work on this next week and make a PR. We can compare both approaches and discuss what to include and what not to include in the final solution.

basically I would use RUSTFLAGS, CFLAGS, and PKG_CONFIG_PATH which are all the environment variables I needed.

happysalada · 2022-03-21T00:08:28Z

I just hit some linkers error on ubuntu, I see why those LD variables are important.
I had to add librdkafka to the LD_LIBRARY_PATH.
After that it complained about

/lib/x86_64-linux-gnu/libpthread.so.0: version `GLIBC_PRIVATE` not found (required by _nix_path_glibc-2-33-177/lib/librt.so.1)

I tried adding ${pkgs.glibc.dev}/include into the LD_INCLUDE_PATH but to no avail. I'm not sure how to fix this one. If anyone has an idea, I'm interested to try.

(this is just for ubuntu, for darwin, the previous fixes where enough).

happysalada · 2022-03-21T02:59:09Z

so here is how I understand the situation.
If I add in the rustflags "${pkg.glibc}/lib" then rust compiles against that version of glibc. (I get a compiler error without that variable. However If I add this library to my LD_LIBRARY_PATH then at runtime, the system will need to find that library. I think it's usually done through the RPATH. in nixpkgs, ld has a wrapper to set the rpath. If we link against a library that the system doesn't have by default in it's RPATH (like on ubuntu any nix libs won't be included by default), we need to have a way to set the RPATH.
The previous error I get is the system's library trying to look for nix glibc and failing (I think).
We might have to keep the hooks that govern LD.

happysalada · 2022-03-22T13:26:21Z

I'm not sure what to do next to tackle the linker problem.
I've found a way to make it work on darwin, but I'm sure it's not how nix traditionally does it (I've verified in the normal shell, the rustflags variable is not set).

happysalada · 2022-03-30T12:43:34Z

I have some small updates on this.
using lld_13 and RUSTFLAGS="-C linkg-arg=-fuse-ld=lld" solved all the linker issues I had on darwin.

On ubuntu I still run into a glibc compilation error no matter what I do.
if anyone ever finds a solution to that one, I'm interested.

happysalada · 2022-04-02T03:13:24Z

alright, I've got more news.

what is needed here is the following

LIBRARY_PATH set to $DEVSHELL_DIR/lib
RUSTFLAGS set to -F framework=$DEVSHELL_DIR/Library/Frameworks. The problem with that is that someone needs to be able to set rustflags if they want to. Would doing something with an eval like `eval = "-F framework=$DEVSHELL_DIR/Library/Frameworks:$RUSTFLAGS" work ? (RUSTDOCFLAGS need to be set to the same value)
CFLAGS set to -I $DEVSHELL_DIR/include -iframework $DEVSHELL_DIR/Library/Frameworks. Same here someone might want to set that value, so wondering if the trick with eval would work.

if that would work, I'm happy to make a PR.

PS: the reason why it was working before is that it was using the clang already installed on my computer. As soon as I had clang_13 in my shell everything started breaking down without the above fixes).

The other problem I had was that I had to bring some weird libs that I've never heard of before like darwin.apple_sdk.libs.libDER. This makes me thing that it's not the right way to solve this problem, but at least the issues are fixed.

zimbatm · 2022-04-02T10:24:29Z

Thanks for taking the time to untangle this mess.

Yeah, a PR where eval is used on RUSTFLAGS and CFLAGS sounds good.

rust: add hasLibraries and hasIncludes

83c10b1

zimbatm approved these changes Jun 2, 2021

View reviewed changes

Mic92 reviewed Jun 3, 2021

View reviewed changes

extra/language/rust.nix Show resolved Hide resolved

	description = "Use this when another language dependens on a dynamic library";
	description = "Use this when another language depends on a dynamic library";

	description = "Rust dependencies from nixpkgs";
	description = "Dependencies on C headers to provided by nixpkgs";

Conversation

bbigras commented Jun 2, 2021

Uh oh!

zimbatm left a comment

Choose a reason for hiding this comment

Uh oh!

Mic92 Jun 3, 2021

Choose a reason for hiding this comment

Uh oh!

Lord-Valen Aug 13, 2023

Choose a reason for hiding this comment

Uh oh!

Mic92 Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mic92 Jun 3, 2021

Choose a reason for hiding this comment

Uh oh!

Lord-Valen Aug 13, 2023

Choose a reason for hiding this comment

Uh oh!

Mic92 Jun 3, 2021

Choose a reason for hiding this comment

Uh oh!

bbigras commented Jun 3, 2021

Uh oh!

Uh oh!

Mic92 commented Jun 3, 2021

Uh oh!

bbigras commented Jun 3, 2021

Uh oh!

zimbatm commented Dec 13, 2021

Uh oh!

happysalada commented Mar 17, 2022

Uh oh!

happysalada commented Mar 19, 2022

Uh oh!

zimbatm commented Mar 20, 2022

Uh oh!

happysalada commented Mar 20, 2022

Uh oh!

happysalada commented Mar 21, 2022

Uh oh!

happysalada commented Mar 21, 2022

Uh oh!

happysalada commented Mar 22, 2022

Uh oh!

happysalada commented Mar 30, 2022

Uh oh!

happysalada commented Apr 2, 2022

Uh oh!

zimbatm commented Apr 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Mic92 Jun 3, 2021 •

edited

Loading