use `file.copy` instead of `file.rename`

[LiNk-NY]

Hi Lori, @lshep

I got this error message when copying a file from my /home/$USER/Downloads to
another partition on our server.

In file.rename(fpath, rpath) :
  cannot rename file '/home/mramos/Downloads/hnsc_tcga.tar.gz' to '/data/16tb/cbio/2e7912cc3e62_hnsc_tcga.tar.gz', reason 'Invalid cross-device link'

This has been seen before in vtest issue 14
and it looks like the solution is to use file.copy.

Could this be changed in BiocFileCache?
Thanks!

Regards,
Marcel

[lshep]

Can you give me an example of the code you are trying to run that causes this?

[mtmorgan]

one consequence of this is that it, well, copies the file, which could be very expensive; maybe try file.rename and on failure file.copy? Or determine a priori whether file.copy is need (separate file systems; not sure how one would do that...)

[LiNk-NY]

Hi Lori, @lshep

Here is some code to reproduce the issue.
This would require that you have a separate partition installed on your system or a USB drive.

in a shell

cd ~/Downloads && curl -O http://download.cbioportal.org/coadread_mskcc.tar.gz

in R

library(BiocFileCache)
bfc <- BiocFileCache("/media/lshep/USB")
bfcadd(bfc, "coadread_mskcc", fpath = "~/Downloads/coadread_mskcc.tar.gz", download = FALSE, action = "move")

@mtmorgan Using file.copy when the rename fails sounds like a good alternative.

Thanks,
Marcel

[lshep]

And why couldn't you create the cache in the partition where the files are?

You can always use exportbfc to make the bfc and selection of files portable to put on a drive

[LiNk-NY]

I can create the cache there but I can't move a file across partitions using bfcadd or file.rename (internally).

[LiNk-NY]

@lshep
The USB drive example was just an easy way to reproduce the issue.
We are actually using a 16 TB partition for storing data. The original code creates the cache in
this partition without a problem.

Thanks!

[lshep]

Just on a quick test I don't think file.copy works either - I think this is a limitation of R and the system:

> temp = tempfile()
> temp
[1] "/tmp/RtmpJn1pnt/file238138cf692d"
> file.rename(temp, "/home/lori/loristempfile")
[1] FALSE
Warning message:
In file.rename(temp, "/home/lori/loristempfile") :
  cannot rename file '/tmp/RtmpJn1pnt/file238138cf692d' to '/home/lori/loristempfile', reason 'Invalid cross-device link'
> file.copy(temp, "/home/lori/loristempfile")
[1] FALSE

[LiNk-NY]

Hi Lori, @lshep

tempfile doesn't create a file, it just gives you a possible name.
I'm not sure how you're getting that error. I've tried to reproduce but
I don't get the same warning.

> temp <- tempfile()
> temp
[1] "/tmp/RtmpTVASgz/file1c963193eee2"
> file.rename(temp, "~/tempfile")
[1] FALSE
Warning message:
In file.rename(temp, "~/tempfile") :
  cannot rename file '/tmp/RtmpTVASgz/file1c963193eee2' to '/home/mr148/tempfile', reason 'No such file or directory'

If no file exists, you can't copy or rename it.

[lshep]

Ah yes you are right. But I'm still not quite convinced that this is a good idea. As it has the potential to pollute two different different user locations -
I guess I don't understand why you can't change the action in bfcadd to action="copy" which would user file.copy instead of file.rename. Then instead of a user potentially polluting their system unintentionally - at least it is intentional

[LiNk-NY]

This is mostly an edge case for users who have separate partitions in their file systems
and don't cache in their home folder.

Yes, I can use action="copy" but I prefer "move" for the same reason one would
use mv instead of cp. I'm caching large files and I don't want 2 copies of them.

I liked Martin's idea of using file.copy when file.rename fails and additionally
removing the original file after the copy for action="move".

I've used action = "copy" in the meantime and deleted the files.

Thanks @lshep.

[HenrikBengtsson]

This is mostly an edge case for users who have separate partitions in their file systems
and don't cache in their home folder.

I don't think this is an edge case. On environments with a shared global file system (e.g. HPC environment), working toward local disk can be several orders of magnitude faster (sic!) than working towards the global file system. This is why many of these systems set TMPDIR to point to a local disk on the current machine. For the same reason, one might want to host cache files on a local disk (e.g. under TMPDIR) rather than on the global file system. This can be achieved by setting R_USER_CACHE_DIR or XDG_CACHE_HOME, which tools::R_user_dir(..., which = "cache") respects.

I liked Martin's idea of using file.copy when file.rename fails and additionally removing the original file after the copy for action="move".

I second this. Maybe the default could be action = c("move", "copy") to indicate attempts are made in that order, i.e. "copy" is a fallback to "move".

[hpages]

The fundamental problem is that file.rename() mimics the behavior of the rename Unix system call which is a low-level operation that only works to rename files within a given file system.

Unix command mv is higher-level and works across partitions. It actually does exactly what Martin is suggesting: When used to move files from one file system to another, it copies the files, then delete the originals. This is of course much more expensive than moving files within a file system where no copies need to be made.

Has someone ever requested a file.move() function to be added to base R on R-devel? Otherwise I'd be surpised if no package has implemented a file.move()-like function that does exactly that. @HenrikBengtsson Maybe something for R.utils?