Return Styles: Pseud0ch, Terminal, Valhalla, NES, Geocities, Blue Moon.

Pages: 1-

simple file copy program

Name: Anonymous 2010-01-18 23:35

need a bash script or python to copy files in batches of 500 and put them into subdirectories

there can be any number of files (I have 15k), and the filenames are irrelevant, and the names of the subdirectories that are created to house the files is irrelevant

here's what I was trying to get work in python but it
errors out at file 339, don't know why



def BreakUp(indir):
   
    biglist = glob.glob('*.mp3')
    dircount = 0
    filecount = 0

    while biglist:
        dircount += 1
        os.mkdir('subdir'+str(dircount))
        for i in range(500):   
            filecount += 1
            try:
                src = biglist[filecount].translate("(copy)")
                os.system ("mv"+ " " + src + " subdir"+ str(dircount))          
                biglist.remove(src) 
            except IndexError:
                raise IndexError, "could not return item " + str(filecount) + " from list " + str(biglist)
        continue
               
           
       

if __name__ == '__main__':
    BreakUp(".")

Name: Anonymous 2010-01-18 23:37

What is file 339?

Name: Anonymous 2010-01-19 0:08

>>1
python
Found your problem, bro.  Try this:
perl -e'mkdir$_ for 0..(@f=<*.mp3>)/500;rename$_,int($n++/500)."/$_"for@f'

Name: Anonymous 2010-01-19 0:12

>>3

thanks.  that looks very compact!

I don't really can read it and therefore do not want to run it without knowing what it does.

hows f get incremented?

sorry I'm not that familiar with perl

Name: Anonymous 2010-01-19 0:25

@f is the array of the filenames from globbing *.mp3

Name: Anonymous 2010-01-19 0:28

>>4
Pattern matching and regular expressions are things perl does really fucking well. That's why f increments, it assumes that you want to run through a list of 500 items and apply the function f to it each item.

Name: Anonymous 2010-01-19 2:59

>>3
I'm learning about perl and regular expressions.  I'm going to try and understand what the line does.  It would be appreciated if you could provide feedback on it.

perl -e gets whatever the stream is and compiles it without needing to save the script.
mkdir$_ creates a new directory, the name of it is the special variable
for 0..(@f=<*.mp3>)/500; This line I have some difficulty understanding.
rename$_ tells mkdir$_ that the directory will be what's following.
,int($n++/500) is what $_ becomes.  It counts how many groups of 500 have been so far and makes it that.
."/$_"for@f' renames the file in the current n position in @f to have the directory location included in the name, EG ./516.mp3 will become ./1/516.mp3 and ./142.mp3 will become ./0/142.mp3 .

Willing to help?

Name: Anonymous 2010-01-19 3:30

>>7
I believe the fors in this snippet apply to the preceding statements, and $_ is the loop variable.
@f would be an array, but used in scalar context (with division) takes its length as its value. Also . would be string concatenation.

Name: Anonymous 2010-01-19 3:42

Right.  Different than I thought entirely.  Thinking is required on my behalf.

Name: Anonymous 2010-01-19 3:52

>>7
Fine…here's the same thing expanded a bit, with the loops in block form instead of using statement modifiers.

@f = <*.mp3>;          #  @f becomes an array of filenames matching the glob.
for (0 .. @f / 500) {  #  @f taken in scalar context = length of @f,
                       #    so 0 through floor(@f / 500) are the directories we'll need.
                       #    Loop across them setting $_ to each number.
    mkdir $_;          #  Make a directory each time through the loop.
}
for (@f) {  #  This loop sets $_ to each individual filename.
    #  $n is undefined at first, which is zero when taken in a numerical context.
    #  It gets incremented each time through the loop, so floor($n / 500) tells us
    #    what directory we should be moving things into.
    rename $_, (int($n++ / 500) . "/" . $_);  #  Rename "file" to "dir/file".
    #  There is no proper "move" operator (the File::Copy module handles that),
    #    but rename is equivalent as long as it stays on the same filesystem.
}

Name: Anonymous 2010-01-19 3:56

That makes lots of sense. 
Now to try to wrap my head around the shorter version with the actual way it works.

Thank you for explaining it.

Name: Anonymous 2010-01-19 4:45

you can do it in BATCH

unless, of course, you're a UNIX user.

Name: Anonymous 2010-01-19 11:36

I will never, ever learn perl. I would lick Bjarne's balls, thanking him for sepples, before I ever learn perl. And I would never give thanks for sepples.

Fuck you, Larry. Fuck you.

Name: Anonymous 2010-01-19 12:03

>>13
What? Don't you want the language you're using to do what you want? I bet you use Python too.

Name: Anonymous 2010-01-19 13:55

>>14
Don't you want the language you're using to do what you want?
What makes you think I want a language to look like a cat walked on the keyboard, Larry?

Name: Anonymous 2010-01-19 16:53

>>15
Just because python is too retarded to allow you to write ugly code does not mean perl is bad. Maybe you're just too much of a pussy for it.

Name: Anonymous 2010-01-19 16:59

>>16
Not a pythonista, Larry, but even if I were, the forced indentation of code is a bit more preferable than the forced obfuscation of code.

Name: Anonymous 2010-01-19 16:59

>>16
But python is excellent for masochists

Name: Anonymous 2010-01-19 17:11

What's sad is that for a simple administrative task like renaming some files in a systematic way I'd go for Python, just like >>1-chan did.
However, today >>3-sama opened my eyes and I saw the light!

Name: Anonymous 2010-01-19 19:42

>>3,10,19
Just to dispel the myth that this is much harder in Python, here's the identical code in Python: (I did not test it)

import os
f = [i for i in os.listdir('.') where i[:-4].lower() == ".mp3"]
for i in range(0, len(f)/500):
  os.mkdir(str(i))
n = 0
for i in f:
  os.rename(i, str(n/500) + "/" + f)
  n += 1

Notable overhead as compared to perl: namespaced os functions; no built-in globbing operation; forced int to string conversion; and no post-increment.

Note that I added lower() to the .mp3 filter. The path concatenation should also be done with os.path.join() instead of doing string concatenation with a forward slash.

Name: >>3 2010-01-19 19:47

>>20
YHBT

Name: Anonymous 2010-01-19 19:48

>>1
OP: 339 is the maximum number of entries allowed in a Python `list".

Name: Anonymous 2010-01-19 19:53

>>22
Don't know if trolling - they only way that could be true is if it created a stack frame for each iteration. I don't think FIOC does that, in fact Guido if quite stack-allergic. In any case xrange should be used instead of range.

Name: Anonymous 2010-01-19 19:56

>>22
What the fuck. This is a ridiculous troll.

Name: Anonymous 2010-01-19 20:26

>>24
Are you trolling me?  This is a well-known limitation of the Python execution model.  If you need more than 339 elements you simply use a list-of-lists (or a list-of-lists-of-lists for very large sets), and nest your loops accordingly.

Name: Anonymous 2010-01-19 20:31

>>25

Name: Anonymous 2010-01-19 20:32

>>24
>>> l = [i for i in range(0,339)]
>>> l.append(339)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Excpetion: StackOverflow

Name: Anonymous 2010-01-19 20:37

>>25,27
I fell for it ;_; Or they fixed it in python 2.6

Name: Anonymous 2010-01-19 20:41

>>27
Excpetion

Name: Anonymous 2010-01-19 20:57

>>29
No Excpetions!

Name: PHP FAGGOT 2010-01-19 21:10

NOW WITH PHP!
php -r "foreach(glob('*.mp3') as $k=>$v){$f=(int)($k/500);$t=file_exists($f)?@rename($v,$f.'/'.$v):mkdir($f) and @rename($v,$f.'/'.$v);}"

Name: Anonymous 2010-01-19 21:21

>>29
Indeed, not only does python have FIOC but it is of very poor quality.

Name: Anonymous 2010-01-19 21:29

>>31
You know a lot about PHP. Are you ashamed? Do you feel dirty?

Name: Anonymous 2010-01-20 1:53



import os, glob

def BreakUp(indir,group_size):
    biglist = glob.glob(indir + '/*')
    dircount = 0
    for f, eachfile in enumerate(biglist):
        if f % group_size == 0:
            dircount += 1
            os.mkdir(indir + '/subdir'+str(dircount))
        os.system ("mv " + eachfile + "  " + indir + "/subdir"+ str(dircount))         
       

if __name__ == '__main__':
    os.system('rm -r ./test')
    os.system('mkdir ./test')
    os.system('cd ./test ; for i in $(seq 9000) ; do touch $i ; done ') ;
    BreakUp("./test",500)

Name: PHP FAGGOT 2010-01-20 2:04

>>33.
No, because I am PHP FAGGOT.

Name: Anonymous 2011-02-03 0:45

Name: Anonymous 2011-02-04 16:06

Don't change these.
Name: Email:
Entire Thread Thread List
All trademarks and copyrights on this site are owned by their respective parties. All comments and posts are the responsibility of their own posters.
- Tinychan + 2ch Mode -