Return Styles: Pseud0ch, Terminal, Valhalla, NES, Geocities, Blue Moon. Entire thread

[BEGINNER] Trying to write efficient code

Name: Anonymous 2010-05-07 12:03

I am writing small, simple script that deals with a flat-file database. It only uses 1 text file for this and it is quite small at the moment. However, this text file has the potential to become big because as my script is used, it continually adds new information to this text file.

To keep it simple, this is what my script does: It adds a new line to the top of the db file.

I have two options here:

1. Load the entire db file into a list (memory),
unshift new line,
then write it back into the file.

2. Create a temporary file,
write new line to temp file,
using a while loop (line-by-line) write the entire db file into the temp file,
then replace (rename) the original db file with the newer temp file.

It seems to me that option 2 is better because it doesn't load the entire file into memory. Of course, I'm just a novice and drawbacks are not as obvious to me.

So I guess my questions are: Am I correct in my assumption that option 2 is better? Or am I missing something? Is one option more cpu-intensive than the other?

Name: Anonymous 2010-05-09 2:11

>>1
Actually, both options do virtually similar things if you think about it.  Where "x" are the number of characters in the file and "y" is the length of the new entry:
1:
a. Read entire file (time: x reads)
b. Append new entry (time: y writes)
c. Write entire file (time: x + y writes)
Total: 2x + 2y

2:
a. Write new entry (time: y)
b. Read first line (cumulative time: x reads)
c. Read first line (cumulative time: x writes)
Time: 2x + y


Timing is also important: if you do (1) it would probably be faster to load all data once at the start of the program, append continuously until the program closes, then write in one shot when done.  The trade-off is that you will need to push around a lot of volatile data in memory at any given time, especially if you expect to add sorted data.  (2), on the other hand, will not work efficiently unless you only create temporary files throughout the program's run and only compile all created files only at a fixed interval; and, even then, that will still bog your program down in file writing and management processes and render new data added to it inconvenient to search.

Think of it this way: if your program is expected to run at a consistent rate, it will have to consistently call file reading and writing services using either your methods at face value.  So, no, (2) will probably be not very efficient for a constantly running and polling program, at least not one with an exceptional growth curve, but (1) presents its own concerns.

Newer Posts
Don't change these.
Name: Email:
Entire Thread Thread List