I am painfully aware that many blog posts end up turning into mini-rants about one bit of software development or another, and with that in mind I am cautious in writing yet another. However, I think it's worthy of a little venting just this once.

Recently I have spent quite a bit of time explaining to people why software along the lines of the following end up being slow, memory hungry and generally painful:

open FILE, "<somefile.txt";
my @filecontents = <MYFILE>;
foreach my $line (@filecontents) {
    # Do something with $line
}

I then explain that the following is much better:

fopen FILE, "<somefile.txt";
while (my $line = <FILE>) {
    # Do something with $line
}

So far, so obvious, one would think. Apparently not. It seems that the fine art of memory conservation is long dead, with most programmers opting for the "simple" approach of reading the entire file into memory and, once it's there, working with it line by line. I have even had a Microsoft certified trainer advise that just reading the entire file is the way to go because it's "easier, and memory is basically unlimited anyway". Oh my.

Unfortunately, this isn't limited to file access. The problem also spills over into other similar areas such as SQL, XML parsing and just about any other area in which sequential processing is used. This problem is often compounded by the developer running the script or process on a local, unloaded box, seeing no issues and then assuming it'll be fine when a thousand of such scripts run at the same time.

The point, if I have one, is just that resources are certainly not free. One can hardly consider reading only what you need, rather than the lazy man's option of "let the operating system handle it", to be premature optimisation. More like the timely application of common sense.