How fast can a BufferedReader read lines in Java?

Your version still has the problem that since Java 7 the substring method allocates a new String, which in unnecessary when it is only borrowed to the parseLine function and not needed afterwards. What you probably want is a view that reuses the same underlying data. With the following I get around 2 GB/s:

    public void readString(String data) throws IOException {
        int lastIdx = 0;
        for (int idx = data.indexOf('\n'); idx > -1; idx = data.indexOf('\n', lastIdx)) {
            parseLine(subSequenceView(data, lastIdx, idx));
            lastIdx = idx + 1;
        }
        parseLine(subSequenceView(data, lastIdx, data.length()));
    }

    CharSequence subSequenceView(CharSequence base, int beginIndex, int endIndex) {
        return new StringView(base, beginIndex, endIndex - beginIndex);
    }

    static class StringView implements CharSequence {
        final CharSequence base;
        final int offset;
        final int length;

        StringView(CharSequence base, int offset, int length) {
            if (length < 0) throw new IllegalArgumentException("length must be >= 0");
            this.base = base;
            this.offset = offset;
            this.length = length;
        }

        @Override
        public char charAt(int n) {
            if (n < 0 || n >= length)
                throw new IndexOutOfBoundsException(n);
            return base.charAt(offset + n);
        }

        @Override
        public int length() {
            return length;
        }

        @Override
        public CharSequence subSequence(int beginIndex, int endIndex) {
            return new StringView(base, offset + beginIndex, endIndex - beginIndex);
        }
    }

Is it possible he edited the code? I don’t see the append in the blog post.

this still does a fair amount of processing, curious what the throughput is with just straight reads into a char[] from FileReader

I'm confused.

Where is the second lines call?

That's a pretty big error if you're correct. What does it say about the language when a CS professor falls for this on a 40 line file?

Java NIO does exactly what you describe. IO is chunked optimally, the entire buffer is pulled, the API exposes a select() mechanism just like any libc, and the user is expected to frame its own received data.

edit: Lemire is showing a lack of what Fowler describes as "mechanical sympathy".

https://martinfowler.com/articles/lmax.html#QueuesAndTheirLa...

I'm reminded to add, in the vein of the author's complaint, that there is a similar ridiculousness in Erlang land, that cannot be circumnavigated so easily: reading/writing to zlib-compressed files using Erlang's file:open(..., [compressed]) option—or generating/parsing zlib-compressed ETF binaries using erlang:binary_to_term(..., [compressed]))—holds [the moral equivalent of†] a global lock. Only one process can be zlib-compressing or zlib-decompressing a chunk of data at once, no matter how many cores your system has.

This means that, even when your data set compresses so well that you'd theoretically gain a ton of speed by having the data streamed from disk compressed, and then decompressed during parsing—this doesn't apply in practice, since you're introducing an artificial bottleneck in your IO reads.

I'm not actually sure if this is a bug in Erlang, per se, or if it's just the intended behavior and compressed file IO was never intended to be used for performance, only for e.g. embedded devices with tiny ROMs.

(If people here think it's a bug, I'll probably go to the effort at some point to profile the performance impact and submit it as a bug on https://bugs.erlang.org.)

† What it's actually doing, is that all zlib compression/decompression passes get sent to a zlib "port driver" as messages. Port drivers can handle multiple requests in-flight at once (they're the in-process equivalent to sockets—each Erlang process's port against the port-driver is its own "connection") but the zlib port driver shim is coded to expose zlib as a single-threaded, blocking, request-response style of server, rather than one that accepts connections in parallel and instantiates a separate zlib context for each separate connection it receives.

Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day.

edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf) since all he's concerned about is the relative performance of different software.

There is a difference between "I'm being dumb because I don't know what I'm doing" and "I'm being lazy because I've done it 1,000 times and the target audience knows what I mean".

I can make a lot of things cpu bound with crappy implementation. I'm not going to write a blog post about any of them.

I'm no Java ninja, but a few things jump out of https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c... :

- at least one heap allocation for every line. After it finds the EOL it first uses 'new String' followed by '.toString()

- the C++ version will almost certainly be backing on to memchr() behind the scenes, which will be using SIMD instructions where it makes sense (e.g. large enough scan size, probably true in this case). the Java version is a manual bytewise-coded loop.

- the C++ version is reusing its output buffer, no reallocations assuming the same string length or less

No idea about encodings in Java, maybe that is playing a role too

There is a layer of classes, so presumably there are multiple buffers and extra copying. It's also converting the underlying encoding to UTF-16, allocating String objects, and copying the data into them.

Do you have example of alternatives for the BufferedReader with the NIO APIs?

I do a lot of work with large GZIP that are read line by line using the standard IO (i.e. GzipInputStrem(FileInputStream)) etc) but your comment has really made me second guess my choice of doing that...

Do you have a quick github project example?

Where do you see string concatenation with the plus operator in the posted code?

Agreed. I think you are being downvoted because you forgot to paste in your benchmark results.

Myself I've worked with gigabyte sized CSV files without issues, so it was likely your implementation rather than the fault of the Java language.

> Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up

Let us all hope that Project Valhalla, the effort to add value types to Java, comes to a swift completion. It would be very helpful in these sorts of scenarios.

Though at this point I'd wonder (as you did) why I'm using Java in the first place given the resource constraints of the system, I've had some success using byte[] arrays and using `yourInputStreamHere.read(someBuffer, yourOffset, YOUR_PAGE_SIZE)` in a few less constrained but relatively more performance critical areas.

Depending on your exact needs, this can sometimes result in more or less constant memory consumption; it's possible to reuse the array on each read. You can also take it a step further by finding the indexes where you would split the array (e.g. the index of the next comma for a CSV) and using methods that operate on an array plus start and end indexes. That sort of strategy can allow you to avoid additional copies the array, at the cost of somewhat annoying and less maintainable code that is definitely not Java-esque.

There are also less severe solutions that won't buffer the whole file in memory, using StringBuilder or other techniques, etc. depending on what you're doing.

This all might be for naught though; I'm assuming the records have to be parsed into some structure, so tricks like the above might not be good enough given Java's seemingly insatiable hunger for heap space.

While I'm not the language's greatest fan, this is one area where Go has been able to shine. Having value types as a possibility makes solving this sort of problem a typically less expensive proposition.

Interesting comments so let's have some answers:

1. 128 M ram total memory, 20-30 M free at best. CPU was slow and we were running off a SD card. Not exactly an enterprise monster.

2. Of course I was parsing it line by line. But running OOM triggered the garbage collector so many times that it took like 3-4 minutes to finish. I didn't mean that the system actually stopped because of running out of ram.

3. Why didn't you do it with the NDK/some other solution? Because the PM just gave up on the feature when it wasn't done in a day. It was just real time search completion whith we didn't really need there, it was just "nice to have".

I’m parsing much larger datasets than those on android, in less than 18M RAM usage.

You just have to use NIO and use direct buffers instead of naively reading and allocating strings.

Android gives bad name to Java with its Dalvik and ART partial implementations.

Why didn't you just write it using the NDK?