Well, first off, CCP is simply appending full UCS2 strings to the file. This causes the BOM (byte order mark) to appear several times in the file.
While this is technically not a bug, it is undesirable as it stops you from simply loading the file and decoding it in 1 go. The quick fix is to simply remove all BOMs.
Here's how you could go about iterating over log lines:
fn = r"X:\Path\to\some\eve\chatlog.txt"
log = open(fn, "rb").read()
log = log.replace("\xff\xfe", "").decode("utf_16")
# log is now a normal unicode string with the whole log.
# you can then iterate over the lines like this:
import StringIO # don't use cStringIO, doesnt work on unicode.
for line in StringIO.StringIO(log):
line = line.rstrip("\r\n")
Alternatively, you could split on the BOMs in the log and end up with the separate sessions. Could be useful :)