Skip to content

Commit be60f89

Browse files
committed
Remove redundant whitespace check in TokenizerStateMachine
1 parent 0ada8a7 commit be60f89

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

tokenizer/src/main/java/edu/illinois/cs/cogcomp/nlp/tokenizer/TokenizerStateMachine.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -336,7 +336,7 @@ public void process(char token) {
336336
if (getCurrent().isAbbr())
337337
return; // previous was upper case, acronym and word
338338
// continues
339-
else if (Character.isWhitespace(c) && Character.isLowerCase(nextnextChar))
339+
else if (Character.isLowerCase(nextnextChar))
340340
return; // when the next char is white space and the next next char
341341
// is lowercase, we know that the next word is not start of
342342
// a sentence, so we continue.

0 commit comments

Comments
 (0)