0.14.0 - Moved twg2.parser.text package to new jtext-tokenizer library, renamed classes/methods with the word 'Fragment' to 'Token'; i.e. CodeFragmentType -> CodeTokenType, see CHANGELOG for full details.

Author: TeamworkGuy2

.classpath

@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<?xml version="1.0" encoding="UTF-8"?>
 <classpath>
 	<classpathentry kind="src" path="src"/>
 	<classpathentry kind="src" path="plugins"/>
@@ -44,6 +44,8 @@
 	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jstreamish/bin/jstreamish.jar" sourcepath="/JStreamish"/>
 	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-parser/bin/jtext_parser.jar" sourcepath="/JTextParser"/>
 	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-template/bin/jtext_template.jar" sourcepath="/JTextTemplate"/>
+	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-tokenizer/bin/jtext_tokenizer-with-tests.jar" sourcepath="/JTextTokenizer"/>
+	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-tokenizer/bin/jtext_tokenizer.jar" sourcepath="/JTextTokenizer"/>
 	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-util/bin/jtext_util.jar" sourcepath="/JTextUtil"/>
 	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtree-walker/bin/jtree_walker.jar" sourcepath="/JTreeWalker"/>
 	<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtuples/bin/jtuples.jar" sourcepath="/JTuples"/>

CHANGELOG.md

@@ -4,7 +4,28 @@ This project does its best to adhere to [Semantic Versioning](http://semver.org/
 
 
 --------
-###[0.13.0](N/A) - 2016-10-26
+###[0.14.0](N/A) - 2016-10-30
+#### Changed
+__Reduced library complexity/scope by moving twg2.parser.text conditions/tokenizers to separate [jtext-tokenizer](https://github.com/TeamworkGuy2/JTextTokenizer) library__
+* Moved twg2.parser.text package to jtext-tokenizer library
+* Moved twg2.parser.Inclusion to jtext-parser library
+* Renamed classes *Fragment -> *Token:
+  * CodeFragment -> CodeToken
+  * CodeFragmentType -> CodeTokenType
+  * DocumentFragment -> TextToken and renamed methods:
+    * getTextFragment() -> getToken()
+	* getFragmentType() -> getTokenType()
+  * DocumentFragmentText -> TextFragmentRefToken
+  * CodeFragmentEnumSubSet -> CodeTokenEnumSubSet
+* Renamed twg2.parser.codeParser.tools.performance -> twg2.parser.codeParser.analytics
+* TokenizeStepDetails -> TokenizeStepLogger and now implements TypedLogger from jtext-tokenizer library
+
+#### Removed
+* Removed twg2.parser.text package (moved to jtext-tokenizer library)
+
+
+--------
+###[0.13.0](https://github.com/TeamworkGuy2/JParseCode/commit/76734b17d16c67a89df7245a2cea2a1133c3b6b0) - 2016-10-26
 #### Added
 * Added ParameterSig and ParameterSigResolved 'parameterModifiers' field
 * Added KeywordUtil parameterModifiers() and isParameterModifier()

README.md

@@ -1,6 +1,6 @@
 JParseCode
 ==============
-version: 0.13.0
+version: 0.14.0
 
 In progress C#/Java/TypeScript parser tools built atop [JTextParser] (https://github.com/TeamworkGuy2/JTextParser), [Jackson] (https://github.com/FasterXML/jackson-core/) (core, databind, annotations) and half a dozen other utility libraries. 
 
@@ -9,7 +9,7 @@ In progress C#/Java/TypeScript parser tools built atop [JTextParser] (https://gi
 * A code first parser aimed at manipulating the resulting AST and writing it back as source code or JSON.  With the goal of allowing simple language constructs like interfaces and data models to be transpiled to different languages. 
 
 ### Not Goals:
-* NOT to create another compiler for C#, Java, or JS/TS. This project's parser expects valid code as input, the few error messages that are present are NOT design to highlight syntax errors in the input. 
+* NOT to create another compiler for C#, Java, or JavaScript/TypeScript. This project's parser expects valid code as input, the few error messages that are present are NOT design to highlight syntax errors in the input. 
 * NOT to create a valid AST for each supported language. Rather an AST like structure that supports the lowest common denominator between the targeted languages. This means many AST compromises are inevitable due to differences in language specs. 
 
 
@@ -65,7 +65,7 @@ Java code to parser SimpleCs.cs (simple_cs_source_string is a string containing
 	CodeFileSrc<CodeLanguage> simpleCsAst = ParseCodeFile.parseCode("SimpleCs.cs", CodeLanguageOptions.C_SHARP, simple_cs_source_string);
 	WriteSettings ws = new WriteSettings(true, true, true, true);
 
-	for(Map.Entry<SimpleTree<DocumentFragmentText<CodeFragmentType>>, IntermClass.SimpleImpl<CsBlock>> block : CodeLanguageOptions.C_SHARP.getExtractor().extractClassFieldsAndMethodSignatures(simpleCsAst.getDoc())) {
+	for(Map.Entry<SimpleTree<CodeToken>, ClassAst.SimpleImpl<CsBlock>> block : CodeLanguageOptions.C_SHARP.getExtractor().extractClassFieldsAndMethodSignatures(simpleCsAst.getDoc())) {
 		CodeFileParsed.Simple<String, CsBlock> fileParsed = new CodeFileParsed.Simple<>("SimpleCs.cs", block.getValue(), block.getKey());
 
 		StringBuilder sb = new StringBuilder();

bin/jparse_code-with-tests.jar

@@ -1,5 +1,5 @@
 {
-	"version" : "0.13.0",
+	"version" : "0.14.0",
 	"name" : "jparse-code",
 	"description" : "An in-progress suite of parsing/transpilation tools for C#, Java, and TypeScript code.  Generates simple JSON ASTs.",
 	"homepage" : "https://github.com/TeamworkGuy2/JParseCode",
@@ -25,6 +25,7 @@
 		"json-stringify": "*",
 		"jstreamish": "*",
 		"jtext-parser": "*",
+		"jtext-tokenizer": "*",
 		"jtext-util": "*",
 		"jtree-walker": "*",
 		"jtuples": "*"

bin/jparse_code.jar

@@ -4,7 +4,7 @@
 import twg2.annotations.Immutable;
 import twg2.ast.interm.classes.ClassSig;
 import twg2.parser.codeParser.BlockType;
-import twg2.parser.fragment.CodeFragment;
+import twg2.parser.fragment.CodeToken;
 import twg2.treeLike.simpleTree.SimpleTree;
 
 /**
@@ -14,11 +14,11 @@
 @Immutable
 public class BlockAst<T_BLOCK extends BlockType> {
 	private final @Getter ClassSig.SimpleImpl declaration;
-	private final @Getter SimpleTree<CodeFragment> blockTree;
+	private final @Getter SimpleTree<CodeToken> blockTree;
 	private final @Getter T_BLOCK blockType;
 
 
-	public BlockAst(ClassSig.SimpleImpl declaration, SimpleTree<CodeFragment> blockTree, T_BLOCK blockType) {
+	public BlockAst(ClassSig.SimpleImpl declaration, SimpleTree<CodeToken> blockTree, T_BLOCK blockType) {
 		this.declaration = declaration;
 		this.blockTree = blockTree;
 		this.blockType = blockType;

package-lib.json

@@ -12,8 +12,8 @@
 import twg2.parser.codeParser.AccessModifier;
 import twg2.parser.codeParser.extractors.DataTypeExtractor;
 import twg2.parser.codeParser.tools.NameUtil;
-import twg2.parser.fragment.CodeFragment;
-import twg2.parser.fragment.CodeFragmentType;
+import twg2.parser.fragment.CodeToken;
+import twg2.parser.fragment.CodeTokenType;
 import twg2.parser.output.WriteSettings;
 import twg2.text.stringEscape.StringEscapeJson;
 import twg2.treeLike.simpleTree.SimpleTree;
@@ -24,11 +24,11 @@
  */
 @Immutable
 public class FieldDef extends FieldSig {
-	private final @Getter SimpleTree<CodeFragment> initializer;
+	private final @Getter SimpleTree<CodeToken> initializer;
 
 
 	public FieldDef(String name, List<String> fullName, TypeSigSimple fieldType, List<AccessModifier> accessModifiers,
-			List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeFragment> initializer) {
+			List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeToken> initializer) {
 		super(name, fullName, fieldType, accessModifiers, annotations, comments);
 		this.initializer = initializer;
 	}
@@ -68,12 +68,12 @@ public String toString() {
 	/** Write a field initializer to a JSON field named 'initializer' if the value is a number, boolean, string, or null literal, else write it to a field named 'initializerExpression'
 	 * @throws IOException
 	 */
-	public static void initializerToJson(SimpleTree<CodeFragment> astNode, boolean preClosingComma, Appendable dst, WriteSettings st) throws IOException {
-		CodeFragment data = null;
+	public static void initializerToJson(SimpleTree<CodeToken> astNode, boolean preClosingComma, Appendable dst, WriteSettings st) throws IOException {
+		CodeToken data = null;
 		boolean isNumOrBoolOrNull = false;
 		if(astNode != null && !astNode.hasChildren() && (data = astNode.getData()) != null &&
-				(data.getFragmentType() == CodeFragmentType.STRING ||
-				(isNumOrBoolOrNull = (data.getFragmentType() == CodeFragmentType.NUMBER || DataTypeExtractor.isBooleanLiteral(data) || DataTypeExtractor.isNullLiteral(data))))) {
+				(data.getTokenType() == CodeTokenType.STRING ||
+				(isNumOrBoolOrNull = (data.getTokenType() == CodeTokenType.NUMBER || DataTypeExtractor.isBooleanLiteral(data) || DataTypeExtractor.isNullLiteral(data))))) {
 			if(preClosingComma) {
 				dst.append(", ");
 			}

src/twg2/ast/interm/block/BlockAst.java

@@ -11,7 +11,7 @@
 import twg2.io.json.stringify.JsonStringify;
 import twg2.parser.codeParser.AccessModifier;
 import twg2.parser.codeParser.tools.NameUtil;
-import twg2.parser.fragment.CodeFragment;
+import twg2.parser.fragment.CodeToken;
 import twg2.parser.output.WriteSettings;
 import twg2.treeLike.simpleTree.SimpleTree;
 
@@ -21,11 +21,11 @@
  */
 @Immutable
 public class FieldDefResolved extends FieldSigResolved {
-	private final @Getter SimpleTree<CodeFragment> initializer;
+	private final @Getter SimpleTree<CodeToken> initializer;
 
 
 	public FieldDefResolved(String name, List<String> fullName, TypeSigResolved fieldType, List<AccessModifier> accessModifiers,
-			List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeFragment> initializer) {
+			List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeToken> initializer) {
 		super(name, fullName, fieldType, accessModifiers, annotations, comments);
 		this.initializer = initializer;
 	}