Added emoji freevar

Strings of emojis are tokenized separately from anything else, and added to an emoticon freevar, which is mixed in with regular emoticons like :P. This breaks old-style freevars like $name$ and $noun$ so some legacy support for compatibility is left in but eventually $name$ should be made into an actual new freevar. Emoji data is from gemoji (https://github.com/github/gemoji).

1 files changed, 4 insertions, 1 deletions

@@ -112,8 +112,11 @@ private:
   
         int maxK;
         std::map<kgram, std::map<int, token_data> > stats;
-  word hashtags {"#hashtag"};
+  
+  // Words
   std::map<std::string, word> words;
+  word hashtags {"#hashtag"};
+  word emoticons {"👌"};
 };
 
 void printKgram(kgram k);