Weighed token casing and presence of periods

Tokens which differ only by casing or the presence of an ending period are
now considered the same token. When tokens are generated, they are cased
based on the prevalence of Upper/Title/Lower casing of the token in the
input corpus, and similarly, a period is added to the end of the word based
on how often the same token was ended with a period in the input corpus.

1 files changed, 9 insertions, 3 deletions

@@ -14,15 +14,21 @@ class kgramstats
 {
 public:
         kgramstats(string corpus, int maxK);
-        map<string, int>* lookupExts(kgram tk);
-        int getMaxK();
         vector<string> randomSentence(int n);
         
 private:
+        typedef struct
+        {
+                int all;
+                int titlecase;
+                int uppercase;
+                int period;
+        } token_data;
         int maxK;
-        map<kgram, map<string, int>* >* stats;
+        map<kgram, map<string, token_data*>* >* stats;
 };
 
 void printKgram(kgram k);
+std::string canonize(std::string f);
 
 #endif
\ No newline at end of file