Package weka.core.tokenizers
Class CharacterDelimitedTokenizer
java.lang.Object
weka.core.tokenizers.Tokenizer
weka.core.tokenizers.CharacterDelimitedTokenizer
- All Implemented Interfaces:
Serializable,Enumeration<String>,OptionHandler,RevisionHandler
- Direct Known Subclasses:
NGramTokenizer,WordTokenizer
Abstract superclass for tokenizers that take characters as delimiters.
- Version:
- $Revision: 10203 $
- Author:
- fracpete (fracpete at waikato dot ac dot nz)
- See Also:
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionReturns the tip text for this propertyGet the value of delimiters (not backquoted).String[]Gets the current option settings for the OptionHandler.Returns an enumeration of all the available options..voidsetDelimiters(String value) Set the value of delimiters.voidsetOptions(String[] options) Sets the OptionHandler's options using the given list.Methods inherited from class weka.core.tokenizers.Tokenizer
globalInfo, hasMoreElements, nextElement, runTokenizer, tokenize, tokenizeMethods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface java.util.Enumeration
asIteratorMethods inherited from interface weka.core.RevisionHandler
getRevision
-
Constructor Details
-
CharacterDelimitedTokenizer
public CharacterDelimitedTokenizer()
-
-
Method Details
-
listOptions
Returns an enumeration of all the available options..- Specified by:
listOptionsin interfaceOptionHandler- Overrides:
listOptionsin classTokenizer- Returns:
- an enumeration of all available options.
-
getOptions
Gets the current option settings for the OptionHandler.- Specified by:
getOptionsin interfaceOptionHandler- Overrides:
getOptionsin classTokenizer- Returns:
- the list of current option settings as an array of strings
-
setOptions
Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).- Specified by:
setOptionsin interfaceOptionHandler- Overrides:
setOptionsin classTokenizer- Parameters:
options- the list of options as an array of strings- Throws:
Exception- if an option is not supported
-
getDelimiters
Get the value of delimiters (not backquoted).- Returns:
- Value of delimiters.
-
setDelimiters
Set the value of delimiters. For convenienve, the strings "\r", "\n", "\t", "\'", "\\" get automatically translated into their character representations '\r', '\n', '\t', '\'', '\\'. This means, one can either usesetDelimiters("\r\n\t\\");orsetDelimiters("\\r\\n\\t\\\\");.- Parameters:
value- Value to assign to delimiters.- See Also:
-
delimitersTipText
Returns the tip text for this property- Returns:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-