Java File Tokenizer

[asmit]

Can somebody help me!!?

I need to read a txt file filled with words (separated by spaces and/or line breaks) and grab each word (tokenize the file and save each token in an array). For now, I just want to print each token to the screen!

My creation of my InputStream looks like:

InputStream inputS = new FileInputStream(filename);

And my java method looks like:

public static void tokenize(InputStream inputS) throws IOException

{

StringTokenizer st = new StringTokenizer(inputS);

while (st.hasMoreTokens())

{

System.out.println(st.nextToken());

}

}

But i'm getting an error saying that it cannot resolve:

StringTokenizer st = new StringTokenizer(inputS);

Can someone help me tokenize and output the results of the file?

[Winston]

You really need to start using the Java API, there's no given constructor for the StringTokenizer class which accepts an InputStream as a parameter, http://java.sun.com/j2se/1.4.2/docs/api/ja...gTokenizer.html

You need to create a new BufferedReader and feed in the input stream and do successive readline calls and pass it into a new StringTokenizer object

public static void tokenize(InputStream inputS) throws IOException
{

 ? ?BufferedReader reader = new BufferedReader(inputS);
 ? ?while(reader.ready())
 ? ?{
 ? ? ? ? ? String lineRead = reader.readLine();
 ? ? ? ? ? StringTokenizer st = new StringTokenizer(lineRead);
 ? ? ? ? ? while (st.hasMoreTokens())
 ? ? ? ? ? {
 ? ? ? ? ? ? ? ?System.out.println(st.nextToken());
 ? ? ? ? ? }
 ? ?}

}

This should work.

[asmit]

  Winston said:

You really need to start using the Java API, there's no given constructor for the StringTokenizer class which accepts an InputStream as a parameter, http://java.sun.com/j2se/1.4.2/docs/api/

You need to create a new BufferedReader and feed in the input stream and do successive readline calls and pass it into a new StringTokenizer object

public static void tokenize(InputStream inputS) throws IOException
{

BufferedReader reader = new BufferedReader(inputS);
while(reader.ready())
{
String lineRead = reader.readLine();
StringTokenizer st = new StringTokenizer(lineRead);
while (st.hasMoreTokens())
{
System.out.println(st.nextToken());
}
}

}

This should work.

584975946[/snapback]

thanks a lot! i'll give that a try!

[asmit]

Why won't it let me create a new BufferedReader from my inputstream (inputS)

C:\JAVA ASSIGNMENTS\red_black.java:198: cannot resolve symbol

symbol : constructor BufferedReader (java.io.InputStream)

location: class java.io.BufferedReader

BufferedReader reader = new BufferedReader(inputS);

[asmit]

  asmit said:

Why won't  it let me create a new BufferedReader from my inputstream (inputS)

C:\JAVA ASSIGNMENTS\red_black.java:198: cannot resolve symbol

symbol  : constructor BufferedReader (java.io.InputStream)

location: class java.io.BufferedReader

  BufferedReader reader = new BufferedReader(inputS);

584975991[/snapback]

never mind! I wrapped the bufferedreader in a new InputStreamREader and it works!

Thaks for you help!!

[Winston]

  asmit said:

never mind!  I wrapped the bufferedreader in a new InputStreamREader and it works!

Thaks for you help!!

584976009[/snapback]

Oh yeah oops lol, well that's what you get for coding in the morning, glad it helped.

[DaNIsH]

Just a quick word, if you read the StringTokenizer API, the top of the page comments note that use of StringTokenizer is discouraged.

The recommended way is String.split();

[_kane81]

Correction - the use of StringTokenizer with InputStream class has been deprecated

they do not discourage use of this class, only this class with InputStream

http://java.sun.com/j2se/1.4.2/docs/api/ja...mTokenizer.html

  Quote

StreamTokenizer(InputStream is)

          Deprecated. As of JDK version 1.1, the preferred way to tokenize an input stream is to convert it into a character stream, for example:

  Reader r = new BufferedReader(new InputStreamReader(is));

  StreamTokenizer st = new StreamTokenizer®;