I am parsing a string in C++ using the following:

    using namespace std;

    string parsed,input=&quot;text to be parsed&quot;;
    stringstream input_stringstream(input);

    if (getline(input_stringstream,parsed,&#39; &#39;))
    {
         // do some processing.
    }

Parsing with a single char delimiter is fine. But what if I want to use a string as delimiter.

Example: I want to split:
    
    scott&gt;=tiger

with `&gt;=` as delimiter so that I can get scott and tiger.   
        
  

Parse (split) a string in C++ using string delimiter (standard C++)

Java has a convenient split method:

    String str = &quot;The quick brown fox&quot;;
    String[] results = str.split(&quot; &quot;);

Is there an easy way to do this in C++?

How do I tokenize a string in C++?

Some ways to iterate through the characters of a string in Java are:

 1. Using `StringTokenizer`?
 2. Converting the `String` to a `char[]` and iterating over that.
 
What is the easiest/best/most correct way to iterate? 

What is the easiest/best/most correct way to iterate through the characters of a string in Java?

Suppose I have the string `1:2:3:4:5` and I want to get its last field (`5` in this case). How do I do that using Bash? I tried `cut`, but I don&#39;t know how to specify the last field with `-f`.

How to split a string in shell and get the last field

I am looking for a clear definition of what a &quot;tokenizer&quot;, &quot;parser&quot; and &quot;lexer&quot; are and how they are related to each other (e.g., does a parser use a tokenizer or vice versa)? I need to create a program will go through c/h source files to extract data declaration and definitions. 

  I have been looking for examples and can find some info, but I really struggling to grasp the underlying concepts like grammar rules, parse trees and abstract syntax tree and how they interrelate to each other. Eventually these concepts need to be stored in an actual program, but 1) what do they look like, 2) are there common implementations.

 I have been looking at Wikipedia on these topics and programs like Lex and Yacc, but having never gone through a compiler class (EE major) I am finding it difficult to fully understand what is going on.





Looking for a clear definition of what a &quot;tokenizer&quot;, &quot;parser&quot; and &quot;lexers&quot; are and how they are related to each other and used?

I just learned about Java&#39;s Scanner class and now I&#39;m wondering how it compares/competes with the StringTokenizer and String.Split. I know that the StringTokenizer and String.Split only work on Strings, so why would I want to use the Scanner for a String? Is Scanner just intended to be one-stop-shopping for spliting?

Scanner vs. StringTokenizer vs. String.Split

I&#39;ve got the following code:

    std::string str = &quot;abc def,ghi&quot;;
    std::stringstream ss(str);

    string token;

    while (ss &gt;&gt; token)
    {
        printf(&quot;%s\n&quot;, token.c_str());
    }

The output is:

&gt; abc  
def,ghi

So the `stringstream::&gt;&gt;` operator can separate strings by space but not by comma. Is there anyway to modify the above code so that I can get the following result?

&gt; **input**: &quot;abc,def,ghi&quot;  
&gt;
&gt; **output**:   
abc  
def  
ghi


How to use stringstream to separate comma separated strings

From something like this:

    print(get_indentation_level())

        print(get_indentation_level())

            print(get_indentation_level())

I would like to get something like this:

    1
    2
    3
Can the code read itself in this way?

All I want is the output from the more nested parts of the code to be more nested. In the same way that this makes code easier to read, it would make the output easier to read. 

Of course I could implement this manually, using e.g. `.format()`, but what I had in mind was a custom print function which would `print(i*&#39; &#39; + string)` where `i` is the indentation level. This would be a quick way to make readable output on my terminal. 

Is there a better way to do this which avoids painstaking manual formatting?

Can a line of Python code know its indentation nesting level?

What is the best way to tokenize/split a NSString in Objective-C?

NSString tokenize in Objective-C

I&#39;m just starting to use NLTK and I don&#39;t quite understand how to get a list of words from text. If I use `nltk.word_tokenize()`, I get a list of words and punctuation. I need only the words instead. How can I get rid of punctuation? Also `word_tokenize` doesn&#39;t work with multiple sentences: dots are added to the last word.

How to get rid of punctuation using NLTK tokenizer?

I know this has been answered to some degree with PHP and MYSQL, but I was wondering if someone could teach me the simplest approach to splitting a string (comma delimited) into multiple rows in Oracle 10g (preferably) and 11g. 

The table is as follows:

    Name | Project | Error 
    108    test      Err1, Err2, Err3
    109    test2     Err1

I want to create the following:

    Name | Project | Error
    108    Test      Err1
    108    Test      Err2 
    108    Test      Err3 
    109    Test2     Err1

I&#39;ve seen a few potential solutions around stack, however they only accounted for a single column (being the comma delimited string). Any help would be greatly appreciated. 


Splitting string into multiple rows in Oracle

I&#39;m trying to use Apache Lucene for tokenizing, and I am baffled at the process to obtain Tokens from a `TokenStream`.

The worst part is that I&#39;m looking at the comments in the JavaDocs that address my question.

http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/analysis/TokenStream.html#incrementToken%28%29

Somehow, an `AttributeSource` is supposed to be used, rather than `Token`s.  I&#39;m totally at a loss.  

Can anyone explain how to get token-like information from a TokenStream?

How to get a Token from a Lucene TokenStream?

In my JavaEE application, I&#39;m using the Atom-based [Google Sites API](https://developers.google.com/google-apps/sites/) to retrieve content from a non-public Google Site. In essence, we&#39;re using the Google Site as a lightweight CMS, and from within the application I use the API to retrieve the site contents to feed my online help system. I&#39;ve had this setup for a while and it&#39;s working without a hitch.

The issue
---------

In my application, I need to add full-text search functionality to the online help system. I knew this feature request would come along at some point, so when deciding on Google Sites to host my content, I checked whether the Sites API supports full-text search. [It does](https://developers.google.com/google-apps/sites/docs/1.0/developers_guide_protocol#ContentFeedQueriesFullText). For example, the following URL will search the entire site `my-site` for pages containing the keyword `user`.

    https://sites.google.com/feeds/content/my.doma.in/my-site?q=user

This works, and gives me the expected result pages. But it does so **only for content written in Western languages**, or, more specifically, languages in which tokens/words are separated by whitespace and punctuation. When I run a similar search on my Japanese content, searching for the keyword `ユーザー`:

    https://sites.google.com/feeds/content/my.doma.in/my-site?q=%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC

I will only get result pages in which the search term appears as a bare string, i.e. delimited by either white-space or punctuation. Since Japanese is a language written in [scriptio continua](http://en.wikipedia.org/wiki/Scriptio_continua), this is not sufficient. Pages that contain, for example:

&gt; ご自身の**ユーザー**基本情報の確認

will not show up in the results. So it seems that the search index that is used behind the scenes is created based on &quot;Western&quot; lexical rules, and that Japanese content is not correctly tokenized. However, when I search for the same keyword from the Google Site&#39;s *Search this site* field, I do get the correct results. I conclude that **a correctly tokenized index exists, but it seems to be impossible to use it for an API-based search**.

What I&#39;ve tried so far
----------------------

To remedy this situation, these are the avenues that I&#39;ve explored so far:

 - I&#39;ve tried looking for language settings in Google Sites itself. There&#39;s a general UI language setting which was already set to Japanese and has no impact on the API query results. There are no per-page or per-template language settings to force the indexer/tokenizer&#39;s hand.
 - I&#39;ve tried quoting the search string with double quotes (`&quot;ユーザー&quot;`).
 - I&#39;ve tried including wildcards (`*ユーザー*`).
 - I&#39;ve tried using additional language parameters to the URL that are common in other Google APIs: `lang`, `hl` (interface language), `rl` (results language),..
 - I&#39;ve tried creating a Google [Custom Search Engine](https://www.google.jp/cse/), but it seems impossible to get it to work on a non-public Google Site.

So...
-----

I&#39;m quickly running out of ideas here. In a worst case scenario, I will end up having to retrieve, tokenize, and index all of the content myself and make it searchable that way. Since this will require a substantial effort, I would like to know if anyone has encountered the same issue and has found an acceptable workaround or solution.

----------

Update 1
--------

I have yet to find an elegant solution for this issue, so I raised a defect on the Google Apps APIs issue tracker: https://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3780

Update 2
--------

After some going back and forth, Google&#39;s engineers have acknowledged that the problem indeed exists as described, and have *&quot;filed the issue internally&quot;*. The defect ticket has been stuck in **triaged** state ever since. If you, like me, are interested in seeing this issue resolved, please take a moment to star/vote for it on [Google&#39;s issue tracker](https://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3780).

Google Sites API full-text search does not work for non-Western languages

I am using nltk, so I want to create my own custom texts just like the default ones on nltk.books. However, I&#39;ve just got up to the method like

    my_text = [&#39;This&#39;, &#39;is&#39;, &#39;my&#39;, &#39;text&#39;]

I&#39;d like to discover any way to input my &quot;text&quot; as:

    my_text = &quot;This is my text, this is a nice way to input text.&quot;

Which method, python&#39;s or from nltk allows me to do this. And more important, how can I dismiss punctuation symbols?

How do I tokenize a string sentence in NLTK?

I want to split each line of a pipe on spaces, and then print each token on its own line.

I realise that I can get this result using:

    (cat someFileInsteadOfAPipe).split(&quot; &quot;)

But I want more flexibility. I want to be able to do just about anything with each token. (I used to use [AWK][1] on Unix, and I&#39;m trying to get the same functionality.)

I currently have:

    echo &quot;Once upon a time there were three little pigs&quot; | %{$data = $_.split(&quot; &quot;); Write-Output &quot;$($data[0]) and whatever I want to output with it&quot;}

Which, obviously, only prints the first token. Is there a way for me to for-each over the tokens, printing each in turn?

Also, the `%{$data = $_.split(&quot; &quot;); Write-Output &quot;$($data[0])&quot;}` part I got from a blog, and I really don&#39;t understand what I&#39;m doing or how the syntax works.

I want to google for it, but I don&#39;t know what to call it. Please help me out with a word or two to Google, or a link explaining to me what the `%` and all the `$` symbols do, as well as the significance of the opening and closing brackets.

I realise I can&#39;t actually use `(cat someFileInsteadOfAPipe).split(&quot; &quot;)`, since the file (or preferable incoming pipe) contains more than one line.

Regarding some of the answers:

If you are using [`Select-String`][2] to filter the output before tokenizing, you need to keep in mind that the output of the `Select-String` command is not a collection of strings, but a collection of `MatchInfo` objects. To get to the string you want to split, you need to access the `Line` property of the `MatchInfo` object, like so:

    cat someFile | Select-String &quot;keywordFoo&quot; | %{$_.Line.Split(&quot; &quot;)}

  [1]: http://en.wikipedia.org/wiki/AWK
  [2]: https://technet.microsoft.com/en-us/library/hh849903.aspx



Split string with PowerShell and do something with each token

I am using `split()` to tokenize a String separated with `*` following this format:

    name*lastName*ID*school*age
    %
    name*lastName*ID*school*age
    %
    name*lastName*ID*school*age

I&#39;m reading this from a file named &quot;entrada.al&quot; using this code:

    static void leer() {
    
        try {
            String ruta=&quot;entrada.al&quot;;
            File myFile = new File (ruta);
            FileReader fileReader = new FileReader(myFile);
    
            BufferedReader reader = new BufferedReader(fileReader);
    
            String line = null;
    
            while ((line=reader.readLine())!=null){
                if (!(line.equals(&quot;%&quot;))){
                    String [] separado = line.split(&quot;*&quot;); //SPLIT CALL
                    names.add(separado[0]);
                    lastNames.add(separado[1]);
                    ids.add(separado[2]);
                    ages.add(separado[3]);
                }
            }
    
            reader.close();
        }

And I&#39;m getting this exception:  
&gt;Exception in thread &quot;main&quot; java.util.regex.PatternSyntaxException: Dangling meta character &#39;*&#39; near index 0
*

My guess is that the lack of a `*` after age on the original text file is causing this. How do I get around it?


Tokenizing Error: java.util.regex.PatternSyntaxException, dangling metacharacter &#39;*&#39;

I need to write a procedure to normalize a record that have multiple tokens concatenated by one char. I need to obtain these tokens splitting the string and insert each one as a new record in a table. Does Oracle have something like a &quot;split&quot; function?

Is there a function to split a string in PL/SQL?

I am used to the c-style `getchar()`, but it seems like there is nothing comparable for java. I am building a lexical analyzer, and I need to read in the input character by character.

I know I can use the scanner to scan in a token or line and parse through the token char-by-char, but that seems unwieldy for strings spanning multiple lines. Is there a way to just get the next character from the input buffer in Java, or should I just plug away with the Scanner class?

The input is a file, not the keyboard.

How do I read input character-by-character in Java?

How do I convert a comma separated string to a array?

I have the input &#39;`1,2,3&#39;` , and I need to convert it into an array.


All Tokenize Solutions on Gang of Coders

Total of 19 Tokenize Solutions