How to Tokenize a String in C++

Jinku Hu Feb 02, 2024

C++ C++ String

Use the find and substr Functions to Tokenize a String in C++
Use the std::stringstream and getline Functions to Tokenize a String in C++
Use the istringstream and copy Algorithm to Tokenize a String in C++

This article will explain several methods of how to tokenize a string in C++.

Use the `find` and `substr` Functions to Tokenize a String in C++

std::string class has a built-in find function to search for a sequence of characters in a given string object. The find function returns the first character’s position found in the string and returns npos if not found. The find function call is inserted in the if statement to iterate over string until the last string token is extracted.

Notice that a user can specify any string type delimiter and pass it to the find method. The tokens are pushed to the vector of strings and already processed part removed with the erase() function on each iteration.

#include <algorithm>
#include <iostream>
#include <iterator>
#include <string>
#include <vector>

using std::cin;
using std::cout;
using std::endl;
using std::string;
using std::vector;

string text = "Think you are escaping and run into yourself.";
int main() {
  string delim = " ";
  vector<string> words{};

  size_t pos = 0;
  while ((pos = text.find(delim)) != string::npos) {
    words.push_back(text.substr(0, pos));
    text.erase(0, pos + delim.length());
  }
  if (!text.empty()) words.push_back(text.substr(0, pos));
  for (const auto &str : words) {
    cout << str << endl;
  }

  return EXIT_SUCCESS;
}

Output:

Think
you
are
escaping
and
run
into
yourself.

Use the `std::stringstream` and `getline` Functions to Tokenize a String in C++

stringstream can be utilized to ingest a string to be processed and use getline to extract tokens until the given delimiter is found. Note that this method only works with single-character delimiters.

#include <algorithm>
#include <iostream>
#include <iterator>
#include <sstream>
#include <string>
#include <vector>

using std::cin;
using std::cout;
using std::endl;
using std::istringstream;
using std::string;
using std::stringstream;
using std::vector;

int main() {
  string text = "Think you are escaping and run into yourself.";
  char del = ' ';
  vector<string> words{};

  stringstream sstream(text);
  string word;
  while (std::getline(sstream, word, del)) words.push_back(word);

  for (const auto &str : words) {
    cout << str << endl;
  }

  return EXIT_SUCCESS;
}

Output:

Think
you
are
escaping
and
run
into
yourself.

Use the `istringstream` and `copy` Algorithm to Tokenize a String in C++

Alternatively, we can use the copy function from the <algorithm> header and extract string tokens on white space delimiters. In the following example, we only iterate and stream tokens to the standard output. To process the string with the copy method, we insert it in istringstream and utilize its iterators.

#include <algorithm>
#include <iostream>
#include <iterator>
#include <sstream>
#include <string>
#include <vector>

using std::cin;
using std::cout;
using std::endl;
using std::istringstream;
using std::string;
using std::stringstream;
using std::vector;

int main() {
  string text = "Think you are escaping and run into yourself.";
  string delim = " ";
  vector<string> words{};

  istringstream iss(text);
  copy(std::istream_iterator<string>(iss), std::istream_iterator<string>(),
       std::ostream_iterator<string>(cout, "\n"));

  return EXIT_SUCCESS;
}

Output:

Think
you
are
escaping
and
run
into
yourself.

Author: Jinku Hu

Founder of DelftStack.com. Jinku has worked in the robotics and automotive industries for over 8 years. He sharpened his coding skills when he needed to do the automatic testing, data collection from remote servers and report creation from the endurance test. He is from an electrical/electronics engineering background but has expanded his interest to embedded electronics, embedded programming and front-/back-end programming.

LinkedIn Facebook

Use the find and substr Functions to Tokenize a String in C++

Use the std::stringstream and getline Functions to Tokenize a String in C++

Use the istringstream and copy Algorithm to Tokenize a String in C++

Related Article - C++ String

Use the `find` and `substr` Functions to Tokenize a String in C++

Use the `std::stringstream` and `getline` Functions to Tokenize a String in C++

Use the `istringstream` and `copy` Algorithm to Tokenize a String in C++