How to Use strtok Function in C

1. Use the strtok Function to Tokenize String With Given Delimiter
2. Use strtok_r Function to Tokenize String With Two Delimiters

This article will demonstrate multiple methods about how to use the strtok function in C.

Use the strtok Function to Tokenize String With Given Delimiter

The strtok function is part of the C standard library defined in the <string.h> header file. It breaks the given string into tokens split by the specified delimiter.

strtok takes two arguments - a pointer to the string to be tokenized as the first parameter and the delimiter string as the second one. Note that, delimiter string is processed as a set of characters, denoting to the separate delimiter.

The function returns the pointer to a null-terminated string, which represents the next token. Mind though, when the same string is parsed with multiple strtok calls, the first pointer argument must be NULL after the initial function call.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[]) {
  char *str1, *token;
  int j;

  if (argc != 3) {
    fprintf(stderr, "Usage: %s string delim\n", argv[0]);
    exit(EXIT_FAILURE);
  }

  for (j = 1, str1 = argv[1];; j++, str1 = NULL) {
    token = strtok(str1, argv[2]);
    if (token == NULL) break;
    printf("%d: %s\n", j, token);
  }

  exit(EXIT_SUCCESS);
}

Sample Command:

./program "Temporary string to be parsed" " "

Output:

1: Temporary
2: string
3: to
4: be
5: parsed

Use strtok_r Function to Tokenize String With Two Delimiters

Alternatively, another version of the function is available called strtok_r, which is a reentrant variant better suited for multi-threaded programs.

Both of these functions modify the string passed as the first argument, and they can’t be on constant strings.

The following example code tokenizes the given string into a two-level hierarchy. The first consists of tokens split on the delimiters passed as the second argument to the program. The inner loop splits each token from the initial tokenization, and finally, the program outputs the results to the console.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[]) {
  char *str1, *str2, *token, *subtoken;
  char *saveptr1, *saveptr2;
  int j;

  if (argc != 4) {
    fprintf(stderr, "Usage: %s string delim subdelim\n", argv[0]);
    exit(EXIT_FAILURE);
  }

  for (j = 1, str1 = argv[1];; j++, str1 = NULL) {
    token = strtok_r(str1, argv[2], &saveptr1);
    if (token == NULL) break;
    printf("%d: %s\n", j, token);

    for (str2 = token;; str2 = NULL) {
      subtoken = strtok_r(str2, argv[3], &saveptr2);
      if (subtoken == NULL) break;
      printf(" --> %s\n", subtoken);
    }
  }

  exit(EXIT_SUCCESS);
}

Sample Command:

./program "Temporary string to be parsed" " " "aeio"

Output:

1: Temporary
 --> T
 --> mp
 --> r
 --> ry
2: string
 --> str
 --> ng
3: to
 --> t
4: be
 --> b
5: parsed
 --> p
 --> rs
 --> d