Capturing stdout from a system() command optimally

I'm trying to start an external application through system() - for example, system("ls"). I would like to capture its output as it happens so I can send it to another function for further processing. What's the best way to do that in C/C++?

From the popen manual:

#include <stdio.h>
  
FILE *popen(const char *command, const char *type);
  
int pclose(FILE *stream);

Try the popen() function. It executes a command, like system(), but directs the output into a new file. A pointer to the stream is returned.

  FILE *lsofFile_p = popen("lsof", "r");

  if (!lsofFile_p)
  {
    return -1;
  }

  char buffer[1024];
  char *line_p = fgets(buffer, sizeof(buffer), lsofFile_p);
  pclose(lsofFile_p);

EDIT: misread question as wanting to pass output to another program, not another function. popen() is almost certainly what you want.

System gives you full access to the shell. If you want to continue using it, you can redirect it's output to a temporary file, by system("ls > tempfile.txt"), but choosing a secure temporary file is a pain. Or, you can even redirect it through another program: system("ls | otherprogram");

Some may recommend the popen() command. This is what you want if you can process the output yourself:

FILE *output = popen("ls", "r");

which will give you a FILE pointer you can read from with the command's output on it.

You can also use the pipe() call to create a connection in combination with fork() to create new processes, dup2() to change the standard input and output of them, exec() to run the new programs, and wait() in the main program to wait for them. This is just setting up the pipeline much like the shell would. See the pipe() man page for details and an example.

The functions popen() and such don't redirect stderr and such; I wrote popen3() for that purpose.

Here's a bowdlerised version of my popen3():

int popen3(int fd[3],const char **const cmd) {
        int i, e;
        int p[3][2];
        pid_t pid;
        // set all the FDs to invalid
        for(i=0; i<3; i++)
                p[i][0] = p[i][1] = -1;
        // create the pipes
        for(int i=0; i<3; i++)
                if(pipe(p[i]))
                        goto error;
        // and fork
        pid = fork();
        if(-1 == pid)
                goto error;
        // in the parent?
        if(pid) {
                // parent
                fd[STDIN_FILENO] = p[STDIN_FILENO][1];
                close(p[STDIN_FILENO][0]);
                fd[STDOUT_FILENO] = p[STDOUT_FILENO][0];
                close(p[STDOUT_FILENO][1]);
                fd[STDERR_FILENO] = p[STDERR_FILENO][0];
                close(p[STDERR_FILENO][1]);
                // success
                return 0;
        } else {
                // child
                dup2(p[STDIN_FILENO][0],STDIN_FILENO);
                close(p[STDIN_FILENO][1]);
                dup2(p[STDOUT_FILENO][1],STDOUT_FILENO);
                close(p[STDOUT_FILENO][0]);
                dup2(p[STDERR_FILENO][1],STDERR_FILENO);
                close(p[STDERR_FILENO][0]);
                // here we try and run it
                execv(*cmd,const_cast<char*const*>(cmd));
                // if we are there, then we failed to launch our program
                perror("Could not launch");
                fprintf(stderr," \"%s\"\n",*cmd);
                _exit(EXIT_FAILURE);
        }

        // preserve original error
        e = errno;
        for(i=0; i<3; i++) {
                close(p[i][0]);
                close(p[i][1]);
        }
        errno = e;
        return -1;
}

The most efficient way is to use stdout file descriptor directly, bypassing FILE stream:

pid_t popen2(const char *command, int * infp, int * outfp)
{
    int p_stdin[2], p_stdout[2];
    pid_t pid;

    if (pipe(p_stdin) == -1)
        return -1;

    if (pipe(p_stdout) == -1) {
        close(p_stdin[0]);
        close(p_stdin[1]);
        return -1;
    }

    pid = fork();

    if (pid < 0) {
        close(p_stdin[0]);
        close(p_stdin[1]);
        close(p_stdout[0]);
        close(p_stdout[1]);
        return pid;
    } else if (pid == 0) {
        close(p_stdin[1]);
        dup2(p_stdin[0], 0);
        close(p_stdout[0]);
        dup2(p_stdout[1], 1);
        dup2(::open("/dev/null", O_WRONLY), 2);

        /// Close all other descriptors for the safety sake.
        for (int i = 3; i < 4096; ++i) {
            ::close(i);
        }

        setsid();
        execl("/bin/sh", "sh", "-c", command, NULL);
        _exit(1);
    }

    close(p_stdin[0]);
    close(p_stdout[1]);

    if (infp == NULL) {
        close(p_stdin[1]);
    } else {
        *infp = p_stdin[1];
    }

    if (outfp == NULL) {
        close(p_stdout[0]);
    } else {
        *outfp = p_stdout[0];
    }

    return pid;
}

To read output from child use popen2() like this:

int child_stdout = -1;
pid_t child_pid = popen2("ls", 0, &child_stdout);

if (!child_pid) {
    handle_error();
}

char buff[128];
ssize_t bytes_read = read(child_stdout, buff, sizeof(buff));

To both write and read:

int child_stdin = -1;
int child_stdout = -1;
pid_t child_pid = popen2("grep 123", &child_stdin, &child_stdout);

if (!child_pid) {
    handle_error();
}

const char text = "1\n2\n123\n3";
ssize_t bytes_written = write(child_stdin, text, sizeof(text) - 1);

char buff[128];
ssize_t bytes_read = read(child_stdout, buff, sizeof(buff));

In Windows, instead of using system(), use CreateProcess, redirect the output to a pipe and connect to the pipe.

I'm guessing this is also possible in some POSIX way?

The functions popen() and pclose() could be what you're looking for.

Take a look at the glibc manual for an example.

Actually, I just checked, and:

1. popen is problematic, because the process is forked. So if you need to wait for the shell command to execute, then you're in danger of missing it. In my case, my program closed even before the pipe got to do it's work.

2. I ended up using system call with tar command on linux. The return value from system was the result of tar.

So: if you need the return value, then not no only is there no need to use popen, it probably won't do what you want.

In this page: capture_the_output_of_a_child_process_in_c describes the limitations of using popen vs. using fork/exec/dup2/STDOUT_FILENO approach.

I'm having problems capturing tshark output with popen.

And I'm guessing that this limitation might be my problem:

> It returns a stdio stream as opposed to a raw file descriptor, which > is unsuitable for handling the output asynchronously.

I'll come back to this answer if I have a solution with the other approach.

I'm not entirely certain that its possible in standard C, as two different processes don't typically share memory space. The simplest way I can think of to do it would be to have the second program redirect its output to a text file (programname > textfile.txt) and then read that text file back in for processing. However, that may not be the best way.