Wait for stdout completed (C)

[seeplus]

For C. Is there a way to wait for all output sent to stdout to be displayed before continuing? I have a situation where I use puts() to display some initialisation text before starting my own i/o code using Windows console functions. I'm getting the situation whereby text displayed via my functions are partly interleaved with the output from using stdout. I can cure this with a Sleep(50) after the last use of puts() but I don't like using Sleep() like this.

Effectively I have working:

...
puts("System initialisation. Please wait...");
Sleep(50);
ioInit();
ioPut("i/o running OK");

but would like:

...
puts("System initialisation. Please wait...");
//Sleep(50);
waitForCompletion(stdout);    // Or something like this
ioInit();
ioPut("i/o running OK");

Any ideas?

PS. fflush(stdout) doesn't work as that returns before stdout has completed the flush. And neither does WaitForSingleObject() for console output (only works for console input).

[dhayden]

To be clear, you want the puts() to finish output.

fflush(stdout) doesn't work as that returns before stdout has completed the flush.

I don't think that's true. Can you post some sample code that we can compile to demonstrate the problem?

[kigar64551]

In general, you process can write data/text to the stdout (or stderr) stream, from where it eventually will be passed on to whatever the stdout (or stderr) stream happens to be connected to – might be a terminal, a file, or a pipe. Anyway, what happens with the output once it leaves your process, that is not under your control! If your stdout is connected to a terminal, then it is totally up to the terminal process (on Windows that would be conhost.exe) when and how it will show the output from your process, or how that output might be "interleaved" with outputs from another process – in cases where multiple processes are connected to the same terminal.

Calling fflush() only flushes the internal buffer of the FILE* stream, which means that all data that is still buffered inside your process will be sent to the destination (e.g. to the terminal, file or pipe) immediately. For example, if the FILE* is connected to a "regular" file, then calling fflush() only ensures that all pending data – from the FILE*'s internal buffer – is passed on to the OS (e.g. via write() syscall), but you still have to call OS-specific function (e.g. fsync() or _commit() syscall) in order to ensure the data is actually written down to the disc as soon as possible. Similarly, if the FILE* is connected to a terminal, then calling fflush() only ensures that all pending data is passed on to the terminal process, but it can not ensure that the text is actually rendered to the screen by the terminal process immediately.

https://static.lwn.net/images/2011/jm-data-flow.png

Not sure how your ioPut() is implemented, but if it calls WriteFile() or WriteConsole() – assuming that we are on Windows – then you simply pass by stdout and all of the buffering that might be done by the underlying FILE*. Still, the end result should be pretty much the same. After all, the FILE* implementation on Windows is going to call WriteFile() (or WriteConsole()) in the end too...

Does the implementation of your ioPut() use threads?

[coder777]

Try _flushall() to flush the c stream under windows:

https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/flushall?view=msvc-170

[kigar64551]

Try _flushall() to flush the c stream under windows:

That simply flushes all open FILE* streams, so should make no difference (except greater overhead) to just fflush(stdout), when its explicitly the stdout stream that you want to flush. But, either way, it only flushes the internal buffer of the FILE* stream (see above).

[seeplus]

Can you post some sample code that we can compile to demonstrate the problem?

Unfortunately not. 1) It's not public code and 2) It's a few thousand lines of code for the io functions.

but if it calls WriteFile() or WriteConsole()

Yes - ultimately it uses WriteConsole() to display character(s).

Does the implementation of your ioPut() use threads?

Yes. That's part of what ioInit() does. the display from our io is from a separate thread - which is probably the issue as WriteConsole() is not thread safe - that's why I need I probably need to wait for any existing output to complete - but how to know when done?

I've also found no solution searching the web - so I'm thinking there's no way to do this. Sleep(50) works fine. I'll just leave it and include a suitable code comment.

This is it working as expected with the Sleep(50):

abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYS

Foobar Foobar Foobar

and this is it not working (with no Sleep(50) ):

Fabcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYS
oobar Foobar Foobar

- Note that the actual output text has been changed to protect the innocent!

The inititial \nF from the the Foobar line is displayed before the abcdef line.

[kigar64551]

Yes. That's part of what ioInit() does. the display from our io is from a separate thread - which is probably the issue as WriteConsole() is not thread safe - that's why I need I probably need to wait for any existing output to complete - but how to know when done?

Well, WriteFile() (or _write()) and WriteConsole() are thread safe – in the sense that you are allowed to call them from concurrent threads without synchronization and still nothing will break/crash. The crucial point is: There simply is no guarantee on how the data from concurrent WriteFile() or WriteConsole() invocations that go to the same file handle will be "multiplexed".

I think that a FILE* stream ultimately calls WriteFile() or WriteConsole() too – once its buffer is full or is flushed – but fwrite() and friends use a lock (critical section) to synchronize those calls! That's why there is the non-synchronized version _fwrite_nolock().

If your ioPut() uses threads and it calls WriteFile() or WriteConsole() without explicit synchronization, then it kind of "defeats" the synchronization that is built into FILE* streams, because your WriteFile() will not be synchronized to the FILE* stream's one.

You would have to synchronize your WriteFile() or WriteConsole() with the same lock (critical section) as the FILE* stream uses internally. Problem is: The FILE structure intentionally is opaque, i.e. you can not (easily) access its internal fields, such as the lock.

[seeplus]

Well, WriteFile() (or _write()) and WriteConsole() are thread safe – in the sense that you are allowed to call them from concurrent threads without synchronization and still nothing will break/crash. The crucial point is: There simply is no guarantee on how the data from concurrent WriteFile() or WriteConsole() invocations that go to the same file handle will be "multiplexed".

Yes - in that sense.

If your ioPut() uses threads and it calls WriteFile() or WriteConsole() without explicit synchronization, then it kind of "defeats" the synchronization that is built into FILE* streams, because your WriteFile() will not be synchronized to the FILE* stream's one.

Yes - that's why I have to wait until FILE* streams is fully completed. The code currently accomplishes this by using a Sleep(50) (and has been for quite a while) but I don't like having an arbitrary Sleep() in the code - that's why I want to replace it with... - but I can't see an alternative...

[kigar64551]

Yes - that's why I have to wait until FILE* streams is fully completed. The code currently accomplishes this by using a Sleep(50) (and has been for quite a while) but I don't like having an arbitrary Sleep() in the code - that's why I want to replace it with... - but I can't see an alternative...

And there is absolutely no guarantee that the Sleep() works at all - even if it happens to work most of the time.

I think that the only proper solution is to synchronize every output to the terminal in every thread on the same lock (critical section).

If, inside your ioPut() code, you synchronize every WriteFile() or WriteConsole() invocation on a CRITICAL_SECTION, and if you also synchronize every access to stdout or stderr (including fflush()) on that same CRITICAL_SECTION, then it should work.

CRITICAL_SECTION g_lock;

int main()
{
    InitializeCriticalSection(&g_lock);

    // ...

    EnterCriticalSection(&g_lock);
    fputs("System initialisation. Please wait...\n", stdout);
    fflush(stdout);
    LeaveCriticalSection(&g_lock);

    // ...

    ioInit();
    ioPut("i/o running OK");
}

DWORD ThreadProc(LPVOID lpParameter)
{
    // ...

    EnterCriticalSection(&g_lock);
    WriteFile(hStdOut, ...);
    LeaveCriticalSection(&g_lock);
}

But why do you need your "custom" I/O code at all?

If it is really needed, then why do you mix it with the standard output FILE* streams and not use your "custom" I/O everywhere?

[seeplus]

We don't 'mix' our io with standard output. We only use standard output prior to our io being initialised and used. Once our ioInit() has been successful we never again use standard console output (or input). We only get thus issue once on startup when we first output some text via standard io before switching to our io. Yes, Sleep(50) isn't guaranteed to work - but it has for quite a while (as far as I know some years). I'm currently doing some upgrade to this software, saw the Sleep() statement, didn't like it. Hence my quest to try to improve upon it. But it seems this is a forlorn quest...

Yes, using a CriticalSection would solve the issue. But as the problem only exists once at the start it's a bit of an overkill.

Update. No, using a critical section doesn't solve the issue as the critical section exits for FILE* before the output has completed - the original issue!