Simulator, Thread Design, and You

Hey all,

I was hoping I could get some advice on designing the proper threading for our project.

A little background info - our company does simulators of complex electrical/mechanical systems. These simulators could have fuel systems, electrical systems, pneumatic pressure systems, etc. I suppose you can just think of a car and that's close enough. We have done many of these simulators and have no problems with the systems themselves. However, we get requests from customers that have drastically different parts. Think of simulating one model of a car, including the ability to fail every individual component and system, then having to do a completely different model. For far too many years, we've rewritten our code base and had very poor code re-usability.

Currently, we are trying to make a centralized code base that can be used as our "core". We want to be able to switch from one model to another with minimal fuss and greatly improve our outdated design. Our method to this madness was to represent all objects as individual componenets derived from a similar base class. These components use an event driven model to pass data between each other dynamically, allowing a great number of systems to use this without actually having to know what the component is. We also have other base classes for appropriate systems, such as "IElectricalComponent". This mostly follows the Template design methodology:
http://www.dofactory.com/Patterns/PatternTemplate.aspx

With this, we should be able to build a new model by simply defining the specific parts of this model to the abstract factory, link up all the data messages, and let the systems run themselves.

The problem we're trying to overcome, however, is the sheer number of possible inputs. We get some of our data from a PLC using real switches and buttons to simulate the real thing, some of our data comes in from individual touch screens, some of it comes in from serial data streams from the real components, and some of it comes in from a control loader. We cannot guarantee until we get the contract how each simulator will behave. Because of this, we have very dynamic connections (and try our hardest to use UDP if possible). Our current model that we're replacing used a single thread for each connection, and did a lot of recreating code to define what that connection did. Because of our new model, we were trying to make this more scripted and dynamic.

The data mapping is not the problem, but the threading is. Since I cannot guarantee what is in each connection, I am uncertain where to put my semaphores/mutexes. Should each system (such as electrical) have it's own mutex, and each connection may have to lock several depending upon it's data? Should each connection have only one semaphore, which would require a giant global mutex lock on all systems while data is being set? Is a mutex generally fast enough that each component should have it's own data setting mutex and have hundreds of them possibly being set on a large chunk of data?

Any advice on how to thread this would be appreciated.
our company does simulators of complex electrical/mechanical systems
Yes, I did some simulation too back then. It's really fun.

http://www.dofactory.com/Patterns/PatternTemplate.aspx
I'd discourage from using it. It tends to make things more complicated then easier. Those ppl want to sell...

The problem we're trying to overcome, however, is the sheer number of possible inputs
Nah, that's not an issue. Having a nice base class suffice

I am uncertain where to put my semaphores/mutexes
This shows your design dilemma.

I'm in a similar situation (about 16 inputs, several protocolls, (embedded) systems). It's a real communication system (means the role of client/server changes). The design is rather straight forward -> process driven.

For such a design it is essential that you separate different concepts. For instance what (protocol) you receive shouldn't have anything to do with how (socket/serial port) you reveive it
I'd discourage from using it. It tends to make things more complicated then easier. Those ppl want to sell...

Nah, that's not an issue. Having a nice base class suffice


That's kinda' what I meant about template design pattern. The base class is very similar to an interface, but it has too much implementation code to be a true abstract interface. That's what I mean by template.

For such a design it is essential that you separate different concepts. For instance what (protocol) you receive shouldn't have anything to do with how (socket/serial port) you reveive it


Oh, I fully agree! I've long ago setup some connection classes to abstract away the type of connection. Also I'm creating serialized message classes that are filled in with separate protocols to even abstract away the data storage.

The problems of having all this data is in the thread safety, and still the main focus of my issues. I don't mind having a lot of data if you can predict where it's coming from or organize them by system, but with ANY data coming in from one single connection, I'm still very uncertain how to setup thread safety critical sections, etc.

One simulator will have a battery voltage coming in through the PLC alongside all the electrical bus switches. Another simulator will receive the battery voltage from our modeling computer and the electrical bus switches are coming in from a separate machine attached to a touch screen. Without being able to predict from sim to sim, I need a flexible design. So the question still stands, where would you put the mutex for thread safety? One global one attached to a data manager that locks and unlocks generally anything assigned to it? Each system gets a mutex and controls thread safety on all components that derive from the system base class? Every component gets it's own mutex?
Last edited on
A base class for handling the incoming data might look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
class CDevice // This is overwritten for each device you want to deal with (PLC and what not)
{
  void Open()
  {
    lock mutex
    OnOpen();
  }
  void Close()
  {
    lock mutex
    Close();
  }

  void Read(void *data, size_t data_size, size_t *data_received)
  {
    lock mutex
    OnRead(...);
  }
  void Write(void *data, size_t data_size)
  {
    lock mutex
    OnWrite(..);
  }

  protected:
    virtual On...() // Within this On... functions (overwritten by the derived class -> e.g OnRead(...) { socket recv }) you can safely deal with the data 

protected:
  mutex;
};

The device itself has nothing to do with threads but is used by thread. In this scenario you need a reader for each incomming channel that collects the data (e.g. in a Device::CBuffer), checks for consistence, and then passing the data to the actual data handler (which might be a thread). The reader don't need to know what the nature of the channel ist. For testing purposes you can even pass a file device

For passing the objects I certainly recommend using smart pointer/shared_ptr

But really: where and when you use mutexes depends on your design. Just a global mutex is certainly nothing good...

By the way
and try our hardest to use UDP if possible
Why not using the safe TCP/IP connection?
Complex simulation stuff + mutexes + third-party-code = expect big problems
Third-party-code + C++ = lack of module isolation = expect even bigger problems

Obviously you are using the wrong tool for the very hard job.


So the question still stands, where would you put the mutex for thread safety


Nowhere. Don't use mutexes. Don't use shared mutable state. It will either crash, or perform badly, and most probably both. Use asynchronous message passing. Use immutable data structures to store state. Use good actor framework. Use a language that is suited for these kind of things. Start from reading these:

http://www.stanford.edu/~ouster/cgi-bin/papers/threads.pdf
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf


And then, learn this:
http://akka.io/
Last edited on
coder777:

I see, so your suggestion is a single data in/out mutex per connection.

This would only work if data could not be written from multiple connections however. For example, if the PLC and a separate machine were both to have the ability to control the Battery switch state, the two threads could receive messages at the same time to set the same data. Those locks on the connections would only work if they were shared, and starts reposing the question of having locks on systems instead.

Also, I should point out that the only reason why a single master mutex idea is applicable is the requirements of the system. Data/logic needs to respond exceptionally fast to any input, but the inputs are not necessarily happening quickly or often. We receive a packet once every 50 ms on our fastest port, but need to respond in less than 10 ms. Some of our slower connections are only communicating at 10 Hz. The amount of time spent in a write lock is minimal and not often done, so the chances of threads waiting to write is severely reduced.

Why not using the safe TCP/IP connection?

Ha, I asked this same question multiple times when I first started this job. In the end, there are simply too many external devices that unreliably broadcast their data (GPS are notorious for this), and essentially only do so through UDP. Also, our real-time operating system isn't a huge fan of TCP/IP for speed sake. It simply has a fairly crappy custom card which I assume was due to no expectations of customer's using it.

rapidcoder:

Don't use mutexes... Use asynchronous message passing.

As far as I've seen in C++ (which we can't avoid due to the nature of the VxWorks OS), asynchronous message passing still requires a critical section. This is due to the queue itself. You can't increase and decrease the size of the message queue across threads without thread safety. The queue.size variable could become corrupted. Most asynchronous message passing methods are in other languages that circumvent this behind the scenes. Certainly correct me if I'm wrong on this.

Use immutable data structures to store state.

I don't see how that's possible to always use immutable types in a C++ environment in a simulation with many dynamically changing variables. Perhaps I misunderstand you on this one?

Use a language that is suited for these kind of things.

Stuck in C++, it's the only thing our kernel will run. Also, language should never be a pitfall if you know how to design.

Anyway, I read through those and they don't say anything useful at all that applies to my problem. Yes you should avoid threading where possible, but being that we have many real-time systems trying to communicate and some of them have VERY particular timing requirements, CPU concurrency is a requirement.

I do appreciate the Akka suggestion. It's a fine model, but our OS won't support it's libraries. This executable is practically running on an embedded system, but an oddly powerful one. C++ and stdlib only unfortunately.
Finally we're getting to some details

As far as I've seen in C++ [...] asynchronous message passing still requires a critical section.

There are many lockless and fine-locked message queue designs for C++, but even with coarse (container-wide) locks, I don't see a problem:

if the PLC and a separate machine were both to have the ability to control the Battery switch state, the two threads could receive messages at the same time to set the same data. Those locks on the connections would only work if they were shared,


This sounds like you have a message queue which consists of the events targeted to the Battery thread, where PLC thread and "separate machine" thread are adding events, and Battery is removing events. The queue has to guarantee consistency, one way or another, so whatever algorithm is used, is encapsulated in it. The threads have nothing special to do.

By the way, as much as I dislike ACE framework (and VxWorks -- I worked for their competitor), I hear ACE works there, take a look for some design ideas.
Last edited on
Topic archived. No new replies allowed.