std::vector and mmapped memory with two processes

[cmisip]

I am working on a project where (If I understand correctly ), two separate processes are communicating via mmap memory. One process, the capture daemon saves avframe framebuffer data into a buffer and sideframedata into a vector while the other process, the processing daemon reads the data and performs operation. Both processes use the same named image_buffer[] array. I think because the daemon's and their variables are in different memory spaces, changes to the circular_buffer by the capture daemon are not seen in the process daemon. Therefore, the need to mmap the data of the capture daemon and then read the mmap file with the process daemon. This works well with the frame buffer information. However, I have a std::vector that I create in the capture daemon and when I read it in the process daemon, recreating the vector causes a segfault. The vector is one AVFrameSideData that is stored in a uint8_t buffer in the heap. It itself is a sequence of AVMotionVector (The actual number of elements is AVframeSideData->size/sizeof(AVMotionVector). I can extract the AVFrameSideData->size and therefore discover the number of AVMotionVector elements. But creating a vector from this buffer in the process daemon causes a segfault :

AVFrameSideData* mbuff = (AVFrameSideData *)image_buffer[index].image->mv_buffer;

  
    if (mbuff) {
    const AVMotionVector *mvs = (const AVMotionVector *)mbuff->data;
    int vec_count=0;
                    for (int i = 0; i < mbuff->size / sizeof(*mvs); i++) {
                        const AVMotionVector *mv = &mvs[i];  //SEGFAULT
                        Info("Vertex %d,%d",mv->dst_x,mv->dst_y );
                        vec_count++;
                        
                    }
    Info("VEC_COUNT %d", vec_count++);
  }

Another way to do it:

      AVFrameSideData* mbuff = (AVFrameSideData *)image_buffer[index].image->mv_buffer;

            AVMotionVector *mv = (AVMotionVector*)mbuff->data;
               int mvcount = mbuff->size / sizeof(AVMotionVector);
               std::vector<AVMotionVector> mvlistv=std::vector<AVMotionVector>(mv,mv+mvcount); //SEGFAULT
               Info("CAPTURE %d", mvlistv.size());
    

Is this because, in the process daemon, the vector element addresses refer to memory space of the capture daemon to which the process daemon has no access thereby resulting in a segfault?

If this is true, I need to send the actual dereferenced data through the mmap file. At the very least, I will need to minimally process the vectors in the capture daemon to select motion vectors with a displacement from src to dst and save just the point coordinate of each of those motion vectors. So each motion vector would be a Point struct with an x and y value.

How does one go about building a fixed size data buffer containing the Points in the capture daemon, that I could read in the process daemon and separate out into individual Point objects. The number of AVMotionVectors is not known but there is a limit ( that I need to discover or guess). The data can be stored in:

image_buffer[index].image->mv_buffer;

mv_buffer must contain sequential values of type Point that I can access by jumping forward 1 Point size at a time and I need to know how many Points there are.

Thanks,
Chris

[JLBorges]

> Is this because, in the process daemon, the vector element addresses refer to memory space of
> the capture daemon to which the process daemon has no access thereby resulting in a segfault?

Yes. The memory for the elements of the vector need to be in memory that is shared between the two processes. In addition, since the shared memory may be mapped at different addresses in different processes, we would need a smart pointer hat can take care of this nicety.

See: Mapping Address Independent Pointer: offset_ptr
http://www.boost.org/doc/libs/1_64_0/doc/html/interprocess/offset_ptr.html

And: Containers in managed memory segments
http://www.boost.org/doc/libs/1_64_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.containers_explained

Consider using boost:interprocess::vector for this purpose.

[cmisip]

I am trying to integrate code into another program so I am constrained as far as possible implementations. The capture daemon and the process daemon are in some sort of a race and the process daemon should never catch up with the capture daemon. So the capture daemon is creating image_buffer[index].image and when it gets to index 50, it steps on index=0 again. The process daemon will start at an earlier index and follow processing its copy of the the image_buffer data. So data in the capture daemon memory space is really volatile and is not expected to live long. The vector data specific to an image_buffer[index] may no longer exist in capture space memory or interprocess memory by the time process daemon needs it If I understand things correctly. Is there another implementation that is possible?

Maybe it helps to mention the program I am trying to modify. It's open source after all. ZoneMinder. I can see (get the size member and therefore figure out how many motion vectors there are ) the AVFrameSideData in zm_monitor.cpp at Monitor::Analyse but segfault in building the std::vector<AVMotionVectore> for the reasons above. If I can do this successfully, I can pass the data to an overload of DetectMotion().

Thanks,
Chris

[cmisip]

Here is the code to build the uint16_t buffer in zm_ffmpeg_camera.cpp

uint16_t* mvect_buffer=image.VectBuffer();
       
        
        AVFrameSideData *sd=NULL;

        sd = av_frame_get_side_data(mRawFrame, AV_FRAME_DATA_MOTION_VECTORS);
        
        if (sd) {
            
            if (mvect_buffer ==  NULL ){
                   Error("Failed requesting vector buffer for the captured image.");
                   return (-1); 
            }
if (mvect_buffer) {
                //save the number of motion vectors
                uint16_t  mvcount=sd->size/sizeof(AVMotionVector);
                memcpy(mvect_buffer,&mvcount, sizeof(uint16_t));
                int offset=sizeof(uint16_t);
                //save the rest
                const AVMotionVector *mvs = (const AVMotionVector *)sd->data;
                int vec_count=0;
                for (int i = 0; i < sd->size / sizeof(*mvs); i++) {
                        const AVMotionVector *mv = &mvs[i];
                        
                        //Check if different source and dest 
                      //  if ((mv->src_x == mv->dst_x) && (mv->src_y == mv->dst_y))
                      //     continue;
             
                        //Check if vector had significant value
                     //   int magnitude = abs(mv->dst_x-mv->src_x) + abs(mv->dst_y-mv->src_y);
                     //   if (magnitude == 0)
                      //     continue;
                        
                        memcpy(mvect_buffer+offset,&mv->dst_x,2);
                        offset+=2;
                        memcpy(mvect_buffer+offset,&mv->dst_x,2);
                        offset+=2;
                        vec_count++;
                        
                }
               
               Info("FFMPEG VEC_COUNT %d", vec_count);
           
               
            }      

And here to read it in zm_monitor.cpp

unsigned int Monitor::DetectMotion(uint16_t * sd, Event::StringSet &zoneSet ) {
 std::vector<Coord> vertex_points;
    uint16_t size=0;
     //first 16bit value is size
    memcpy(&size,sd,2);
    if ((size>0) && (sd)) {
       
        
        
        memcpy(&size,sd,2);
        int offset=sizeof(uint16_t);
        for (int i = 0; i < size; i++) {
                        
            int x,y;
                        memcpy(&x,sd+offset,2);
                        offset+=2;
                        memcpy(&y,sd+offset,2);
                        offset+=2;
                        vertex_points[i]=Coord(x,y);
                        Info("Vertex %d,%d",vertex_points[i].X(),vertex_points[i].Y() );
                        
        }
       
       Info("DetectMotion size of vertex %d compared with vector size of %d", size, vertex_points.size());
   }else
       Info("No mvect data");
    
    return vertex_points.size();

}    

mv_buffer is created this way:
mv_buffer = (uint16_t *) malloc(mv_size);

and freed with:
free(mv_buffer)

The mv_size is computed this way:

//maximum size bytes required:
//store at 0 byte the size in uint16_t, and each pair of uint16_t for x, and y values
        //for 1920x1080 with 4x4 size macroblocks, this would be ((((1920/4)+1)*(1080/4))*2)+2  
        //static const unsigned int mv_size=259742; 
        static const unsigned int mv_size=259742;

I have not tested it yet. Trying to figure out why zoneminder keeps killing the capture process. The capture process is timed and if it does not exit, it is killed.

What do you think of the code?
Thanks,
Chris

[cmisip]

I did try the code. I am getting the correct size of the vector by reading the first 2 bytes of sd in DetectMotion. It is the same value for size in zm_ffmpeg_camera.cpp. I had to rework things so everything is stored in 8 bit. However, things still fail on creation of std::vector. I don't really need the vector since I can iterate through the contents of the mv_buffer but I wonder why it would segfault with std::vector creation. Things seem to be working. If I wave my hand in front of the camera, I could get motion vectors and alarm based on the original Detectmotion that worked on pixel data.

  if ( !(image_count % (motion_frame_skip+1) ) ) {
            // Get new score.
            motion_score = last_motion_score = DetectMotion( mvect_buffer, zoneSet );
            Info("VECTOR motion score : %d", motion_score);
            motion_score = last_motion_score = DetectMotion( *snap_image, zoneSet );
            Info("PIXEL motion score : %d", motion_score);
          }

Sample output: No Motion

 2017-06-12 07:49:23.951746	zma_m1		19390	INF	VECTOR motion score : 0	zm_monitor.cpp	1300
2017-06-12 07:49:23.918714	zma_m1		19390	INF	No mvect data	zm_monitor.cpp	3060
2017-06-12 07:49:23.886023	zma_m1		19390	INF	PIXEL motion score : 0	zm_monitor.cpp	1302
2017-06-12 07:49:23.843082	zma_m1		19390	INF	VECTOR motion score : 0	zm_monitor.cpp	1300

Sample output: Motion

2017-06-12 07:50:35.491782	zma_m1		19390	INF	PIXEL motion score : 38	zm_monitor.cpp	1302
2017-06-12 07:50:35.434041	zma_m1		19390	INF	VECTOR motion score : 218	zm_monitor.cpp	1300
2017-06-12 07:50:35.317943	zma_m1		19390	INF	PIXEL motion score : 29	zm_monitor.cpp	1302
2017-06-12 07:50:35.266826	zma_m1		19390	INF	VECTOR motion score : 254	zm_monitor.cpp	1300
2017-06-12 07:50:35.162679	zma_m1		19390	INF	Monitor-1: 2455 - Gone back into alarm state	zm_monitor.cpp	1505
2017-06-12 07:50:35.128407	zma_m1		19390	INF	PIXEL motion score : 42	zm_monitor.cpp	1302
2017-06-12 07:50:35.071188	zma_m1		19390	INF	VECTOR motion score : 579	zm_monitor.cpp	1300
2017-06-12 07:50:35.003768	zma_m1		19390	INF	Monitor-1: 2454 - Gone into alert state

zm_ffmpeg_camera.cpp

AVFrameSideData *sd=NULL;

        sd = av_frame_get_side_data(mRawFrame, AV_FRAME_DATA_MOTION_VECTORS);
        
        if (sd) {
            
            if (mvect_buffer ==  NULL ){
                   Error("Failed requesting vector buffer for the captured image.");
                   return (-1); 
            }
            
            if (mvect_buffer) {
                
                int offset=sizeof(uint16_t);
                //save the rest
                const AVMotionVector *mvs = (const AVMotionVector *)sd->data;
                int vec_count=0;
                for (int i = 0; i < sd->size / sizeof(*mvs); i++) {
                        const AVMotionVector *mv = &mvs[i];
                        
                        //Check if different source and dest 
                        if ((mv->src_x == mv->dst_x) && (mv->src_y == mv->dst_y))
                           continue;
             
                        //Check if vector had significant value
                        int magnitude = abs(mv->dst_x-mv->src_x) + abs(mv->dst_y-mv->src_y);
                        if (magnitude == 0)
                           continue;
                        
                        //CONVERT int16 to two int8 here and save
                        uint8_t x8bit[2]={ mv->dst_x & 0xff, mv->dst_x >> 8 };
                        memcpy(mvect_buffer+offset,&x8bit,2);
                        offset+=2;
                        uint8_t y8bit[2]={ mv->dst_y & 0xff, mv->dst_y >> 8 };
                        memcpy(mvect_buffer+offset,&y8bit,2);
                        offset+=2;
                        
                        vec_count++;
                        
                }
                
                //save the number of motion vectors
                //uint16_t  mvcount=sd->size/sizeof(AVMotionVector);
                //convert vec_count in 8bit here and save
                uint8_t vc8bit[2]={ vec_count & 0xff, vec_count >> 8 };
                memcpy(mvect_buffer,&vc8bit, 2);
               
              // Info("FFMPEG VEC_COUNT %d", vec_count);
           
               
            }      
        } 
        
        

Code for DetectMotion

unsigned int Monitor::DetectMotion(uint8_t * sd, Event::StringSet &zoneSet ) {
 
  //std::vector<Coord> vertex_points;  
  
  //minimum manhattan distance to consider a motion_vector with displacement
    int min_vector_size=1; //FIXME, need to be a config option
    //number of vectors clustered together as minimum "filter"
    int min_vectors_filter=2; //FIXME, need to be a config option
    //minimum manhattan pixel distance between vectors to consider them as belonging together in a cluster
    int min_vector_cluster_distance=10;  //FIXME, need to be a config option
    
  
    uint16_t size=0;
    
    int offset=0;
    //first 16bit value is size
    if (sd) {
        uint8_t size8bit[2];
        memcpy(&size8bit,sd,2);
        size=((uint16_t)size8bit[1] << 8) | size8bit[0];
        offset=sizeof(uint16_t);
    }
    if (size>0) {
        for (int i = 0; i < size; i++) {
                        
            uint16_t x,y;
            uint8_t x8bit[2];
                        memcpy(&x8bit,sd+offset,2);
                        x=((uint16_t)x8bit[1] << 8) | x8bit[0];
                        offset+=2;
            uint8_t y8bit[2];            
                        memcpy(&y8bit,sd+offset,2);
                        y=((uint16_t)y8bit[1] << 8) | y8bit[0];
                        offset+=2;
                        
                        
                        //vertex_points[i]=Coord(x,y);
                        //Info("Vertex %d,%d",vertex_points[i].X(),vertex_points[i].Y() );
                        
                        
                         //Info("Vertex %d,%d",x,y);
                        
                       
        }
       
       //Info("DetectMotion size of vertex %d compared with vector size of %d", size, vertex_points.size());
   } else
       Info("No mvect data");
    
    //return vertex_points.size();
    return size;

}    
  

Any ideas?

Thanks,
Chris

[cmisip]

I am marking this as solved. Solution to vector copy into another memory space that I am using right now is to dereference all the values stored in the vector, save in a mmap file and recreate the vector in the other memory space with memcpy. The use of the mmap file is the method that the project maintainer is using. It seems to be working with my code modifications and the code has progressed from the snippet here. I however have another question in another thread regarding memcpy that is related to this.

Thanks for all the help,
Chris