libextractor  1.11
About: GNU libextractor is a library used to extract meta-data from files of arbitrary type.
  Fossies Dox: libextractor-1.11.tar.gz  ("unofficial" and yet experimental doxygen-generated source code documentation)  

extractor_ipc.h File Reference

IPC with plugin (OS-independent API) More...

Include dependency graph for extractor_ipc.h:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures

struct  InitMessage
 
struct  StartMessage
 
struct  UpdateMessage
 
struct  SeekRequestMessage
 
struct  MetaMessage
 

Macros

#define MAX_META_DATA   32 * 1024 * 1024
 
#define MAX_SHM_NAME   255
 
#define MESSAGE_INIT_STATE   0x00
 
#define MESSAGE_EXTRACT_START   0x01
 
#define MESSAGE_UPDATED_SHM   0x02
 
#define MESSAGE_DONE   0x03
 
#define MESSAGE_SEEK   0x04
 
#define MESSAGE_META   0x05
 
#define MESSAGE_DISCARD_STATE   0x06
 
#define MESSAGE_CONTINUE_EXTRACTING   0x07
 

Typedefs

typedef void(* EXTRACTOR_ChannelMessageProcessor) (void *cls, struct EXTRACTOR_PluginList *plugin, enum EXTRACTOR_MetaType meta_type, enum EXTRACTOR_MetaFormat meta_format, const char *mime, const void *value, size_t value_len)
 

Functions

struct EXTRACTOR_SharedMemoryEXTRACTOR_IPC_shared_memory_create_ (size_t size)
 
void EXTRACTOR_IPC_shared_memory_destroy_ (struct EXTRACTOR_SharedMemory *shm)
 
unsigned int EXTRACTOR_IPC_shared_memory_change_rc_ (struct EXTRACTOR_SharedMemory *shm, int delta)
 
ssize_t EXTRACTOR_IPC_shared_memory_set_ (struct EXTRACTOR_SharedMemory *shm, struct EXTRACTOR_Datasource *ds, uint64_t off, size_t size)
 
uint64_t EXTRACTOR_datasource_get_pos_ (struct EXTRACTOR_Datasource *ds)
 
struct EXTRACTOR_ChannelEXTRACTOR_IPC_channel_create_ (struct EXTRACTOR_PluginList *plugin, struct EXTRACTOR_SharedMemory *shm)
 
void EXTRACTOR_IPC_channel_destroy_ (struct EXTRACTOR_Channel *channel)
 
ssize_t EXTRACTOR_IPC_channel_send_ (struct EXTRACTOR_Channel *channel, const void *data, size_t size)
 
ssize_t EXTRACTOR_IPC_process_reply_ (struct EXTRACTOR_PluginList *plugin, const void *data, size_t size, EXTRACTOR_ChannelMessageProcessor proc, void *proc_cls)
 
int EXTRACTOR_IPC_channel_recv_ (struct EXTRACTOR_Channel **channels, unsigned int num_channels, EXTRACTOR_ChannelMessageProcessor proc, void *proc_cls)
 

Detailed Description

IPC with plugin (OS-independent API)

Author
Christian Grothoff

@detail The IPC communication between plugins and the main library works as follows. Each message begins with a 1-character opcode which specifies the message type. The main library starts the plugins by forking the helper process and establishes two pipes for communication in both directions. First, the main library send an 'INIT_STATE' message to the plugin. The start message specifies the name (and size) of a shared memory segment which will contain parts of the (uncompressed) data of the file that is being processed. The same shared memory segment is used throughout the lifetime of the plugin.

Then, the following messages are exchanged for each file. First, an EXTRACT_START message is sent with the specific size of the file (or -1 if unknown) and the number of bytes ready in the shared memory segment. The plugin then answers with either: 1) MESSAGE_DONE to indicate that no further processing is required for this file; the IPC continues with the EXTRACT_START message for the next file afterwards; 2) MESSAGE_SEEK to indicate that the plugin would like to read data at a different offset; the main library can then either a) respond with a MESSAGE_DISCARD_STATE to tell the plugin to abort processing (the next message will then be another EXTRACT_START) b) respond with a MESSAGE_UPDATED_SHM which notifies the plugin that the shared memory segment was moved to a different location in the overall file; the target of the seek should now be within the new range (but does NOT have to be at the beginning of the seek) 3) MESSAGE_META to provide extracted meta data to the main library. The main library can then either: a) respond with a MESSAGE_DISCARD_STATE to tell the plugin to abort processing (the next message will then be another EXTRACT_START) b) respond with a MESSAGE_CONTINUE_EXTRACTING to tell the plugin to continue extracting meta data; in this case, the plugin is then expected to produce another MESSAGE_DONE, MESSAGE_SEEK or MESSAGE_META round of messages.

Definition in file extractor_ipc.h.

Macro Definition Documentation

◆ MAX_META_DATA

#define MAX_META_DATA   32 * 1024 * 1024

How long do we allow an individual meta data object to be? Used to guard against (broken) plugns causing us to use excessive amounts of memory.

Definition at line 77 of file extractor_ipc.h.

◆ MAX_SHM_NAME

#define MAX_SHM_NAME   255

Maximum length of a shared memory object name

Definition at line 82 of file extractor_ipc.h.

◆ MESSAGE_CONTINUE_EXTRACTING

#define MESSAGE_CONTINUE_EXTRACTING   0x07

Sent from LE to plugin to make plugin continue extraction. (sent in response to META data).

Definition at line 314 of file extractor_ipc.h.

◆ MESSAGE_DISCARD_STATE

#define MESSAGE_DISCARD_STATE   0x06

Sent from LE to plugin to make plugin discard its state (extraction aborted by application). Only one byte. Plugin should get ready for next 'StartMessage' after this. (sent in response to META data or SEEK requests).

Definition at line 308 of file extractor_ipc.h.

◆ MESSAGE_DONE

#define MESSAGE_DONE   0x03

Sent from plugin to LE to tell LE that plugin is done analyzing current file and will send no more data. No message format as this is only one byte.

Definition at line 210 of file extractor_ipc.h.

◆ MESSAGE_EXTRACT_START

#define MESSAGE_EXTRACT_START   0x01

Sent from LE to a plugin to tell it extracting can now start. The SHM will point to offset 0 of the file.

Definition at line 128 of file extractor_ipc.h.

◆ MESSAGE_INIT_STATE

#define MESSAGE_INIT_STATE   0x00

Sent from LE to a plugin to initialize it (opens shm).

Definition at line 87 of file extractor_ipc.h.

◆ MESSAGE_META

#define MESSAGE_META   0x05

Sent from plugin to LE to tell LE about metadata discovered.

Definition at line 258 of file extractor_ipc.h.

◆ MESSAGE_SEEK

#define MESSAGE_SEEK   0x04

Sent from plugin to LE to tell LE that plugin needs to read a different part of the source file.

Definition at line 216 of file extractor_ipc.h.

◆ MESSAGE_UPDATED_SHM

#define MESSAGE_UPDATED_SHM   0x02

Sent from LE to a plugin to tell it that shm contents were updated.

Definition at line 166 of file extractor_ipc.h.

Typedef Documentation

◆ EXTRACTOR_ChannelMessageProcessor

typedef void(* EXTRACTOR_ChannelMessageProcessor) (void *cls, struct EXTRACTOR_PluginList *plugin, enum EXTRACTOR_MetaType meta_type, enum EXTRACTOR_MetaFormat meta_format, const char *mime, const void *value, size_t value_len)

Handler for a message from one of the plugins.

Parameters
clsclosure
pluginplugin of the channel sending the message
meta_typetype of the meta data
meta_formatformat of the meta data
mimemime string send from the plugin
value'data' send from the plugin
value_lennumber of bytes in 'value'

Definition at line 436 of file extractor_ipc.h.

Function Documentation

◆ EXTRACTOR_datasource_get_pos_()

uint64_t EXTRACTOR_datasource_get_pos_ ( struct EXTRACTOR_Datasource ds)

Query datasource for current position

Parameters
dsdata source to query
Returns
current position in the datasource or UINT_MAX on error

Definition at line 259 of file extractor_ipc_gnu.c.

References EXTRACTOR_datasource_seek_(), and EXTRACTOR_SharedMemory::pos.

◆ EXTRACTOR_IPC_channel_create_()

◆ EXTRACTOR_IPC_channel_destroy_()

◆ EXTRACTOR_IPC_channel_recv_()

int EXTRACTOR_IPC_channel_recv_ ( struct EXTRACTOR_Channel **  channels,
unsigned int  num_channels,
EXTRACTOR_ChannelMessageProcessor  proc,
void *  proc_cls 
)

Receive data from any of the given IPC channels (blocking). Wait for one of the plugins to reply.

Parameters
channelsarray of channels, channels that break may be set to NULL
num_channelslength of the 'channels' array
procfunction to call to process messages (may be called more than once)
proc_clsclosure for 'proc'
Returns
-1 on error (i.e. no response in 10s), 1 on success

Receive data from any of the given IPC channels (blocking). Wait for one of the plugins to reply. Selects on plugin output pipes, runs 'receive_reply' on each activated pipe until it gets a seek request or a done message. Called repeatedly by the user until all pipes are dry or broken.

Parameters
channelsarray of channels, channels that break may be set to NULL
num_channelslength of the channels array
procfunction to call to process messages (may be called more than once)
proc_clsclosure for proc
Returns
-1 on error, 1 on success

Receive data from any of the given IPC channels (blocking). Wait for one of the plugins to reply. Selects on plugin output pipes, runs 'receive_reply' on each activated pipe until it gets a seek request or a done message. Called repeatedly by the user until all pipes are dry or broken.

Parameters
channelsarray of channels, channels that break may be set to NULL
num_channelslength of the 'channels' array
procfunction to call to process messages (may be called more than once)
proc_clsclosure for 'proc'
Returns
-1 on error, 1 on success

Definition at line 467 of file extractor_ipc_gnu.c.

References EXTRACTOR_PluginList::channel, EXTRACTOR_Channel::cpipe_out, EXTRACTOR_IPC_channel_destroy_(), EXTRACTOR_IPC_process_reply_(), EXTRACTOR_PluginList::libname, LOG, LOG_STRERROR, MAX_META_DATA, EXTRACTOR_Channel::mdata, EXTRACTOR_Channel::mdata_size, NULL, EXTRACTOR_Channel::ov_read, EXTRACTOR_Channel::plugin, EXTRACTOR_PluginList::round_finished, EXTRACTOR_PluginList::seek_request, and EXTRACTOR_Channel::size.

Referenced by do_extract().

◆ EXTRACTOR_IPC_channel_send_()

ssize_t EXTRACTOR_IPC_channel_send_ ( struct EXTRACTOR_Channel channel,
const void *  data,
size_t  size 
)

Send data via the given IPC channel (blocking).

Parameters
channelchannel to communicate with the plugin
bufdata to send
sizenumber of bytes in buf to send
Returns
-1 on error, number of bytes sent on success (never does partial writes)

Definition at line 428 of file extractor_ipc_gnu.c.

References EXTRACTOR_Channel::cpipe_in, LOG_STRERROR, NULL, EXTRACTOR_Channel::old_buf, and EXTRACTOR_Channel::ov_write.

Referenced by do_extract(), EXTRACTOR_IPC_channel_create_(), process_plugin_reply(), send_discard_message(), send_update_message(), and write_plugin_data().

◆ EXTRACTOR_IPC_process_reply_()

ssize_t EXTRACTOR_IPC_process_reply_ ( struct EXTRACTOR_PluginList plugin,
const void *  data,
size_t  size,
EXTRACTOR_ChannelMessageProcessor  proc,
void *  proc_cls 
)

Process a reply from channel (seek request, metadata and done message)

Parameters
pluginplugin this communication is about
bufbuffer with data from IPC channel
sizenumber of bytes in buffer
procmetadata callback
proc_clscallback cls
Returns
number of bytes processed, -1 on error

Definition at line 42 of file extractor_ipc.c.

References EXTRACTOR_metatype_get_max(), EXTRACTOR_METATYPE_UNKNOWN, SeekRequestMessage::file_offset, LOG, MAX_META_DATA, MESSAGE_DONE, MESSAGE_META, MESSAGE_SEEK, MetaMessage::meta_format, MetaMessage::meta_type, MetaMessage::mime_length, NULL, EXTRACTOR_PluginList::round_finished, EXTRACTOR_PluginList::seek_request, EXTRACTOR_PluginList::seek_whence, MetaMessage::value_size, and SeekRequestMessage::whence.

Referenced by EXTRACTOR_IPC_channel_recv_().

◆ EXTRACTOR_IPC_shared_memory_change_rc_()

unsigned int EXTRACTOR_IPC_shared_memory_change_rc_ ( struct EXTRACTOR_SharedMemory shm,
int  delta 
)

Change the reference counter for this shm instance.

Parameters
shminstance to update
deltavalue to change RC by
Returns
new RC

Definition at line 196 of file extractor_ipc_gnu.c.

References EXTRACTOR_SharedMemory::rc.

Referenced by EXTRACTOR_extract(), and EXTRACTOR_plugin_remove().

◆ EXTRACTOR_IPC_shared_memory_create_()

struct EXTRACTOR_SharedMemory* EXTRACTOR_IPC_shared_memory_create_ ( size_t  size)

◆ EXTRACTOR_IPC_shared_memory_destroy_()

void EXTRACTOR_IPC_shared_memory_destroy_ ( struct EXTRACTOR_SharedMemory shm)

Destroy shared memory area.

Parameters
shmmemory area to destroy
Returns
NULL on error

Definition at line 211 of file extractor_ipc_gnu.c.

References EXTRACTOR_SharedMemory::map, NULL, EXTRACTOR_SharedMemory::ptr, EXTRACTOR_SharedMemory::shm_id, EXTRACTOR_SharedMemory::shm_name, EXTRACTOR_SharedMemory::shm_ptr, and EXTRACTOR_SharedMemory::shm_size.

Referenced by EXTRACTOR_plugin_remove().

◆ EXTRACTOR_IPC_shared_memory_set_()

ssize_t EXTRACTOR_IPC_shared_memory_set_ ( struct EXTRACTOR_SharedMemory shm,
struct EXTRACTOR_Datasource ds,
uint64_t  off,
size_t  size 
)

Initialize shared memory area from data source.

Parameters
shmmemory area to initialize
dsdata source to use for initialization
offoffset to use in data source
sizenumber of bytes to copy
Returns
-1 on error, otherwise number of bytes copied

Definition at line 231 of file extractor_ipc_gnu.c.

References EXTRACTOR_datasource_read_(), EXTRACTOR_datasource_seek_(), LOG, EXTRACTOR_SharedMemory::ptr, EXTRACTOR_SharedMemory::shm_ptr, and EXTRACTOR_SharedMemory::shm_size.

Referenced by do_extract().