Issue
This Content is from Stack Overflow. Question asked by Googlebot
I want to process data fetched from the MySQL database simultaneously. I pass the data to each thread process:
#include <mysql.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <pthread.h>
#include "thpool.h" // https://github.com/Pithikos/C-Thread-Pool
#define THREADS 10
struct fparam
{
int id;
char *data;
};
void process(void *arg)
{
struct fparam *args = arg;
// Processing ID and Data here
printf("%d - %sn", args->id, args->data);
}
int main(int argc, char **argv)
{
threadpool thpool = thpool_init(THREADS);
// MySQL connection
MYSQL_RES *result = mysql_store_result(con);
int num_fields = mysql_num_fields(result);
struct fparam items[100]; // 100 is for the representation
MYSQL_ROW row;
int i = 0;
while ((row = mysql_fetch_row(result)))
{
items[i].id = atoi(row[0]);
items[i].data = row[1];
thpool_add_work(thpool, process, (void *)(&items[i]));
i++;
}
mysql_free_result(result);
mysql_close(con);
thpool_wait(thpool);
thpool_destroy(thpool);
exit(0);
}
When there are many rows, items
gets too big to fit in the memory (not just heap).
How can I limit the number of rows stored in the memory and delete them when they have been processed?
I think a key issue that we do not know if process
function is faster or fetching the rows from the database.
Solution
Use a queue, a list where you add items at one end and take them out of the other.
You can write your own; a linked list can be used as a queue adding items to one end and removing them from the other. Or use an existing implementation such as the one provided by GLib.
This Question was asked in StackOverflow by Googlebot and Answered by Schwern It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.