[SOLVED] Passing MySQL fetched rows to thread pool in C

Issue

This Content is from Stack Overflow. Question asked by Googlebot

I want to process data fetched from the MySQL database simultaneously. I pass the data to each thread process:

#include <mysql.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <pthread.h>
#include "thpool.h" // https://github.com/Pithikos/C-Thread-Pool

#define THREADS 10

struct fparam
{
  int id;
  char *data;
};

void process(void *arg)
{
  struct fparam *args = arg;
  // Processing ID and Data here
  printf("%d - %sn", args->id, args->data);
}

int main(int argc, char **argv)
{
  threadpool thpool = thpool_init(THREADS);

  // MySQL connection

  MYSQL_RES *result = mysql_store_result(con);

  int num_fields = mysql_num_fields(result);
  struct fparam items[100]; // 100 is for the representation

  MYSQL_ROW row;
  int i = 0;
  while ((row = mysql_fetch_row(result)))
  {
    items[i].id = atoi(row[0]);
    items[i].data = row[1];
    thpool_add_work(thpool, process, (void *)(&items[i]));
    i++;
  }

  mysql_free_result(result);
  mysql_close(con);

  thpool_wait(thpool);
  thpool_destroy(thpool);

  exit(0);
}

When there are many rows, items gets too big to fit in the memory (not just heap).

How can I limit the number of rows stored in the memory and delete them when they have been processed?

I think a key issue that we do not know if process function is faster or fetching the rows from the database.



Solution

Use a queue, a list where you add items at one end and take them out of the other.

You can write your own; a linked list can be used as a queue adding items to one end and removing them from the other. Or use an existing implementation such as the one provided by GLib.


This Question was asked in StackOverflow by Googlebot and Answered by Schwern It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.

people found this article helpful. What about you?