We have moved our forum to GitHub Discussions. For questions about Phalcon v3/v4/v5 you can visit here and for Phalcon v6 here.
  1. Home
  2. General

What's the best approach for Exporting huge data sets to CSV?

I have created a functionality for exporting datatable records to CSV, which works smoothly. However for large data sets(let say 1M records) it takes 2-3 minutes.

However I don't want the user to wait 3 or max minutes for csv generation. So What can I do to run the CSV generation in background and once it's completed I show the user the download link or download starts automitacally ?How can we achive this in phalcon?Any source or turoirialrecommendation will be better.

/**
 * Generate CSV file
 */
public static function generateCSVFile($csvFilePath, $fieldHeading ,$data) {
    $baseDir = getcwd();
    $csvFile = fopen($baseDir.$csvFilePath, 'w');
    fputcsv($csvFile, $fieldHeading);
    foreach ($data as $key => $fields) {
        fputcsv($csvFile, $fields);
    }
    fclose($csvFile);
}


10.4k
edited Nov '19

To save file faster try to increase buffer for writing above size of the target file:

https://www.php.net/manual/en/function.stream-set-write-buffer.php

as for download by user directly, set appropriate headers 1st, then create file pointer for output:

$filename = "filename";
header("Content-type: text/csv");
header("Content-Disposition: attachment; filename={$filename}.csv");
header("Pragma: no-cache");
header("Expires: 0");
$fp = fopen('php://output', 'w');


125.7k

@tztztztz 's method is the easiest and most straightforward (and if it answers your question, please click the "Accept Answer" button).

Depending on your situation, another solution may be to generate the file asynchronously. I've never used it, but Beanstalkd is a work queue you can add jobs to, which run separately from the main page request thread. You'd then need to use AJAX or Websockets to update the user's page once the file is generated and available for download. @tztztztz's method doesn't generate a file on the server - so less to clean up if you can go that way.