Upload Large CSV File using Queue Job Batching in Laravel

Hello Artisan,

In my previous tutorial, I was showing that how to upload million records data in Laravel using array chunks. Hello devs, In this example, I will extend the previous tutorial like I will use Laravel queue job batching to upload million records without server braking response.

So in this example, I will upload the same 50k CSV data which is very large I think, and show you how we can upload it in the database using Laravel queue job batching with array chunk. So from this example, you will learn how we use queue and job batching to upload big data in the database in the Laravel application.

If we use this queue job batching background to upload large CSV files then of course it will boost our uploading performance must. So in this Laravel large CSV file upload suing job batching is going to be a very special tutorial for you.

I will share step by step with you so that you can understand it better. I already created a tutorial called Laravel large file upload using array chunk. So before starting this example, please read this content. 

Recommended: Uploading Million Records in Laravel using Array Chunk Example

Before starting this Laravel job batching tutorial, first, download this 50k CSV file data from this link.

Preview:

Step 1: Download Laravel

I am going to start this Laravel job batching from scratch so that you can understand it better. I will show you step by step guide on this Laravel chunk upload example with queue job batching. So download a fresh Laravel project by the following command.

laravel new upload-csv

Step 2: Create Migration

In this step, we need a table to store our large data in the database. So run the below command to create a sale model.

php artisan make:model Sale -m

And update the Sale model like below:

app\Models\Sale.php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;

class Sale extends Model
{
    use HasFactory;

    protected $guarded = [];
}

And now update our sales table like below:

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class CreateSalesTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('sales', function (Blueprint $table) {
            $table->id();
            $table->string('Region');
            $table->string('Country');
            $table->string('Item Type');
            $table->string('Sales Channel');
            $table->string('Order Priority');
            $table->string('Order Date');
            $table->string('Order ID');
            $table->string('Ship Date');
            $table->string('Units Sold');
            $table->string('Unit Price');
            $table->string('Unit Cost');
            $table->string('Total Revenue');
            $table->string('Total Cost');
            $table->string('Total Profit');
            $table->timestamps();
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::dropIfExists('sales');
    }
}

And now run php artisan migrate command to migrate our database.

Step 3: Create Route

Now in this step, we have to create two routes cause we need a file upload form and then we have to upload it into the server.

routes/web.php

use Illuminate\Support\Facades\Route;
use App\Http\Controllers\SaleController;

/*
|--------------------------------------------------------------------------
| Web Routes
|--------------------------------------------------------------------------
|
| Here is where you can register web routes for your application. These
| routes are loaded by the RouteServiceProvider within a group which
| contains the "web" middleware group. Now create something great!
|
*/

Route::get('/', [SaleController::class,'index'])->name('upload');
Route::post('/', [SaleController::class,'upload_csv_records']);

Step 4: Create Controller

In this step, we have to create our controller and has to be implemented our two methods. So update sale controller like below:

App\Http\Controllers\SaleController.php

namespace App\Http\Controllers;

use App\Models\Sale;
use App\Jobs\SaleCsvProcess;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Bus;

class SaleController extends Controller
{
    public function index()
    {
        return view('welcome');
    }

    public function upload_csv_records(Request $request)
    {
        if( $request->has('csv') ) {

            $csv    = file($request->csv);
            $chunks = array_chunk($csv,1000);
            $header = [];
            $batch  = Bus::batch([])->dispatch();

            foreach ($chunks as $key => $chunk) {
            $data = array_map('str_getcsv', $chunk);
                if($key == 0){
                    $header = $data[0];
                    unset($data[0]);
                }
                $batch->add(new SaleCsvProcess($data, $header));
            }
            return $batch;
        }
        return "please upload csv file";
    }
}

Step 5: Create View

In this step, we need to create our blade view to show our file upload form. So create it to complete the Laravel chunk CSV file upload example with queue job batching.

resources/views/welcome.blade.php

Step 6: Create Job

Now in this step, we have to create our job to process our large CSV file. To create it by the following command.

php artisan make:job SaleCsvProcess

And update it like:

App\Jobs\SaleCsvProcess.php

namespace App\Jobs;

use App\Models\Sale;
use Illuminate\Bus\Queueable;
use Illuminate\Bus\Batchable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Contracts\Queue\ShouldBeUnique;

class SaleCsvProcess implements ShouldQueue
{
    
    use Batchable, Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
    
    public $header;
    public $data;

    public function __construct($data, $header)
    {
        $this->data = $data;
        $this->header = $header;
    }

    public function handle()
    {
        foreach ($this->data as $sale) {
            $sellData = array_combine($this->header,$sale);
            Sale::create($sellData);
        }
    }
}

Now we need to set up our job before using it. You know that Laravel's job batching feature allows you to easily execute a batch of jobs and then perform some action when the batch of jobs has completed executing. We need queue:batches-table to use it.

php artisan queue:batches-table

And then run php artisan migrate to get job_batches table. 

Step 7: Set up Queue

You know that we are going to use the Laravel queue with job batching for uploading large data in Laravel. So do step by step to set up laravel queue.

php artisan queue:table

//and the

php artisan migrate

Recommended: Laravel 8.x Queues Example with Redis and Horizon

Don't forget to instruct your application to use the database driver by updating the QUEUE_CONNECTION variable in your application's .env file:

.env

QUEUE_CONNECTION=database

All are set to go. To start our queue worker after uploading a file and for processing every job, we need to run the below command. So run this command php artisan queue:work:

php artisan queue:work

Now if you upload a CSV file and then you will see the below output in your queue terminal:

Now all are set to go. You can test now. Hope it can help to upload large CSV file data in the Laravel application.