Hello Artisan,
In my previous tutorial, I was showing that how to upload million records data in Laravel using array chunks. Hello devs, In this example, I will extend the previous tutorial like I will use Laravel queue job batching to upload million records without server braking response.
So in this example, I will upload the same 50k CSV data which is very large I think, and show you how we can upload it in the database using Laravel queue job batching with array chunk. So from this example, you will learn how we use queue and job batching to upload big data in the database in the Laravel application.
If we use this queue job batching background to upload large CSV files then of course it will boost our uploading performance must. So in this Laravel large CSV file upload suing job batching is going to be a very special tutorial for you.
I will share step by step with you so that you can understand it better. I already created a tutorial called Laravel large file upload using array chunk. So before starting this example, please read this content.
Recommended: Uploading Million Records in Laravel using Array Chunk Example
Before starting this Laravel job batching tutorial, first, download this 50k CSV file data
from this link.
Preview:
Step 1: Download Laravel
I am going to start this Laravel job batching from scratch so that you can understand it better. I will show you step by step guide on this Laravel chunk upload example with queue job batching. So download a fresh Laravel project by the following command.
laravel new upload-csv
Step 2: Create Migration
In this step, we need a table to store our large data in the database. So run the below command to create a sale model.
php artisan make:model Sale -m
And update the Sale model like below:
app\Models\Sale.php
namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
class Sale extends Model
{
use HasFactory;
protected $guarded = [];
}
And now update our sales
table like below:
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
class CreateSalesTable extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
Schema::create('sales', function (Blueprint $table) {
$table->id();
$table->string('Region');
$table->string('Country');
$table->string('Item Type');
$table->string('Sales Channel');
$table->string('Order Priority');
$table->string('Order Date');
$table->string('Order ID');
$table->string('Ship Date');
$table->string('Units Sold');
$table->string('Unit Price');
$table->string('Unit Cost');
$table->string('Total Revenue');
$table->string('Total Cost');
$table->string('Total Profit');
$table->timestamps();
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
Schema::dropIfExists('sales');
}
}
And now run php artisan migrate
command to migrate our database.
Step 3: Create Route
Now in this step, we have to create two routes cause we need a file upload form and then we have to upload it into the server.
routes/web.php
use Illuminate\Support\Facades\Route;
use App\Http\Controllers\SaleController;
/*
|--------------------------------------------------------------------------
| Web Routes
|--------------------------------------------------------------------------
|
| Here is where you can register web routes for your application. These
| routes are loaded by the RouteServiceProvider within a group which
| contains the "web" middleware group. Now create something great!
|
*/
Route::get('/', [SaleController::class,'index'])->name('upload');
Route::post('/', [SaleController::class,'upload_csv_records']);
Step 4: Create Controller
In this step, we have to create our controller and has to be implemented our two methods. So update sale controller like below:
App\Http\Controllers\SaleController.php
namespace App\Http\Controllers;
use App\Models\Sale;
use App\Jobs\SaleCsvProcess;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Bus;
class SaleController extends Controller
{
public function index()
{
return view('welcome');
}
public function upload_csv_records(Request $request)
{
if( $request->has('csv') ) {
$csv = file($request->csv);
$chunks = array_chunk($csv,1000);
$header = [];
$batch = Bus::batch([])->dispatch();
foreach ($chunks as $key => $chunk) {
$data = array_map('str_getcsv', $chunk);
if($key == 0){
$header = $data[0];
unset($data[0]);
}
$batch->add(new SaleCsvProcess($data, $header));
}
return $batch;
}
return "please upload csv file";
}
}
Step 5: Create View
In this step, we need to create our blade view to show our file upload form. So create it to complete the Laravel chunk CSV file upload example with queue job batching.
resources/views/welcome.blade.php
Step 6: Create Job
Now in this step, we have to create our job to process our large CSV file. To create it by the following command.
php artisan make:job SaleCsvProcess
And update it like:
App\Jobs\SaleCsvProcess.php
namespace App\Jobs;
use App\Models\Sale;
use Illuminate\Bus\Queueable;
use Illuminate\Bus\Batchable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Contracts\Queue\ShouldBeUnique;
class SaleCsvProcess implements ShouldQueue
{
use Batchable, Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $header;
public $data;
public function __construct($data, $header)
{
$this->data = $data;
$this->header = $header;
}
public function handle()
{
foreach ($this->data as $sale) {
$sellData = array_combine($this->header,$sale);
Sale::create($sellData);
}
}
}
Now we need to set up our job before using it. You know that Laravel's job batching feature allows you to easily execute a batch of jobs and then perform some action when the batch of jobs has completed executing. We need queue:batches-table
to use it.
php artisan queue:batches-table
And then run php artisan migrate
to get job_batches table.
Step 7: Set up Queue
You know that we are going to use the Laravel queue with job batching for uploading large data in Laravel. So do step by step to set up laravel queue.
php artisan queue:table
//and the
php artisan migrate
Recommended: Laravel 8.x Queues Example with Redis and Horizon
Don't forget to instruct your application to use the database
driver by updating the QUEUE_CONNECTION
variable in your application's .env
file:
.env
QUEUE_CONNECTION=database
All are set to go. To start our queue worker after uploading a file and for processing every job, we need to run the below command. So run this command php artisan queue:work
:
php artisan queue:work
Now if you upload a CSV file and then you will see the below output in your queue terminal:
Now all are set to go. You can test now. Hope it can help to upload large CSV file data in the Laravel application.
#laravel #laravel-8x #file-upload #laravel-chunk #laravel-job-batching #laravel-queue