Uploading Million Records In Laravel Using Array Chunk Example

Hello Artisan,

Hope you are doing well. In this Laravel brand new tutorial, I am going to show you how to manage or handle large data in Laravel before uploading it to the server. You know that uploading large data from a large CSV file is sometimes needed. How do you handle this situation?

In this example, I will share with you from scratch that how to handle large data in Laravel using array chunks. You know that the chunk method breaks the large group of data set into a smaller group of data set (chunks). That means if I upload 50k records then before uploading we will break it into a smaller part of data and then upload it with this smaller part one after another.

So if you don't know Laravel chunk upload example then this Laravel chunk upload huge or large data in the server tutorial is going to be the perfect example. So let's start our uploading millions of records in Laravel using array chunk.

I already posted a tutorial on Laravel large CSV file upload using queue job batching. You can read this tutorial, It is a more performance booster for large file uploading.

Recommended: Upload Large CSV File using Queue Job Batching in Laravel

Now download this 50k CSV data to test this Laravel large CSV file upload example using array chunk 50k csv file

Step 1: Download Laravel

I am going to start from scratch so that you can understand better. I will show you step by step guide on this Laravel chunk upload example. So download a fresh Laravel project by the following command.

laravel new upload-csv

Step 2: Create Migration

In this step, we need a table to store our huge data in the database. So run the below command to create a sale model.

php artisan make:model Sale -m

And update the Sale model like below:

app\Models\Sale.php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;

class Sale extends Model
{
    use HasFactory;

    protected $guarded = [];
}

And now update our sales table like below:

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class CreateSalesTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('sales', function (Blueprint $table) {
            $table->id();
            $table->string('Region');
            $table->string('Country');
            $table->string('Item Type');
            $table->string('Sales Channel');
            $table->string('Order Priority');
            $table->string('Order Date');
            $table->string('Order ID');
            $table->string('Ship Date');
            $table->string('Units Sold');
            $table->string('Unit Price');
            $table->string('Unit Cost');
            $table->string('Total Revenue');
            $table->string('Total Cost');
            $table->string('Total Profit');
            $table->timestamps();
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::dropIfExists('sales');
    }
}

And now run php artisan migrate command to migrate our database.

Step 3: Create Route

Now in this step, we have to create two routes cause we need a file upload form and then we have to upload it into the server.

routes/web.php

use Illuminate\Support\Facades\Route;
use App\Http\Controllers\SaleController;

/*
|--------------------------------------------------------------------------
| Web Routes
|--------------------------------------------------------------------------
|
| Here is where you can register web routes for your application. These
| routes are loaded by the RouteServiceProvider within a group which
| contains the "web" middleware group. Now create something great!
|
*/

Route::get('/', [SaleController::class,'index'])->name('upload');
Route::post('/', [SaleController::class,'upload_csv_records']);

Step 4: Create Controller

In this step, we have to create our controller and has to be implemented our two methods. So update sale controller like below:

App\Http\Controllers\SaleController.php

namespace App\Http\Controllers;

use App\Models\Sale;
use Illuminate\Http\Request;

class SaleController extends Controller
{
    public function index()
    {
        return view('welcome');
    }

    public function upload_csv_records(Request $request)
    {
        if($request->has('csv'))
        {
            $csv = file($request->csv);

            //chunking file
            $chunks = array_chunk($csv,1000);
            $path = resource_path('temp');

            //convert chunk to new csv file
            foreach ($chunks as $key => $chunk) {
                $name = "/tmp{$key}.csv";
                file_put_contents($path.$name,$chunk);
            }

            //getting all the file inside the directories
            $files = glob("$path/*.csv");

            $header = [];
            
            foreach ($files as $key => $file) {

                $data = array_map('str_getcsv', file($file));

                if($key == 0){
                    $header = $data[0];
                    unset($data[0]);
                }
                
                foreach ($data as $sale) {
                    $sellData = array_combine($header,$sale);
                    Sale::create($sellData);
                }

                unlink($file);
            }

            return "stored";

        }
        return "please upload csv file";

    }
}

Step 5: Create View

In this step, we need to create our blade view to show our file upload form. So create it to complete Laravel chunk example.

resources/views/welcome.blade.php

All are set to go. Just do one more thing before testing. We need to create a temp directory inside our resources folder.

See the preview after uploading a file using Laravel array chunk.

Now run the server and test it. Hope it can help you.