More on big deletes:
http://mysql.rjweb.org/doc.php/deletebig
> 400GB data is already there in one drive.
> 300GB/week
> E drive is having 500GB
By the time I finish writing this response, you will be out of disk space? !!
Problems are compounding:
* With the table in ibdata*, you can spread it across 4 drives.
* With innodb_file_per_table = 1 (and not PARTITIONing), you cannot spread the table across drives.
* With RAID striping, the above issues vanish, but changing to Striping means reinstalling the OS. Well, maybe you could RAID 3 drives, leaving the OS, etc, on the current drive. (This may be part of the "quick" solution.)
* PARTITIONs are effectively a bunch of separate tables acting as a single table.
* Table maintenance (should you ever need it) _usually_ needs enough space to copy over the table being ALTERed. So it is generally unwise to fill up the disk as much as you are threatening to do. Related to that...
* If the 'purge' code fails to run, you quickly out of disk space, with no way to recover. That is, you need to make really sure it works before going into production.
Let's see the SHOW CREATE TABLE. When building terabyte-sized tables, tweaking the the datatypes becomes very important -- Saving one byte in one column of a billion rows saves 1GB. BIGINT, when it is not really needed wastes 4 bytes per row compared to INT. Etc.
> innodb_buffer_pool_size = 1G
How much RAM? Usually this should be set to 70% of RAM.
You have not mentioned trouble keeping up with 300GB/week = 500MB/sec. How many rows is that?
My advice (pending answers to above questions) is
* Study the schema to see if the disk footprint can be made smaller. (In one case, I showed the DBA how to shrink it by a factor of 4. Hmmmm... If that could work for you, much of this discussion vanishes!)
* PARTITION BY RANGE + Purging by DROP PARTITION (You apparently have not experienced how sloooow the equivalent DELETE would be.)
* innodb_file_per_table = 1 (to make the DROP much more efficient)
* RAID striping instead of ibdata* things (This is probably required for the file_per_table, together with other details.) Another option is to buy a 2TB drive.
The rest of the settings you showed (plus others) will need reviewing again later.
What will your SELECTs look like? If you have not tested them yet, you could be in a rude surprise. A table scan of 400GB will probably take hours; beware! Please show me some SELECTs. Between the SELECTs and the CREATE TABLE, I may have a lot more important suggestions.
I must emphasize that tables in the terabyte range need a lot of non-trivial analysis. Here is one of my blogs on such:
http://mysql.rjweb.org/doc.php/datawarehouse