I am under research of how to improve the performance of the below situation.
1.Transferring 1 million 50KB files from client to server.
2.I use a thread to list the files,one thread to read and send to server.
3.I send as 1440 bytes to server.
1. I store the 1440 bytes in a buffer and once it reached 100MB i will do a write operation to the hard disk as a binary file. (one thread stores in memory and other thread does I/O operation). Simultaneously one thread will update the mysql database with file details.
This takes about 3 hours for me to complete the backup process.
I just need to optimize it to the core.. Geeks, pls suggest me some ideas to improve the performance.. I will be sooo thankful.
You should always try to improve the slowest part. Storing the data on the server isn't your bottleneck, it'll be the data transfer. So focus on removing transfer delays, and you'll find the largest improvements there.