Running out of IO

I started today with just over 1.3M links found this morning. I am going to jack the robots up leave them for a while.

I am quickly running out of IO on my system. I need a shit load of RAM because the database tables and indexes are getting very big or at least in terms of my system they are getting really big.

So far I have found 12.4 Million links and I have confirmed 1.3 Million of them. The indexes alone are approaching 2Gb in size so I also need more disks. The 18Gb Fujitsu SCSI u160 68 pin MAN3184MP is getting tight on usable space. I also need to start thinking about moving the logs to a disk on their own and to start splitting up the tables onto seperate disks to avoid having the SCSI do all the work.

12.4M links_found
1.3M home_page

By Harry on September 25, 2003 11:52 PM | Permalink | Comments (0)