freemen replied on Mon, Jun 15 2009 12:14 PM

rated by 0 users

Good question here. Does anyone at SW have an answer?

kweise replied on Mon, Jun 15 2009 1:54 PM

rated by 0 users

I'd like to know the answer too.  I'm deleting expired flow data once a day as well using the default 1000 as the number of expired IPs to remove at a time.  I currently have 5,905,348 expired IPs. 

Thanks!

freemen replied on Mon, Jun 15 2009 2:13 PM

rated by 0 users

One wonders if the solution will be to delete all the left over expired IPs with a SQL query via the Database Manager.

ET replied on Tue, Jun 16 2009 3:37 AM

rated by 0 users

But you can always set up in [Admin]->[NTA Settings]->[Database Maintenance]->"Process all expired IPs", right?

Internaly this operation is performed by portions, so if db maintenance reach time threshold reserved for clean-up (15 minutes by default), it ends anyway.

Or is there some other problem?

Thanks

kweise replied on Tue, Jun 16 2009 12:57 PM

rated by 0 users

I can't speak for the others, but the problem for me has to do with there not being a lot of information in the admin guide about what each setting does and not really wanting to make a bunch of changes (especially in the database area) without having a good understanding of what I'm doing.  I would conceed that each option is fairly self explanitory, but I don't feel like I understand the impact to the database's performance if I were to change something for the worse.  For example, under Maximum minutes to process expired IPs, never stop processing expired IPs is an option, but it also says it isn't recommended.  That being the case, I'm probably not going to select that option, but it also makes me wonder how long I should let it run before stopping it.  The default being 15 minutes makes me think it must be a somewhat intensive task and is something I probably don't want to run for too long.  Where I'm stuck is how long is too long.  I have over 6 million expired IPs, so obviously the defaults aren't doing it for me.  I can't find anything in the documentation that gives me a good feel for what changing the selection to process all expired IPs for say 120 minutes would do performance wise to my Orion environment.

Like I said, I can't speak for anyone else and honestly, I haven't spent much time trying to research this (other than looking it up in the admin guide.)

Thanks,
Kevin

ET replied on Wed, Jun 17 2009 2:55 AM

rated by 0 users

I understand your concerning here. It's not very clear that these 2 choices are tightly connected. So the worst and not recommended option here is:

• Process all expired IPs
• Never stop processing expired IPs (Not recommented)

Because this could block your SQL for a long time (unpredictible time).

And yeah, this clean-up could be time consumptive, and it depends on your IP's range, number of expired IP's and how powerfull your SQL is. 

I agree here, that the default boundaries are too low for your case, especially this 1000 IPs per day. It sounds more better for me to set up [Process all expired IPs] and limit clean-up operation for e.g. 5 minutes. If you have powerfull SQL it will cover all expired IP's and your SQL will not be stressed for a long time in the worst case.

kweise replied on Wed, Jun 17 2009 6:34 AM

rated by 0 users

Thanks ET.  I've changed my maintenance settings to process all expired IPs and set the maximum minutes to process IPs to 5 minutes.  I'll see how that runs overnight and make any adjustments necessary.  Thanks again for your help!

MTorok replied on Wed, Jun 17 2009 9:35 AM

rated by 0 users

kweise,

I reached out to development regarding your request. The following recommendation was suggested: select Process all expired IPs and specify a Custom number of minutes of 60. The resource consumption should be relatively low, as performance enhancements in version 3.1 SP2 introduced a separate NetflowEndpoints table.

You might want to try that, if ET's settings don't fix you up.

HTH,

Michael

kweise replied on Wed, Jun 17 2009 1:59 PM

rated by 0 users

Michael,

We have a pretty beefy SQL box (at least that's what the DBA's tell me, I don't have access to it.)  I set the custom number of minutes to 60 and set the maintenance to run at 4am.  I'll let that run tonight and see where I stand in the morning.  My only real concern is we aren't the only database on the server, so I didn't want to impact the performance of the other databases sharing that box.  There shouldn't be much going on between 4 and 5am so I'm comfortable trying it for 60 minutes at a time.

Thanks for your help!

Kevin

kweise replied on Thu, Jun 18 2009 6:29 AM

rated by 0 users

Well, I managed to somehow fat finger the custom number of minutes when I tried to change it from 5 to 60 minutes.  So, my job kicked off at 4am this morning but only ran for 5 minutes.  However, it did manage to drop me down to only having about 3.4 million expired IPs.  So, I bumped up the custom number of minutes back to the default of 15 and based on the numbers, I should have everything cleaned up when the job runs again tomorrow morning. 

Thanks for all the help!

kweise replied on Fri, Jun 19 2009 7:46 AM

rated by 0 users

The last of my 3.4 million expired IPs were cleared out this morning in just over 14 minutes.  I'm currently showing 185 expired IPs.  Thanks Michael and ET for all your help!

robertcbrowning replied on Wed, Dec 2 2009 5:52 AM

rated by 0 users

I had millions of expired IPs too.  Now set to All IPs, 60 mins and I'm down to 4000 now.  Thanks guys.