mcbridea replied on Thu, Jun 11 2009 11:54 AM

rated by 0 users

Hi mhh351,

We are investigating ways to off-load some of the work we are asking SQL to do. It is a priority for us.

Andy

mhh351 replied on Thu, Jun 11 2009 1:45 PM

rated by 0 users

If I may be so bold, may I suggest a stripped down version of the NTA function be a no cost license. It would not be usable with any other software. It's only function would be to collect and send flow information.

Have this collector software set up in lesser devices, such as good but old PCs or old but sufficient servers placed in various parts of the network. This would be in a "probe" sort of style. It also gets you ready, at Solarwinds, for what I think you are visualizing!

Add the function that NTA will look for these devices based on criteria, either supplied by IP address and name or have the collector "call home" to be recognized by the NTA program "Job". Your criteria should be proprietary to the point that you select a TCP Port as well.

Another thought would be for the collectors, no matter whether they are remote or standard, to send to the central site SQL server on their own.

I still vote for a separate database and maybe even database server as a requisite set up. Stay away from NPM at all costs. This sort of traffic and the NPM traffic will not coexist well due to the constant chatter or each kind of traffic.

These collectors then dump the data at specified times (2 to 5 minutes) directly to the database. The NTA program can then get the information directly from the DB without further processing. The only thing that NTA would have to do, is recognize new flows and/or devices for its functions and displays.

The initial recognition would be somewhat slow so as to line up with the Node ID in the NPM database (NCM, too) and getting it applied to the NPM Web Page for that device.

I hope that gets you going and helps with the thought process.

mhh351 replied on Fri, Jun 12 2009 12:02 PM

rated by 0 users

Any Thoughts?

pserwe replied on Sat, Jun 13 2009 1:04 AM

rated by 0 users

I'll second the motion, although I'll go a step further..  Netflow should dump data into it's OWN SQL database.  No reason the website can't pull data from multiple sources.  Hell, it does already when we really don't want it to, like NCM.

Peter

Donald_Francis replied on Fri, Jun 19 2009 6:26 PM

rated by 0 users

We run netflow with about 250 interfaces and we slam the heck out of our SQL box and I suspect it is due to netflow.

I would be very handy if it was a seperate DB so you could place it on another SQL box or at least on a diff set of drives.

It would also be great if instead of their being a DB update for every flow if they were cached for some short period of time and wirtten to the DB in blulk.

mcbridea replied on Mon, Jun 22 2009 10:29 AM

rated by 0 users

Hi Donald,

We are investigating ways to off load some work from SQL. No Dates to promise

Andy

mhh351 replied on Thu, Jul 9 2009 8:52 AM

rated by 0 users

Please see my second post on this where I stated "if I may be so bold" for ideas on how others do it.

mcbridea replied on Thu, Jul 9 2009 3:20 PM

rated by 0 users

I hear you guys. These are all options we are considering.

Andy

jeff.stewart replied on Sat, Jul 11 2009 8:12 AM

rated by 0 users

I do like the thought of these ideas, but additional software and hardware cost come in to play.  I will say that v3.5 has made huge gains in performance than other version.  If the performance keeps increasing with each version I see no need for the ideas mentioned. 

mhh351 replied on Mon, Oct 26 2009 2:17 PM

rated by 0 users

Just a kindly comment. When we run a Worldwide network, having it take all of this Netflow data across the WAN links becomes costly froma data standpoint. Consolidating that data and sending it in compressed or pre-processed form is much better. OTher vendors do that and the efficiency is much greater. Links that are already busy and costly should be used as efficiently as possible. If this was just for domestic sites, we would not care as much.

As for the amount of data going to ONE database, that is just not a good design. Each Database should have its own purpose if not its own server. If one looks at the amouont of data transactions that NPM does already to the DB server, one would not want to add that kind of traffic or amount of data to the DB from Netflow types of transactions.

Response time is very important to improve upon and your point is a good one. When we try to scale the overall application/database relationship, we want to make sure that other applications such as NPM do not have to suffer for either the size of the database or for the other polling/gathering that is going on.

Thanks for the discussion!

chris.lapoint replied on Mon, Oct 26 2009 3:23 PM

rated by 0 users

Thanks for the continued comments and discussion.  I can tell you that scalability and simplified visualization of data are absolutely key themes for our upcoming releases.   Our goal is to take an evolutionary approach to address the use-cases described above.  As Jeff noted above, we made some significant strides in the 3.5 release and our upcoming 3.6 release will continue to improve scalability and performance.

For those continuing the discussion, it is extremely helpful if you can list out your use-cases (i.e. what specific problems are you trying to solve) as mhh351 did above in the first paragraph.  This way we can prioritize and ensure that as we look at ways to evolve NTA architecture for the big win (e.g. separate database, standalone remote collector, etc.), we can also knock off some smaller wins along the way for you.

Keep the great feedback coming!

-Chris

kdouglas1 replied on Fri, Mar 12 2010 1:05 PM

rated by 0 users

Any update on this?

We're in a competitive situation trying to decide on the direction for our Enterprise wide NMS.  While Orion wins hands down for usability, my past experience with NTA (a key component of our solution) has been less than stellar when trying to collect information from a fair number of interface (collecting from six GigE interface on a beefy box allowed Orion to perform well, but in my new environment I'm looking at about 100 interfaces). 

The competitor supports separate NetFlow collectors that don't impact the performance of the core application.  They can support two or three separate collectors that optimize the date before sending it to the core application. 

I'm concerned about recommending Orion given the performance issues I've seen.  Off load the collector function AND add NTA collector function to the Scalability Engines.  

Ken

chris.lapoint replied on Fri, Mar 12 2010 1:57 PM

rated by 0 users

Ken, I've sent you an email offline.   I'd like to understand your use-case better and then we can decide how best to move forward.   Then, I'll close the loop with the rest of the community as appropriate.