myITforum.com Community Forum myITforum.com Community Forum

Home  Forums  Blogs  Live Support chat  Search Articles  Wiki  FAQ  Email Lists  Register  Login  My Profile  Inbox  Address Book  My Subscription  My Forums 

Photo Gallery  Member List  Search  Calendars  FAQ  Ticket List  Log Out

All Forums RSS Feed Subscription:


  


Resolved !! Help! Offermgr 5.5hrs to "initialize" before replicating adverts

 
View related threads: (in this forum | in all forums)

Logged in as: Guest
  Printable Version
All Forums >> [Management Products] >> Microsoft Systems Management Server >> SMS 2003 >> Resolved !! Help! Offermgr 5.5hrs to "initialize" before replicating adverts Page: [1]
Login
Message << Older Topic   Newer Topic >>
Resolved !! Help! Offermgr 5.5hrs to "initialize&q... - 8/1/2008 3:38:35 PM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		Can anyone out there confirm behavior I’m seeing?

We've recently started experiencing an issue in which adverts do not replicate to lower tiers. When we look at offermgr.log it is scrolling with messages such as:

Examining collection: Z1X02A1F
CCollectionSource_SQL::CreatePrivateCopy - Inserted table _CPY_OfferManagerTemp_11642 with 417 rows
Found 0 member (legacy clients,users,user groups) changes for collection Z1X02A1F
Flushed the offer info cache data to disk. 

This happens whenever SMS Executive is restarted and the process takes 5.5 hrs to complete. During this time if an advert is created it will not replicate to the lowers sites until the process has completed.  We currently have 6k adverts, so I understand what its doing, its checking all the collections that has an advert assigned to it, but why hold up the replication of adverts?  Its not a hardware issue, the box is Quad core Xeon 3.6GHz with 16GB of ram and an external SAS array Raid 10, its a real fast box.  I believe this has always been happening we just never noticed it until recently, can someone else confirm they see the same issue on a primary with 1000's of adverts ?

Thank you.

< Message edited by rspinelli682 -- 9/24/2008 2:42:44 PM >
Post #: 1
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 9:53:50 AM   
iglatz


Posts: 848
Score: 23
Joined: 9/21/2001
From: Erding, Bavaria - Germany
Status: offline 		I had a similar issue at a customer site where all collections were updated at the same time in the morning. This caused offer manager to be slowed down for about 1h. All adverts that were created during this time of the day, showed up about an hour later in the advertisement status. Adverts defined later only took a couple of minutes.
Do you have a lot of collections being updated quite frequently that might cause the server being occupied with updates?

(in reply to rspinelli682)
Post #: 2
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 11:37:35 AM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		Iglatz, we do have a lot of collections and there all updated different times during the day.  The issue is that after a restart, no matter what time we restart the executive, the "reinitialization" takes 5.5 hrs because of the number of adverts we have.  I believe this reinitialization is by design, but it's a bad design for a firm with 1000's of adverts, like us.  I was hoping someone could look in offermgr.log after doing a restart and see if they saw the same issue of scrolling with info as below:

Examining collection: Z1X02A1F
CCollectionSource_SQL::CreatePrivateCopy - Inserted table _CPY_OfferManagerTemp_11642 with 417 rows
Found 0 member (legacy clients,users,user groups) changes for collection Z1X02A1F
Flushed the offer info cache data to disk. 

I know the easy answer is to reduce the number of adverts, but for a firm of our size with 170k clients, that’s easier said then done.

Thank you.

(in reply to iglatz)
Post #: 3
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 11:51:32 AM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		I see the same thing in our offermgr.log and we're roughly the same size as you with ~200K clients and 10K ads.  But I gotta say I don't know that we've ever had a problem with that behavior.  We only really reboot those servers once per month and it's on a weekend so by the time anyone would create an ad, it would be a day or two later so I can't speak to it as the source of the replication delay, though it would make sense.

_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 4
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 11:58:42 AM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		John, that’s good info to know. 

You should see the same behavior though whenever smsbackup service kicks off since it stops and starts the executive not just on a restart of the box.  We didn’t notice the behavior either since most of our adverts are done during the day, so by the time people started creating adverts everything was working.  We have now started creating adverts overnight with the India team and that’s when we noticed it.  I bet if you tried to create an advert while offermgr was going thru the process you wouldnt have the ad replicate to lower tiers until it was completed. 

(in reply to jnelson993)
Post #: 5
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 12:48:19 PM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		Good to know. 

Hmmm, I wonder if ConfigManager has this same deal going on?


_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 6
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 1:10:58 PM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		According to MS, SCCM 2007 has the same issue.  We've opened a case and are trying to get MS to come out and say its a bug.  I see this as being a product scaling issue in my opinion.  Not many sites have as many clients as we/you do, so they would never see it.

(in reply to jnelson993)
Post #: 7
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 1:16:19 PM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		I'm not sure of all the details of how they're doing it under the hood, but it seems to me that one possibility would be to at least make this multi-threaded so it does it's work quicker.  Don't know if that's possible, but on my big 32GB 8way box it wouldn't take long :)


_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 8
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 1:26:18 PM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		John, if this wouldn't be to much to ask, do you think you could see how long after sms executive is restarted after the backup has completed, you start seeing messages like below:

Skip checking the membership for collections XYZ033BB, XYZ033B9, XYZ033B8, XYZ033B6

This is when we have found that the primary is "ready" to replicate adverts.  I'm assuming you have the majority of your adverts on your Tier 2 servers ?  You would need to bump up offermgr.log (which I'm sure you already know) to capture the info.

If you can't do it its no big deal, but would be great for me so I compare apples to apples.

Thanks!


(in reply to jnelson993)
Post #: 9
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 1:30:57 PM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		I'll see what I can find out.  All of our ads are on our central administration site and trickle down to the regional primaries (no administration happens directly from those servers, it's all done from the central).

Obviously that server is busy enough that the logs from last night are already overwritten.  I'll check first thing in the morning. How big do you think is big enough for that log?


_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 10
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/4/2008 1:40:39 PM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		Well i didnt know a good size, so I just did the good old default of 50Mb and that got me a full 2 days. 

I wish we had the same setup as you guys, but we have regions (EMA/APA) who want their own control, so we set them up with Tier 2 servers. 

I appreciate this, next time your in NYC I owe you a beer or 7 :)


(in reply to jnelson993)
Post #: 11
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/6/2008 8:24:51 AM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		Looks like it's taking about 25 minutes.

_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 12
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/6/2008 10:10:17 AM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		25 minutes ? Man i would take that in a heartbeat.  You mind listing your config ?  SQL 2000 or SQL 2005 ?  What type of box, array config ?  How many collections do you have the adverts going to ?  If you have 10k adverts but there only going to 100 collections (which im sure you don't have) the process is fast, if you have 6k different collections thats where the fun begins.

Weird, we have a pretty good box and array running SAS drives w/Raid 10, how could it be taking so long in our config.

< Message edited by rspinelli682 -- 8/6/2008 10:15:49 AM >

(in reply to jnelson993)
Post #: 13
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/6/2008 3:25:49 PM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		Server 2003
SQL 2005
SMS 2003 SP3 plus latest hotfix
HP Proliant DL760 G2
8GB RAM
SAN connected for SMS/DB folders

Hmm, I wonder if someone has done some cleaning in advance of our SCCM rollout. 
I only see 397 active ads and 2484 collections.




_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 14
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/6/2008 3:49:21 PM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		John, in my testing expired ads still counted, didn't matter if the ad was expired or not, its really how many collections have adverts (expired or not) targeted to them.  When you say active advertisements do you mean not expired?

If you run the following query, how many collections do you come up with?

select count(distinct(collectionID)) as intCollCount from v_advertisement

We have 5230 unique collections with adverts targeted to them out of the 9600 total collections in SMS.  If you have 2484 collections with adverts being targeted to them, 25 mins is still alot better then what we're getting.  My guess is you have 2484 total collections, but a smaller number that actually have adverts targeted to them, but that’s just my guess, that query above will tell you the real deal.

We're still running SQL 2000 here haven't upgraed to SQL2005 yet, its in the works.  Im jealous you guys are getting ready to rollout SCCM2007, we're starting our testing/lab work now.

If you could answer this last question, I won’t harass you anymore and my free drinks offer when you’re in NYC still stands.

Thanks!

< Message edited by rspinelli682 -- 8/6/2008 3:53:06 PM >

(in reply to jnelson993)
Post #: 15
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/6/2008 3:54:49 PM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		ah, I got ya.  403. 

Man, don't our production folks do anything anymore? :)

_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 16
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/6/2008 4:51:24 PM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		John, very much appreciated!  I have a case open with MS, so at least I know its not just us having a problem, its a scaling isssue.  I'll update this post again when I get more info.

I wish our production had only 403 active collections! :)


(in reply to jnelson993)
Post #: 17
RE: Help! Offermgr 5.5hrs to "initialize" bef... - 8/6/2008 5:07:56 PM   
jnelson993


Posts: 900
Score: 127
Joined: 2/18/2005
From: Minneapolis, MN
Status: offline 		Well, just stop sending packages...it'll make your job a lot easier :)


_____________________________

Number2 (John Nelson)
MyITForum - Blog
MyITForum - Forum Posts

(in reply to rspinelli682)
Post #: 18
RESOLVED! RE: Help! Offermgr 5.5hrs to "initialize... - 9/24/2008 2:28:47 PM  1 votes
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline 		We actually got a resolve from MS on this.  The overhead was being caused since offer manager is spending a large amount of time on producing legacy(NT4 clients) support files for the CAP (these are the log entries that we see in offer manager for each collection about inserting tables and flushing offer info cache).  We don't have any legacy clients in our environment, only advanced clients so applying the workaround below fixed the issue for us.  We made the change below on our primaries and the process that took 4 hrs 47 mins took 1 minute 26 seconds after the change.  If you have legacy clients still in your organization you can't use the workaround below.

To make this configuration change on a site for testing here are the steps:

- Make sure you have a backup.
- Stop all SMS Services for the site.
- Make a backup of the sitectrl.ct0 from sms\inboxes\sitectrl.ct0 file
- Use text editor on sitectrl.ct0
- Find "Create CAP Instructions" string, this will be in the SMS_OFFER_MANAGER section of the site control file.
- Change "PROPERTY <Create CAP Instructions><><><1>"to "PROPERTY <Create CAP Instructions><><><0>" to disable the work to support std clients
- Save the file and restart services, check offer manager log, the lines about creating private copies, inserting tables, flushing offer info cache etc should be omitted and the offer manager startup should be much faster.
 

Some other points about this change:
- This will prevent you from having working std clients in the future, they simply won’t be able to get SWDist or anything that depends on SWDist.
- This needs to be tested as it is not a common configuration setting (with SMS 2003) to be sure we don’t have any unexpected results. There shouldn’t be downside if you don’t have standard/legacy clients but we can always turn it back on if it causes a problem. I would suggest after enabling this on a site or branch of sites that we create a couple of test adverts to make sure everything is working normally before proceeding.

Thank you.

(in reply to jnelson993)
Post #: 19
RE: RESOLVED! RE: Help! Offermgr 5.5hrs to "initia... - 9/25/2008 7:44:35 AM   
rzander

 

Posts: 44
Score: 11
Joined: 4/2/2002
From: Switzerland
Status: offline 		Thanks for posting this "tweak"... The offermgr startup time takes now 2minutes instead of 3hours !!!

(in reply to rspinelli682)
Post #: 20
RE: RESOLVED! RE: Help! Offermgr 5.5hrs to "initia... - 9/25/2008 11:52:51 AM   
rspinelli682

 

Posts: 266
Score: 12
Joined: 1/24/2006
Status: offline
quote:

ORIGINAL: rzander

Thanks for posting this "tweak"... The offermgr startup time takes now 2minutes instead of 3hours !!!


Glad it worked out for you.  Its the least I can do for the creator of SMS Client Center, its an amazing tool.  I'm looking forward to SMS Client Center v2.

I'm sure you already know this, but I'll say it anyway, to really take full advantage of this tweak it needs to be applied on all primaries, so none of them have this long delay.

(in reply to rzander)
Post #: 21
Page:   [1]
All Forums >> [Management Products] >> Microsoft Systems Management Server >> SMS 2003 >> Resolved !! Help! Offermgr 5.5hrs to "initialize" before replicating adverts Page: [1]
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts



  
Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI

0.266