(SOVLED) – Audio dropouts with Dell E6410, Nvidia NVS and Tascam US-144

Update 10th Feb 2011 :

Another useful very useful utility for actually identifying which driver is causing the issue is LatencyMon from http://www.resplendence.com/latencymon

 

Background

When I recently purchased a new Dell E6410 laptop, I also shifted from Windows XP to Windows 7. When it arrived, I ditched the fairly average Seagate Momentus 7200 160Gb replaced it with a fairly large 500gb hybrid drive and installed a fresh copy of Windows 7 32bit without unwanted software or utilities. I have 4Gb ram installed so wouldn’t be loosing out too much by running 32bit, the main reason for not going 64bit was that I wanted to continue to use live audio software, particularly Ableton Live with an external audio interface with low latency ASIO drivers. 64bit drivers are being released all the time, but I was hoping for an easier time by keeping things 32 bit.

The existing audio interface was a Tascam US-122, and this was not officially supported under Windows7. While the drivers could be tricked into installing by running the installer application in Vista compatibility mode the results were far from stellar. It worked, but the drop-outs were pretty bad, up to a second at a time, and frequent. Worst still, the drivers would hang the system on shutdown.

The Tascam US-122 was sold swiftly on ebay and replaced with a more current model with Windows 7 driver support – the Tascam US-144 mk2.

The laptop also has: (it becomes relevant later)

• NVidia NVS 3100M graphics
• NVidia Audio
• Intel Gigabit lan
• Intel 6300-N Wireless

Audio drop-outs and stutter

The driver installation was straightforward, and the US-144 was up and running within a minute. I immediately fired up Ableton changed the audio output configuration, stuck with the default ‘medium’ latency settings and hit play. Everything fired up and the audio was clean.

For about 20 seconds.

The audio would then drop out for up to a second at a time, and relatively frequently. There was a vague connection with what I was doing at the time, but not one I could really figure out. It seemed to often coincide with an action on my part, but I couldn’t make it happen to order either, and then sometimes it would happen without any apparent interaction from me. This wasn’t just with Ableton either, just running an mp3 media player would also suffer the same way.

A tried several things to remedy this:

• All latency settings
• Different USB ports
• Using latest Tascam drivers
• Disabling Antivirus (NOD32 AV)
• Using ProcMon to look for other process threads or anything else appearing when the dropouts occur
• Looking through Tascam and other forums

All without any success. I was particularly hopeful when firing up ProcMon that I would find the culprit process that was interferring with the audio stream, but found no obvious correlation with events and the drop outs.

I decided to cut my losses and return the unit immediately with a view to trying a different vendors product. I gravitated towards the M-audio and also the Presonus product lines with M-Audio having the edge. While on the M-audio forums, I found a page entitled Pops, clicks, crackles, dropouts, or distortion – USB Audio Troubleshooting Guide . It mentions a utility –  Thesycon’s DPC Latency Checker which specifically looks at the system latency as seen by a driver. Running the utility immediately showed that there were some DPC latency issues with frequent spikes of over 17,000 microseconds – over eight times the problem threshold of 2000 microseconds. The article discusses IRQ issues and recommends turning unused devices off, so I opened up device manager and disabled everything that wasn’t required, or could be temporarily disabled, this included:

[the underlined items I left permanently off]

• Intel 82577LM Gigabit Network connection
• Intel Centrino Ultimate-N 6300 AGN
• Microsoft Virtual WiFi Miniport Adapter
• High Definition Audio Device (+ 4x NVIDIA High Definition Audio devices)
• Bluetooth
• ECP Printer Port LPT1
• Intel Active Maangement Technology SOL COM3
• Microsoft USBCCID Smartcard Reader WUDF
• Dell ControlVault w/o Fingerprint Sensor
• Integrated webcam

From the system BIOS I also disabled the serial port, LPT1 and bluetooth module. With all this disabled and after a reboot the DPC graph still didn’t look good and the large 17,000+ spikes remained. I found another forum post that linked NVidia’s graphics card  ‘powermizer’ feature to the PDC latency spikes. The powermizer feature ramps the GPU clock up and down according to graphics load, allegedly when the clock changes, the DPC issue strikes.

I fired the video adapter GPU monitoring utility – TechPowerUp GPU-Z. Running this alongside the PDC checker  allows you to check for event correlations. Sure enough, the GPU clock would ramp up and down, and particularly when dropping to the most idle state the DPC spike would occur. It wasn’t guaranteed though, sometimes there was no spike, but when there was a spike it was always around 17,000 microseconds and occured in a GPU clock change. At this point I no longer owned the Tascam audio interface, it was already in the post being returned, but the pattern of dropouts were a match to the audio drop-outs I had been experiencing with it. I never had issues with the on-board NVidia audio, but the latency of that device was far too big to use with Ableton.

I uploaded a video to YouTube showing this correlation (best viewed full screen):

The solution

Nvidia Powermizer switched off.

Reading further I found mention of a utility to disable NVidia’s PowerMizer feature with PowerMizer Switch v1.2. I ran this ‘as administrator’, selected “turn off” and rebooted. I also changed system power settings to ‘max performance’ and tried again., and this time was not running on battery either. I’m please to report that at this point the problem disappeared and DPC latency fell drastically and no spikes occurred. In the video I’m also running CPU load tester and a benchmark concurrently, and at some points rapidly switching a window from maximized to minimized to try to invoke a problem. It didn’t happen – no more latency issues. See video:

While this resolved the problem, it isn’t a 100% black and white case of blaming the graphics driver as I’ve changed more than just the PowerMizer setting – I’ve also switched power modes and am running on AC, not battery.  But truth be told, it looks like NVidia is the most likely root cause.

I should also point out that the issue of DPC latency is ablaze across Nvidia’s own forums. One particular forum thread spanning over 1 year and 23 pages long specifically refers to “notebook powermizer issues”. Trawling through the official nvidia responses in that thread shows there has already been an official “bugfix” released last year. Despite that “fix” and subsequent driver updates between then and now (16th Jan 2011) I still ran into this issue even using the latest drivers for my NVS3100M ( v260.99 ). So whatever it is, it seems far from fixed. I’m just glad that disabling PowerMizer gave me the result I wanted.

Hopefully someone else may find this useful in resolving or improving their latency and audio drop out problems.

Sitecore Staging. Onwards and upwards.

There isn’t anything new here, more a case of just wrapping up some posts and tying some events together.

In July 2009 I posted about some significant issues with the Sitecore staging module. Two months later in Sept 2009 Alex Shyba announced a new shared source “Stager” module. The new open-source module took a different approach and fully implemented partial cache clearing in a staged server setup. This was great news, and it worked really well – issues of heavy cache  and session clearing were now history.  “Kudos for making this module” was the bottom line in a blog post from Mark Cassidy, but he also raised the important issue of support.  As a shared source module, the new stager wasn’t supported by Sitecore, making it a difficult choice to upgrade. Fast forward another three months to December 2009 and it was again announced that the “Stager” shared source module has now become the de-facto “Sitecore Staging Module”, replacing the old module and thus becoming fully supported.

Fast forward again six months to July 2010 and the release of Sitecore 6.3 which raises the bar with in-built scaling ability. I haven’t had the opportunity to use it, but the feature list is impressive and includes the ability for multiple content master servers. So in 12 months, things have changed drastically for the better for multi-server Sitecore environments. Hopefully I’ll get a chance to use it sometime soon!

Problems with the Sitecore 6 staging module part 2 – extranet user loses session

[Update Dec 2010: See this post for details of important changes since this post for first published]

 

Session state lost after publishing.

This post follows on from Part 1.

The symptom:

I guess this is pretty self-explanitory, but it depends how you are arriving at this problem. You might be using real accounts, virtual users, or whatever. The effect is that your extranet user, i.e. public website user, appears to lose their session some time after logging in. They are effectively logged-out. You won’t notice this with anonymous users, only users that have logged in. This may also initially appear as a security problem, perhaps the user is suddenly unable to access content they should have access to.

The problem:

Pretty much the same as in Part 1.  If the Sitecore staging module is set to the recommended “Full” cache clearing mode, and a publishing operation occurs, all extranet users are logged off after the staging module .xml work files are processed. As explained previously, there is a short delay between the publishing operation starting and the web service actually being called from the master.

The cause:

When the Sitecore staging module is set to the recommended “Full” cache clearing mode, the slave server clears all caches simultaneously, but it also calls Context.ClientData.RemoveAll() which effectively clears the session for extranet users.  This affects the current available version of the staging module  for Sitecore 6 as of July 2009 (v5.4.0 rev 080625). In contrast, the current version for sitecore 5 has been fixed as of July 17th, and should no longer have this issue.

The solution:

The staging mode in “partial” cache clearing mode does not invoke Context.ClientData.RemoveAll(), so the simple solution is to use the “partial” cache setting. Again the problem is as stated before – this is not the recommended setting, and this is reiterated in the troubleshooting section. There is also no explanation of the scenarios where partial cache clearing is not enough, just that in the “partial setting does not clear the entire item cache”. I had previously thought that this would affect publishing operations that include template changes, but I haven’t explored that notion further.

Finally, despite this rather large bug being known, it is not documented at all other than in forum posts. Nor is it in the release notes.   A slightly similar but unrelated  issue (almost certainly less common too) is  covered in point 6.2.3 of the installation document.

Problems with Sitecore and the staging module part 1.

[Update April 2011: This post and part 2 are now irrelevant for Sitecore 6.x  See this post for details of important changes since this post was first published]

This has been a bugbear for some time now. I’ve had firsthand experience of all the issues I’m going to describe and it was not an easy journey to figure out what was actually happening. The reason for this post is to have some kind of searchable record of the problem online that is indexed. This is because Sitecore’s own forums (where this issue has appeared many times) appear to not be indexed by any search engines, and to help other developers who run into this problem.

For the record, and in case anyone does this see this post in that way, I’m not anti-Sitecore,  I like the product (otherwise I wouldn’t bother doing this), I like the level of support too (they must have some  patience!), I just think the staging module is pretty flawed in some areas. Finally, some of these issues are already known, but don’t appear in any list of ‘known issues’. This is immensely frustrating!

The Sitecore staging module is an ‘official’ (i.e. supported) module that allows you to separate your live, public-facing servers from your CMS server that editors directly create content on. From now I will refer to the internal CMS server as  ‘master’ and all public facing web servers as ‘slave’ – this is the same naming convention used in the staging module documentation.  A typical configuration will consist of one master server located locally within your office, and one or more slave servers located externally with a hosting company.

There are three issues that occur when using the staging module that I’m intending to eventually highlight, these are:

1. Cache clearing
2. Extranet users are logged-out after each publishing operation.
3. Publishing delay

Cache clearing

The symptom:

Your live website slows to a halt at regular intervals, depending on how often you publish. CPU utilization of the w3wp.exe process will often suddenly ramp up to 100% for some time (probably somewhere within 5-90 seconds)  and the SQL server process will be doing the same, or be worse.  After this time, your site returns to normal. Page requests to your site may hang for the entire duration and if they don’t close the browser, will eventually get a response.  This happens after each publishing operation + staging task (scheduled) publishing delay.

The problem:

After publishing content in a staging environment, the master server calls a webservice on each slave server to clear caches. The recommended setting for this is “Full” cache clearing rather than “partial”. You can verify if staging is the cause of all this by performing the following steps:

1. On the master server, check the folder /sitecore modules/staging/workdir/cache for the presence of any .xml files. If you haven’t published recently, this should be empty. (If there are loads of .xml files you probably have another problem).
2. Publish something from the master server to a slave server
3. Immediately check the /workdir/cache folder again (it should now contain one .xml file)
4. Wait for the .xml file to disappear (it’s now being processed)

As soon as the .xml file disappears, the master server should hit the webservice on the slave server to clear caches. If at this point, your site slows down and the CPU and SQL load jumps unacceptably, then this is very likely to be a problem caused by staging and the aggressive cache-clearing invoked by the ‘Full’ setting.

The cause:

The documentation does state that ‘full’ method is slow, however I think it underplays how much impact this can have. This setting is configured on the master server and is used in the webservice call to the slave after a publishing operation to clear the caches on the slave. The main difference is that ‘partial’ clears the front-end HTML cache (i.e. your rendered outputs) and the Data cache. Using this mode, your site will probably hardly blink unless you have very intensive renderings. The ‘full’ mode however does what it says in the documentation, and clears all caches, and Sitecore has a lot of them. You can see a list of these in /sitecore/admin/cache.aspx . It also does some other additional clearing operations and then does all this all again for the shell website (the sitecore shell at /sitecore is just another website). The whole effect is that web server and SQL server load can spike in a big big way. In my particular case, the resulting server load and time-delay is comparable to the same load and delay that occurs after recycling the application pool, or in other words, every time I publish the site feels like it’s rebooting, and that seems to be pretty much what is happening. The ‘rebooting’ analogy also lends itself to the #2 point about extranet users (post to follow). If you publish frequently, have a large number of items, or heavy load then this makes things much worse.

The solution:

The simple solution is the try the “partial” caching mode instead.  Sitecore say that in some scenarios items will not be published correctly and therefore only the “full” setting is recommended. This is the crux of the problem. You’re damned if you do, and damned if you don’t. I really think that Sitecore needs to come up with a better way to handle staging, either by fixing this module, or a different approach altogether. So far we’re stuck between a method that isn’t recommended and doesn’t always work, and a method that is recommended but can easily cripple your performance whenever you publish.

More to follow…. in part two.

How to copy a project from VSS and remove bindings.

If you want to duplicate the solution to add to a new source control system. A simple option is to:

1. Duplicate the local working folder to another folder.
2. Open the new copy solution in Visual Studio
3. File > Source Control > Change Source Control
4. Check / Select all items
5. Choose [Unbind] for each project to remove any association between the copy folder, and VSS.
6. Enable options to show hidden and system files in Windows Explorer.
7. Delete all files ending with *.scc, *.vspscc, *.user, *.vssscc, *.suo
8. Add the copy folder into the new source control system.

Problems with this include loosing all history information. Going forwards, you’re starting at the current state with nothing to roll back to.

Sitecore with Gridview – rowupdating event doesn’t fire

Whenever using Databound server controls with Sitecore need to be aware that Sitecore  meddles with some controls. This nasty issue which can have you banging your head against the wall for hours (days?) has been previously blogged by Mark Cassidy but I guess it depends on what symptom you’re having first, and wether you relate it back to this obscure setting. In my case, I was previously aware of this issue but only relating to apparent loss of viewstate on controls. Unfortunately I didn’t think of it as affecting events too.

In my case most features worked, Edit buttons would invoke row editing mode, cancel button works fine, rowCommand event generally worked, but it wasRowUpdating that simply wasn’t working. Adding System.Web.UI.WebControls.GridView to <typesThatShouldNotBeExpanded> in web.config resolves this issue. your web.config would then look like :

<typesThatShouldNotBeExpanded>
<type>System.Web.UI.WebControls.Repeater</type>
<type>System.Web.UI.WebControls.GridView</type>
<type>System.Web.UI.WebControls.DataList</type>
</typesThatShouldNotBeExpanded>

This issue does now have a fleeting mention on http://sdn.sitecore.net/Scrapbook….. but requires login to view it.

While looking for that link, I also came across another blog post detailing the same issue and frustrations I’ve had with Gridview events that may have more detail.

GridView problem – Item has already been added. Key in dictionary: ‘Timestamp’ Key being added: ‘Timestamp’

In general ASP.net, this can be caused by calling DataBind() more than once on a databound control. Also be aware that this is influenced by automatic databind being on too.

Relating this to Sitecore CMS, you should be aware that Sitecore ships with the following line in web.config:

<!– AUTOMATIC DATA BIND
Indicates if the data bind function is run automatically
–>
<setting name=“AutomaticDataBind“ value=“false“ />

This means that you will need to either call Databind whenever you need it, or you can simply set this value to true.  This AutomaticDataBind setting will also cause problems with other types of databound controls, such as templated controls. It’s generally simpler to set it to ‘true’..

Sitecore preview bug – looking at web database

Sitecore v6 introduced another preview tool accessible from [Presentation] > [Preview] as opposed to [Publish] > [Preview]. The new preview mode opens another tab in the content editor which shows the preview of the current item.

This is all fine and well except that it is permanently looking at the web database rather than master database. This can be observed by creating a new item, saving it, and then selecting preview. You get a 404 error, unless you publish it first.

Unfortunately this nice feature is 80% broken and can only be used for looking at published content.

Update: This was a registered bug in February 2009 with a workaround available from Sitecore support, but this still has not been released and I believe the bug still exists in Sitecore 6.1

Sitecore – Could not load file or assembly System.Data.SQLite

After installing Sitecore v6, you get the following error when viewing the website or sitecore UI:

Sitecore Could not load file or assembly ‘System.Data.SQLite, Version=1.0.48.0, Culture=neutral, PublicKeyToken=db937bc2d44ff139’ or one of its dependencies. An attempt was made to load a program with an incorrect format.

This is actually detailed in page 20 of the sitecore troubleshooting doc, but I missed it first time round. This is caused by the presence of System.Data.SqlLite in the bin folder when running on 64-bit OS. An alternative dll is available for 64bit OS from Sitecore.

Solution: If you’re are not using SQLite simply the delete the System.Data.SqlLite.dll from the bin folder and recycle your app to get back up and running. If you are using SQLite, download the alternative dll for 64bit OS.

Sitecore ContextItem.Database.SelectItems() and languages.

While poking around in the shared source RSS module for Sitecore, I found that several aspects of the “static feeds” feature loose language context during publishing. This only effects static feeds whereby the rss.xml file(s) are written to disk and linked-too directly. The processes is kicked-off my hooking into the publishing-end event.

While the similar method Context.Database.GetItem retrieves an item in the same language as the context item, it appears that SelectItems(selector, predicate) does not. So if you are in a language other than ‘en’ English, you will probably retrieve the ‘en’ version of your non-‘en’ language items, resulting in large areas of missing data.

I’m not sure if this only effects SelectItems()  when run outside of the context of a web request, or if it happens all the time, but the work-around is to explicitly change your context.language before calling SelectItems().

ContextItem.Database.SelectItems(selector, predicate);     …..will return ‘en’ language items regardless.

Instead you must use:

Sitecore.Globalization.Language.Current = ContextItem.Language;
ContextItem.Database.SelectItems(selector, predicate);    

….which then retrieves items in the same language of contextItem.