When deploying acceleration solutions I frequently have a conversation that goes something like this:

Customer: "We're not seeing the performance improvements we were hoping for."

Me:  "What improvements are you getting?"

Customer:  "Only a second."

Me: "What was the response time without acceleration?"

Customer: "5 seconds."

Whether or not the response time improvement is seen as good enough varies from person to person.  From my perspective with a page load time of only 5 seconds to start a 20% reduction to 4 seconds is great.  However if the page load time was 20 seconds to start with and the improvement was only 1 second that isn't as good.  A 1 second difference may be all it takes to increase customer satisfaction or sales.  A recent study by Aberdeen Group called "The Performance of Web Applications: Customers Are Won or Lost in One Second" found that a "one second delay in response times of Web applications can impact customer satisfaction by up to 16%."

The higher the response time is initially the greater the improvement is likely to be.  I would be shocked to see a 50% reduction in response time on a page that takes 5 or 6 seconds to load, but not at all surprised to see that level of improvement on a page that takes 20 seconds.  The other factor to take into consideration is that not all pages are created equal.  Some sections of an application may see massive reductions while others not as much, this is natural. 

I did some testing recently of a Sharepoint application.  The customer was primarily interested in the response times of document downloads, the results showed that when the latency was greater than 100 ms the response times dropped by 50% for the larger documents.  This was a very substantial savings seeing response times drop from over 40 seconds for some documents to less than 20 seconds.  Browsing of the pages within the application saw improvements of 20-30% as well but given that the response times were less than 10 seconds to start with this was not seen as impressive.  My recommendation is always to focus on the areas that are the slowest this is where you'll see the greatest improvements and these are likely where the majority of complaints are coming from. 

The other recommendation is to consider where the testing is being conducted from and how users are accessing the application.  If you are doing all the testing from a LAN or a location that has less than 100ms of latency chances are you may not see much significant gains.  Testing from other locations can provide you with a very different view on the applications performance and the improvement gains.  While a page may take only 5 seconds from one location users with slower link speeds or more latency can see that same page take 10 or 15 seconds.