VW504.00/507.00 is only 0//5W30 oils and among those oils has most stringent wear requirements. However, it allows SAPS to be very high, all the way to 1.5. Still, it is not possible to achieve DPF protection with such high SAPS, so companies never go above 0.8 in VW504.00/507.00. There is no doubt that low SAPS oils are using less additives, but generally they are because of very stringent requirements "cooked" with very sophisticated additives and high quality base oils. Oil company that I worked for as one of field test guys could cook high SAPS oil very cheaply. Low-SAPS oil was very expensive to develop and it is more expensive to make. We for example drove VW Golf V 1.9 TDI and 2.0TDI and Polo 1.2TDI for 400k km's before dissembling engines. Some tests included 20.000km with no turning off engine in open road environment, 20,000km in mixed and 20,000km in strictly urban driving. We would just recycle drivers constantly. Then stop/go traffic, very short turn off/turn on runs, extremely cold starts, or extremely hot starts etc, etc. After 400k km, engines and cars would be inspected for wear and tear. Then oil was sent to VW for approval.
Commercial type of oils like Rotella T6, Delvac, Delo etc. are not on par in sophistication with oils such as VW504.00/507.00 etc. Commercial use has different demands and oil companies in general have easier times to meet those demands since we are talking about different engines, different emission schedules, larger DPF/SCR systems that are easier to fit into big equipment vehicles. Those commercial vehicles usually have very large oil sumps, block heaters etc. Oil companies cannot develop expensive oil, especially on European market for such vehicles because it cuts into profit. Here oil is cheap, in Europe liter of VW504.00/507.00 is some $15-20. It hits profit if you are running let's say 200-300 trucks. That is why trucks are on different emission schedule and they could use higher SAPS oils, higher NOACK oils etc. since engines are more rudimentary.
As for MB229.5/1 I believe deposit wise that is most stringent spec. I think LL-04 is most stringent oxidation wise. That is why it is always good to buy oil that meets all those specs.
But to think how some UOA with some wear numbers will suddenly prove how some other non-approved oil is better is ridiculous. Manufacturers before sending engine on the market do everything possible to find weakness to that engine and then they constantly tests engines during sale to see are there any weak spots that are missed during initial testing. On top of that, data from service departments, failed engines etc. are always inspected and some practices when it comes to maintenance are updated (like for example FCA V6 diesel in JEEP and RAM). Those companies which neglect all that end up like Chrysler I guess.
Interesting, however I have a couple comments regarding some of your statements. I don't think that's true regarding the base oils in 507. Top Tec 4200 is a group III product. The base oil is not "high quality". It also has a rather high NOACK %. I would think there are also other brands that sell 507 oil composed of G III bases as well. In this example, the requirements for 507 must not be very stringent if it allows for this. It would also mean that it's ingredients are inferior to CJ-4 Delvac and Ravenol.
The new E6 HDEO Lubes being sold now, specifically Delvac LE 5w30 and Delvac ESP 5w30 are not cheap. LE is $35 and ESP is $45 a gallon. That is not "cheap", it is the same price as 507 oil, so I don't see your point about not being able to market "expensive low ash HD oils".
Lastly in 2016 FCA did finally change the spec on the Ecodiesel to an HDEO due to numerous oil related failures, however they did not change any of the emissions equipment. So it seems that the extra SAPS wasn't an issue.
VAG knows that the 502 oil they spec'd for the RS4 wasn't up to the task. It couldn't deal with the fuel dilution, nor could it protect the engine during the recommended OCI. What did VW/Audi do to solve this? NOTHING WHATSOEVER. They still have done nothing.
BMW has made similar mistakes. Either by using an oil that could not adequately protect the engine, and/or by having an extremely long OCI which further exacerbated their oil's shortcomings. Manufacturers are not always proactive about solving problems, especially if they happen outside the warranty window.
Some issues develop over the time of exploitation. I know that MB had issues with oil coolers, and that is resolved later. Sometimes although companies put a lot of miles on the cars, you cannot replicate real life conditions over several years.
I personally do oil changes every 5,500 miles using Mobil1 5W30 ESP and Valvoline 5W40 MST (I found it on sale so still have it in stock). When it comes to oil evaporation, use oil with lowest NOACK. So far according to one web site Mobil1 5W30 ESP has lowest NOACK of available oils for modern diesels. Oils like Mobil1 Delvac or Shell Rotella T6 do not have nowhere near NOACK like Mobil1 5W30 ESP. That is the thing with oil approvals. Since you are doing a lot of city driving you want light oil like Mobil1 5W30 ESP, or some other MB229.51 0/5W30 oil. Do not go to 5W40 or heavy duty oils, especially 5W40 HDD oils. You want your oil to reach operating temperature as fast as possible.
What is the NOACK of Delvac ESP 5w40, specifically the CJ-4 formula ? I've looked everywhere I can think of but can't find it. Can you forward your resources?
Just because oil is made out of Group III base stock does not mean that base stock is in lower quality. Mobil1 5W30 ESP is prime example of that where NOACK is (supposedly though) 5.6%. It is final product that matters, not whether oil is using primarily GR III, IV or V. Each group has its own advantages, and they are usually mixed.
As for Delvac NOACK, I really cannot remember where I have seen their numbers, thought I remember precisely that T6 is 12.4%. Your reference how Delvac is $35 and expensive is not accurate in global terms. That is actually NOT expensive oil. Oil in the U.S. is cheap, including Delvac. Same oil in Europe is probably $20 more.
As for FCA, I stated long time ago here, when they decided to move to Shell Rotella that DPF will be long term issue. However, what is long term? That is FCA decision to make. Probably they calculated that DPF and other emission related components will last more then 120,000 miles. After that, marketing calculation is that it will not affect company. And it was not oil related issue. Heavier oil (remember, FCA also said C3 5W40 is acceptable, as well as ACEA A3/B3 B4) is used to hide engineering issue. In the end, I believe that major role in that decision as the fact that a). FCA has deal with Shell, and Shell does not have 5W40 C3 oil in the U.S., and b). availability of T6.
As for VW502.00, I am not sure any ACEA A3/B3 B4 oil was able to deal with fuel dilution issues in first generation FSI engines. If you are referring to the fact that people used T6 in those engines and other FSI engines such as EA113, that is because of weight, which was 14.2cst, not because somehow T6 is magical oil since it is HDD oil. Most VW502.00 5W40 oils are light W40 oils in low 13cst range. Why VW did not do anything? Again, marketing, what is damage from that? In the end we are talking about company that thought they can scam whole world with so called "clean" diesel.
Do not get me wrong, companies are in the business of making money, and they will calculate whether they will recall some engines, other components or not. Also oil companies are in the business of making money, and if they think they can make oil that meets whatever spec. with cheapest base oil, they will do it, including HDD oils.
However, there is no any incentive for VW, BMW, MB, whatever, to recommend more expensive oil for their consumers if cheaper oil will do the job. It is all about money, bang for a buck etc.