HDTP Beta, Shifter and New Vision

Discuss every aspect of HDTP here.

Moderator: HDTP Team

Forum rules
Please do not feed the trolls.
medzernik
NSF
Posts: 76
Joined: Thu Jun 28, 2012 8:37 am
Location: Slovakia

Re: HDTP Beta, Shifter and New Vision

Post by medzernik »

bjorn98009_91 wrote:
Y|yukichigai wrote:
It just loads all HDTP meshes, with a manual exclusion coded in for the MJ12 troopers. I may start getting fancy later, but for now at the state HDTP is in I think this should be enough.
Yeah, I figured it was something like that. ^^
medzernik wrote:
I lowered AA and few other things in preferences menu. By pure accident might i add. But it works and i am very VERY happy now! Hopefully i am slow enough so that when i go to paris the next revision is out ^^.
I see. I've really tried to optimize the maps all I can, but it's hard with outdoor maps. I still might have one or two cards up my sleeve though. But yeah, too high settings might just break the engine in some of the most demanding maps. The first Liberty Island is such a map due too that it is huge, outdoors and we've loaded it with loads of stuff.
Yep, Also is the NYC docks with that ship revised? or it is not included/not reworked?
User avatar
bjorn98009_91
Silhouette
Posts: 688
Joined: Thu May 08, 2008 8:17 am
Location: Hufvudstaden, Sweden
Contact:

Re: HDTP Beta, Shifter and New Vision

Post by bjorn98009_91 »

The shipyard and the ship are revised. In fact all levels before Paris has been revised, so that's up to Graveyard.
Producer and Quality Assurance Manager for Deus Ex: Revision.
medzernik
NSF
Posts: 76
Joined: Thu Jun 28, 2012 8:37 am
Location: Slovakia

Re: HDTP Beta, Shifter and New Vision

Post by medzernik »

bjorn98009_91 wrote:The shipyard and the ship are revised. In fact all levels before Paris has been revised, so that's up to Graveyard.
Awesome can't wait to get there, New York has been incredible so far and unatco is just a gem! From what i've seen so far, paris is a gem too. Also will i be able to install the new revision and use my old saves?
User avatar
bjorn98009_91
Silhouette
Posts: 688
Joined: Thu May 08, 2008 8:17 am
Location: Hufvudstaden, Sweden
Contact:

Re: HDTP Beta, Shifter and New Vision

Post by bjorn98009_91 »

You should be able to do that. For instance you could save just before Graveyard is over, before the level transition and when you install the full version of Revision it should load the appropriate map (Paris).

Edit: However, it is recommended that you replay the mod anyway, since we are continuing to polish old maps. We will for instance make changes to UNATCO, since we are not 100% satisfied over how it looks now.
Producer and Quality Assurance Manager for Deus Ex: Revision.
User avatar
kocmo
Thug
Posts: 45
Joined: Mon Mar 17, 2014 11:51 am

Re: HDTP Beta, Shifter and New Vision

Post by kocmo »

Revision consistently crashes for me while trying to save game in the very first mission (NYC Island), when I hold/throw flares. Revision.log is being truncated mid-word, with nothing useful in it earlier. I have posted technical description of the problem and the solution in the next post.
Last edited by kocmo on Mon Mar 17, 2014 3:59 pm, edited 2 times in total.
User avatar
kocmo
Thug
Posts: 45
Joined: Mon Mar 17, 2014 11:51 am

Re: HDTP Beta, Shifter and New Vision

Post by kocmo »

Description of installed environment:
  • my own mod unifiedDeusEx, which combines BioMod with Shifter 1.9 RC1 and dxEnhanced (and adds a lot of other improvements and bugfixes :-))
  • Revision Demo 1.3.1
  • New Vision 1.5
  • HDTP release 2 beta3
A frankenstein mix of all the latest & greatest, in short. Point is, it pulls really a lot of resources, I measure Revision.exe memory usage on NYC Island at 1100 MB during regular gameplay.

Code: Select all

Mon Feb 17 06:32:57.906 2014 (UTC - 8:00): (10ac.19bc): Stack overflow - code c00000fd (first chance)

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

FAULTING_IP: 
Core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9
1015044c eb54            jmp     Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4d03f (101504a2)

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000010126751 (Core!FRegistryObjectInfo::FRegistryObjectInfo+0x00000000000232ee)
   ExceptionCode: c00000fd (Stack overflow)

DEFAULT_BUCKET_ID:  STACK_OVERFLOW
PROCESS_NAME:  Revision.exe
ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.
EXCEPTION_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.
PRIMARY_PROBLEM_CLASS:  STACK_OVERFLOW
BUGCHECK_STR:  APPLICATION_FAULT_STACK_OVERFLOW_ZEROED_STACK

STACK_TEXT:  
WARNING: Stack unwind information not available. Following frames may be wrong.
001b3074 1015044c 0028d530 0fd75500 1057a628 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x232ee
001b30b8 10369b02 0fd75524 0028d530 0fd75500 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4cfe9
001b30e8 1015e6b5 0028d530 001b33e0 00cd5b70 Engine!AInventory::operator new+0x65a9c
001b311c 10166bbe 1057a628 1012ad38 0028d530 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x5b252
001b3158 10126e65 0028d530 00cd5b70 0fd7542c Core!FRegistryObjectInfo::FRegistryObjectInfo+0x6375b
001b3408 1015044c 0028d530 0fd75000 1057a628 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x23a02
001b344c 10369b02 0fd75024 0028d530 0fd75000 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4cfe9
001b347c 1015e6b5 0028d530 001b3774 00cd5b70 Engine!AInventory::operator new+0x65a9c
001b34b0 10166bbe 1057a628 1012ad38 0028d530 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x5b252
001b34ec 10126e65 0028d530 00cd5b70 0fd74f2c Core!FRegistryObjectInfo::FRegistryObjectInfo+0x6375b
001b379c 1015044c 0028d530 0fd74b00 1057a628 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x23a02
001b37e0 10369b02 0fd74b24 0028d530 0fd74b00 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4cfe9
001b3810 1015e6b5 0028d530 001b3b08 00cd5b70 Engine!AInventory::operator new+0x65a9c
001b3844 10166bbe 1057a628 1012ad38 0028d530 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x5b252
001b3880 10126e65 0028d530 00cd5b70 0fd74a2c Core!FRegistryObjectInfo::FRegistryObjectInfo+0x6375b
... (and so on, and so forth) ...

FOLLOWUP_IP: 
Core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9
1015044c eb54            jmp     Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4d03f (101504a2)

SYMBOL_STACK_INDEX:  1
SYMBOL_NAME:  core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9

MODULE_NAME: Core
IMAGE_NAME:  Core.dll
FAILURE_BUCKET_ID:  STACK_OVERFLOW_c00000fd_Core.dll!FRegistryObjectInfo::FRegistryObjectInfo
BUCKET_ID:  X64_APPLICATION_FAULT_STACK_OVERFLOW_ZEROED_STACK_DETOURED_core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9
FAILURE_ID_HASH_STRING:  um:stack_overflow_c00000fd_core.dll!fregistryobjectinfo::fregistryobjectinfo
Revision.log:

Code: Select all

Log: Garbage: objects: 52918->42655; refs: 1074080
Log: Garbage: objects: 52501->0; refs: 1074080
Altogether, looks like a genuine stack overflow while processing a linked list (and not like infinite recursion, which was my original thought). There's some information about FRegistryObjectInfo at https://github.com/stephank/surreal/blo ... UnObjBas.h

I set Revision.exe to LARGEADDRESSAWARE via EDITBIN.exe, that didn't help. Shouldn't hurt either, though :-)
So I raised stack limit of Revision.exe to 2MB via EDITBIN, double the default value, and this solved the problem.

I recommend to use this solution on a machine with 64-bit operating system and at least 3GB of RAM. To use it on 32-bit Windows, you might need to boot it with /3GB flag.

Commands to patch Revision.exe:

Code: Select all

EDITBIN.EXE /LARGEADDRESSAWARE Revision.exe
EDITBIN.EXE /STACK:2097152 Revision.exe
I have also included fixed Revision.exe in my mod unifiedDeusEx.
Last edited by kocmo on Thu Mar 20, 2014 11:10 pm, edited 2 times in total.
medzernik
NSF
Posts: 76
Joined: Thu Jun 28, 2012 8:37 am
Location: Slovakia

Re: HDTP Beta, Shifter and New Vision

Post by medzernik »

kocmo wrote:Description of installed environment:
  • my own mod unifiedDeusEx, which combines BioMod with Shifter 1.9 RC1 and dxEnhanced (and adds a lot of other improvements and bugfixes :-))
  • Revision Demo 1.3.1
  • New Vision 1.5
  • HDTP release 2 beta3
A frankenstein mix of all the latest & greatest, in short. Point is, it pulls really a lot of resources, I measure Revision.exe memory usage on NYC Island at 1100 MB during regular gameplay.

Code: Select all

Mon Feb 17 06:32:57.906 2014 (UTC - 8:00): (10ac.19bc): Stack overflow - code c00000fd (first chance)

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

FAULTING_IP: 
Core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9
1015044c eb54            jmp     Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4d03f (101504a2)

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000010126751 (Core!FRegistryObjectInfo::FRegistryObjectInfo+0x00000000000232ee)
   ExceptionCode: c00000fd (Stack overflow)

DEFAULT_BUCKET_ID:  STACK_OVERFLOW
PROCESS_NAME:  Revision.exe
ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.
EXCEPTION_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.
PRIMARY_PROBLEM_CLASS:  STACK_OVERFLOW
BUGCHECK_STR:  APPLICATION_FAULT_STACK_OVERFLOW_ZEROED_STACK

STACK_TEXT:  
WARNING: Stack unwind information not available. Following frames may be wrong.
001b3074 1015044c 0028d530 0fd75500 1057a628 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x232ee
001b30b8 10369b02 0fd75524 0028d530 0fd75500 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4cfe9
001b30e8 1015e6b5 0028d530 001b33e0 00cd5b70 Engine!AInventory::operator new+0x65a9c
001b311c 10166bbe 1057a628 1012ad38 0028d530 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x5b252
001b3158 10126e65 0028d530 00cd5b70 0fd7542c Core!FRegistryObjectInfo::FRegistryObjectInfo+0x6375b
001b3408 1015044c 0028d530 0fd75000 1057a628 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x23a02
001b344c 10369b02 0fd75024 0028d530 0fd75000 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4cfe9
001b347c 1015e6b5 0028d530 001b3774 00cd5b70 Engine!AInventory::operator new+0x65a9c
001b34b0 10166bbe 1057a628 1012ad38 0028d530 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x5b252
001b34ec 10126e65 0028d530 00cd5b70 0fd74f2c Core!FRegistryObjectInfo::FRegistryObjectInfo+0x6375b
001b379c 1015044c 0028d530 0fd74b00 1057a628 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x23a02
001b37e0 10369b02 0fd74b24 0028d530 0fd74b00 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4cfe9
001b3810 1015e6b5 0028d530 001b3b08 00cd5b70 Engine!AInventory::operator new+0x65a9c
001b3844 10166bbe 1057a628 1012ad38 0028d530 Core!FRegistryObjectInfo::FRegistryObjectInfo+0x5b252
001b3880 10126e65 0028d530 00cd5b70 0fd74a2c Core!FRegistryObjectInfo::FRegistryObjectInfo+0x6375b
... (and so on, and so forth) ...

FOLLOWUP_IP: 
Core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9
1015044c eb54            jmp     Core!FRegistryObjectInfo::FRegistryObjectInfo+0x4d03f (101504a2)

SYMBOL_STACK_INDEX:  1
SYMBOL_NAME:  core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9

MODULE_NAME: Core
IMAGE_NAME:  Core.dll
FAILURE_BUCKET_ID:  STACK_OVERFLOW_c00000fd_Core.dll!FRegistryObjectInfo::FRegistryObjectInfo
BUCKET_ID:  X64_APPLICATION_FAULT_STACK_OVERFLOW_ZEROED_STACK_DETOURED_core!FRegistryObjectInfo::FRegistryObjectInfo+4cfe9
FAILURE_ID_HASH_STRING:  um:stack_overflow_c00000fd_core.dll!fregistryobjectinfo::fregistryobjectinfo
Revision.log:

Code: Select all

Log: Garbage: objects: 52918->42655; refs: 1074080
Log: Garbage: objects: 52501->0; refs: 1074080
Altogether, looks like a genuine stack overflow while processing a linked list (and not like infinite recursion, which was my original thought). There's some information about FRegistryObjectInfo at https://github.com/stephank/surreal/blo ... UnObjBas.h

I set Revision.exe to LARGEADDRESSAWARE via EDITBIN.exe, that didn't help. Shouldn't hurt either, though :-)
So I raised stack limit of Revision.exe to 2GB via EDITBIN, double the default value, and this solved the problem.

This solution is likely to work only on machines with 64-bit operating system and at least 3GB of RAM.

Commands to patch Revision.exe:

Code: Select all

EDITBIN.EXE /LARGEADDRESSAWARE Revision.exe
EDITBIN.EXE /STACK:2097152 Revision.exe
I have also included fixed Revision.exe in my mod unifiedDeusEx.
Hello, I had a similar problem not too long ago. The engine is simply overloaded, thus when reloading models while saving, it crashes. Try lowering GFX in preferences. You must get through the mission somehow, then it's fine!
User avatar
kocmo
Thug
Posts: 45
Joined: Mon Mar 17, 2014 11:51 am

Re: HDTP Beta, Shifter and New Vision

Post by kocmo »

medzernik wrote:Hello, I had a similar problem not too long ago. The engine is simply overloaded, thus when reloading models while saving, it crashes. Try lowering GFX in preferences. You must get through the mission somehow, then it's fine!
Instead of taking compromises and settling down for less, why not pursue a proper solution :-) ? Besides, I'm a sucker for visuals.

Seriously, having to mark 32-bit game binaries LARGEADDRESSAWARE is a very common thing during the last few years, once you start loading them up with mods. Off the top of my head, I can remember Fallout 3 / New Vegas, and the entire S.T.A.L.K.E.R family. 32-bit S.T.A.L.K.E.R CoP won't even run stably on its own out of the box, without any mods. 2^32 = 4294967296 = 4 GB, so let 32-bit binaries use it all :-)
User avatar
bjorn98009_91
Silhouette
Posts: 688
Joined: Thu May 08, 2008 8:17 am
Location: Hufvudstaden, Sweden
Contact:

Re: HDTP Beta, Shifter and New Vision

Post by bjorn98009_91 »

Interesting, I already knew of the large address aware feature, but the stack option was news to me. I'll make a separate 64-bit only Revision.exe so people that have 64-bits can select that and get the enhancements.

We've improved the memory scenario in Liberty Island, removed some trees and tried to optimize the maps bsp-wise. When I run it memory usage is around 790MB. We will also be doing texture culling to remove any unneeded/unused textures in the maps, that should lower RAM usage as well. Although not quite sure what goes into VRAM and regular RAM.
Producer and Quality Assurance Manager for Deus Ex: Revision.
User avatar
kocmo
Thug
Posts: 45
Joined: Mon Mar 17, 2014 11:51 am

Re: HDTP Beta, Shifter and New Vision

Post by kocmo »

bjorn98009_91 wrote:Interesting, I already knew of the large address aware feature, but the stack option was news to me. I'll make a separate 64-bit only Revision.exe so people that have 64-bits can select that and get the enhancements.
Hi, responded to you on ModDB. There's no need to ship separate binaries of Revision.exe for 32-bit and 64-bit Windows, you can ship a single one, marked LARGEADDRESSAWARE and with increased stack limit. If 32-bit Windows denies it memory allocation, Revision.exe should hopefully close with an "Out of memory" error. Theoretically :-) Of course, an unmodified Revision.exe would've already crashed from a stack overflow earlier.

In other words, a patched Revision.exe is not expected to behave worse than unpatched one under any scenario, and should behave better under many scenarios. (And, of course, creating a true 64-bit DeusEx.exe is impossible, since other Unreal Engine DLLs are 32-bit).

Not sure of the VRAM memory management details, I know Microsoft improved resource management from Direct3D 9 to Direct3D 10, but I don't remember the specifics. It could be dependent on the behavior of particular Deus Ex renderer. For what it's worth, during my Revision playthrough GPU reported VRAM usage reach up to ~1050 MB. I play with Direct3D 10 renderer, New Vision, and 4x antialiasing enabled.
Last edited by kocmo on Tue Mar 18, 2014 3:39 pm, edited 1 time in total.
User avatar
bjorn98009_91
Silhouette
Posts: 688
Joined: Thu May 08, 2008 8:17 am
Location: Hufvudstaden, Sweden
Contact:

Re: HDTP Beta, Shifter and New Vision

Post by bjorn98009_91 »

Ah. That's great to know. I've just updated my installer directory with the new .exe.

We are going to use the D3D9 renderer instead of D3D10, mainly due to bugs in it (detail textures being overridden doesn't work that well and sometimes polygons on world geometry goes black), also WinXp will then not be supported.
Producer and Quality Assurance Manager for Deus Ex: Revision.
Cybernetic pig
Illuminati
Posts: 2284
Joined: Thu Mar 08, 2012 3:21 am

Re: HDTP Beta, Shifter and New Vision

Post by Cybernetic pig »

Wow, really? That's quite the trade off. Why not offer both?

Little tip: All black textures I've come across can be fixed with deco: cage light. Remove it's collision (all props false/0) and set physics_none, then if necessary hide the light wherever such as inside geometry. If light brightness is < 32 it fails to light up the bugged textures but anything > works fine. Of course this may not be ideal but giving up DX10 sounds upsetting.
User avatar
kocmo
Thug
Posts: 45
Joined: Mon Mar 17, 2014 11:51 am

Re: HDTP Beta, Shifter and New Vision

Post by kocmo »

Cybernetic pig wrote:Wow, really? That's quite the trade off. Why not offer both?
I second that, especially as Direct3D 10 renderer is the only one to support override DDS textures.

P.S.: Bjorn, aren't you already shipping Direct3D 10 renderer with Revision 1.3.1, moreover, a customized one (changed HDR shaders)?

Regarding bugs in Direct3D 10 renderer, since its source code is available, the problem should not be unsolvable in principle :-)
User avatar
bjorn98009_91
Silhouette
Posts: 688
Joined: Thu May 08, 2008 8:17 am
Location: Hufvudstaden, Sweden
Contact:

Re: HDTP Beta, Shifter and New Vision

Post by bjorn98009_91 »

kocmo wrote:
Cybernetic pig wrote:Wow, really? That's quite the trade off. Why not offer both?
I second that, especially as Direct3D 10 renderer is the only one to support override DDS textures.

P.S.: Bjorn, aren't you already shipping Direct3D 10 renderer with Revision 1.3.1, moreover, a customized one (changed HDR shaders)?

Regarding bugs in Direct3D 10 renderer, since its source code is available, the problem should not be unsolvable in principle :-)
Yes, I know. But it's buggy (I guess it can be fixed, but who has the time or the know-how). It's also performing worser. I just tied the NFS-Unatco fight in our version of Hell's Kitchen, I get 19 FPS with the D3D10 renderer and 31 FPS with D3D9. It's sad, but right now we don't have time for that. If someone wants to download and select the D3D10 renderer that's totally possible, but we won't bundle it since we aren't satisfied with the results.
Producer and Quality Assurance Manager for Deus Ex: Revision.
User avatar
kocmo
Thug
Posts: 45
Joined: Mon Mar 17, 2014 11:51 am

Re: HDTP Beta, Shifter and New Vision

Post by kocmo »

bjorn98009_91 wrote:Yes, I know. But it's buggy (I guess it can be fixed, but who has the time or the know-how). It's also performing worser.
I, ugh, had some ideas towards improving said renderer, and was going to look at it eventually. Not that I've done any 3D programming in the past 15 years, but life is an opportunity for learning. In any case, I won't be able to get to it until summer, if at all this year.
Post Reply