Home | Download | Discussion | Help | Site Map | New Posts | Sign in
MTS has all free content, all the time. Find out how YOU can help to keep it running. Tell me how...
Advertisment:

Picked upload! XML Extractor for The Sims 4

by scumbumbo Posted 28th Aug 2015 at 7:15 AM - Updated 7th Feb 2016 at 7:52 PM by scumbumbo : v3.5.1 extracts JAZZ resources
67 Comments / replies (Who?) - 15 Feedback Posts, 10 Thanks Posts
Picked Upload! This is a picked upload! It showcases some of the best talent and creativity available on MTS and in the community.
Updated 02/07/2016 - Version 3.5.1 has an option to extract the ASM/JAZZ (type 0x02D5DF13 resources) XML from the game packages. These are placed in the AsmJazz subfolders of a game pack's extract. Note that many of the game pack deltas don't have any of these resources, so if you don't own that game pack only those which have been changed by a game patch and thus are present in the delta can be extracted.

ASM/JAZZ extracted from a single package will have no white space, similar to the output from the process previously posted by Plasticbox. When performing a full extract, the XML is reformatted nicely with line breaks and indentation.

Version 3.5.0 is available which adds major improvements to speed when performing a full extract by using RAM instead of temporary files, and allowing game packs to be processed in parallel using multiple threads. Selecting the maximum number of simultaneous threads to use is easily selectable in a combobox, it defaults to 1/2 of your available "logical CPUs" which typically will mean one thread per CPU core. Running more threads than suggested typically will not gain significant speed increases (only a second, two or three in my testing).

On a relatively fast quad-core machine with 16 GB of RAM, my time to perform an extract was reduced from an average of 3 minutes 10 seconds, to 1 minute 8 seconds! If my math is right, that's a 65% improvement in speed!

Note - this may actually be slower for machines with low RAM, even if you select only to run an extract with only a single thread. The old version will continue to be available until it is known how the new version performs on machines with less than 4GB of RAM. Anyone with four or less gigs of RAM please let me know how well it performs for you!

Description

This is a utility program to extract the game's XML files for use in creating mods. XML can be extracted from the installed game packages or a single combined tuning file. The program can also easily be used to extract a single binary tuning file from the game packages and convert that to a combined XML tuning file.

The program is designed to perform these functions quickly by only loading those parts of the game packages that are required for producing the XML and looking ahead to quickly build required lookup tables.

When extracting the XML in Full Automatic mode (the recommended and default mode) the game's strings and references to other XML resource names can be included in comments. This feature is not available when working with a single combined tuning file. If you have multiple game language packages installed, you can select which language you want the XML comments to use.

Also in Full Automatic, the program can optionally output three tab delimited text files with all strings and XML resource names. The two reference files are suitable for opening in a spreadsheet package for easy searching, and the XML File Index can be loaded into the companion XML File Finder program to quickly locate files for viewing and editing.

There are three naming conventions available for the final XML files:
  • Type, Group and Instance (TGI) for quick import into S4PE (e.g. S4_E882D22F_00000000_000000000001A2BC.xml)
  • TGI plus resource name, formatted to automatically create a _KEY Name when imported in S4PE (e.g. S4_B61DE6B4_00000001_000000000001C8C5_object_drink_Cider.xml)
  • Native TS4 naming format using the TGI (e.g. 0x00000000!0x00000000000029c3.0x2451c101)
  • Or just the resource name

Requirements

The program requires Windows XP or higher with the .Net Framework 4.5.1.

The program requires access to an installed copy of the game in order to extract the XML from the packages. To work on a machine that does not have the game installed you would need to copy the required packages, binary or combined tuning files to the PC you need to work on. These files cannot be provided by the author for copyright reasons.

Basic Usage
  1. Launch the program. This step is important!
  2. Select the Full Automatic Extract mode. This should be selected by default.
  3. The program should automatically locate your Sims 4 install folder. If not, you can browse manually for it (this setting is saved when you exit the program).
  4. Choose the options for the extract. Again, these options will be saved for the next time you use the program.
  5. Under Output Options, select the Destination Folder for the XML files. You should create a new empty work folder for your first time. Subfolders will automatically be created for each game pack (see screenshot).
  6. By default, the program will ask you before overwriting any files. You can also choose automatic overwrite at this time, which is useful if you're performing a full extract overr an existing extract (to avoid being prompted billions and billions of times). If you forget, you can select to "Stop Asking Me" when the overwrite dialog appears.
  7. If you choose to rename existing files, any file will be automatically renamed with (Copy #) appended to that file rather than overwriting. You will be warned if you choose this option as it is very slow if doing a full extract over an existing extract (30,000+ files would need to be renamed).
  8. Click the Begin Extracting button and wait. The program only updates it's user interface once for every file, so for the long steps it may appear to hang. Just wait and it will move on. The entire extraction process should take less than 3 minutes on a fast PC. Including strings, XML references and the file index will take only a few extra seconds!

File Finder Usage

In order to use the file finder, you must have extracted the full XML using the "Create XML File Index" option. This option is on by default.

If you installed using the MSI file, a hot key is registered with Windows to launch the File Finder application using the Ctrl-Alt-F keys. Note that this key sequence will not be active until Windows Explorer is restarted - the easiest way to do this is to logout and log back in after installing. You can change this key sequence if desired by editing the properties of the XML File Finder application in your Start menu.

The first time you launch the file finder, you will need to load an index file produced by the extractor. This file should be in the output folder of the XML extract in a text file called "XML File Index". Since the full path to the XML files is included inside the file index, if you move the XML files to another location on your hard drive you will have to recreate the index file using the extractor.

Once an index file is loaded, simply type the search terms you wish to look for in the Search For field. There is no need to hit Enter to search, if you pause typing for 1/2 second the search will being. You can continue typing to modify the search. Searches are not case sensitive.

So for instance, when looking for interactions dealing with the piano, you might start by typing "interaction". The program will find (currently) 7,720 matches, which is a bit much to look through, so add a space and type "piano" which yields 53 matches. You can keep adding terms until you find what you are looking for, so if you then added "pract" to the search terms you would see 6 matches.

The search can find files using portions of the TGI, so if you're looking for the 0x7B2E action (loot_Buff_Computer_HighScore_Typing) you could find this using the terms from the name, or by just typing "7b2e" (which also finds the interaction 0x17B2E).

The instance is also stored for searching in decimal form, so if you see the broadcaster type "129459" in an XML file which doesn't have comments, you can easily find that file by just typing "129459" into your search terms to find "broadcaster_Jealousy_ClubEncouraged".

Once you have a short enough list of files that you can find what you're looking for, you can simply right-click the file in the list to select either "Open in Windows Explorer" or "Open in XML Editor".

Version History
  • Version 3.3.1 - Initial release
  • Version 3.3.2 (August 29, 2015)
    • Added language choice for string comments if multiple language packages available.
    • Added native XML naming convention
  • Version 3.3.3 (January 14, 2016)
    • Bugfix: Missing or invalid XML extracts will get placed into an error directory rather than crashing
    • Bugfix: Empty XML attribute values will correctly generate an empty string on output
  • Version 3.4.0 (January 17, 2016)
    • XML Extractor can produce a file index when performing a full extract
    • Added XML File Finder program to load file index and quickly search for XML files
  • Version 3.4.1 (January 18, 2016)
    • Bugfix: Warn if a game package contains no binary tuning instead of crashing.
  • Version 3.4.2 (February 3, 2016)
    • Option to group results in File Finder by resource type
    • Results in File Finder are sorted by name
    • The input focus in the File Finder defaults to the search terms input
  • Version 3.5.0 (February 5, 2016)
    • Uses RAM instead of temporary files for the conversion process.
    • Can process each game pack in a separate thread to use multiple CPU cores.
    • Maximum number of simultaneous threads is easily selectable.
    • Added package name for SP05
  • Version 3.5.1 (February 7, 2016)
    • Option added to unpack ASM/JAZZ resources.
    • Exiting the program during a full extract is now (a slight bit) cleaner.

Additional Credits
  • Velocitygrass for the original XML Extractor and source code.
  • SimGuruModSquad for obtaining permission to share the game's PackedXmlDocument source code with me.
  • Too many people to mention by name at Mod The Sims who have tested and given suggestions. But thanks to all of you, feel free to visually insert your name here!
  • Caphyon Ltd. for the freeware Advanced Installer used to create the install file.
  • Microsoft for the Visual Studio Express IDE for .Net.

Operating System: Windows
Utility Type: Miscellaneous
Plugin Type: Not Applicable

You must have the expansion or stuff packs listed below installed to use this custom content. Please see the post text for any exceptions.

Sims 4
Key: - File was updated after upload was posted
Filename - Tip: You can click the magnifying glass to see the archive contents Size Downloads Date
XML Extractor for The Sims 4 (MSI Installer).zip | Description: v3.4.2 (no threads) - Installer for the XML Extractor 264.1 KB 1,001 3rd Feb 2016
XML Extractor for The Sims 4 (EXE Only).zip | Description: v3.4.2 (no threads) - The EXE files only if you dont want an installer 145.8 KB 927 3rd Feb 2016
TS4 Combined XML Extractor (VS 2013 Source Code).rar | Description: v3.4.2 (no threads) - Visual Basic source code for the project (v 3.4.2) 197.3 KB 481 3rd Feb 2016
Threaded XML Extractor for The Sims 4 (MSI Installer).zip | Description: Latest Version - Installer for the XML Extractor 268.7 KB 25 7th Feb 2016
Threaded XML Extractor for The Sims 4 (EXE Only).zip | Description: Latest Version - The EXE files only if you dont want an installer 150.1 KB 50 7th Feb 2016
Threaded TS4 Combined XML Extractor (VS 2013 Source Code).rar | Description: Latest Version - Visual Basic source code for the project (v 3.5.1) 203.8 KB 21 7th Feb 2016
Don't forget to Thank the creator after downloading! Use the Thanks button located above the screenshots above.
Basic Download and Install Instructions:

1. Download: Click the download link to save the .rar or .zip file(s) to your computer.

2. Extract: Use 7zip (recommended), WinRAR (Windows) or Stuffit (OS X) to extract the file(s) from the .rar or .zip file(s). OS X also has a built-in utility for that (just rightclick).

3. Download and unzip either the MSI installer or EXE-only version of the program.

Running the MSI installer will automatically create a program folder and shortcut in the Start menu. If a new version of the program is released, it should upgrade over the original program automatically. If the installation is somehow corrupted, a repair option (and uninstall) is available in Windows Control Panel.

The EXE-only file may be placed anywhere you like, no installation is required and no additional files will be created.

Need more information?
 
Page 1 of 3
67 Comments / Replies (Who?) - 49 Feedback Posts, 15 Thanks Posts
Hide Thanks Posts for this thread (Show only feedback)
Top Secret Researcher
THANKS POST
#2 Old 28th Aug 2015 at 7:23 AM
I know I will need this someday. Thank you!
Smeg Head
THANKS POST
#3 Old 28th Aug 2015 at 7:33 AM
If anything is going to get me started on the modding trail again, it will be this. Nice one, scumbumbo.
Lab Assistant
THANKS POST
#4 Old 28th Aug 2015 at 11:48 AM
Thank you so much for this program, I 'm going to last, it's great
Test Subject
#5 Old 28th Aug 2015 at 5:26 PM
Does anyone even use the 0x<group>!0x<instance>.0x<type> format that the game can natively read from outside .package files? I'm not sure I've seen anyone besides myself using it.
Not exactly a request to add it, mind; I have my own set of scripts to handle converting Sims 4-related file names, and they already handle the format you have used. It's more that I'm curious about whether other people make use of it or not.
Field Researcher
THANKS POST
#6 Old 28th Aug 2015 at 5:59 PM
A must-have for modders! Thank you! :D
Field Researcher
#7 Old 28th Aug 2015 at 6:34 PM Last edited by egureh : 28th Aug 2015 at 6:48 PM.
Hi scumbumbo, I am curious about how does this extractor choose string file.
I am using English version Windows 10, my Origin client setting and the language setting of TS4 are both en-US.
But I got a copy of Chinese strings after extracting... and all the string comments of xml files are in Chinese.
The only Chinese relevant setting I made to my system is Country and Region, which I set to China, will this affect the extractor?
Mad Poster
THANKS POST
#8 Old 29th Aug 2015 at 3:03 AM
Thanks! I'm sure I'll be making use of this! :)
Top Secret Researcher
Original Poster
#9 Old 29th Aug 2015 at 3:16 AM
Quote:
Originally Posted by DarkWalker
Does anyone even use the 0x<group>!0x<instance>.0x<type> format that the game can natively read from outside .package files?

I'm familiar with that numbering but didn't realize it would work outside of a package file. I wonder if that format can be reloaded in game then like you can with a loose file script?

It would be easy enough for me to add that naming selection since it sounds like it could be useful. The next time I update the program that will be included.
Top Secret Researcher
Original Poster
#10 Old 29th Aug 2015 at 3:27 AM
Quote:
Originally Posted by egureh
Hi scumbumbo, I am curious about how does this extractor choose string file.
I am using English version Windows 10, my Origin client setting and the language setting of TS4 are both en-US.
But I got a copy of Chinese strings after extracting... and all the string comments of xml files are in Chinese.
The only Chinese relevant setting I made to my system is Country and Region, which I set to China, will this affect the extractor?

Hmmm, interesting - you must have both the Chinese and English string packages? The program just looks for all of the Strings_*.package files and if a string hash is already in the index then it is ignored, so in the case of multiple string package files whichever of the filenames comes first would be the language the strings end up in. It doesn't look at the game settings at all.

I'll give it some thought about how best to alter that situation.
Field Researcher
#11 Old 29th Aug 2015 at 4:05 AM
Quote:
Originally Posted by scumbumbo
Hmmm, interesting - you must have both the Chinese and English string packages? The program just looks for all of the Strings_*.package files and if a string hash is already in the index then it is ignored, so in the case of multiple string package files whichever of the filenames comes first would be the language the strings end up in. It doesn't look at the game settings at all.
I'll give it some thought about how best to alter that situation.

So I understand, I do have all the 17 language packages, and the program alphabetically reads Strings_CHT_CN.package first.
Yah, I try to delete other 16 string packages, leaving only Strings_ENG_US.package, and the extracting result is perfectly fine now.
Thank you! :D
Top Secret Researcher
Original Poster
#12 Old 29th Aug 2015 at 4:10 AM
@DarkWalker - Regarding the 0x<group>!0x<instance>.0x<type> format, the files I'm seeing from EA that use this convention do not have a "0x" before the type, they just use 0x<group>!0x<instance>.<type>. I'm going to assume that the notation you originally typed is what works for you and include that 0x before type, but if that was a typo let me know and I'll fix that.
Top Secret Researcher
Original Poster
#13 Old 29th Aug 2015 at 4:11 AM
Quote:
Originally Posted by egureh
So I understand, I do have all the 17 language packages, and the program alphabetically reads Strings_CHT_CN.package first.
Yah, I try to delete other 16 string packages, leaving only Strings_ENG_US.package, and the extracting result is perfectly fine now.
Thank you! :D

No worries, glad it's working for you now but I have a plan of attack already on a proper fix so you don't have to go through all that

ETA - @egureh - How did you get all the different string files like that? Would be useful for me for testing to be able to do the same. I'll Google it, but if you see this first....
Field Researcher
#14 Old 29th Aug 2015 at 4:46 AM
Quote:
Originally Posted by scumbumbo
No worries, glad it's working for you now but I have a plan of attack already on a proper fix so you don't have to go through all that
ETA - @egureh - How did you get all the different string files like that? Would be useful for me for testing to be able to do the same. I'll Google it, but if you see this first....

Well the story is a bit long, thanks to EA we do not have Origin server in China mainland, the download speed in Origin client is kind of terrible.
So I use a sniffer tool to get the direct download link and download the game via a download tool, which could reach my max network speed.
And this will download the whole game as a zip file, ignoring language setting, so here we go, I got all the 17 string packages.
Anyway, as long as you purchased a world wild version, you could get the multilanguage zip file. Hmm, seems the story is not that long.
Top Secret Researcher
Original Poster
#15 Old 29th Aug 2015 at 5:01 AM
Quote:
Originally Posted by egureh
Well the story is a bit long

Interesting

Sounds like more work than I need to do though. I was just curious how well the extract worked with the strings in a different language, and you've really answered that anyway - it sounds like that does work.
Field Researcher
#16 Old 29th Aug 2015 at 5:05 AM
Quote:
Originally Posted by scumbumbo
Interesting
Sounds like more work than I need to do though. I was just curious how well the extract worked with the strings in a different language, and you've really answered that anyway - it sounds like that does work.

Yep, I guess if the program only reads Strings_ENG_US.package then there is no such problem.
Top Secret Researcher
Original Poster
#17 Old 29th Aug 2015 at 5:15 AM
Quote:
Originally Posted by egureh
Yep, I guess if the program only reads Strings_ENG_US.package then there is no such problem.

The next version will scan the installed Strings_* packages at startup and give you a combobox to select which language package you want to extract the strings from.

With both that and the new file name pattern, I will probably get it posted ASAP, maybe later tonight yet.
Field Researcher
#18 Old 29th Aug 2015 at 5:42 AM
Quote:
Originally Posted by scumbumbo
The next version will scan the installed Strings_* packages at startup and give you a combobox to select which language package you want to extract the strings from.
With both that and the new file name pattern, I will probably get it posted ASAP, maybe later tonight yet.

Sounds cool! :D
Test Subject
#19 Old 29th Aug 2015 at 7:17 AM
Quote:
Originally Posted by scumbumbo
@DarkWalker - Regarding the 0x<group>!0x<instance>.0x<type> format, the files I'm seeing from EA that use this convention do not have a "0x" before the type, they just use 0x<group>!0x<instance>.<type>. I'm going to assume that the notation you originally typed is what works for you and include that 0x before type, but if that was a typo let me know and I'll fix that.


Well, I've always used 0x before the type when representing it as a hex number, I didn't even know it could be left out.

It's possible, though, to use non-numeric types by adding them to the resources.cfg file, in which case the 0x should be left out; I do this for my personal use, as it makes finding the correct file to edit easier. I can provide instructions and examples later (when I'm not posting from my mobile) if you are interested.
Top Secret Researcher
Original Poster
#20 Old 29th Aug 2015 at 7:24 AM
Version 3.3.2 with the above mentioned additions has been uploaded. I have not updated the source code as these changes are fairly minor.
Field Researcher
THANKS POST
#21 Old 29th Aug 2015 at 7:55 AM
Thanks for updating!
Lab Assistant
THANKS POST
#22 Old 6th Sep 2015 at 6:33 AM
Hey thank you!
Test Subject
THANKS POST
#23 Old 23rd Sep 2015 at 2:38 AM
A 1000 thanks for this! It's absolutely perfect and I love it.
just a girl
THANKS POST
#24 Old 13th Oct 2015 at 10:41 PM
Thank you very much! You made it so easy to use, the thing does all by itself =)
Mad Poster
THANKS POST
#25 Old 4th Dec 2015 at 9:42 PM Last edited by Deaderpool : 4th Dec 2015 at 10:30 PM.
Is it just me or is this broken as-of today's patch? Not even sure if @Scumbumbo is still around, but thought I'd ask.
Page 1 of 3