Using ZIPs as WAD replacement

From 3DGE Wiki
Jump to: navigation, search
Doomwiki
For more information on this article, visit the PK3 page on the Doom Wiki.

EDGE allows using several compressed archive formats instead of the traditional WADs. Most notably:

EPK
EDGE uses a modified ZIP container named EPK (Edge PaCK), which is the recommended extension for EDGE-specific mods. The main data-file for EDGE now uses this container format (edge.epk). It is a basically a ZIP/PK3 container renamed so other ports do not accidentally load EPK files. Though this is preferred, any archive format EDGE loads will still assume that data is stored in the proper namespaces.
MiniWikipediaLogoIcon.pngZIP
A .zip file used as the container for a source port mods are usually given the .pk3 extension, instead of .zip. While there is no technical differences, this helps avoid potential confusion from users thinking the mod should be extracted from it, and also prevents Windows from presenting it as a directory. EDGE supports several ZIP archiving methods: stored (no compression), shrunk, imploded, MiniWikipediaLogoIcon.pngdeflated (the most commonly used), MiniWikipediaLogoIcon.pngbzip2 and MiniWikipediaLogoIcon.pngLZMA algorithms. Note that many ZIP utility only partially implement the ZIP standard and as a result their support is limited to stored and deflated files. This partial implementation also affects some editing tools which normally support the PK3 format, such as EDGE Builder, Doom Builder 2, and Slumped. EDGE employs PhysicsFS for archive handling.
MiniWikipediaLogoIcon.png7z
A .7z file used as the container for a EDGE mod is usually given the .pk7 extension, for similar reasons. This format offers much better compression ratio, however its reduced support, especially by modding utilities, make it less interesting overall. It also increases the memory footprint on the engine, so caution must be taken when choosing between a 7z and a PK3 format. A possibility would be to develop a mod as a PK3, then convert it to a PK7 once finished, to decrease the size of the archive to be distributed. However, keep in mind that decompression is quite slower than for zips, so loading times are increased.
PAK
A .PAK file is used as an alternative container for EDGE mods; it follows Quake's PAK convention strictly. This also works for Quake II PAKs, as well as .SIN packfiles. It is nearly functionally identical to the WAD container format. EDGE mods wishing to use a custom IWAD replacement as a container over .WADs should prefer .PAK files.

Compressed archives have many advantages over PWADs and PAKs, even for storing content for Doom:

  • Smaller file size due to compression
  • Greatly reduced memory usage during play
  • Proper use of directories instead of WAD namespaces (directories also shares the distinction of being referred to as namespaces)
  • You no longer need dedicated WAD management tools like XWE for manipulating your data.
  • More flexibility due to the use of full path names. Currently there is only limited support but it is planned to change everything that references lump names to support full paths.

How to

You can embed WADs inside the top level root of an archive. Any WAD file found in the root directory will be added to the lump directory as well. This allows loading many of the ZIPs distributed through /idgames without the need to extract them. If several wads are included in an archive, they will be loaded in alphabetical order. If maps wish to stay in the WAD format, they need to be given the explicit lump name (MAPxx, ExMx, or whatever custom format you define in Levels.ddf). They can also be extracted verbatim as uncompressed data (as in, the /maps directory would contain all map data and node information like a top level PWAD would).

WARNING: Loading WAD files inside of a ZIP or a PK3 can potentially cause memory issues to arise. EDGE will unpack each WAD file and use more memory to access the contents. This effect quickly adds up, especially in larger mods (with the exception of map wads in the maps/ folder, provided they follow the format explicitly -- see below). The bigger the mod is, the more likely it may run out of memory during play and crashing EDGE, or for memory corruption to occur. Thus, it is highly recommended to completely unpack the entire contents of any WADs inside, or part of the mod, into the sorted directories.
Furthermore, EDGE supports and promotes subdirectories under the main ones for sorting purposes (i.e. Sprites/Weapons/Cannon, Sprites/Monsters/Technodemon, etc). No extra effort is needed to indicate the sprites are present, as EDGE automatically scans all subdirectories as long as they reside in their proper namespaces. If Images are contained this way, some users wouldn’t even need to define the name entry in Images.DDF.

Unlike WADs which do not have a true directory structure, the placement of data within ZIPs is much more strictly enforced in order to allow the engine to recognize it as a default lump. Any special data must be placed in the respective directory inside the ZIP or it will not be found. For the lump name, the first 8 characters of the file name are used, the extension is stripped. So for example, to place a COALHUD into a ZIP file, you have to name your file coalhud1.TXT, COALHUD1.LMP (or whatever extension you like) and place it in "./scripts". The following sub directories are used to assign data to the existing WAD namespaces in EDGE:

Directory Description
acs/ contains ACS libraries normally found between A_START and A_END Not yet implemented!
cameras/ contains CAMDAT files for camera-man scripts.
colormaps/ contains Boom and EDGE colormaps normally found between C_START and C_END.
doom_ddf/

her_ddf/ wolf_ddf/

rott_ddf/

OPTIONAL! Can contain all DDF, RTS, and COAL scripts here. This is not explicitly needed however, since they can be added in the root directory of the ZIP all the same. These folders also exists in the root EDGE directory by default, so this is only needed if you do not want to reference your script files externally. It is recommended to use the /scripts namespace instead as EDGE will automatically use /scripts instead of the default external DDF directories.
filter/ (New from 2.1.0 Test3) contains lumps and directories that will only be loaded with specific games. Not yet implemented!
flats/ contains flats normally found between FF_START and FF_END
graphics/ All special graphics like title pictures or font characters must go here. This namespace does not exist in WADs.
hires/ contains high-resolution textures normally found between HI_START and HI_END
maps/ contains levels in form of WADs. These WADs must only contain the data for one single level (including GL nodes lumps if required). Any other data in such a WAD will be ignored. Note that the file name and not the map label inside the WAD determines how the map is named in the game, unless otherwise specified by Levels.DDF. EDGE will automatically build the GL nodes via glBSP if a map is found.
models/ contains MD2, MD3, or MD5 models. This namespace does not exist in WADs.
music/ contains all data that is used as music. This namespace does not exist in WADs.
patches/ contains patches normally found between PP_START and PP_END
scripts/ contains DDF/RTS/COAL script files. They can be named by their 8 character name, or their full names as seen in EDGE's root directory, under /doom_ddf.
skins/ contains all skins to be loaded for 3D models. This namespace does not exist in WADs.
sounds/ contains all sound files that are referenced by DDFSFX. This namespace does not exist in WADs.
sprites/ contains sprites normally found between S_START and S_END (also known as SS_START, SS_END)
textures/ contains textures normally found between TX_START and TX_END
voices/ contains Strife dialog sounds normally found between V_START and V_END
voxels/ contains voxel objects normally found between VX_START and VX_END

(Note: Any lump found in the root directory is added to the global namespace.)

Tips

  • SLADE or similar programs are recommended for creating the actual .zip file, to ensure programs like WinZIP or WinRAR do not create them with unsupported file compression. Otherwise, they may appear to be damaged, which may hinder modifications of the epk with SLADE, Slumped, etc.
  • SLADE is also capable of reading and writing the .epk archive directly, where WinZIP and WinRAR natively cannot; WinRAR/7Z can still open epks though.
  • The ZIP file's directory is sorted alphabetically before adding it to the lump directory; any WAD files in the .zip are then loaded afterwards. Keep that in mind when creating data that relies on file name ordering. For most things, there is no need to depend on ordering, with the exception of texture animations of type range that animate textures from the flats/ or textures/ subdirectories.
  • Any file not in one of the reserved directories is not added to the WAD directory and can only be used by code that looks for full path names. Currently that is only the music code but it will be gradually extended so that everything that references a lump name can do it.
  • Sprite lumps for the \ frame in a WAD (such as VILE\* for one of the Arch-Vile's healing frames) can be put in a ZIP file, the backslash character just has to be changed to a caret character (^). So, VILE^1 to VILE^8 in a ZIP file will be interpreted as the VILE\1 to VILE\8 lumps. This replacement only works for sprites, any other lump name should not contain backslashes anywhere. As it is possible to use multiple sprite names for an actor with DDF, frames beyond Z need not be used however.
  • It is strongly recommended to use the .EPK extension, not .zip or .pk3, when creating a file meant to be loaded directly into EDGE. The average user is used to opening a .zip file and extracting a .wad from it, so naming the actual game file a .zip will often cause unwanted confusion.
  • The contents of the files inside the ZIP must be identical to the lumps in a WAD. This is especially important for graphics. Even though that is the most likely format you will encounter, importing them with a WAD manager will convert them to Doom's internal format. XWE converts graphics encoded in Doom's internal format to .BMP during export. SLumpEd can export these graphics as raw Doom lumps. In order for EDGE to recognize graphics as such, they have to be either in Doom's internal format or a supported image format such as PNG.
  • Beware of hidden files! If you add a full directory, or directory tree, to a EPK, it is possible that unwanted system files (such as Windows' thumbs.db file that tends to be created in any folder containing pictures) will be included as well. They will needlessly increase the file size and can possibly trigger warning messages when loading EDGE. Slumped frequently crashes when browsing through an EPK's resources, as it expects thumbs.db to be a graphic and obtain incoherent values.


Compatibility

Currently the only editors which support this format are EDGE Builder, Doom Builder 2, SLumpEd and Slade 3, but of course you can also use any Zip tool available if you do not need access to Doom specific features like texture definitions or graphics offsets.

ZDoom, Eternity, Vavoom and Doomsday largely use PK3 files for their resources; however their subdirectory organization differ. Adapting a mod from one of these ports to work in EDGE may therefore require repackaging, in addition to conversion of enhanced features to their EDGE equivalent. The Eternity Engine, which uses the PKE format, adopted a model mostly similar to ZDoom's, but different enough for incompatibility to be possible in both ways. EDGE is also similar to ZDoom's.

See also