(TODO)

These are global settings applied to the entire VFX. Parameters such as Revised Scale and Revised Color can be used to adjust the size and color of an entire VFX without having to manually change the values of each individual component.

Also of note are the DrawLayer parameter, which controls when the VFX is drawn (such as BeforeSky or BeforeCloud), and the Clipbox settings.

Welcome to the XIV Dev Wiki. The idea is for this to be a single place where everything is located in a easy to access and searchable manner. Meta discussion and questions should be via issues on the GitHub repo that this documentation is pulled from: https://github.com/xivdev/docs/.

Final Fantasy XIV © 2010-2022 SQUARE ENIX CO., LTD. All Rights Reserved. We are not affiliated with SQUARE ENIX CO., LTD. in any way.

Every VFX has exactly 1 Scheduler.

Schedulers control when Timelines are started, and have 2 components: a list of items, and a list of triggers. A scheduler can have an arbitrary number of items, but must have 12 triggers. Both items and triggers follow the same format:

bool Enabled
int StartTime
int TimelineIndex

Triggers

What the triggers represent is not understood, but they are probably each for a specific type of action, such as sheathing a weapon, getting hit, etc. Unused triggers have TimelineIndex = -1.

Items

Items function like triggers, but will start automatically without any additional input. If you are having trouble viewing a VFX, such as one which was originally used for a cutscene, it can often be useful to disable all the triggers, and instead exclusively use items to start timelines.

RSF is very similar to RSV except it works on file level.

Pseudocode

The procedure to generate a RSF file is as follows:

def create_rsf(plaintext):
    deflated_bytes = zlib.compress(plaintext, level=9)
    
    # Server stores this part and will send it to the client when zoning-in
    header = deflated_bytes[:64]
    
    # Client stores this part
    body = deflated_bytes[64:]

*[Last updated FFXIV 6.58]

This folder, called FFXIV_CHRxxxx where xxxx is a unique ID for the character, contains all of the client side data related to one or another of the characters played.

Following is a list of the data files found in this folder.

Note: most of the information in here is taken from without doublechekcing for now.

TODO: figure out the format for each file.

1. The client uses to generate an arbitrary timestamp, which it converts roughly into milliseconds by dividing it by 10,000.

2. An IPC message is sent by the client which includes the timestamp encoded as a 32-bit integer value.

3. An IPC message is sent back by the server with the same timestamp.

A timeline consists of a list of items, and a list of clips.

Items represent the creation of an emitter, while clips represent other actions, such as deleting particles. Although clips are not well understood, the are probably used for things such as hiding a vfx when a weapon is sheathed.

Clips must be triggered from items using the ClipNumber parameter. Usually, items which trigger clips will have EmitterIndex = -1.

Items are also where emitters are attached to binders and effectors using BinderIndex, EmitterIndex, and EffectorIndex

Emitters, as their name suggests, emit objects (either other emitters, or particles). They are triggered through timeline items, and can be moved in 3D space just like any other object. Unlike particles, emitters themselves are not visible. Emitters can also trigger a sound to play.

When an emitter creates another object, it can influence the properties of its child (such as color, scale, or position). Whether or not an emitter will influence its children is determined by parameters such as InfluenceCoordScale.

There are several types of emitters:

• Point

Effectors apply miscellaneous effects to emitters, particles, or the camera. In order to be used, they must either be triggered through a , or an .

There are several types of effectors:

• Point light

• Directional light

Excel is the relational data storage system that the game uses to store information, similarly to a relational database. The structure of EXL, EXH and EXD files are documented here.

Data files are xxx.dat files generally located in the XIV client folder and sub folders at <..>\Documents\My Games\FINAL FANTASY XIV - A Realm Reborn\.

The FFXIV Character Creator saves presets, when saving a character appearance at the end of character creation, in a file called FFXIV_CHARA_XX.dat where XX is the preset slot number from 01 to 40.

All data related to characters that have been used on the current machine is saved in a folder called

Binders determine where a vfx should be created, and its tracking behavior (should it follow a moving character).

There are several types of binders:

• Point (the most common, attach to a character, for example)

• Camera

(TODO)

Life controls how long a particle lives, and can be somewhat counter-intuitive. Life = -1 means that the emitter lives forever, so it's animations will only trigger once. If you want an emitter to continuously loop (that is, die and get re-created), set it a value greater than -1.

(TODO)

Few things to note:

• Just like Super Mario 64, y component of coordinate represents altitude. Think z component as depth as in Z-buffer.

(TODO)

Particles are the visible component of a VFX, and are created exclusively by emitters. They are similar to emitters in that they can be positioned in 3D space.

There are several types of particles:

• Decal

• Decal Ring

• Disc

• Laser

• Light Model

• Line

• Model

• Parameter

• Polygon

• Polyline (line made of )

• Powder (used for small particles, such as the dust which some weapons create)

• Windmill

• Quad (a 1x1 square)

Particles can have textures mapped onto them using several different types of nodes (covered later).

Sometimes, a particle does not require complex animations, and can use the SimpleAnimations node instead (in order for it to work, the SimpleAnimationsEnabled must be True) This is often used for Powder particles, which often have very basic animations due to their size and number.

There are several types of texture nodes which can be applied to a particle. Most texture nodes will have a TextureIndex. This determines which texture will be used as a color, normal map, etc. TextureColor1 is especially important as it has MaskTextureIndex, which determines which parts of the particle will be transparent.

Most texture nodes will also have a UVSetIndex (UV Sets are discussed later).

UV Sets are contained within particles, and are used to manipulate textures. This primarily involves scaling, rotating, and offsetting a texture. For example, most aura or fire VFXs will use a UV Set to animate a texture so that it "scrolls" across the particle, creating the illusion of a moving flame or energy aura.

This is a list of various open source resources and projects related to FFXIV. Please if your project is listed here and you'd like to remove it.

Upon launching the game, the retail launcher passes handful of information (e.g. login token, client language, etc...) to the game that usually looks like this:

• Encrypt UTF-8 plain text using Blowfish; you need to pad the buffer with zero as needed

  • Key is GetTickCount() & 0xFFFF_0000

The packet sent from the server upon entering the instance is as follows:

// sent for each rsv string
struct RsvPkt {
  UInt32 value_size;    // len(without_nul_char(value))
  char key_str[32];     // null terminated
  char value_str[1024]; // null terminated
}

Pseudocode

Links

• https://github.com/NotAdam/Lumina/blob/0530a97d21/src/Lumina/Excel/RSV/RsvProvider.cs

Animation lock is an internal timer that player has to wait certain amount of time before they are allowed to use any actions.

Basics

• Timer counts toward zero. (i.e. it represents remaining time)

• Many GCDs have 0.1s animation lock. This timer is while casting. It is resumed after player finishes casting on client side.

• oGCDs and instant skills have 0.5s initial animation lock.

• Server can also set animation lock in response to actions used from the client. (usually by the packet commonly called "effect packet") However, it simply discards any previous timer running. Therefore, game does not compensate for player's ping. (i.e. time already spent during initial animation lock)

• You can't target ground while animation lock is active.

When those three above combined, it can greatly deter game-play experience in certain condition. This will be discussed below.

• Some actions like limit break, foods and duty actions may lock for different amount of time. For example, movement skill locks for 0.8 seconds and items have 2.0 seconds.

Since animation lock from the server overwrites curent animation lock to new value, it is this reason why double weaving between GCDs extremely unliable when player is connected to the datacentre across continent.

• If the client fail to receive response during initial animation lock, client is allowed to use next action. TODO.

While Swiftcast is one of notorious offender, this property is not limited to Swiftcast only. For example, Red Mage's Dualcast just works same as hardcasting Swiftcast.

TODO: rescue lb3

TODO: it is possible to cancel oGCD 0.6s animlock by letting GCD overwrites it on high ping environment. (something something related to hardcasting swiftcast)

In most zones, weather is pseudorandom and can be derived from the current time. Each zone has a weather table, which contains the possible weather states for the zone and the stacked probability of that weather being active.

Target value

The value that the pseudorandom number generator produces is referred to as the target value. This value can be calculated deterministically for any given instant. It will always be in the range [0, 99].

Calculation

Given a UNIX timestamp unixSeconds:

int32_t CalculateTarget(int32_t unixSeconds) {
    // Get Eorzea hour for weather start
    int32_t bell = unixSeconds / 175;

    // For these calculations, 16:00 is 0, 00:00 is 8 and 08:00 is 16.
    // This rotates the time accordingly and holds onto the leftover time
    // to add back in later on.
    uint32_t increment = ((uint32_t)(bell + 8 - (bell % 8))) % 24;

    // Take Eorzea days since the unix epoch
    uint32_t totalDays = (uint32_t)(unixSeconds / 4200);

    uint32_t calcBase = (totalDays * 0x64) + increment;
    uint32_t step1 = (calcBase << 0xB) ^ calcBase;
    uint32_t step2 = (step1 >> 8) ^ step1;
    return (int32_t)(step2 % 0x64);
}

Weather lookup

Once the target value has been calculated, the WeatherRate row for the current TerritoryType is retrieved. The rates of each entry in the row are accumulated until the target value is less than the current value of the rate accumulator. At this point, the current weather rate entry is selected, and the corresponding weather ID denotes the Weather row that should be used.

Territories can be queried by name using their PlaceName attribute. Territories have a WeatherRate that describes the weather rate row they use. Using the process described above, the current Weather being used can be determined.

Some zones, such as the original version of Diadem, use the message to control what weather the player experiences. It is not known what zones currently use this message for weather.

The character creator saves it's presets in binairy files called FFXIV_CHARA_XX.dat where XX is the preset slot number. These files are located in the <..>\Documents\My Games\FINAL FANTASY XIV - A Realm Reborn\ folder.

Note: The investigation into this format has been made in a very cursory manner. There is no garantie of accuracy.

[Last updated: FFXIV 7.0 Benchmark]

The file is read in little endian format.

The file contains 4 sections:

We'll only cover version 3 for now.

Intro

This patch format has been extant since FFXIV 1.0, but has had a few modifications since then, especially from 2.0 onward.

A patch file name generally starts with either H or D, for HIST and DIFF, respectively, followed by a version string, such as 2013.08.09.0001.0000. This is optionally followed by a lowercase letter (starting at 'a') in case the patch file corresponding to that version string would be larger than 1.5GB for HIST files. e.g.: H2013.08.09.0001.0000a.patch, D2019.05.29.0000.0000.patch.

Patch Naming Convention

Patch files follow the following naming convention:

<D|H><YEAR>.<MONTH>.<DAY>.<PART>.<REVISION>.patch

Where D|H indicates that a patch is a delta patch or a history patch respectively.

Some example patch names are as follows:

• Byte order, unless explicitly mentioned, is big endian order. That is, 0x00AB_CDEF in hexadecimal will be encoded as 00 AB CD EF in file - not EF CD AB 00 which is little endian.

All patch files start with 91 5A 49 50 41 54 43 48 0D 0A 1A 0A.

The ZiPatch signature is followed by a contiguous array of chunks with the following structure

Where Header.Size indicates the size of Chunk.Payload, and Header.Name is a 4-character string indicating the type of the chunk, e.g., "FHDR", "APLY", and "SQPK".

Following Header, there's a Chunk describing the chunk contents, which will be detailed in the following pages. Following the chunk payload, there is a CRC32 of Header.Name with Payload.

This contiguous array of chunks terminates with a chunk of Name "EOF_".

FHDR is always the first chunk in the file, containing a few unused fields and some information about the current patch file (such as number of commands, etc.)

FhdrChunk.Name has been observed to be HIST for history patches, or DIFF for delta patches.

The two currently known Options, Ignore Missing and Ignore Mismatch are normally set to false via two APLY chunks set right after the FHDR. This means that the official patcher fails if a file it expects there is missing or some data it expects to be there is not.

Creates directory at Path. Path is relative to the game folder plus the patch type being applied (boot or game).

e.g.: If Path = "movie\ex1", and we are patching 'game' at "C:\Games\SquareEnix\FINAL FANTASY XIV - A Realm Reborn\", we would create the folder "ex1" at location "C:\Games\SquareEnix\FINAL FANTASY XIV - A Realm Reborn\game\movie\"

Similar to ADIR, but the folder at Path is deleted instead. It only deletes empty folders.

Indicates no chunks should be processed after this.

Primitives

[StructLayout(LayoutKind.Explicit, Size = 5)]
struct SqpkChunk
{
    [FieldOffset(0x0)] Uint32BE Size;
    [FieldOffset(0x4)] SqpkOperation Opcode;
    [FieldOffset(0x5)] SqpkPayload Payload;
}

enum SqpkOperation : byte
{
    A, // Add data. Used to overwrite blocks in dat files. Can also delete old data
    D, // Delete data. Used to delete blocks (overwrite with 0x00) in dat files
    E, // Expand data. Used to insert empty blocks into dat files
    F, // File operations. Adding files, deleting files, etc.
    H, // Header Update. Updates headers of dat or index files
    I, // Index Add/Delete. Present in patch files but unused
    X, // Patch Info. Present in patch files but unused
    T, // Target Info. Present in patch files but only one field is used
}

SqpkChunk is a 'container' chunk which executes one of several operations in SqpkOperation. These will be detailed in the following subsections.

Important to note that any fields prepended with "Block" will mean that the number is in blocks, or units of 128-bytes. e.g., BlockOffset means that the offset in bytes would be calculated by BlockOffset << 7 or BlockOffset * 128.

These operations often contain fields indicating which file the operation will apply to. These adhere to the following structure:

[StructLayout(LayoutKind.Explicit, Size = 8)]
struct SqpkFile
{
    [FieldOffset(0x0)] UInt16BE MainId;
    [FieldOffset(0x2)] UInt16BE SubId;
    [FieldOffset(0x4)] UInt32BE FileId;

    // Expansion => (byte)(SubId >> 8);
    // DatFileName => $"{MainId:x2}{SubId:x4}.win32.dat{FileId}"
    // IndexFileName => $"{MainId:x2}{SubId:x4}.win32.index{(FileId == 0 ? string.Empty : FileId.ToString())}"
}

When referring to block headers, the following struct will be relevant:

[StructLayout(LayoutKind.Explicit, Size = 128)]
struct BlockHeader
{
    [FieldOffset(0x0)] UInt32LE Size;
    [FieldOffset(0x4)] UInt32LE Type;
    [FieldOffset(0x8)] UInt32LE FileSize;
    [FieldOffset(0xC)] UInt32LE TotalBlocks;
    [FieldOffset(0x10)] UInt32LE UsedBlocks;
}

struct BlockCount {
  [FieldOffset(0x00)] UInt32BE Value;

  UInt32 InBytes => Value * 128;
}

struct SqpkFilePath {
  [FieldOffset(0x00)] UInt16BE Main;
  [FieldOffset(0x02)] UInt32BE Sub;
  [FieldOffset(0x04)] UInt32BE File;
}

struct SqpkFileRange {
  [FieldOffset(0x00)] BlockCount Offset;
  [FieldOffset(0x04)] BlockCount Count;
}

Types

Type A - Add Data

[StructLayout(LayoutKind.Explicit)]
struct SqpkAddData
{
    [FieldOffset(0x0)] byte reserved[3];
    [FieldOffset(0x3)] SqpkFile File; // Dat file
    [FieldOffset(0xB)] UInt32BE BlockOffset;
    [FieldOffset(0xF)] UInt32BE BlockCount;
    [FieldOffset(0x13)] UInt32BE BlockDeleteCount;
    [FieldOffset(0x17)] byte Data[BlockCount << 7]; // TODO: Not valid C#
}

This operation writes SqpkAddData.Data to the dat File, at BlockOffset. It will then write BlockDeleteCount blocks of empty (0x00) 128-bytes.

Type D - Delete Data

[StructLayout(LayoutKind.Explicit, Size = 19)]
struct SqpkDeleteData
{
    [FieldOffset(0x0)] byte reserved[3];
    [FieldOffset(0x3)] SqpkFile File; // Dat file
    [FieldOffset(0xB)] UInt32BE BlockOffset;
    [FieldOffset(0xF)] UInt32BE BlockCount;
}

This operation writes BlockCount empty (0x00) 128-byte blocks at BlockOffset, with the first of those blocks containing a BlockHeader struct with:Size = 128, Type = FileSize = UsedBlocks = 0, TotalBlocks = BlockCount

If Delete Data encounters EOF while writing or seeking to BlockOffset, it will fail.

Expand Data follows the same structure as Delete Data, but the behaviour differs. When Expand Data seeks past the end of file or writes past the end of file, it does not fail, since this is its intended use.

Notes:

• FilePath is a null terminated utf-8 string. Length of the string is designated by FileHeader.FilePathSize

Notes:

• If blkHeader.Compressed is true then the payload is a deflated stream.

• If blkHeader.Compressed is false then the payload is a raw bytes stream.

• Yes, AlignedBlockSize is very dodgy. Apparently they are rounded up to next multiple of 128 except they're not.

This is currently no-op.

This is currently no-op.

Game packets are composed of 3 distinct parts, the packet header, the segment header and a (sometimes) optional IPC header.

Packet Header

struct FFXIVARR_PACKET_HEADER
{
  uint64_t magic[2];
  uint64_t timestamp;
  uint32_t size;
  uint16_t connectionType;
  uint16_t segmentCount;
  uint8_t unknown_20;
  uint8_t isCompressed;
  uint32_t unknown_24;
};

Only present when the parent segment type is set to SEGMENTTYPE_IPC.

Decoding packets is reasonably simple and assuming you have a buffer that you write data in to for each connection, it's something like the following:

A lot of detail is omitted for brevity, but it's generally pretty straightforward.

A more comprehensive example of packet parsing can be found in Sapphire:

• (and some container-level parsing)

AVFX files are organized as a nested series of blocks, with each block following the format:

The contents of a block can further contain other blocks. It is also worth noting that the name of a block is reversed from its actual meaning. For example, the "top-level" block for an AVFX file is "XFVA"(AVFX reversed). Names which are fewer than 4 character are also padded out to 4 bytes. Blocks are also padded out to be 4-aligned, so even if Sizeis 3, there will be an extra 00 before the next block starts.

Note: the names of the blocks are reversed for readability. Also, the notation TmLn[], for example, simply means several sequential TmLn blocks, one directly after the other. There is no "list" structure in AVFX files.

Excel is a binary data format for storing relational data. It's composed of 3 different files:

1. Excel List (.exl) is used to provide the excel index for the game, and allows certain sheets to be accessed by an ID rather than a name.

2. Excel Header (.exh) is used to define the data layout, variant, segmentation and available languages in the event of localisation being present.

3. is the raw data itself and just contains an offset table and then data for each row. The structure of this file changes depending on which variant is defined in the header.

Probably the simplest format in the entire game given it's just a text file.

The first line of the file is its 'header' and defines its type, EXLT and it's version following that. Each sequential line after the 'header' is a path, relative to the exd/ category, and its immutable ID. The immutable ID is optional and in cases where not relevant (eg. quest dialogue files) it's value is -1.

For example, here's an example of the content of a list file:

From these entries, you can build something like the following:

From here, the only thing you can do is parse headers as you will not be able to accurately figure out data paths without first reading the header due to language and row segmentation.

As a note, any excel entry in this file with an ID set get their header pre-cached by the game on startup.

The Excel Header defines the schema and how you read data files. The first structure in this file is it's header which contains information you'll need to parse the rest of the file.

The magic is always EXHF. If it's not, the file is probably not the file you're trying to read.

DataOffset isn't relevant to this file at all, but is required when loading certain data from Excel Data files such as strings. It points to the end of the fixed size data of a row. For example, a row can be made of a bunch of integral types which have a known length at compile time, however a string length is variable. This offset allows you to then seek to the end of the row and access any additional data that may be on the end. If this is weird, don't worry, because it'll make sense later.

ColumnCount is how many columns exist within the file and is the first thing you'll read after the header. In other words, you'll need to read sizeof( ColumnDefinition ) * ColumnCount immediately after the header.

Offset is relative to the row offset, which you don't have until you read the offset table from a data file.

Type is the type of data stored. Most don't have any special handling outside of String and PackedBoolX but we'll cover that later.

PageCount is the number of 'pages' an excel data sheet is split into. Many sheets only have a single page and every row will exist in a single page, however larger sheets such as Quest or Item have many pages. This information is another structure and immediately follows the column definition data.

startId is the row id where a page starts. You'll need this to build file paths for data files.

rowCount is how many parent rows a sheet contains. As a quick example before we cover it more in depth later is that given a row id, you can calculate which page a row is on with row >= startId && row < startId + rowCount - 1.

Finally, the last thing you'll need to read out of a header file is the languages. These are needed to generate paths to data files along with the paging information.

You can read out the languages as is.

Used when reading data. Default requires no extra processing and you can just iterate over the offset list inside a data file. SubRows makes each row contain it's own rows. As a better example, it's like having a compound key on a database table. Instead of one column being the identifier for a row, you have two instead.

This is the total count of all rows across every page. The game uses this field when it internally queries for a row count.

Once you have the 3 pieces of critical information from the headers, namely:

1. The column count

2. The page count

3. The languages

You have everything you need to find data files! How exciting. The files follow one of 2 path formats, which makes this pretty easy. Make sure not to discard the data from the header, there's still some other information you'll need from it.

If the no language is set, your paths follow the following format:

Otherwise, if one or many languages are set (as in, not None), the following format applies:

This should be relatively obvious, but here's a few examples:

The above also applies for quest sheets and so on which exist in subfolders.

The data file contains a single page of sheet data and as mentioned before, you will need the header file to read data correctly.

The value of magic should always be EXDF.

indexSize is how large the row offset index is, in terms of total size. To convert that to a number of entries, you'd do indexSize / sizeof( ExcelDataOffset ).

Immediately following the ExcelDataHeader, the root row offsets are stored. The reason it's called the 'root row offset' is because on variant 2 sheets (or sheets with subrows), this offset won't point to data that can be read following the column data.

rowId is the absolute row id, so a simple way to map these in whatever you're doing is to convert the list of data offsets in this file to a key value map, where the key is the rowId and the value is the offset. Then you can directly index rowIds on any given data page.

offset is the absolute offset to where the data is located in the file. It can be used as is.

Once you seek to a row offset by following the offset list after the header, the first thing you'll encounter is the row header.

The dataSize is the entire size of the row, including any data for subrows (if they exist). You can use this field to copy out the exact amount of data for a row (or subrows) to then later parse if you choose not to do it in place.

rowCount is always 1 on variant 1 sheets and you can ignore that field entirely if you choose to. However, on variant 2 sheets, the rowCount is how many subrows belong to a row.

Therefore, on variant 1 sheets, immediately after the row header is your row data. You can then read columns directly out of the data given a column offset that you read from the header.

On the other hand, variant 2 sheets, following the header is the first subrow, or subrow 0. You can calculate the offset to n subrow with the following:

To clear up a few things:

1. The rowOffset is the offset from the offset list.

2. rowOffset + 6 is the skipping the size of the row header. You could just store the offset once you read the row header and use that.

3. header.dataOffset

There are some intricacies to reading certain types of columns which we'll cover in the next section.

Reading data out of a row depends a bit on the type of the column that you're reading. Most don't require any additional logic and you can read them out of the row and use them as is.

All of the types are as follows:

As mentioned previously, variable length data is stored past the end of the fixed length data segment of a row. So to access a string, you need to read a uint32 where there string column is. You can then obtain the offset of the string by doing the following:

Where rowOffset is the offset of the first column of a row, the dataOffset from the header and the uint32 that you read out from the column.

Packed bools are always 8 bits/1 byte long. To each type is basically which bit you need to bitwise and against to read the correct bool out.

A simple way of doing this for all of them is something like the following:

Alternatively, you can handle each one indivdually, but it's better to be lazy and do it the lazy way instead.

Read and use it as is. It just works™ - no magic required. Just make sure the endianness is correct.

SqPack(s) are formed from the following concepts, in which order 'matters':

1. Repositories

2. Categories

4. Data Files

Repositories are essentially a collection of categories and there's not much to know here. As of writing, there's 4, one for the base game (ffxiv) and one for each expansion (ex1, ex2, etc.). Consider the following directory structure:

Each folder inside sqpack/ is it's own repository. As an aside, when the game tries to access files, it distinguishes which repository to find a file in by parsing the file path and pulling out 2 segments, the repository name and the category.

For example:

The first two paths explicitly define their repository in their file path, and it's always the second segment - if it's omitted it defaults to the ffxiv repository. Category is always the first segment and must be present for a path to resolve.

Categories are just logical separations of game data. The following categories exist:

Every game path will start with one of the above names and it defines which index to search to find a file.

Indexes and regular data files are effectively SqPack files and subsequently have a common header, SqPackHeader. It looks like the following:

Indexes (and data) starts at size so you want to seek to the value of size before you read anything out of the file.

The index data is located directly after the header and depends on which variant of index file you load. The retail client ships with both variants of index files, which we'll mostly refer to as index and index2 to make the difference obvious. Contrary to the retail client shipping with both index variants, benchmarks only ship with index2 files. The reason is unknown.

Immediately following the SqPackHeader there's a SqPackIndexHeader (which is only present in index files):

The actual SqPackIndexHeader is 0x400bytes large, but for the purposes of this, we're only interested in the first 16 bytes. From the indexDataOffset and indexDataSize, you can determine where to start reading from and how many index elements exist inside an index. indexDataOffset is an absolute offset to where the index data is located, and indexDataSize is the collective size of every IndexHashTableEntry that's in a file. This entry is slightly different in the case of index2 files, so we'll generally cover the two with a focus on index1 files for now.

There's a couple notable differences between the C++ and C# version, so we'll just explain the C++ version and the latter will make sense too.

The hash is a u64 that contains two u32s: the lower bits are the filename CRC32, the higher bits are the folder CRC32.

Generally speaking, calculating a hash works like this:

1. Convert the path to lowercase

2. Find the last instance of / and split the string with the last / existing in the first group. The filename needs to have no directory separators

3. Calculate the CRC32 of both path segments

The dataFileId is to identify which file (on disk) contains the file. Larger categories are split across multiple files (each is capped at 2,000,000,000 bytes, or 2 GB), so this is used to distinguish between 020000.win32.dat0 and 020000.win32.dat1 for example, where dataFileId would be 0 and 1 respectively for files located in either dat.

The offset is the absolute number of 8 byte aligned segments that the file is located at within a specific dat file. In a given dat file, a file is located at offset * 0x8 which gives you the absolute offset to start reading a file from.

The main difference between index and index2 is that the entire path is encoded into one CRC32 hash and does not split the path by folder and filename. Outside of that, everything is identical to index.