Is this EXT2 parsing implementation correct?

[francescob]

I need to write a small application that checks whether a file exists on an EXT2/3 partition. I looked for some simple directory parsing implementation and came across this one from from https://github.com/TheCodeArtist/ext2-parser/blob/master/ext-shell.c:

void ls(int fd, int base_inode_num)
{
char* name;
int curr_inode_num;
int curr_inode_type;
 
debug("data block addr\t= 0x%x\n", inodes[base_inode_num-1].i_block[0]);
 
struct os_direntry_t* dirEntry = malloc(sizeof(struct os_direntry_t));
assert (dirEntry != NULL);
assert(lseek(fd, (off_t)(inodes[base_inode_num-1].i_block[0]*1024), SEEK_SET) == (off_t)(inodes[base_inode_num-1].i_block[0]*1024));
assert(read(fd, (void *)dirEntry, sizeof(struct os_direntry_t)) == sizeof(struct os_direntry_t));
 
while (dirEntry->inode) {
 
name = (char*)malloc(dirEntry->name_len+1);
memcpy(name, dirEntry->file_name, dirEntry->name_len);
name[dirEntry->name_len+1] = '\0';
 
curr_inode_num = dirEntry->inode;
curr_inode_type = dirEntry->file_type;
 
lseek(fd, (dirEntry->rec_len - sizeof(struct os_direntry_t)), SEEK_CUR);
assert(read(fd, (void *)dirEntry, sizeof(struct os_direntry_t)) == sizeof(struct os_direntry_t));
 
if (name[0] == '.') {
if ( name[1]=='.' || name[1]=='\0')
continue;
} else {
debug("rec_len\t\t= %d\n", dirEntry->rec_len);
debug("dirEntry->inode\t= %d\n",dirEntry->inode);
printInodeType(curr_inode_type);
printInodePerm(fd, curr_inode_num);
printf("%d\t", curr_inode_num);
printf("%s\t", name);
printf("\n");
}
}
 
return;
}

It seems to start reading the first directory inode data block and keeps reading all the directory entry structures however I don't understand how is this supposed to work if the directory contains more entries that can be stored in a single inode data block. Is this implementation incomplete/wrong, am I reading it incorrectly or is there some EXT2/3 behavior I'm not aware of? Shouldn't the function read the directory inode data blocks before starting to read the entries?

[Karl L.]

Why are you trying to read the filesystem metadata directly? It seems potentially problematic and unnecessarily complicated. You can check whether a file exists on any filesystem Linux understands - whether that be EXT2, EXT3, EXT4, XFS, BTRFS, UDF, HFS, HFS+, UFS, FAT32, or NTFS - with a few simple commands. For example:

$ sudo mount -o ro /dev/sdb1 /mnt
$ find /mnt -name '*somefile.txt'
$ sudo umount /mnt

 

If your goal is to search the contents of a read-only image file like the project you linked to seems to suggest, you can also do that fairly simply using existing utilities. For example:

$ sudo kpartx -av some_hard_disk_image.img
$ sudo mount -o ro /dev/mapper/loop0p1 /mnt
$ find /mnt -name '*somefile.txt'
$ sudo umount /mnt
$ sudo kpartx -d some_hard_disk_image.img

[francescob]

  On 23/09/2013 at 20:00, xorangekiller said:

Why are you trying to read the filesystem metadata directly? It seems potentially problematic and unnecessarily complicated. You can check whether a file exists on any filesystem Linux understands - whether that be EXT2, EXT3, EXT4, XFS, BTRFS, UDF, HFS, HFS+, UFS, FAT32, or NTFS - with a few simple commands. For example:

I don't see what is complicated. I did the same for NTFS and FAT32 without any problem, it was actually pretty easier since they aren't splitted up in all those annoying regions. While EXT2/3 are documented as well the documentation is quite bad since it doesn't cover the common implementations behaviors. In the end I just followed how it was implemented in the kernel sources, certainly having being able to read directories in sequence without having to read the data as a file first would have helped a lot. Having to ask users to install cygwin or other pieces of garbage for a minor function, now that would have been complicated. With ~200 lines of code (ignoring the structures) I solved the problem without any other annoying library or additional software.

[Karl L.]

So if I understand you proprerly, you were trying to read data from an EXT2 partition on Windows? In that case even installing Cygwin like you suggest would not have helped. Although I now understand why you couldn't use the sequence of basic commands I posted above, I stand by my assertion that you are going about this the wrong way. It sounds like Windows is not the right tool for the job.

 

That said, I would love to see your source code if you have it working. I'm assuming it is GPLv2 licensed since you took code from the kernel, of course.

[francescob]

  On 27/09/2013 at 00:24, xorangekiller said:

So if I understand you proprerly, you were trying to read data from an EXT2 partition on Windows?

On Linux there are several libraries available for that making the whole ordeal quite pointless.

 

  On 27/09/2013 at 00:24, xorangekiller said:

In that case even installing Cygwin like you suggest would not have helped.

Yes it would because those same libraries can be compiled and used there. But requiring something as horrible as Cygwin to be installed on user machines just to save a few hundred lines of code didn't really seem a good idea.

 

  On 27/09/2013 at 00:24, xorangekiller said:

Although I now understand why you couldn't use the sequence of basic commands I posted above, I stand by my assertion that you are going about this the wrong way. It sounds like Windows is not the right tool for the job.

I don't think that requiring users to reboot back to linux every time they want to find a file is a solution either.

 

  On 27/09/2013 at 00:24, xorangekiller said:

Although I now understand why you couldn't use the sequence of basic commands I posted above, I stand by my assertion that you are going about this the wrong way. It sounds like Windows is not the right tool for the job.

Following an implementation doesn't mean copying any line of code, code that would have been entirely useless anyway since that's a driver code, and drivers are made for asynchronous access therefore full of additional abstractions, thread synchronization and memory paging. Following the implementation meant simply looking which fields of the filesystem structures the EXT2 driver was looking for doing those basic calculations for finding out which data to read next. I found some noticeable differences between how the ext2 driver handles reading the directories compared to that code example I linked above and also to the many confused and incomplete specifications available.

 

I don't want to post the code since it relies on a lot additional code to perform the disk reads and other application tasks but I can describe the steps required to obtain the same, in case anybody with the same problem would ever stumble into this thread:

1. Read the SuperBlock that is always 1024 bytes after the beginning of the partition
2. Calculate the block size by shifting the base block size on the left by the logarithmic block size value in the superblock (a simple 1024 << LOGBLOCKVALUE)
3. Calculate the logical superblock block number by dividing the SuperBlock offset by the block size
4. Calculate the number of Block Groups by dividing the SuperBlock inodes count value by the SuperBlock inodes per group value
5. Read all the blocks group descriptors into an array (they're stored sequentially). The position of the first descriptor is at BLOCKSIZE*(SUPERBLOCK_BLOCKNUMBER + 1).
6. Read the root directory inode (always 2). In order to do that it's required to know which group of inodes the inode belongs to (inodes are compacted in several groups, one for each blocks group), the position of the inodes table, and the position of that inode within the inodes group. The group number is a simple (INODENUMBER-1) (inodes are non-positional and 0 is reserved so in the calculations you always have to subtract 1) divided by the INODES_PER_GROUP value of the Superblock. Then to find the inodes block table position you read the INODES_TABLE_BLOCK_NUMBER of the matching group from the blocks group descriptors array read before and multiply it by the block size. Then to find the byte offset of the inode within that inode table you first find the logical inode number in the table by doing a module division of the (INODENUMBER-1) by the superblock INODES_PER_GROUP value. Then you multiply that value by the INODE_STRUCTURE_SIZE value of the superblock finding out the byte offset inside the table you need to read at.
7. Then you need a function to read directory or files, reading both is actually exactly the same. Every inode has a BLOCKS array containing 15 block numbers, the first 12 items are direct block values, the 13th item is a block number of a block that is entirely used for storing additional block values, the 14th and 15th block numbers are the same but with twice and thrice the indirection. There's a value in the inode (BLOCKSCOUNT) that tells exactly how many blocks there are to read. Blocks numbers equal to zero are non-allocated blocks (used in sparse files).
8. Once obtained the directory data you start reading all the directory structures one by one, moving by the record length value. The file name is adjacent to the structure and the file type is stored in the directory itself. The list should end when the record length is 0. If a directory structure contains an inode with the value 0 it means the item has been deleted (explaining how recovering files on EXT2 was quite the nightmare, you no longer know where it is and there's no other name associated to it).

The longest part is reading the blocks values but I think the whole implementation can be done with 100 lines or less on an unmanaged language especially if you don't care about reading the whole structures.