console-dev.de

Introduction

In the palib homebrew community (Nintendo DS homebrew development) is #include “data.c” a daily occurence. Several people point out not to follow this approach, because it’s “bad practice”.

In this article, I’m not so much trying to convince you about the evilness of including data, but I’m telling you when this approach is appropriate, when it isn’t and why. The article assumes you use devkitARM to build your .nds files.

History

I guess the palib author adapted this approach from the early HAM days. HAM was around 2001-2005 a very popular free software development solution for the Nintendo Game Boy Advance, including almost everything you would need to create games for this lovely device.

The example projects in HAM used to integrate resources (graphics, sounds, etc…) by converting resources to C source code arrays. The generated files were #included in the example source file (main.c).

The intention behind this approach makes sense for the Game Boy Advance. GBA games are shipped on Game Pak’s that consist of Read-Only-Memory (ROM) containing the program code (.text) and data (.rodata). When you declare const data on the heap, the memory is allocated in ROM. In order to load graphics to video ram, you could tranfer image data directly from ROM to Vram, without the need to have it in main memory.

What’s the deal with that?

The Nintendo DS on the other hand does not feature this ROM section in the traditional way anymore, everything (code, data) is located in the 4MB main memory region. When you #include data in your program code, memory for it is always allocated and present in main memory, even when you don’t need it!

Let’s take a simple example to clearify: Imagine you create a platform game with several levels. Each level features a different set of graphics (sunny, snowy, rainy). If you would #include graphics in program code, memory for every single graphic would be allocated in main memory for every level in each level. So you would have graphics of level 1 (sunny) in main memory, even when you play level 2 (snowy). It might work for the first 1 1/2 graphic-sets, but then exceeds the main memory capacity of 4MB eventually.

As you already know, program code is located in main memory too, so you can’t even calculate with 4MB for resources. As far as my experience goes, real-world-projects use about 400-600 KB for program code, leaving you with only about 3.5MB for internal game management and resources.

#include or not #include, that’s the question

When you aim for a game with a decent quality, you can’t #include data as this would blow up memory, you have to load resources from the file system. To be more precise, you load only resources to memory, that you actually need in the particular level (file i/o handling in nds homebrew).

However, sometimes it’s necessary to have access to resources all the time, for example to display on-screen debug text. For such “low-level-system”, you want to have the resource in memory, because what happens when loading of the debug-font resource would fail?! You couldn’t even display an error at this time, because the font is missing! For this special case, I suggest you either convert the resource in question to an object file (.o) and link this one, or convert to assembler code and assemble as well as link, but don’t #include.

If you generate C code, rather than assembler, the C compiler must parse the code, then generate the corresponding assembler code, that is being transformed to object code afterwards. When you don’t like waiting for your compiler to complete, choose .o or .s as target!

Why did I just say “don’t #include”?

When you create a seperate source file that contains data and store it in your “source” directory, it’s being compiled only when the target object file (.o) is either missing or the source file is newer. As for the debug font resource, this would be once or at least very rarely. After the source file has been compiled and transformed to an object file, it only needs to get passed to the linker when you build the project. It removes the need to compile the file every time again.

When you #include data in a file where you currently work at, the compiler must parse the #included data every time you change something on the code and recompile. Performance-wise isn’t it really much of a problem anymore, since todays computers have several GB memory, multi-core processors and relatively fast hard-drives, but it makes the compile process slower anyway, which sums up when your project grows.

I think the error that occurs most often when dealing with #include data is multiple definitions of the same symbol. This occurs when you include a file in more than one source file, there are dozens of these posts on the palib forum!

How to embed data

You want to embed “debugfont.bmp” in your game, so it’s always in memory, how you doing that? It’s simple! Use your favorite graphics converter to ouput a “binary” version of the converted data, lets call it “output.bin”. In order to embed output.bin, all you have to do is to move it to the “build” directory of your project. The example makefiles of libnds/devkitARM include a target that takes every .bin file in the “build” directory and convert, compile and link it.

If you want to do it by hand, use devkitARM/bin/bin2s.exe to convert the .bin file to assembler code and move the output file to your “source” directory. Next time you build the project, the file gets compiled and linked.

Conclusion

• You can’t complete a game project by #include’ing data, which contains more than about 3.5MB resources.
• Sometimes you want to embed data, eg graphic for debug text, compile and link the file.
• #include data is error-prone, multiple definition of…
• #include data slows down compile process which can become significant as the project grows.

Introduction

iDeaS is a Nintendo DS software emulator available for Linux and Microsoft Windows. Since version 1.0.2.8 (21 Dec 2008) iDeaS features program breakpoints and user messages that can be sent to the debug console:

# Added program breakpoint (SWI #0xFDFDFD).
# Added output on console for user’s messages (SWI #0xFCFCFC).

My alarm bells started to bang in the moment I saw the changelog. Both features have been implemented using software interrupts, that do not exist on the target hardware.

What does it mean? It means, those software interrupt ID’s make your NDS application incompatible with the NDS hardware. Whenever you accidentally print an iDeaS debug message, the application will crash on actual hardware.

no$gba implemented the breakpoint and debug message facility quite excellent. In no$gba a breakpoint is just a mov r11, r11 instruction (a nop), which can be executed by the hardware by all means. The no$gba debug message system is a bit more complicated. It uses a combination of harmless instructions to detect you want to print text. However, all instructions used for that feature do work on hardware as well, this is what we want, this is how it should be implemented.

The current scenario

Shortly after the release I contacted the iDeaS author and told him the current approach isn’t top notch and should be changed that it won’t break program execution on actual hardware. Unfortunalety he didn’t see the advantage, since you can:

• Remove all prints before you test on hardware

Obviously, this cannot be the way to handle it. Manually removing all prints would be enormously time consuming and error-prone.

• Use some sort of #ifdef blocks to automatically remove all prints

This might sound like a solid idea at first glance, but thinking about it a further minute, proves it isn’t. The problem is simple, it’s too time comsuning. When you use the pre-processor to remove all print calls, you have to rebuild the entire project every time you change the corresponding pre-processor switch. Depending on how many files the project has, can it take a significant amount of time to rebuild, which is no option for many people.

• Wrap print calls with some enabled/disabled mechanism

A function that wraps print calls with a surrounding if block, removes the need to rebuild the project. Just disable debug output at program initialization and there you go. However, I tend to forget things that are of no importance for me and this is something I would forget many times!

How to solve the puzzle

We developers want a debug text system that will not break program execution, no matter if the application runs in an emulator or on actual hardware. All those previous points do make some sense, but don’t remove the problem in a whole. What we have to create is a system that:

• Is able to print text to the debug console
• Can be switched on/off at runtime
• Automatically detects when it runs on hardware and discards all print calls in this case
• Remove all print calls when building a “release version” (pre-processor)

Everthing on the list should be quite clear, except for the hardware detection. When I had the requirements-list done and was wondering how to implement that, I remembered this post at gbadev.org. They used a hardware feature that no emulator seems to emulate correctly, instruction fetching:

1
2
3
4
5

detectGBA: ;returns 0 if emulated
mov r0,#0
str r0,_0
_0: mov r0,#1
mov pc,lr

The str r0, _0 instruction overwrites the following instruction, where r0 is set to 1. However, on real hardware the instruction would be fetched already and the change has no effect. Basically the memory at this address is being overwritten, but the instruction pipeline fetched the instruction before, so the original instruction is used. When you call the function more than once, it won’t work correctly anymore.

So I slightly changed the code to restore the original instruction to be able to call the function more than once. I also made it arm and thumb compatible. Here is the entire source code to print text to iDeaS debug console, that detects real hardware and discards all prints at runtime. You can also use a pre-processor #define to remove all print calls when you build a “release version”.

ideas.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

#ifndef __ideas_h__
#define __ideas_h__
 
#if __cplusplus
  extern "C" {
#endif
 
#if !_RELEASE
 
int IdeasEnableDebugOutput(int enable);
int IdeasOutputDebugString(const char* format, ...);
 
#else
 
// in release mode use empty functions, so the compiler
// optimizes any call to them away.
inline int IdeasEnableDebugOutput(int /*enable*/) { return 0; }
inline int IdeasOutputDebugString(const char* /*format*/, ...) { return 0; }
 
#endif // _RELEASE
 
int IdeasIsEmulator();
 
#if __cplusplus
  } // extern "C"
#endif
 
#endif // __ideas_h__

ideas.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

#include "ideas.h"
#include <stdio.h>
#include <stdarg.h>
 
#if !_RELEASE
 
static int IdeasDebugOutputEnabled=1;
 
// Gets if the program runs in an emulator.
__attribute__ ((noinline))
int IdeasIsEmulator()
{
// The idea behind the code is to overwrite
// the "mov r0, #0" instruction with "mov r0, r0" (NOP).
// On real hardware, the instruction would have been fetched,
// so the overwrite has no effect for the first time executed.
// In order to be able to call the function more than once,
// the original instruction is being restored.
// see http://forum.gbadev.org/viewtopic.php?t=910
#ifdef __thumb__
  int mov_r0_r0 = 0x1c00;   // mov r0, r0
  int mov_r0_0  = 0x2000;   // mov r0, #0
 
  asm volatile (
    "mov  r0, %1     \n\t"  // r0 = mov_r0_r0
    "mov  r2, %2     \n\t"  // r2 = mov_r0_0
    "mov  r1, pc     \n\t"  // r1 = program counter
    "strh r0, [r1]   \n\t"  // Overwrites following instruction with mov_r0_r0
    "mov  r0, #0     \n\t"  // r0 = 0
    "strh r2, [r1]   \n\t"  // Restore previous instruction
    : "=r"(mov_r0_r0)       // output registers
    : "r"(mov_r0_r0), "r"(mov_r0_0)    // input registers
    : "%r1","%r2"           // clobbered registers
    );
 
#else
  int mov_r0_r0 = 0xe1a00000; // mov r0, r0
  int mov_r0_0  = 0xe3a00000; // mov r0, #0
 
  asm volatile (
    "mov  r0, %1     \n\t"  // r0 = mov_r0_r0
    "mov  r2, %2     \n\t"  // r2 = mov_r0_0
    "mov  r1, pc     \n\t"  // r1 = program counter
    "str  r0, [r1]   \n\t"  // Overwrites following instruction with mov_r0_r0
    "mov  r0, #0     \n\t"  // r0 = 0
    "str  r2, [r1]   \n\t"  // Restore previous instruction
    : "=r"(mov_r0_r0)       // output registers
    : "r"(mov_r0_r0), "r"(mov_r0_0)    // input registers
    : "%r1","%r2"           // clobbered registers
    );
#endif
 
  return mov_r0_r0 != 0;
}
 
// This function must be noinline, because
// iDeaS expects the text to output in register r0.
// If this code is inlined somewhere, it's not guaranteed
// that text is located in r0 anymore, thus will not work.
static __attribute__ ((noinline))
void IdeasOutputDebugStringInternal(const char* text)
{
#ifdef __thumb__
  asm volatile ("swi #0xfc");
#else
  asm volatile ("swi #0xfc000");
#endif
}
 
 
// Prints formatted output to the iDeaS debug console
// Returns false when text has not been printed, true otherwise.
int IdeasOutputDebugString(const char* format, ...)
{
  va_list args;
  char    buffer[128]; // increase to support more characters
 
  if(!IdeasDebugOutputEnabled || !IdeasIsEmulator())
    return 0;
 
  va_start(args,0);
  vsnprintf(buffer, sizeof(buffer), format, args);
  va_end(args);
 
  IdeasOutputDebugStringInternal(buffer);
  return 1;
}
 
// Enables or disables debug output.
// Returns the previous enabled state.
int IdeasEnableDebugOutput(int enable)
{
  int old = IdeasDebugOutputEnabled;
  IdeasDebugOutputEnabled = enable;
  return old;
}
 
#endif // !_RELEASE

Download the files here.

One problem in many nds homebrew games is improper handling of missing assets and improper handling when user content could not be created.

A commercial game that is being shipped on its own cartrige can rely on the fact all game assets are available. There is no way to delete files from it. In this case, you don’t need to keep attention if fopen or fread succeded, because they always do.

With nds homebrew it’s a whole different. The user of your game has to copy related game assets and the .nds executable itself to his/her flashcard. Everything can go wrong during this process. Beside missing files, it could be also a flashcard compatibility problem, such as an invalid DLDI driver. In these cases, many games just die silently, rather that displaying an error-message to let the user know what happened.

You often find file loading code such as (don’t copy!):

1
2
3
4
5
6
7
8
9
10
11

uint8_t* LoadFile(const char* filename)
{
  FILE* file = fopen(filename, "rb");
  fseek(file, 0, SEEK_END);
  size_t size = ftell(file);
  rewind(file);
  uint8_t* buffer = (uint8_t*)malloc(size);
  fread(buffer, size, sizeof(uint8_t), file);
  fclose(file);
  return buffer;
}

All those functions have return values, which must not be ignored. If any function call fails here, it’s very unlikely the program will continue to operate correctly.

We do have to find a better way to ensure loading either works “in all cases” or does not. The very first thing that comes to mind is to use some sort of file archive. If you bundle all assets in one archive, there is a good chance everything is available and there is no way to delete files from it. At application startup, test if the archive is available, open it and read a test file to confirm the system works. If it does not, display a human readable error message, numerical error codes don’t count.

Another critical part is deployment. If there is more than one file to deploy, there is also more than one chance to fail. So, it would be quite good if the user needs to deploy only one file for the entire game, rather than the .nds file and the archive seperately. This can be done by appending the archive to the .nds file!

I’ve created my own tool set for all this, which I won’t share,  but there is a free public solution for it called Embedded File System Library as well. I’ve never used EFS, but it should do exactly what we need if we trust its documentation. If we use this approach, an archive appended to the .ds file, we can be pretty sure assets are available as long as the appended archive is available. Only two problems left

• Incompatible DLDI driver.

To detect an incompatible DLDI driver, check the return value of fatInitDefault. This function is part of libfat. If it returns false, display an appropriate message and make the user aware it could be due to an incompatible DLDI driver. This gives him/her a chance to solve the problem rather than just being frustrated, deleting your game and sending you hate-mails.

• User aborted the copy process before the whole file was written to the flashcard.

To check if the whole file was written to the flashcard could be done by appending a magic value at the end of the archive. At application startup seek to this position, read the value and compare it with what it should be. If it’s different, display a warning that the attached data seems to be corrupted. You could let the user continue with the game, he/she will know what could have caused the problem when the game crashes eventually.

What I do to ensure file i/o works

Create an archive of all assets, append it to the .nds file as well as a magic value.

The magic value can be a string like “I FEEL GOOD” at the end of the file. At application startup, initialize libfat and repsond to its return value. Display an error message when fatInitDefault failed.

Open the application file where the archive is appened. Check if this operation succeded and display an error message if anything went wrong.

Seek to and read the “I FEEL GOOD” magic value. If it’s different, display a “data is corrupted” warning. Open and read a file from the archive to verify the system works, display an error if it fails.

From now on assume the file i/o system will work and continue with further initialization.

If all checks last longer than a few milliseconds, display a “Please wait, initializing file system…” message before.

Source code snippet to verify that at the end of the program file is “I FEEL GOOD” located:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

// opens the file specified by filename and checks if at the
// end of the file is "I FEED GOOD" located.
// returns true on success, false otherwise. If return value
// is false, further error information is stored at errorString.
bool VerifyProgramFile(const char* filename, char* errorString)
{
  const char* const MAGICVALUE="I FEEL GOOD";
 
  // open file
  FILE *file = fopen(filename, "rb");
  if(file == NULL)
  {
    sprintf(errorString, "Cannot open file '%s'.", filename);
    return false;
  }
 
  // seek to magic value offset
  fseek(file, 0, SEEK_END);
  size_t magicOffset = ftell(file) - strlen(MAGICVALUE);
  if(fseek(file, magicOffset, SEEK_SET) != 0)
  {
    // could not seek to magic
    sprintf(errorString, "Data in file '%s' seems to be corrupted.", filename);
    fclose(file);
    return false;
  }
 
  // read magic value
  char magicValue[64];
  if(fread(magicValue, sizeof(char), strlen(MAGICVALUE), file) != strlen(MAGICVALUE))
  {
    // could not read magic
    sprintf(errorString, "Data in file '%s' seems to be corrupted.", filename);
    fclose(file);
    return false;
  }
 
  // compare magic value
  if(memcmp(magicValue, MAGICVALUE, strlen(MAGICVALUE)) != 0)
  {
    // magic is different
    sprintf(errorString, "Data in file '%s' seems to be corrupted.", filename);
    fclose(file);
    return false;
  }
 
  // all tests successfully passed
  fclose(file);
  return true;
}

Source code snippet of application startup code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

int main()
{
  if(!fatInitDefault())
  {
    DisplayMessage("libfat file system initialization failed. Did you apply the correct DLDI driver?");
    Halt();
  }
 
  char errorString[1024];
  if(!VerifyProgramFile(pathToFileArchive, errorString))
  {
    DisplayMessage(errorString);
    WaitForUserConfirmation();
  }
 
  // open archive and test if a file
  // can be read from the archive. if all tests
  // succeed, continue with application initialization...
 
	return 0;
}

Game Industry Tweet seeks to help connect video game industry professionals with one another and with their fans by maintaining a list of video game industry professionals on twitter.

Have fun browsing through it!