Not sure because it should work like that. Some other things I noticed on your code though:
#define PIXEL_SIZE (1) /* change this if you change to another screenmode */
You are using 16bit mode, so that
should be 2.
u16 *address=VRAM+((((512)*PIXEL_SIZE)*y)+x); // Caculates address of pixel.
The formula is wrong plus you're doing pointer arithmetic on 16bit pointers here. (effectively the errors sum up to 0 in your special case)
sceDisplaySetMode(mode,SCR_WIDTH,SCR_HEIGHT);
Does not do what you think it does. In fact you don't ever need to call this function and if you do, the first parameter needs to be 0.
sceDisplaySetFrameBuf(VRAM,BUF_WIDTH,1,1);
This functions third parameter should be your mode variable.
Also, you can replace the pspDebugScreenInit function with the following two functions, giving you control over where the display buffer is supposed to be and what pixelformat to use:
pspDebugScreenSetXY(0,0);
pspDebugScreenInitEx( VRAM, mode, 1 );
Anyway, is there a good reason why you want to use 16bit mode for a pixel fixer application? And why do you change brightness in three stages?
Other than that, I really see no reason why your program should still print wrongly. Could you paste your current main.c code again if it doesn't change?